Script WebLogic and Java Patches

In December, we talked quite a bit about patching Java and WebLogic on the blog and podcast. There was a WebLogic CVE, and then a patch, to apply. If you want a recap on the CVE and patching process, here are the posts:

While applying the patches, I wanted to script the process so patching would be consistent across all our servers. I pulled the scripts into a GitHub project for sharing and reuse. If you haven’t scripted a WebLogic patch, this would be a place to start. The scripts use PowerShell and built for WebLogic 10.3.6. So, they use SmartUpdate instead of OPatch. I also added in a Java patch to the process too. You could pull out the Java patch script to use by itself. One more note: all the patches, Java, and scripts were set to run from the folder e:\installers\weblogic1036-2015-CVE-Patches. If you use these for your environment, or just use them as a template, you’ll want to update those paths for your specific configuration.

There is nothing ground-breaking about these scripts 🙂 I can write scripts, but I’m not the best script developer out there. If you see places where the scripts need improvement, file an issue with the project or submit a pull request! The main goal with this project and post is to get others started with scripting. Scripting, even if the scripts are basic, can benefit administrators. I hope that this quick overview might help someone get started.

Scripts Overview

These scripts are writtin in PowerShell. If PowerShell scripts are not enabled on the server, run this command to allow PowerShell scripts to run:

set-executionpolicy unrestricted

  1. Install new SmartUpdate version (3.3.0)

    installSmartUpdate.ps1

    The silent.xml file is used for a silent install (no prompts). The installation directory is set to e:\oracle. If you want a different directory, change the value for “BEAHOME”. 1. Stop all web servers running on the server .stopPIAServices.ps1 The script looks for any Windows service that containts “*-PIA” in the name. If you have any WebLogic domains were not created by the

    installNTService script, you may need to shut them down by hand.

  2. Prepare and copy files from the weblogic1036-2015-CVE-Patches folder

    prepareFiles.ps1

    This script performs tasks to prepare different files for patching: On our servers, two files needed updates to run the Smart Update utility. registry.xml needed to remove a reference to Tuxedo; bsu.cmd needed an increase in memory to the Java Heap. The registry.xml file also contains a reference to the server where it was installed. The script will change that value based on the new server’s name. The original files are backed up first and a .bkp extension is added to the file name. The script also copies jdk-1.7.0_79 to our e:\java folder. If you want the new java version in a different location, you can change the path in the file.

  3. Apply both WebLogic patches The patches we are applying resolve the December 2015 CVE with WebLogic. If you are using these scripts for future patches, you’ll want to update the patch ID’s in the script.

    applyWebLogicPatches.ps1

    Both patches are applied to WebLogic using the bsu command. The script assumes your patches are in the folder e:\patches\cve-2015-4852. NOTE: On one of our servers, the second patch stalled during the “Checking for Conflicts” step. If the script stalls for more than a few minutes, hit Cntl-C.

  4. Update the JAVA_HOME values

    updateJavaVersion.ps1

    The JAVA_HOME value in the setEnv.cmd script will be updated to the new path. You must update this script for each server. The paths in the script are hard-coded. (The hard coding is an obvious candidate to fix next. Should be able to use the Get-ChildItem cmdlet to find all the setEnv.cmd files.)

  5. Update Registry value for JAVA_HOME

    updateRegistryJavaVersion.ps1

    The JAVA_HOME value in the Registry for each web service will be updated. You must update this script for each server. The paths in the script are hard-coded. (Again, another place for improvement. Need to find a search cmdlet for the Registry. Could look for -PIA in the service name.)

  6. Start all web servers running on the server.

    startPIAServices.ps1

    Again, this looks for all Windows services that have *-PIA in the name and starts them. That’s it.

The scripts are pretty simple, and you can write a wrapper script to run all the sub-scripts. That way you’d have one script to kick off. Or, you could add these into a tool like Rundeck to execute from a centralized place. Once you start down the path of scripting, many opportunities open up to speed up everyday tasks.

8.55 – Cross-Origin Resource Sharing

In PeopleTools 8.55, Cross-Orign Resource Sharing (CORS) is now supported. Don’t know what CORS allows? Here is a good primer on the topic.

The Same-Origin Policy restricts the browser from performing certain actions by scripts or documents based on the origin. The origin is everything in the URL before the path (for example, http://www.example.com). For certain actions, the browser will compare origins and, if they don’t match, won’t allow things to proceed. For example:

  • A parent document can’t access the contents of an that comes from a different origin. This prevents a malicious site from opening up your bank’s website and stealing your credentials, as an example.
  • While one document can send information to another via a form post, AJAX requests across origins are generally disallowed.

The Same-Origin Policy is a vital piece of web security architecture, but it also poses a problem. What happens when you want to allow a site with a different origin to access your content?

Here is a great example of where CORS support can benefit PeopleSoft. In Enterprise Learning Management, you can link to hosted web-based training. Often, that web-based training is on a different domain. With CORS support, you can add in remote sites in the Web Profile and display remote courses in your ELM environment.

To enable Cross-Origin sites, open your Web Profile. There is a new tab, Authorized Site, that lets you list many sites to support.

Let’s Encrypt with PeopleSoft

Let’s Encrypt is a service provided by the Internet Security Research Group to provide free SSL certificates to anyone. The goal of the project is get the entire web encrypted. I mentioned the project in Episode 7 of The PeopleSoft Administrator Podcast and thought it would be a great exercise to try it with PeopleSoft. Let’s Encrypt uses a client on the server to automate the certificate request process. The client will:
* Validate that you own the web server
* Generate a CSR
* Download the certificate
* Apply the certificate to the web server (limited support)
* Automatically renew the certificate

There are a few requirements to use the Let’s Encrypt clients though:

  • The web server needs to accessible by the internet. The Let’s Encrypt site will validate that you own the server by checking for a specific file on the web server.
  • Not all operating systems are supported, yet.
  • Some web server’s have built-in support (IIS, Apache), but others do not (e.g, WebLogic).

We can still generate certificates though, the automatic renewal won’t update the web server though.

Install Let’s Encrypt Client for Windows We’ll use the

letsencrypt-win-simple command line client for Windows. Download the latest release from GitHub and extract the folder to a permanent location.

Generate a new certificate

  1. Run .letsencrypt.exe --accepttos

    Let's Encrypt (Simple Windows ACME Client) ACME Server: https://acme-v01.api.letsencrypt.org/ 
    Config Folder: C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org 
    Loading Signer from C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orgSigner 
    Getting AcmeServerDirectory Loading Registration from C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org
    Registration Scanning IIS 7 Site Bindings for Hosts 
    No IIS bindings with host names were found. Please add one using IIS Manager. A host name and site path are required to verify domain ownership. 
    No targets found. 
    M: Generate a certificate manually. 
    A: Get certificates for all hosts 
    Q: Quit 
    Which host do you want to get a certificate for: 
    
  2. Since we are not running IIS, we’ll generate a certificate manually.

    Which host do you want to get a certificate for: M Enter a host name: 
    
  3. Enter the DNS name for your web server.

    Enter a host name: hr.psadmin.io 
    Enter a site path (the web root of the host for http authentication): 
    
  4. Next, enter the root path for your web server. If you are running WebLogic, that will be PORTAL.war directory on your web server.

    Enter a site path (the web root of the host for http authentication): W:\pt8.55\webserv\peoplesoft\applications\peoplesoft\PORTAL.war
    
  5. Then, the Let’s Encrypt client will create a new file under PORTAL.war.well-knownacme-challenge. That file will be used to validate that you own the web server.

    Authorizing Identifier ps92t855.psadmin.io 
    Using Challenge Type http-01 Writing challenge answer to W:\pt8.55\webserv\peoplesoft\applications\peoplesoftPORTAL.war.well-known/acme-challenge /1c2yN7Y93sJwRUmRGaoG4kT-QynrIcGr4szre-3nTsQ 
    Answer should now be browsable at http://ps92t855.psadmin.io/.well-known/acme-challenge/1c2yN7Y93sJwRUmRGaoG4kT-QynrIcG r4szre-3nTsQ 
    Submitting answer Refreshing authorization Authorization Result: valid 
    Deleting answer 
    
  6. After the web server ownership is verfied, new certificates will generated and copied to your system. The certificates are copied to your %USERPROFILE%AppDataRoamingletsencrypt-win-simple folder in a few formats:

    • .der
    • .pem
    • .pfx

The client will also add the certificates to the Windows Certificate Store for you. To add the certificates to WebLogic, we’ll use the .pem

    Requesting Certificate Request Status: Created 
    Saving Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orghr.psadmin.io-crt.der 
    Saving Issuer Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orgca-009813F47513E5750B43E7431E971E44BD-crt.pem 
    Saving Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orghr.psadmin.io-all.pfx (with no password set) 
    Opened Certificate Store "WebHosting" 
    Adding Certificate to Store 
    Closing Certificate Store 
    WARNING: Unable to configure server software. 
    Creating Task letsencrypt-win-simple httpsacme-v01.api.letsencrypt.org with Windows Task Scheduler at 9am every day. 
    Renewal Scheduled Manual hr.psadmin.io (W:pt8.55webservpeoplesoftapplicationspeoplesoftPORTAL.war) 
    Renew After 2/9/2016 
    Press enter to continue. 

Create a New pskey Keystore Now that we have certificates, let’s create a new

pskey file with the certificates. We’ll use Keystore Explorer to quickly generate the file.

  1. Open Keystore Explorer. (If it’s first time you’ve used it, follow the instructions to download the Unlimited Strength files).
  2. Create a new keystore file.
  3. Select the file type of “JKS”.
  4. Select “Tools > Import Key Pair”.
  5. Select the “OpenSSL” option.
  6. Deselect “Encrypted Private Key”.
  7. For the “OpenSSL Private Key File”, select the file hr.psadmin.io-key.pem.
  8. For the “Certificate(s) File”, select hr.psadmin.io-cert.pem
  9. Click “Import”.
  10. Enter an alias name that is descriptive. I used hr.psadmin.io-2015-12.
  11. Since the prive key was delivered without a password, we’ll want to enter one. Enter a password for the key pair.

Now you have the private and public key for your DNS entry in the keystore. Next, we need to add the root (and intermediate) certificates so that a chain of trust is established.

  1. In Keystore Explorer in our new keystore file, right-click on our certificate. Select “Edit Certificate Chain > Append Certificate”.
  2. Select the file ca-GUID-crt.pem and click “Append”.
  3. Save the file, give the keystore a password, and name the file pskey-2015-12.

Load Keystore into WebLogic After importing the certificates into

pskey-2015-12, we need to copy the file to the web server and tell WebLogic to use the new file. The integrationGateway.properties file will need to know about the new keystore as well.

  1. Copy the pskey-2015-12 file to your web server directory %PS_CFG_HOME%\webserv\peoplesoft\pia\config\keystore.
  2. Log into the WebLogic console.
  3. Navigate to “Environment > Servers > PIA > Keystores”.
  4. Click the “Lock & Edit” button to allow editing.
  5. Click the “Change” button for the Keystores option.
  6. Select “Custom Identity and Custom Trust” and “Save”.
  7. In the “Custom Identity Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  8. In the “Custom Identity Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  9. In the “Custom Trust Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  10. In the “Custom Trust Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  11. Click Save. WebLogic will look at the new keystore file. Next, we need to tell WebLogic certificate it should serve to users.

  12. Click on the “SSL” tab.

  13. Change the “Private Key Alias” to hr.psadmin.io-2015-12.
  14. In the “Private Key Passphrase” boxes, enter the password you gave the hr-psadmin.io-2015-12 keypair.
  15. Click Save.
  16. Click the “Activate Changes” button.

Update integrationGateway.properties

Before we reboot the WebLogic domain, we need to update the integrationGateway.properties file.

  1. On your web server, open the integrationGateway.properties file under %PS_CFG_HOME%\webserv\peoplesoft\applications\peoplesoft\PSIGW.war\WEB-INF.
  2. Find the line secureFileKeystorePath and change file name to pskey-2015-12.
  3. If the password you gave the keystore is different than the previous file, you’ll need to update that parameter in the file.
    1. Open a command prompt and go to %PS_CFG_HOME%\webserv\peoplesoft\bin.
    2. Run the command setEnv.cmd to set the environment variables.
    3. Go to the folder piabin.
    4. Run the command PSCipher to get the encrypted text.
  4. Restart your WebLogic domain.

Test your HTTPS Connection

As WebLogic is starting up, make sure to check the logs to verify that the server started with your new certificate. Once the server has started, open a browser and go test the site. You should see a secure connection in the browser to your site.

How to Patch Java in WebLogic

With the recent attacks on SSL, WebLogic and Java, I wanted to give an overview on how you patch Java for your WebLogic instances.

When you install WebLogic, it asks you for the location of your Java Home. Then, every web server instance you create uses that Java Home. Unless you have patched Java in the past, all of the WebLogic instances on that server will be using the old Java Home.

Download the New JDK

Go to Oracle’s Java Download page and download the latest JDK. Make sure to select the correct codeline that your version of PeopleTools supports.

PeopleTools 8.53-8.55 support Java 1.7 (aka Java 7). It will implicity support any patch on the 1.7 codeline. So, you can install the latest 1.7.0_xx patch and use it with WebLogic and PeopleTools.

Install the new JDK (you don’t need the JRE) to a common location. We use the folder convention e:javajdk-1.7.0_xx to install the JDK.

Update the commEnv Script

By default, the Java Home parameter is set in the %WL_HOME%commEnv script. This script configures environment variables that are common to all WebLogic instances on the server.

You can update the JAVA_HOME in the commEnv script, but it will affect ALL the WebLogic instances on that machine. This might be what you are looking for. But, if you run more that one web server you might want to try the next option.

Update the setEnv Script

In each WebLogic instance you create (hr92dmo, hr92dev, etc), the file %PIA_HOMEbinsetEnv will set the configuration that applies only to a specific domain. This is where I prefer to set JAVA_HOME. In the file, you will find a line that says:

@REM JAVA_HOME is set via commEnv.sh, to override set it here.

Simply add this line to set a JAVA_HOME for a specific web server:

set JAVA_HOME=e:javajdk-1.7.0_79

Now, you can patch your Demo environment and test without affecting other web servers on the server.

Update the Windows Service

If you are on Windows and installed a service for your web server, you will need to change the JAVA_HOME value for the service. You could re-create the service but there is an easier way.

Open the Registry Editor (regedit) and navigate to:

HKLM:SYSTEMCurrentControlSetserviceshr92dmo-PIAParameters 

Under this registry folder, you’ll see a Key name “JavaHome”. Update the value’s path to match your new JAVA_HOME. Restart the service for the change to take affect.

Patching Java for WebLogic is pretty simple. The next step (and upcoming blog post) will be to script these changes, and WebLogic patches, so you can automate your web server patching.

How to Apply WebLogic Patches

Oracle has released a patch for the latest CVE against WebLogic, so I wanted to walk though the steps to apply the patch to WebLogic and show how to use Smart Update. Smart Update is the utility used by WebLogic to apply patches to your installation.

UPDATE 12/8/2015 Thanks to Matt Tremblay for pointing out, WebLogic 12.1.2+ is now using OPatch for WebLogic patching. Look for an second WebLogic patching post soon about using OPatch with WebLogic.

Smart Update 3.3.0

Version 3.3.0 is the latest version and is included with WebLogic 10.3.5 and later. If you launch Smart Update and find that its an older version, go grab version 3.3.0 (Patch 12426828).

Launching Smart Update

On Windows, if you chose to create a Program Group, you can launch Smart Update from the Start Menu under the “Oracle WebLogic” folder. Or, you can launch it from the command line:

%BEA_HOME%utilsbsubsu.cmd

The first time you run Smart Update, it may ask you to provide a %BEA_HOME% path. Give the path to your BEA Home (e.g, e:oracle).

If you receive an error: “Unable to locate any supported product installations” or “The BEA Home directory selected does not contain any supported patch targets”, check out MOS Documents 946541.1 or 1063605.1 for the fix.

Applying Patches

In the Smart Update window, you will see the installed applications in the left pane. Make sure “WebLogic” is selected. On the right, the top pane shows you patches that have been applied. The lower pane displays patches in your download directory that are waiting to be applied.

By default, Smart Update will look for patches under %BEA_HOME%utilsbsucache_dir for patches. To start Smart update and have it look at a different path, use the -patch_download_dir=[path] flag or select File > Preferences to change the directory.

To apply a patch, click the green arrow in the “Downloaded Patches” pane for the patch. Smart Update will check for patch conflicts and the apply the patch.

Command Line

You can also run Smart Update from the command line. This is great when you have multiple servers to patch. Running bsu.cmd -help will give you all the options you need when scripting.

Let’s Fix CVE-2015-4852

Since we have new patches to fix CVE-2015-4852 (T3/Java Deserialization), let’s use those as our example. Go to this page to find the applicable patch (or patches if you are on 10.3.6) to apply.

Extract Patches

Download the patches you need and unzip them. Copy the .jar and .xml files from the patch folders to your web server. I put the patch files under e:patchescve-2015-4852 on the web server. We will tell Smart Update to use this directory. (Since we are running WebLogic 10.3.6, there are two patches to install in our example.).

Apply Patches

Before you apply any patches, make sure to stop any web servers running on the server. If don’t, Smart Update won’t be able to patch .jar files that are in use.

Let’s run Smart Update from the command line. Open a command prompt and navigate to %BEA_HOME%utilsbsu. We need to pass these values to the bsu program:

  • -install
  • -patch_download_dir=e:patchescve-2015-4852
  • -patchlist=EJUW (note, this is not the patch number, but the PSU Patch ID)
  • -prod_dir=e:oraclewlserver_10.3
  • -verbose

So, my command to apply the first CVE patch looks like this:

bsu -install -patch_download_dir=e:patchescve-2015-4852 -patchlist=EJUW -prod_dir=e:oraclewlserver_10.3 -verbose

I had to change the memory settings for Smart Update. In the bsu.cmd file, I modified the set MEM_ARGS line:

set MEM_ARGS=-Xms512m -Xmx1024m -XX:PermSize=64m -XX:MaxPermSize=128m -Xss512k

Smart Update will give you a “Success” message, or an error message. Next, let’s apply the second CVE patch:

bsu -install -patch_download_dir=e:patchescve-2015-4852 -patchlist=ZLNA -prod_dir=e:oraclewlserver_10.3 -verbose

Verify WebLogic Version

To verify WebLogic has the new patches, we can run two commands. The first command is to set the environment with:

%WL_HOME%serverbinsetWLSEnv.cmd

Then, run the command:

java weblogic.version

The output will look similar to this:

[code lang=”text”]
WebLogic Server Temporary Patch for BUG22248372 Tue Nov 24 00:35:04 MST 2015
WebLogic Server 10.3.6.0.12 PSU Patch for BUG20780171 THU JUN 18 15:54:42 IST2015
WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050
[/code]

Another option to check the version of WebLogic is using the Smart Update utility:

bsu -prod_dir=e:oraclewlserver_10.3 -status=applied -verbose -view

The output shows that we have applied the 10.3.6.0.12 and CVE patch. Now, restart all your web servers and start testing!

Maintenance Page with Backdoor Login

A standard requirement when doing PeopleSoft maintenance is to change the normal sign in page. When the system is offline, we often like to display a maintenance page that informs users that the system will be down and prevents them from logging in. This is done easily enough by changing the signin.html page, adding a message and removing the login form.

However, there is one more requirement that is a little tricky. Once the maintenance is complete, there are often tasks that need to be completed before handing the system back to the end users. These can be configuration changes, running batch processes or simply completing validation that everything was applied correctly and is in working order. How can the core team sign on to the system and complete these tasks, all while preventing end user access?

I recently came up with a bit of JavaScript that gave us a backdoor to the system during our last PeopleTools upgrade. What the script does is hide the login form on the sign in page, preventing login.  The trick was that our core team knew the key to the backdoor. After the sign in page would load, when they pressed Ctrl+Space the login form would be reveled. The combination of keys was only known to the team, so they were the only ones able to get in. Keep in mind if you had some crafty end users this could obviously be worked around, but it did the trick for us.

The JavaScript and HTML to accomplish this is listed below.  Adding the script and HTML changes to your signin.html page is pretty straight forward. First include the file containing the JavaScript, or write it inline. You then need to wrap your login form elements in a <div> with an id=loginbox. I would suggest starting before the <div> containing ptLabelUserid and ending after the <div> containing the submit button. Lastly, you need to add hideLogin(true); to the body onload attribute.

Keep in mind the key doesn’t have to be Ctrl+Space, it can be any key combination really.You will see in the comments of the script a link to information about other keycodes that can be used.

Updated: 10/11/2016
Instead of hard coding a true value for your hideLogin() parameter, why not use a Custom Property set in the Web Profile? You can create any Custom Property you would like, for example: login.isLoginHidden and set to true. Then reference the property in your signin.hmtl page like this: hideLogin("<%=login.isLoginHidden%>"). This will allow you to toggle the hide login functionality by updating the web profile and bouncing the server.

Click here to see a working demo. Enjoy!


#3 – HTTPS and WebLogic

In episode 3 of The PeopleSoft Administrator Podcast, Dan and Kyle talk about HTTPS. We discuss what HTTPS is and how to implement HTTPS with WebLogic. Dan shares how to mitigate against the newer SSL attacks, and tips and tricks to help manage certificates and simplify configuring HTTPS. We also review some of our PeopleTools 8.55 predictions after the OpenWorld presentations were released.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Links from this episode: