Advanced WebLogic Configuration with the DPK

Continuing on our DPK series, this week we’ll explore how to do advanced web server domain configuration. The psft_configuration.yaml file gives you many of the configuration options for your web domain. While these settings will help you configure your web domain, there are more changes to WebLogic most administrators make. One of those changes is configuring SSL/Certificates and Logging.

DPK and WSLT

Before the DPK, you could script configuration changes to WebLogic using the WebLogic Scripting language WSLT. WSLT is a great interface to WebLogic, and you can do quite a bit with it.

We use the psft_customizations.yaml file to document the WLST commands in a section called config_settings:. This section is not included in psft_configuration.yaml for the PIA, so it will be a new section for your files. The config_settings: section is included for app server and process scheduler sections. If you look at the file puppet\etc\modules\pt_config\tests\test_pia_create.pp you can see the PeopleTools team is using the config_settings section when creating a test domain.

In the psft_customizations.yaml file, the config_settings: section will go below the webserver_settings: section and above the site_settings: section. Everything under config_settings: is broken down into WSLT commands. To make a change to WebLogic, use the tree structure from your web domains’ config.xml file to identify a section to change. E.g, Servers/PIA/WebServer/PIA/WebServerLog/PIA will be the HTTP access log settings for the PIA server.

Under each WSLT header, you can include a key: value for any configurable option in WebLogic. For our Servers/PIA/WebServer/PIA/WebServerLog/PIA above, we’ll enable the access log:

config_settings:
  Servers/PIA/WebServer/PIA/WebServerLog/PIA:
    LoggingEnabled:                     true

When we run puppet apply, the Puppet code take this configuration and executes the WLST to make the change to WebLogic. After the puppet apply run, you can check the config.xml file to verify the updated configuration.

Using WSLT to Identify Config Values

If you don’t know what the path or values you want to change, you can spin up a WLST session and browse your web domain. To start WLST, run setWLSEnv.cmd script under your WL_HOME\server\bin folder. That will set up the environment variables so Java can find the WLST package.

Next, start up a WLST session with java weblogic.WLST. Your web server needs to be running to connect, and you’ll need the console username and password.

connect('system', 'Passw0rd', 't3://localhost:8000')

Once you connect to your domain, you can use cd() and ls() commands to navigate around the configuration tree. For example, this command will take us to the Access Log settings for the PIA.

cd('Servers/PIA/WebServer/PIA/WebServerLog/PIA')
ls()

You can do much more with WLST, but we’ll save that for another time.

SSL and Certificate Configuration

Now that we can configure a web domain via WLST commands, let’s set up SSL and a certificate for a domain. The DPK will do all the configuration, but it won’t update the pskey file. A good idea would be to prebuild your pskey file (using KeyStore Explorer) and be ready to copy into your domains piaconfig folder after the DPK runs.

This is what the config_settings section of the psft_customizations.yaml looks like to configure SSL:

config_settings:
  Servers/PIA: 
    CustomIdentityKeyStorePassPhrase:   Passw0rd
    CustomTrustKeyStorePassPhrase:      Passw0rd
    KeyStores:                          CustomIdentityAndCustomTrust
  Servers/PIA/SSL/PIA:
    ServerPrivateKeyAlias:              psadminio-2016
    ServerPrivateKeyPassPhrase:         Passw0rd
  Servers/PIA/WebServer/PIA/WebServerLog/PIA:
    LoggingEnabled:                     true

If you have configured SSL through the WebLogic console before, these configuraiton options should look familiar. We tell WebLogic to use a custom keystore (pskey), the password to get into the keystore, the Private Key alias and the password for the Private Key.

In production systems, it’s not a good idea to have your unecrypted password in the yaml files. For Windows, support for encrypting password in the yaml files isn’t there yet. For Linux, eyaml can encrypt your passwords.

This is what a psft_customizations.yaml looks like for a web server domain:

pia_domain_list:
  hr92dmo:
    os_user:               psadm
    ps_cfg_home_dir:       e:/psoft/hr92dmo
    gateway_user:          administrator
    gateway_user_pwd:      Passw0rd
    auth_token_domain:     ".%{::domain}"

    webserver_settings:
      webserver_type:           "%{hiera('webserver_type')}"
      webserver_home:           "%{hiera('weblogic_location')}"
      webserver_admin_user:     system
      webserver_admin_user_pwd: Passw0rd
      webserver_admin_port:     "%{hiera('pia_http_port')}"
      webserver_http_port:      "%{hiera('pia_http_port')}"
      webserver_https_port:     "%{hiera('pia_https_port')}"

    config_settings:
      Servers/PIA: 
        CustomIdentityKeyStorePassPhrase:   Passw0rd
        CustomTrustKeyStorePassPhrase:      Passw0rd
        KeyStores:                          CustomIdentityAndCustomTrust
      Servers/PIA/SSL/PIA:
        ServerPrivateKeyAlias:              psadminio-2016
        ServerPrivateKeyPassPhrase:         Passw0rd
      Servers/PIA/WebServer/PIA/WebServerLog/PIA:
        LoggingEnabled:                     true

    site_list:
      hr92dmo:
        appserver_connections: "%{hiera('pia_psserver_list')}"
        domain_conn_pwd:       "%{hiera('domain_conn_pwd')}"

        webprofile_settings:
          profile_name:        DEV
          profile_user:        PTWEBSERVER
          profile_user_pwd:    Passw0rd

        report_repository_dir: "%{hiera('report_repository_dir')}"

Update Existing Domains

Unlike other parts of the DPK, the WLST commands execute every time you run puppet apply .\site.pp. So, if you want to update a config value in your web domain, the DPK will make those changes for you after the domain was created. With the app server and process schedulers, the DPK doesn’t handle changes well. Once the domain is built, configuration changes are not applied to app and batch domains.

Repeatable Processes

The goal of the DPK, Puppet and other automation software is to create a dependable and repeatable process. Taking advantage of these features in the DPK will get you closer to the ideal of complete automation. The ability to issue WLST commands against WebLogic domains gives admins another tool with the DPK to build a system to your specifications.

Using the DPK to install only WebLogic

With the Deployment Packages, you can install an entire PeopleSoft system. But what if you want to install just one component? You can install the PIA role, but you get WebLogic, Tuxedo, Java, and a PIA domian. In this post, we’ll show how to leverage the DPK with custom Puppet manifests and install only WebLogic (and Java) on a server.

You will need to the DPK Puppet files on the server. Run the bootstrap script and stop at the “Default Initialization” question so nothing gets installed. You can use any parameters you want for the bootstrap script. We’ll be ignoring the site.pp file and using a new manifest file.

I’ll document the custom manifests in this post, but if you want to download the code you can grab it from GitHub. Copy the files in the project into the corresponding folders under the puppet\etc folder on your machine.

psft_customizations.yaml Updates

For the WebLogic install, I wanted the software to install in the e:/oracle folder. I added/updated these values in my psft_customizations.yaml file:

oracle_base:      e:/oracle

jdk_location:           "%{hiera('oracle_base')}/jdk1.7.0_95"
weblogic_location:      "%{hiera('oracle_base')}/bea"

You don’t need to use oracle_base and can ignore these changes if you want WebLogic installed to the default location.

weblogic.pp Manifest

Create a new manfiest under etc\manfiests (or copy the site.pp file) and name it weblogic.pp. In the weblogic.pp manifest, we are doing two things:

  1. Create the oracle_base folder (the DPK will fail if the base folder doesn’t exist)
  2. Call a custom DPK role (we’ll create this next)

Here is the weblogic.pp file:

# Verify the Oracle Base folder is present - DPK doesn't like it when the base folder isn't created.
file { hiera('oracle_base') :
    ensure => 'directory',
}

# Call custom profile to install WebLogic (and JDK)
node default {
  include ::pt_role::pt_tools_weblogic
}

If you’ve looked at the site.pp file you’ll recognize the second section. The first part of the file is Puppet code to create a folder if it doesn’t exist.

If you aren’t familiar with Puppet, the Puppet CookBook is a good place to get familiar with writing manifests.

WebLogic DPK Role

A new DPK role file was created under etc\modules\pt_role\manifests called pt_tools_weblogic.pp. From the post on the sites.pp file, we introduced the idea of DPK roles. The best practice is to only have one role per machine. For the WebLogic-only install, we’ll create a new role so its clear that this server is only running WebLogic.

class pt_role::pt_tools_weblogic inherits pt_role::pt_base {
  notify { "Applying pt_role::pt_tools_weblogic": }
  $ensure   = hiera('ensure')
  contain ::pt_profile::pt_weblogic
  Class['::pt_profile::pt_weblogic'] 
}

The role file uses the ensure attribute from the defaults.yaml file to determine if Puppet should install or remove WebLogic. The default value is to install (ensure = present).

Next, the role calls a new profile called pt_profile::pt_weblogic. Let’s create that file net.

WebLogic DPK Profile

A DPK profile handles the configuration lookup from Hiera and preparing Puppet for the installation (or removal). Our custom profile grabs the installation directory (oracle_base and weblogic_location) from the psft_customizations.yaml file. WebLogic requires Java too, so we configure that in the profile as well.

Create a new profile under etc\modules\pt_profile\manifests and call it pt_weblogic.pp.

# Custom Profile to prepare for WebLogic installation
class pt_profile::pt_weblogic {
  notify { "Applying pt_profile::pt_weblogic": }

  ## Hiera lookups
  $ensure                    = hiera('ensure')
  $env_type                  = hiera('env_type')

  $tools_archive_location      = hiera('archive_location')

  $jdk_hiera             = hiera('jdk')
  $jdk_location          = $jdk_hiera['location']
  $jdk_remove_value      = $jdk_hiera['remove']
  if $jdk_remove_value == false {
    $jdk_remove = false
  }
  else {
    $jdk_remove = true
  }
  notice ("JDK remove is ${jdk_remove}")

  $weblogic_hiera        = hiera('weblogic')
  $weblogic_location     = $weblogic_hiera['location']
  $weblogic_remove_value = $weblogic_hiera['remove']
  if $weblogic_remove_value == false {
    $weblogic_remove = false
  }
  else {
    $weblogic_remove = true
  }
  notice ("Weblogic remove is ${weblogic_remove}")

  $redeploy = hiera('redeploy', false)
  class { '::pt_setup::weblogic_deployment':
    ensure                 => $ensure,
    tools_archive_location => $tools_archive_location,
    inventory_location     => $inventory_location,
    jdk_location           => $jdk_location,
    jdk_remove             => $jdk_remove,
    weblogic_location      => $weblogic_location,
    weblogic_remove        => $weblogic_remove,
    redeploy               => $redeploy,
  }
  contain ::pt_setup::weblogic_deployment
}

WebLogic DPK Deployment

Finally, we’ll call the Puppet code to install Java and WebLogic! We are using two DPK types in this manifest:

  • pt_deploy_jdk
  • pt_deploy_weblogic

This is why the DPK needs to be installed before using these manifests. If the DPK is missing, these calls will fail (and so will some of the hiera lookups). The deployment manifest is under the folder etc\modules\pt_setup\manifests. Create the file weblogic_deployment.pp from the code below or from the GitHub project.

# Custom Puppet Class to deploy WebLogic using the DPK
class pt_setup::weblogic_deployment (
  $ensure                 = present,
  $deploy_pshome_only     = false,
  $tools_archive_location = undef,
  $tools_install_user     = undef,
  $tools_install_group    = undef,
  $oracle_install_user    = undef,
  $oracle_install_group   = undef,
  $db_type                = undef,
  $pshome_location        = undef,
  $pshome_remove          = true,
  $inventory_location     = undef,
  $oracleclient_location  = undef,
  $oracleclient_remove    = true,
  $jdk_location           = undef,
  $jdk_remove             = true,
  $weblogic_location      = undef,
  $weblogic_remove        = true,
  $tuxedo_location        = undef,
  $tuxedo_remove          = true,
  $ohs_location           = undef,
  $ohs_remove             = true,
  $redeploy               = false,
) {

  notice ("Applying pt_setup::weblogic_deployment")

  $jdk_tag          = 'jdk'
  $weblogic_tag     = 'weblogic'

  $jdk_archive_file      = get_matched_file($tools_archive_location, $jdk_tag)
  if $jdk_archive_file == '' {
    fail("Unable to locate archive (tgz) file for JDK in ${tools_archive_location}")
  }
  $weblogic_archive_file = get_matched_file($tools_archive_location, $weblogic_tag)
  if $weblogic_archive_file == '' {
    fail("Unable to locate archive (tgz) file for Weblogic in ${tools_archive_location}")
  }

  $jdk_patches = hiera('jdk_patches', '')
  if ($jdk_patches) and ($jdk_patches != '') {
    notice ("JDK patches exists")
    $jdk_patches_list = values($jdk_patches)
  }
  else {
    notice ("JDK  patches does not exists")
    $jdk_patches_list = undef
  }

  pt_deploy_jdk { $jdk_tag:
    ensure            => $ensure,
    deploy_user       => $tools_install_user,
    deploy_user_group => $tools_install_group,
    archive_file      => $jdk_archive_file,
    deploy_location   => $jdk_location,
    redeploy          => $redeploy,
    remove            => $jdk_remove,
    patch_list        => $jdk_patches_list,
  }

  $weblogic_patches = hiera('weblogic_patches', '')
  if ($weblogic_patches) and ($weblogic_patches != '') {
    notice ("Weblogic patches exists")
    $weblogic_patches_list = values($weblogic_patches)
  }

  pt_deploy_weblogic { $weblogic_tag:
    ensure                    => $ensure,
    deploy_user               => $tools_install_user,
    deploy_user_group         => $tools_install_group,
    archive_file              => $weblogic_archive_file,
    deploy_location           => $weblogic_location,
    oracle_inventory_location => $inventory_location,
    oracle_inventory_user     => $oracle_install_user,
    oracle_inventory_group    => $oracle_install_group,
    jdk_location              => $jdk_location,
    redeploy                  => $redeploy,
    remove                    => $weblogic_remove,
    patch_list                => $weblogic_patches_list,
    require                   => Pt_deploy_jdk['jdk'],
  }

}

The manifests under pt_setup will do the heavy work of installing new software. This custom manifest will deploy Java and WebLogic on our servers. If your bootstrap script didn’t create a dpk\archives folder, the installation will fail. The DPK manifests (and our custom one) will look for the archives before it attempts to install.

Installation

Once your custom manifests are built, go to the puppet\etc\manifests folder. Run the command

puppet apply .\weblogic.pp --trace --debug

to install WebLogic.

Future Improvements

When I did the test, I tried to follow the Roles and Profiles pattern used by the DPK. It might seem complex for a smaller installation. You could combine all this into one manifest, but then it gets ugly to maintain. When we add more software components, the abstraction between configuration and implementation allows us to re-use Puppet code.

This was a test for myself to see if I could break down some of the abstraction in the DPK and write (or modify) manifests to control more of the process. Plus, we have DPK role that we can use for WebLogic-only deployments 🙂

#26 – WebLogic Filters

This week we talk about HR Image 17, the new Security Automation tool, and share some comments from David Kurtz about PeopleSoft on 12c. Then, Kyle dives into WebLogic Servlet Filters and shares how filters can be used with PeopleSoft.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Script WebLogic and Java Patches

In December, we talked quite a bit about patching Java and WebLogic on the blog and podcast. There was a WebLogic CVE, and then a patch, to apply. If you want a recap on the CVE and patching process, here are the posts:

While applying the patches, I wanted to script the process so patching would be consistent across all our servers. I pulled the scripts into a GitHub project for sharing and reuse. If you haven’t scripted a WebLogic patch, this would be a place to start. The scripts use PowerShell and built for WebLogic 10.3.6. So, they use SmartUpdate instead of OPatch. I also added in a Java patch to the process too. You could pull out the Java patch script to use by itself. One more note: all the patches, Java, and scripts were set to run from the folder e:\installers\weblogic1036-2015-CVE-Patches. If you use these for your environment, or just use them as a template, you’ll want to update those paths for your specific configuration.

There is nothing ground-breaking about these scripts 🙂 I can write scripts, but I’m not the best script developer out there. If you see places where the scripts need improvement, file an issue with the project or submit a pull request! The main goal with this project and post is to get others started with scripting. Scripting, even if the scripts are basic, can benefit administrators. I hope that this quick overview might help someone get started.

Scripts Overview

These scripts are writtin in PowerShell. If PowerShell scripts are not enabled on the server, run this command to allow PowerShell scripts to run:

set-executionpolicy unrestricted

  1. Install new SmartUpdate version (3.3.0)

    installSmartUpdate.ps1

    The silent.xml file is used for a silent install (no prompts). The installation directory is set to e:\oracle. If you want a different directory, change the value for “BEAHOME”. 1. Stop all web servers running on the server .stopPIAServices.ps1 The script looks for any Windows service that containts “*-PIA” in the name. If you have any WebLogic domains were not created by the

    installNTService script, you may need to shut them down by hand.

  2. Prepare and copy files from the weblogic1036-2015-CVE-Patches folder

    prepareFiles.ps1

    This script performs tasks to prepare different files for patching: On our servers, two files needed updates to run the Smart Update utility. registry.xml needed to remove a reference to Tuxedo; bsu.cmd needed an increase in memory to the Java Heap. The registry.xml file also contains a reference to the server where it was installed. The script will change that value based on the new server’s name. The original files are backed up first and a .bkp extension is added to the file name. The script also copies jdk-1.7.0_79 to our e:\java folder. If you want the new java version in a different location, you can change the path in the file.

  3. Apply both WebLogic patches The patches we are applying resolve the December 2015 CVE with WebLogic. If you are using these scripts for future patches, you’ll want to update the patch ID’s in the script.

    applyWebLogicPatches.ps1

    Both patches are applied to WebLogic using the bsu command. The script assumes your patches are in the folder e:\patches\cve-2015-4852. NOTE: On one of our servers, the second patch stalled during the “Checking for Conflicts” step. If the script stalls for more than a few minutes, hit Cntl-C.

  4. Update the JAVA_HOME values

    updateJavaVersion.ps1

    The JAVA_HOME value in the setEnv.cmd script will be updated to the new path. You must update this script for each server. The paths in the script are hard-coded. (The hard coding is an obvious candidate to fix next. Should be able to use the Get-ChildItem cmdlet to find all the setEnv.cmd files.)

  5. Update Registry value for JAVA_HOME

    updateRegistryJavaVersion.ps1

    The JAVA_HOME value in the Registry for each web service will be updated. You must update this script for each server. The paths in the script are hard-coded. (Again, another place for improvement. Need to find a search cmdlet for the Registry. Could look for -PIA in the service name.)

  6. Start all web servers running on the server.

    startPIAServices.ps1

    Again, this looks for all Windows services that have *-PIA in the name and starts them. That’s it.

The scripts are pretty simple, and you can write a wrapper script to run all the sub-scripts. That way you’d have one script to kick off. Or, you could add these into a tool like Rundeck to execute from a centralized place. Once you start down the path of scripting, many opportunities open up to speed up everyday tasks.

#8 – PeopleTools 8.55 Hands-on

Merry Christmas and Happy Holidays! We have an extra long and fun episode this week but you may want to listen to this it twice 🙂

In Episode 8, we dive into PeopleTools 8.55 and get hands-on! We talk about our first experiences with the PeopleTools DPK’s, changes to Change Assistant in 8.55 and Dan’s experiment with the Let’s Encrypt project. We also ask a big question about a new Related Content feature, and then answer it after the break. This feature has the potential to substantially reduce your customizations!

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Let’s Encrypt with PeopleSoft

Let’s Encrypt is a service provided by the Internet Security Research Group to provide free SSL certificates to anyone. The goal of the project is get the entire web encrypted. I mentioned the project in Episode 7 of The PeopleSoft Administrator Podcast and thought it would be a great exercise to try it with PeopleSoft. Let’s Encrypt uses a client on the server to automate the certificate request process. The client will:
* Validate that you own the web server
* Generate a CSR
* Download the certificate
* Apply the certificate to the web server (limited support)
* Automatically renew the certificate

There are a few requirements to use the Let’s Encrypt clients though:

  • The web server needs to accessible by the internet. The Let’s Encrypt site will validate that you own the server by checking for a specific file on the web server.
  • Not all operating systems are supported, yet.
  • Some web server’s have built-in support (IIS, Apache), but others do not (e.g, WebLogic).

We can still generate certificates though, the automatic renewal won’t update the web server though.

Install Let’s Encrypt Client for Windows We’ll use the

letsencrypt-win-simple command line client for Windows. Download the latest release from GitHub and extract the folder to a permanent location.

Generate a new certificate

  1. Run .letsencrypt.exe --accepttos

    Let's Encrypt (Simple Windows ACME Client) ACME Server: https://acme-v01.api.letsencrypt.org/ 
    Config Folder: C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org 
    Loading Signer from C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orgSigner 
    Getting AcmeServerDirectory Loading Registration from C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org
    Registration Scanning IIS 7 Site Bindings for Hosts 
    No IIS bindings with host names were found. Please add one using IIS Manager. A host name and site path are required to verify domain ownership. 
    No targets found. 
    M: Generate a certificate manually. 
    A: Get certificates for all hosts 
    Q: Quit 
    Which host do you want to get a certificate for: 
    
  2. Since we are not running IIS, we’ll generate a certificate manually.

    Which host do you want to get a certificate for: M Enter a host name: 
    
  3. Enter the DNS name for your web server.

    Enter a host name: hr.psadmin.io 
    Enter a site path (the web root of the host for http authentication): 
    
  4. Next, enter the root path for your web server. If you are running WebLogic, that will be PORTAL.war directory on your web server.

    Enter a site path (the web root of the host for http authentication): W:\pt8.55\webserv\peoplesoft\applications\peoplesoft\PORTAL.war
    
  5. Then, the Let’s Encrypt client will create a new file under PORTAL.war.well-knownacme-challenge. That file will be used to validate that you own the web server.

    Authorizing Identifier ps92t855.psadmin.io 
    Using Challenge Type http-01 Writing challenge answer to W:\pt8.55\webserv\peoplesoft\applications\peoplesoftPORTAL.war.well-known/acme-challenge /1c2yN7Y93sJwRUmRGaoG4kT-QynrIcGr4szre-3nTsQ 
    Answer should now be browsable at http://ps92t855.psadmin.io/.well-known/acme-challenge/1c2yN7Y93sJwRUmRGaoG4kT-QynrIcG r4szre-3nTsQ 
    Submitting answer Refreshing authorization Authorization Result: valid 
    Deleting answer 
    
  6. After the web server ownership is verfied, new certificates will generated and copied to your system. The certificates are copied to your %USERPROFILE%AppDataRoamingletsencrypt-win-simple folder in a few formats:

    • .der
    • .pem
    • .pfx

The client will also add the certificates to the Windows Certificate Store for you. To add the certificates to WebLogic, we’ll use the .pem

    Requesting Certificate Request Status: Created 
    Saving Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orghr.psadmin.io-crt.der 
    Saving Issuer Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orgca-009813F47513E5750B43E7431E971E44BD-crt.pem 
    Saving Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orghr.psadmin.io-all.pfx (with no password set) 
    Opened Certificate Store "WebHosting" 
    Adding Certificate to Store 
    Closing Certificate Store 
    WARNING: Unable to configure server software. 
    Creating Task letsencrypt-win-simple httpsacme-v01.api.letsencrypt.org with Windows Task Scheduler at 9am every day. 
    Renewal Scheduled Manual hr.psadmin.io (W:pt8.55webservpeoplesoftapplicationspeoplesoftPORTAL.war) 
    Renew After 2/9/2016 
    Press enter to continue. 

Create a New pskey Keystore Now that we have certificates, let’s create a new

pskey file with the certificates. We’ll use Keystore Explorer to quickly generate the file.

  1. Open Keystore Explorer. (If it’s first time you’ve used it, follow the instructions to download the Unlimited Strength files).
  2. Create a new keystore file.
  3. Select the file type of “JKS”.
  4. Select “Tools > Import Key Pair”.
  5. Select the “OpenSSL” option.
  6. Deselect “Encrypted Private Key”.
  7. For the “OpenSSL Private Key File”, select the file hr.psadmin.io-key.pem.
  8. For the “Certificate(s) File”, select hr.psadmin.io-cert.pem
  9. Click “Import”.
  10. Enter an alias name that is descriptive. I used hr.psadmin.io-2015-12.
  11. Since the prive key was delivered without a password, we’ll want to enter one. Enter a password for the key pair.

Now you have the private and public key for your DNS entry in the keystore. Next, we need to add the root (and intermediate) certificates so that a chain of trust is established.

  1. In Keystore Explorer in our new keystore file, right-click on our certificate. Select “Edit Certificate Chain > Append Certificate”.
  2. Select the file ca-GUID-crt.pem and click “Append”.
  3. Save the file, give the keystore a password, and name the file pskey-2015-12.

Load Keystore into WebLogic After importing the certificates into

pskey-2015-12, we need to copy the file to the web server and tell WebLogic to use the new file. The integrationGateway.properties file will need to know about the new keystore as well.

  1. Copy the pskey-2015-12 file to your web server directory %PS_CFG_HOME%\webserv\peoplesoft\pia\config\keystore.
  2. Log into the WebLogic console.
  3. Navigate to “Environment > Servers > PIA > Keystores”.
  4. Click the “Lock & Edit” button to allow editing.
  5. Click the “Change” button for the Keystores option.
  6. Select “Custom Identity and Custom Trust” and “Save”.
  7. In the “Custom Identity Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  8. In the “Custom Identity Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  9. In the “Custom Trust Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  10. In the “Custom Trust Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  11. Click Save. WebLogic will look at the new keystore file. Next, we need to tell WebLogic certificate it should serve to users.

  12. Click on the “SSL” tab.

  13. Change the “Private Key Alias” to hr.psadmin.io-2015-12.
  14. In the “Private Key Passphrase” boxes, enter the password you gave the hr-psadmin.io-2015-12 keypair.
  15. Click Save.
  16. Click the “Activate Changes” button.

Update integrationGateway.properties

Before we reboot the WebLogic domain, we need to update the integrationGateway.properties file.

  1. On your web server, open the integrationGateway.properties file under %PS_CFG_HOME%\webserv\peoplesoft\applications\peoplesoft\PSIGW.war\WEB-INF.
  2. Find the line secureFileKeystorePath and change file name to pskey-2015-12.
  3. If the password you gave the keystore is different than the previous file, you’ll need to update that parameter in the file.
    1. Open a command prompt and go to %PS_CFG_HOME%\webserv\peoplesoft\bin.
    2. Run the command setEnv.cmd to set the environment variables.
    3. Go to the folder piabin.
    4. Run the command PSCipher to get the encrypted text.
  4. Restart your WebLogic domain.

Test your HTTPS Connection

As WebLogic is starting up, make sure to check the logs to verify that the server started with your new certificate. Once the server has started, open a browser and go test the site. You should see a secure connection in the browser to your site.

How to Apply WebLogic Patches – Part 2

In Part 1, I showed how to use Smart Update to patch WebLogic. Starting with WebLogic 12.1.2, OPatch handles all the pacthing. Let’s walk though using OPatch to update WebLogic to fix the latest vulnerability. OPatch is included in the WebLogic install, so everything you need to apply patches is ready to go.

Windows Path Limit

If you are on Windows and applying patches 21370953 and 22250567, you may run into an error The file name(s) would be too long for the destination folder. The patch contains so may folders that they conflict with the Windows limit of 260 characters for a file name. The work around is to use the jar utility that comes with the JDK to unzip the patch. jar -xvf p21370953_121300_Generic.zip

Set OPatch Environment

OPatch needs to know what ORACLE_HOME you are applying the patch to. Depending on your server configuration, you may need to set ORACLE_HOME to the directory that contains WebLogic.

set ORACLE_HOME=e:\middleware-854

Let’s Fix CVE-2015-4852

Since we have new patches to fix CVE-2015-4852 (T3/Java Deserialization), let’s use those as our example.Go to this page to find the applicable patch (or patches if you are on 10.3.6) to apply.

Extract Patches

Download the patches you need and unzip them. I put the patch files under

e:\patches\cve-2015-4852

on the web server.

set PATCH_TOP=e:\patches\cve-2015-4852
unzip -d %PATCH_TOP p21370953_121300_Generic.zip
unzip -d %PATCH_TOP p22250567_121300_Generic.zip

Apply Patches

Make sure all of your web server instances are shut down. Then, move into the first patch folder so it is your current directory. Once you are in the patch folder, we call OPatch.

cd patches\cve-2015-485221370953
e:\middleware-854\OPatch\opatch apply

At the end of the patch, you should see a OPatch succeeded message. Let’s apply the second patch.

cd patches\cve-2015-485222250567
e:\middleware-854\OPatch\opatch apply 

At the end of the patch, you should see a OPatch succeeded message.

Verify WebLogic Version

To verify WebLogic has the new patches, we use OPatch’s lsinventory command.

e:\middleware-854\OPatch\opatch lsinventory

The output will look similar to this:

Interim patches (2) : Patch 22250567 : applied on Fri Dec 11 07:46:45 CST 2015 
Unique Patch ID: 19584835 
Patch description: "One-off" Created on 22 Nov 2015, 01:36:21 hrs PST8PDT 
Bugs fixed: 22175246, 22200449, 22247869, 21495475 
This patch overlays patches: 21370953 
This patch needs patches: 21370953 as prerequisites 
Patch 21370953 : applied on Fri Dec 11 07:46:45 CST 2015 
Unique Patch ID: 19198495 
Patch description: "WebLogic Server 12.1.3.0.5 PSU Patch for BUG21370953 October 2015" 

The output shows that we have applied the CVE patches. Now, restart all your web servers and start testing!

How to Patch Java in WebLogic

With the recent attacks on SSL, WebLogic and Java, I wanted to give an overview on how you patch Java for your WebLogic instances.

When you install WebLogic, it asks you for the location of your Java Home. Then, every web server instance you create uses that Java Home. Unless you have patched Java in the past, all of the WebLogic instances on that server will be using the old Java Home.

Download the New JDK

Go to Oracle’s Java Download page and download the latest JDK. Make sure to select the correct codeline that your version of PeopleTools supports.

PeopleTools 8.53-8.55 support Java 1.7 (aka Java 7). It will implicity support any patch on the 1.7 codeline. So, you can install the latest 1.7.0_xx patch and use it with WebLogic and PeopleTools.

Install the new JDK (you don’t need the JRE) to a common location. We use the folder convention e:javajdk-1.7.0_xx to install the JDK.

Update the commEnv Script

By default, the Java Home parameter is set in the %WL_HOME%commEnv script. This script configures environment variables that are common to all WebLogic instances on the server.

You can update the JAVA_HOME in the commEnv script, but it will affect ALL the WebLogic instances on that machine. This might be what you are looking for. But, if you run more that one web server you might want to try the next option.

Update the setEnv Script

In each WebLogic instance you create (hr92dmo, hr92dev, etc), the file %PIA_HOMEbinsetEnv will set the configuration that applies only to a specific domain. This is where I prefer to set JAVA_HOME. In the file, you will find a line that says:

@REM JAVA_HOME is set via commEnv.sh, to override set it here.

Simply add this line to set a JAVA_HOME for a specific web server:

set JAVA_HOME=e:javajdk-1.7.0_79

Now, you can patch your Demo environment and test without affecting other web servers on the server.

Update the Windows Service

If you are on Windows and installed a service for your web server, you will need to change the JAVA_HOME value for the service. You could re-create the service but there is an easier way.

Open the Registry Editor (regedit) and navigate to:

HKLM:SYSTEMCurrentControlSetserviceshr92dmo-PIAParameters 

Under this registry folder, you’ll see a Key name “JavaHome”. Update the value’s path to match your new JAVA_HOME. Restart the service for the change to take affect.

Patching Java for WebLogic is pretty simple. The next step (and upcoming blog post) will be to script these changes, and WebLogic patches, so you can automate your web server patching.

How to Apply WebLogic Patches

Oracle has released a patch for the latest CVE against WebLogic, so I wanted to walk though the steps to apply the patch to WebLogic and show how to use Smart Update. Smart Update is the utility used by WebLogic to apply patches to your installation.

UPDATE 12/8/2015 Thanks to Matt Tremblay for pointing out, WebLogic 12.1.2+ is now using OPatch for WebLogic patching. Look for an second WebLogic patching post soon about using OPatch with WebLogic.

Smart Update 3.3.0

Version 3.3.0 is the latest version and is included with WebLogic 10.3.5 and later. If you launch Smart Update and find that its an older version, go grab version 3.3.0 (Patch 12426828).

Launching Smart Update

On Windows, if you chose to create a Program Group, you can launch Smart Update from the Start Menu under the “Oracle WebLogic” folder. Or, you can launch it from the command line:

%BEA_HOME%utilsbsubsu.cmd

The first time you run Smart Update, it may ask you to provide a %BEA_HOME% path. Give the path to your BEA Home (e.g, e:oracle).

If you receive an error: “Unable to locate any supported product installations” or “The BEA Home directory selected does not contain any supported patch targets”, check out MOS Documents 946541.1 or 1063605.1 for the fix.

Applying Patches

In the Smart Update window, you will see the installed applications in the left pane. Make sure “WebLogic” is selected. On the right, the top pane shows you patches that have been applied. The lower pane displays patches in your download directory that are waiting to be applied.

By default, Smart Update will look for patches under %BEA_HOME%utilsbsucache_dir for patches. To start Smart update and have it look at a different path, use the -patch_download_dir=[path] flag or select File > Preferences to change the directory.

To apply a patch, click the green arrow in the “Downloaded Patches” pane for the patch. Smart Update will check for patch conflicts and the apply the patch.

Command Line

You can also run Smart Update from the command line. This is great when you have multiple servers to patch. Running bsu.cmd -help will give you all the options you need when scripting.

Let’s Fix CVE-2015-4852

Since we have new patches to fix CVE-2015-4852 (T3/Java Deserialization), let’s use those as our example. Go to this page to find the applicable patch (or patches if you are on 10.3.6) to apply.

Extract Patches

Download the patches you need and unzip them. Copy the .jar and .xml files from the patch folders to your web server. I put the patch files under e:patchescve-2015-4852 on the web server. We will tell Smart Update to use this directory. (Since we are running WebLogic 10.3.6, there are two patches to install in our example.).

Apply Patches

Before you apply any patches, make sure to stop any web servers running on the server. If don’t, Smart Update won’t be able to patch .jar files that are in use.

Let’s run Smart Update from the command line. Open a command prompt and navigate to %BEA_HOME%utilsbsu. We need to pass these values to the bsu program:

  • -install
  • -patch_download_dir=e:patchescve-2015-4852
  • -patchlist=EJUW (note, this is not the patch number, but the PSU Patch ID)
  • -prod_dir=e:oraclewlserver_10.3
  • -verbose

So, my command to apply the first CVE patch looks like this:

bsu -install -patch_download_dir=e:patchescve-2015-4852 -patchlist=EJUW -prod_dir=e:oraclewlserver_10.3 -verbose

I had to change the memory settings for Smart Update. In the bsu.cmd file, I modified the set MEM_ARGS line:

set MEM_ARGS=-Xms512m -Xmx1024m -XX:PermSize=64m -XX:MaxPermSize=128m -Xss512k

Smart Update will give you a “Success” message, or an error message. Next, let’s apply the second CVE patch:

bsu -install -patch_download_dir=e:patchescve-2015-4852 -patchlist=ZLNA -prod_dir=e:oraclewlserver_10.3 -verbose

Verify WebLogic Version

To verify WebLogic has the new patches, we can run two commands. The first command is to set the environment with:

%WL_HOME%serverbinsetWLSEnv.cmd

Then, run the command:

java weblogic.version

The output will look similar to this:

[code lang=”text”]
WebLogic Server Temporary Patch for BUG22248372 Tue Nov 24 00:35:04 MST 2015
WebLogic Server 10.3.6.0.12 PSU Patch for BUG20780171 THU JUN 18 15:54:42 IST2015
WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050
[/code]

Another option to check the version of WebLogic is using the Smart Update utility:

bsu -prod_dir=e:oraclewlserver_10.3 -status=applied -verbose -view

The output shows that we have applied the 10.3.6.0.12 and CVE patch. Now, restart all your web servers and start testing!

Smart v. Dumb Load Balancing

Setting up Load Balancing is common for production systems. Lately, I have seen a few organizations set up their load balancing so they end undoing all their good intentions. At the top of the infrastructure is a smart device, the load balancer, that can look at traffic and the servers underneath it to see where the new connections should be assigned. You want to the load balancer to decide where traffic goes so your web and app servers can focus on processing requests. Often times, the web server is also set up to load balance connections as well.

In the configuration.properties file, the psserver parameter will automatically use a round robin method to assign connections to any server listed. I call this dumb load balancing because there is no logic to assigning traffic. There are many reasons why multiple servers are listed in the psserver parameter, but there are new changes in PeopleTools that let us make smarter decisions with load balancing. First, let’s cover the most common load balancing algorithms to see why they are smart. Next, we’ll talk about the load balancing for the app server layer and the changes we should make. Then, we’ll put it all together and explore different options for optimal load balancing.

Algorithms

Load balancers have many algorithms to choose how traffic is divided, but these three are the most common:

  • Round Robin The Round Robin method assigned connections down a circular queue. Round Robin is naive about traffic and there are no decisions on which servers are assigned connections. If the server is next in the queue, the load balancer will assign the connection. Round robin does not look at traffic load or performance, so it can lead to unbalance server loads.
  • Least Connection The Least Connection method tracks connections to each pool member and assigns new connections to the server with the fewest connections. This algorithm will keep the connection levels on each server as close as possible. Least connections attempts to balance server load by keeping the connection counts the same. (This article from F5)[https://devcentral.f5.com/articles/back-to-basics-least-connections-is-not-least-loaded] has a great overview on Least Connections.
  • Fastest Connection The Fastest Connection method tracks server response times and assigns new connections to the pool members with the fast responses. Fasted Connection prioritizes speed and can be useful for environments where performance is a top priority. The assumption is the fastest server has the capacity to handle more connections. With Fastest Connection, you want to set a connection limit on the pool members. You don’t want to saturate one server with too many connections. More on Fastest Connection.

App Server

“Load Balancing” When people configure the web server, they want to make sure anyone who connects to that web server can reach an application server.

In the psserver parameter, they list multiple application servers. The web server will use a round robin algorithm to “balance” the load. This also gives the web server failover capabilities. If app server 1 is down, the web server will send all the connections to app server 2 and 3.

Smart v Dumb Load Balancing RoundRobin

That diagram looks busy – it must be a good design! 😉

In earlier versions of PeopleTools and WebLogic, this was the only way to offer failover at the web layer (without putting another load balancer between the web and app server). This worked to get failover from the web to app server layers, but it removes any smart load balancing. The other option was to go 1 to 1 between the web and app server. So web server 1 points only to app server 1, web server 2 point only to app server 2. You’re smart load balancing will work much better, but if app server 2 goes down, any user routed to web server 2 will get error messages. That’s not a good user experience.

Smart v Dumb Load Balancing 1to1

Smart Load Balancing

Starting with PeopleTools 8.50, you can take advantage of Jolt Failover groups. Jolt Failover groups let you keep the smart decisions at the load balancer but still keep app server failover on the web server layer. To use Jolt Failover, use this syntax in the psserver parameter in the configuration.properties file:

server1:9000{server2:9000;server3:9000}

The web server will only assign connections to server1 while it is up and responsive. When the server1 goes down, the web server will move to server2. If server2 goes down, the web server will send connections to server3. The failover group also supports round robin within the failover group. Instead of a semi-colon, use a comma. E.g,

server1:9000{server2:9000,server3:9000}

You can also you weighted connections for round-robin. This is useful if one server is more powerful than others. E.g,

server1:9000{server2:9000#5,server3:9000#3}

In this case, if server1 is down, server2 will receive 5 connections for every 3 connections assigned to server3.

Smart v Dumb Load Balancing Failover

The main advantage of using failover groups is to keep the smart decisions for assigning traffic at the load balancer. You can read more about Jolt Configuration options in PeopleBooks.