#58 – Pagelets and Complaints

This week on the podcast, Dan and Kyle talk about a ransomware attack, load balancer health checks, applying POC patches and complain about minor annoyances in Change Assistant. Kyle shares a story about a misbehaving pagelet and how he investigated the issue.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Monitoring WebLogic and Java

Lately, I have had interest in monitoring WebLogic’s performance. Since Weblogic is built on Java, there are some standard tools we can use to look into the Java Virutal Machine (JVM). We’ll cover two of those tools in this post: JConsole and VisualVM. Both JConsole and VisualVM are included in the Java Development Kit so they are already on your server. These tools will give you information about the JVM used to run WebLogic and can help you tune you web servers.

JMX

To get monitoring data out of WebLogic’s JVM, we need to enable JMX. Java Management Extensions (JMX) is a monitoring technology built into Java. Applications that run on Java can build instrumentation into the application to provide performance data about the application. Even without additional data, the JVM will provide CPU, Memory, Thread and other stats about the heap.

To enable JMX for WebLogic, we’ll update the setEnv.cmd or setEnv.sh file. At the end of the JAVA_OPTIONS line, add these flags:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

There are 4 flags we pass to Java when the JVM is started:

  1. Enabling JMX Remote connections
  2. JMX Remote Port
  3. Requiring SSL Connections
  4. Authentication for JMX

For testing, I’ve turned off SSL and Authentication, but if you are enabling JMX across all your servers I recommend you turn both options on. For the JMX Port, pick a port value that is unique for each instance of the JVM. Each WebLogic and App Server instance will have its own JVM. For more information on configuring JMX, this is a link to the official documentation.

If your are on Windows and updated the setEnv.cmd file, you will want to re-install the Service that starts the PIA domain. The JAVA_OPTIONS parameters are stored in the registry when you create the service. If you update setEnv.cmd, you need to recreate the service (or manually update the registry).

Now that JMX is enabled on our domains, let’s look at a few tools to help us monitor our JVMs.

JConsole

JConsole is a utility included with the JDK download. Under JAVA_HOME\bin you’ll find jconsole.exe To start, we’ll run JConsole from our web server where we enabled JXM (instead of our desktop). Open JConsole and it will ask you to connect to a JMX Process. You have two options: Local Process and Remote Process. We’ll use the remote process option and use these values to connect to our web server: localhost:8888. We don’t need a username or password since we passed the flag jmxremote.authentication=false.

jconsole01

After connecting, you’ll get a message asking about your insecure connection. Click “Insecure” to continue. On the main page, we see 4 graphs related to the JVM.

jconsole03

These graphs give you a good overview of the JVM status. The CPU graph will show you how much of the CPU that JVM is using, and the Threads graph gives a good indication of workload on the JVM. The best part of JMX is the Memory graph. Getting your JVM Heap sized correctly can make a big different in performance. The graph should follow a pattern when Garbage Collection runs.

jconsole04

You don’t want Garbage Collection to run too often, or the usage too high after Garbage Collection. This graph helps with getting the right size for your web server. (You can find more tuning information here.)

VisualVM

VisualVM is another untility included with the JDK download and is also under JAVA_HOME\bin. We’ll start VisualVM on the server as well by running jvisualvm.exe --console new.

visualmv02

When VisualVM opens, we create a new connection by right-clicking on “Local” and selecting “Add JMX Connection”. Fill in the port number and select “Do not require SSL connection”.

visualvm03

VisualVM show us similar data as JConsole, but I think it looks a nicer. Under the Monitor tab, you can also force the JVM to run a Garbage Collection. For the most part, these two applications are similar.

visualvm03b

Remote JMX Connections

We have run both applications on the server to connect to JMX, but these applications are more useful if we can connect to the servers remotely. By default, JMX will only accept local connections. To enable remote connections to JMX, we have to pass this flag:

 -Dcom.sun.management.jmxremote.local.only=false

After you add that parameter to your setEnv.cmd JAVA_OPTIONS line, restart the web server. On a different computer or server, launch VisualVM or JConsole. Create a remote connection to JMX on the server. In the Connection box, enter the server name and port for the JMX instance.

visualvm05

JMX Authentication

Once you get the basic configuration in place, you want to enable authentication to connect to the JMX instance. The default JMX authentication is stored in the JDK folder. That will affect all domains using the JDK folder. Instead, we will use a JMX password file for each web server domain.

  1. Open the file JAVA_HOME\jre\lib\management\jmxremote.access.
  2. Add the line psMonitor readonly to the bottom of the file and save. This line adds a new user named psMonitor and a read-only account to any JMX instances using this JAVA_HOME.
  3. Copy the file JAVA_HOME\jre\lib\management\jmxremote.password.template to PS_CFG_HOME\webserv\jmxremote.password.
  4. Open the new jmxremote.password file.
  5. Add the line psMonitor test123 to the bottom of the file and save. This line sets the password for the psMonitor user. To give each web server domain a different password, set a unique password in this file under each PS_CFG_HOME.
  6. Open the setEnv.cmd file and add these parameters:

    -Dcom.sun.management.jmxremote.password.file=PS_CFG_HOME\webserv\jmxremote.password -Dcom.sun.management.jmxremote.authenticate=true
    
  7. Restart the web server for the new paramters to take affect.

Now that we have a web server configured to run JMX with authentication, we will create another connection in VusualVM to use the username and password.

  1. Right-click on the remote server and select “Add JMX Connection”
  2. Enter the server name and port.
  3. Enter psMonitor for the Username and test123 for the Password.
  4. Select “Do no require SSL connection”
  5. Click OK.

jmxauth01

jmxauth02

Load Balancers and Client IP Addresses

One disadvantage of running PeopleSoft behind a load balancer is that it can hide your user’s real IP address. With a load balancer, WebLogic and PeopleSoft log the load balancer’s IP address instead of the client connecting to your system. Load balancers support an HTTP header called X-Forwarded-For, and will populate that HTTP header with the client’s IP address. We can configure WebLogic to read this header and add that to our HTTP access log.

The psaccesslog table tracks logins into PeopleSoft, but a customization is required to change the logged IP address to read from X-Forwarded-For. We won’t over that here. Instead, we’ll show a configuration change to capture this information.

X-Forwarded-For

The X-Forwarded-For HTTP header is designed to pass in the HTTP originating client IP address through the load balancer. Logging a client IP address helps you identify where users are coming from (geo-ip) and can help with auditing access to your system. Since organizations run many different load balancers, we won’t cover how to enable that. Ask your network administrators to enable and populate X-Forwarded-For and pass it through to your WebLogic servers.

If you want to test the logging without getting a network admin involved, FireFox has a nice add-on, X-Forwarded-For Header that simulates the header for you.

WebLogic Access Log

Out of the box, WebLogic doesn’t capture the X-Forwarded-For HTTP header. To read that header, we can configure the HTTP Access Log to read the header and log it. To do that, we need to enable the Access Log and tell WebLogic what header to look for.

EnableXFF

  1. Open http://server:port/console and login.
  2. Go to “Environment > Servers > PIA > Logging > HTTP”.
  3. Click “Lock & Edit”.
  4. Select the checkbox for “HTTP access log file enabled”.
  5. Save the changes.
  6. Expand the “Advanced” section.
  7. Change the Format to Extended.
  8. Add cs(X-Forwarded-For) to the Extended Logging Format Fields.
  9. Set the Log File Buffer to 0. (This will write entires immediately to the log file.)
  10. Save the changes.
  11. Click the “Release Configuration” button.
  12. Restart the web server.

In the PIA_access.log file, you should see a dash - or an IP address. If the header is blank, the dash will be populated. Otherwise, WebLogic will log the client’s real IP address.

Here a sample line from the PIA_access.log file after we enabled X-Forwarded-For logging:

#Version:   1.0
#Fields:    date time cs-method cs-uri sc-status cs-username cs(user-agent) s-ip c-ip time-taken bytes cs(X-Forwarded-For)
#Software:  WebLogic

2016-11-23  20:49:16    GET /psc/ps/EMPLOYEE/ELM/c/NUI_FRAMEWORK.PT_LANDINGPAGE.GBL 200 -   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"    0.0.0.0:10020   10.0.1.174  5.484   10907   "10.0.1.222"

The last value, "10.0.1.222" is the client IP address (my workstation). The c-ip value (10.0.1.174) is the load balancer IP adress in this scenario.

Advanced WebLogic Configuration with the DPK

Continuing on our DPK series, this week we’ll explore how to do advanced web server domain configuration. The psft_configuration.yaml file gives you many of the configuration options for your web domain. While these settings will help you configure your web domain, there are more changes to WebLogic most administrators make. One of those changes is configuring SSL/Certificates and Logging.

DPK and WSLT

Before the DPK, you could script configuration changes to WebLogic using the WebLogic Scripting language WSLT. WSLT is a great interface to WebLogic, and you can do quite a bit with it.

We use the psft_customizations.yaml file to document the WLST commands in a section called config_settings:. This section is not included in psft_configuration.yaml for the PIA, so it will be a new section for your files. The config_settings: section is included for app server and process scheduler sections. If you look at the file puppet\etc\modules\pt_config\tests\test_pia_create.pp you can see the PeopleTools team is using the config_settings section when creating a test domain.

In the psft_customizations.yaml file, the config_settings: section will go below the webserver_settings: section and above the site_settings: section. Everything under config_settings: is broken down into WSLT commands. To make a change to WebLogic, use the tree structure from your web domains’ config.xml file to identify a section to change. E.g, Servers/PIA/WebServer/PIA/WebServerLog/PIA will be the HTTP access log settings for the PIA server.

Under each WSLT header, you can include a key: value for any configurable option in WebLogic. For our Servers/PIA/WebServer/PIA/WebServerLog/PIA above, we’ll enable the access log:

config_settings:
  Servers/PIA/WebServer/PIA/WebServerLog/PIA:
    LoggingEnabled:                     true

When we run puppet apply, the Puppet code take this configuration and executes the WLST to make the change to WebLogic. After the puppet apply run, you can check the config.xml file to verify the updated configuration.

Using WSLT to Identify Config Values

If you don’t know what the path or values you want to change, you can spin up a WLST session and browse your web domain. To start WLST, run setWLSEnv.cmd script under your WL_HOME\server\bin folder. That will set up the environment variables so Java can find the WLST package.

Next, start up a WLST session with java weblogic.WLST. Your web server needs to be running to connect, and you’ll need the console username and password.

connect('system', 'Passw0rd', 't3://localhost:8000')

Once you connect to your domain, you can use cd() and ls() commands to navigate around the configuration tree. For example, this command will take us to the Access Log settings for the PIA.

cd('Servers/PIA/WebServer/PIA/WebServerLog/PIA')
ls()

You can do much more with WLST, but we’ll save that for another time.

SSL and Certificate Configuration

Now that we can configure a web domain via WLST commands, let’s set up SSL and a certificate for a domain. The DPK will do all the configuration, but it won’t update the pskey file. A good idea would be to prebuild your pskey file (using KeyStore Explorer) and be ready to copy into your domains piaconfig folder after the DPK runs.

This is what the config_settings section of the psft_customizations.yaml looks like to configure SSL:

config_settings:
  Servers/PIA: 
    CustomIdentityKeyStorePassPhrase:   Passw0rd
    CustomTrustKeyStorePassPhrase:      Passw0rd
    KeyStores:                          CustomIdentityAndCustomTrust
  Servers/PIA/SSL/PIA:
    ServerPrivateKeyAlias:              psadminio-2016
    ServerPrivateKeyPassPhrase:         Passw0rd
  Servers/PIA/WebServer/PIA/WebServerLog/PIA:
    LoggingEnabled:                     true

If you have configured SSL through the WebLogic console before, these configuraiton options should look familiar. We tell WebLogic to use a custom keystore (pskey), the password to get into the keystore, the Private Key alias and the password for the Private Key.

In production systems, it’s not a good idea to have your unecrypted password in the yaml files. For Windows, support for encrypting password in the yaml files isn’t there yet. For Linux, eyaml can encrypt your passwords.

This is what a psft_customizations.yaml looks like for a web server domain:

pia_domain_list:
  hr92dmo:
    os_user:               psadm
    ps_cfg_home_dir:       e:/psoft/hr92dmo
    gateway_user:          administrator
    gateway_user_pwd:      Passw0rd
    auth_token_domain:     ".%{::domain}"

    webserver_settings:
      webserver_type:           "%{hiera('webserver_type')}"
      webserver_home:           "%{hiera('weblogic_location')}"
      webserver_admin_user:     system
      webserver_admin_user_pwd: Passw0rd
      webserver_admin_port:     "%{hiera('pia_http_port')}"
      webserver_http_port:      "%{hiera('pia_http_port')}"
      webserver_https_port:     "%{hiera('pia_https_port')}"

    config_settings:
      Servers/PIA: 
        CustomIdentityKeyStorePassPhrase:   Passw0rd
        CustomTrustKeyStorePassPhrase:      Passw0rd
        KeyStores:                          CustomIdentityAndCustomTrust
      Servers/PIA/SSL/PIA:
        ServerPrivateKeyAlias:              psadminio-2016
        ServerPrivateKeyPassPhrase:         Passw0rd
      Servers/PIA/WebServer/PIA/WebServerLog/PIA:
        LoggingEnabled:                     true

    site_list:
      hr92dmo:
        appserver_connections: "%{hiera('pia_psserver_list')}"
        domain_conn_pwd:       "%{hiera('domain_conn_pwd')}"

        webprofile_settings:
          profile_name:        DEV
          profile_user:        PTWEBSERVER
          profile_user_pwd:    Passw0rd

        report_repository_dir: "%{hiera('report_repository_dir')}"

Update Existing Domains

Unlike other parts of the DPK, the WLST commands execute every time you run puppet apply .\site.pp. So, if you want to update a config value in your web domain, the DPK will make those changes for you after the domain was created. With the app server and process schedulers, the DPK doesn’t handle changes well. Once the domain is built, configuration changes are not applied to app and batch domains.

Repeatable Processes

The goal of the DPK, Puppet and other automation software is to create a dependable and repeatable process. Taking advantage of these features in the DPK will get you closer to the ideal of complete automation. The ability to issue WLST commands against WebLogic domains gives admins another tool with the DPK to build a system to your specifications.

Using the DPK to install only WebLogic

With the Deployment Packages, you can install an entire PeopleSoft system. But what if you want to install just one component? You can install the PIA role, but you get WebLogic, Tuxedo, Java, and a PIA domian. In this post, we’ll show how to leverage the DPK with custom Puppet manifests and install only WebLogic (and Java) on a server.

You will need to the DPK Puppet files on the server. Run the bootstrap script and stop at the “Default Initialization” question so nothing gets installed. You can use any parameters you want for the bootstrap script. We’ll be ignoring the site.pp file and using a new manifest file.

I’ll document the custom manifests in this post, but if you want to download the code you can grab it from GitHub. Copy the files in the project into the corresponding folders under the puppet\etc folder on your machine.

psft_customizations.yaml Updates

For the WebLogic install, I wanted the software to install in the e:/oracle folder. I added/updated these values in my psft_customizations.yaml file:

oracle_base:      e:/oracle

jdk_location:           "%{hiera('oracle_base')}/jdk1.7.0_95"
weblogic_location:      "%{hiera('oracle_base')}/bea"

You don’t need to use oracle_base and can ignore these changes if you want WebLogic installed to the default location.

weblogic.pp Manifest

Create a new manfiest under etc\manfiests (or copy the site.pp file) and name it weblogic.pp. In the weblogic.pp manifest, we are doing two things:

  1. Create the oracle_base folder (the DPK will fail if the base folder doesn’t exist)
  2. Call a custom DPK role (we’ll create this next)

Here is the weblogic.pp file:

# Verify the Oracle Base folder is present - DPK doesn't like it when the base folder isn't created.
file { hiera('oracle_base') :
    ensure => 'directory',
}

# Call custom profile to install WebLogic (and JDK)
node default {
  include ::pt_role::pt_tools_weblogic
}

If you’ve looked at the site.pp file you’ll recognize the second section. The first part of the file is Puppet code to create a folder if it doesn’t exist.

If you aren’t familiar with Puppet, the Puppet CookBook is a good place to get familiar with writing manifests.

WebLogic DPK Role

A new DPK role file was created under etc\modules\pt_role\manifests called pt_tools_weblogic.pp. From the post on the sites.pp file, we introduced the idea of DPK roles. The best practice is to only have one role per machine. For the WebLogic-only install, we’ll create a new role so its clear that this server is only running WebLogic.

class pt_role::pt_tools_weblogic inherits pt_role::pt_base {
  notify { "Applying pt_role::pt_tools_weblogic": }
  $ensure   = hiera('ensure')
  contain ::pt_profile::pt_weblogic
  Class['::pt_profile::pt_weblogic'] 
}

The role file uses the ensure attribute from the defaults.yaml file to determine if Puppet should install or remove WebLogic. The default value is to install (ensure = present).

Next, the role calls a new profile called pt_profile::pt_weblogic. Let’s create that file net.

WebLogic DPK Profile

A DPK profile handles the configuration lookup from Hiera and preparing Puppet for the installation (or removal). Our custom profile grabs the installation directory (oracle_base and weblogic_location) from the psft_customizations.yaml file. WebLogic requires Java too, so we configure that in the profile as well.

Create a new profile under etc\modules\pt_profile\manifests and call it pt_weblogic.pp.

# Custom Profile to prepare for WebLogic installation
class pt_profile::pt_weblogic {
  notify { "Applying pt_profile::pt_weblogic": }

  ## Hiera lookups
  $ensure                    = hiera('ensure')
  $env_type                  = hiera('env_type')

  $tools_archive_location      = hiera('archive_location')

  $jdk_hiera             = hiera('jdk')
  $jdk_location          = $jdk_hiera['location']
  $jdk_remove_value      = $jdk_hiera['remove']
  if $jdk_remove_value == false {
    $jdk_remove = false
  }
  else {
    $jdk_remove = true
  }
  notice ("JDK remove is ${jdk_remove}")

  $weblogic_hiera        = hiera('weblogic')
  $weblogic_location     = $weblogic_hiera['location']
  $weblogic_remove_value = $weblogic_hiera['remove']
  if $weblogic_remove_value == false {
    $weblogic_remove = false
  }
  else {
    $weblogic_remove = true
  }
  notice ("Weblogic remove is ${weblogic_remove}")

  $redeploy = hiera('redeploy', false)
  class { '::pt_setup::weblogic_deployment':
    ensure                 => $ensure,
    tools_archive_location => $tools_archive_location,
    inventory_location     => $inventory_location,
    jdk_location           => $jdk_location,
    jdk_remove             => $jdk_remove,
    weblogic_location      => $weblogic_location,
    weblogic_remove        => $weblogic_remove,
    redeploy               => $redeploy,
  }
  contain ::pt_setup::weblogic_deployment
}

WebLogic DPK Deployment

Finally, we’ll call the Puppet code to install Java and WebLogic! We are using two DPK types in this manifest:

  • pt_deploy_jdk
  • pt_deploy_weblogic

This is why the DPK needs to be installed before using these manifests. If the DPK is missing, these calls will fail (and so will some of the hiera lookups). The deployment manifest is under the folder etc\modules\pt_setup\manifests. Create the file weblogic_deployment.pp from the code below or from the GitHub project.

# Custom Puppet Class to deploy WebLogic using the DPK
class pt_setup::weblogic_deployment (
  $ensure                 = present,
  $deploy_pshome_only     = false,
  $tools_archive_location = undef,
  $tools_install_user     = undef,
  $tools_install_group    = undef,
  $oracle_install_user    = undef,
  $oracle_install_group   = undef,
  $db_type                = undef,
  $pshome_location        = undef,
  $pshome_remove          = true,
  $inventory_location     = undef,
  $oracleclient_location  = undef,
  $oracleclient_remove    = true,
  $jdk_location           = undef,
  $jdk_remove             = true,
  $weblogic_location      = undef,
  $weblogic_remove        = true,
  $tuxedo_location        = undef,
  $tuxedo_remove          = true,
  $ohs_location           = undef,
  $ohs_remove             = true,
  $redeploy               = false,
) {

  notice ("Applying pt_setup::weblogic_deployment")

  $jdk_tag          = 'jdk'
  $weblogic_tag     = 'weblogic'

  $jdk_archive_file      = get_matched_file($tools_archive_location, $jdk_tag)
  if $jdk_archive_file == '' {
    fail("Unable to locate archive (tgz) file for JDK in ${tools_archive_location}")
  }
  $weblogic_archive_file = get_matched_file($tools_archive_location, $weblogic_tag)
  if $weblogic_archive_file == '' {
    fail("Unable to locate archive (tgz) file for Weblogic in ${tools_archive_location}")
  }

  $jdk_patches = hiera('jdk_patches', '')
  if ($jdk_patches) and ($jdk_patches != '') {
    notice ("JDK patches exists")
    $jdk_patches_list = values($jdk_patches)
  }
  else {
    notice ("JDK  patches does not exists")
    $jdk_patches_list = undef
  }

  pt_deploy_jdk { $jdk_tag:
    ensure            => $ensure,
    deploy_user       => $tools_install_user,
    deploy_user_group => $tools_install_group,
    archive_file      => $jdk_archive_file,
    deploy_location   => $jdk_location,
    redeploy          => $redeploy,
    remove            => $jdk_remove,
    patch_list        => $jdk_patches_list,
  }

  $weblogic_patches = hiera('weblogic_patches', '')
  if ($weblogic_patches) and ($weblogic_patches != '') {
    notice ("Weblogic patches exists")
    $weblogic_patches_list = values($weblogic_patches)
  }

  pt_deploy_weblogic { $weblogic_tag:
    ensure                    => $ensure,
    deploy_user               => $tools_install_user,
    deploy_user_group         => $tools_install_group,
    archive_file              => $weblogic_archive_file,
    deploy_location           => $weblogic_location,
    oracle_inventory_location => $inventory_location,
    oracle_inventory_user     => $oracle_install_user,
    oracle_inventory_group    => $oracle_install_group,
    jdk_location              => $jdk_location,
    redeploy                  => $redeploy,
    remove                    => $weblogic_remove,
    patch_list                => $weblogic_patches_list,
    require                   => Pt_deploy_jdk['jdk'],
  }

}

The manifests under pt_setup will do the heavy work of installing new software. This custom manifest will deploy Java and WebLogic on our servers. If your bootstrap script didn’t create a dpk\archives folder, the installation will fail. The DPK manifests (and our custom one) will look for the archives before it attempts to install.

Installation

Once your custom manifests are built, go to the puppet\etc\manifests folder. Run the command

puppet apply .\weblogic.pp --trace --debug

to install WebLogic.

Future Improvements

When I did the test, I tried to follow the Roles and Profiles pattern used by the DPK. It might seem complex for a smaller installation. You could combine all this into one manifest, but then it gets ugly to maintain. When we add more software components, the abstraction between configuration and implementation allows us to re-use Puppet code.

This was a test for myself to see if I could break down some of the abstraction in the DPK and write (or modify) manifests to control more of the process. Plus, we have DPK role that we can use for WebLogic-only deployments 🙂

#26 – WebLogic Filters

This week we talk about HR Image 17, the new Security Automation tool, and share some comments from David Kurtz about PeopleSoft on 12c. Then, Kyle dives into WebLogic Servlet Filters and shares how filters can be used with PeopleSoft.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Script WebLogic and Java Patches

In December, we talked quite a bit about patching Java and WebLogic on the blog and podcast. There was a WebLogic CVE, and then a patch, to apply. If you want a recap on the CVE and patching process, here are the posts:

While applying the patches, I wanted to script the process so patching would be consistent across all our servers. I pulled the scripts into a GitHub project for sharing and reuse. If you haven’t scripted a WebLogic patch, this would be a place to start. The scripts use PowerShell and built for WebLogic 10.3.6. So, they use SmartUpdate instead of OPatch. I also added in a Java patch to the process too. You could pull out the Java patch script to use by itself. One more note: all the patches, Java, and scripts were set to run from the folder e:\installers\weblogic1036-2015-CVE-Patches. If you use these for your environment, or just use them as a template, you’ll want to update those paths for your specific configuration.

There is nothing ground-breaking about these scripts 🙂 I can write scripts, but I’m not the best script developer out there. If you see places where the scripts need improvement, file an issue with the project or submit a pull request! The main goal with this project and post is to get others started with scripting. Scripting, even if the scripts are basic, can benefit administrators. I hope that this quick overview might help someone get started.

Scripts Overview

These scripts are writtin in PowerShell. If PowerShell scripts are not enabled on the server, run this command to allow PowerShell scripts to run:

set-executionpolicy unrestricted

  1. Install new SmartUpdate version (3.3.0)

    installSmartUpdate.ps1

    The silent.xml file is used for a silent install (no prompts). The installation directory is set to e:\oracle. If you want a different directory, change the value for “BEAHOME”. 1. Stop all web servers running on the server .stopPIAServices.ps1 The script looks for any Windows service that containts “*-PIA” in the name. If you have any WebLogic domains were not created by the

    installNTService script, you may need to shut them down by hand.

  2. Prepare and copy files from the weblogic1036-2015-CVE-Patches folder

    prepareFiles.ps1

    This script performs tasks to prepare different files for patching: On our servers, two files needed updates to run the Smart Update utility. registry.xml needed to remove a reference to Tuxedo; bsu.cmd needed an increase in memory to the Java Heap. The registry.xml file also contains a reference to the server where it was installed. The script will change that value based on the new server’s name. The original files are backed up first and a .bkp extension is added to the file name. The script also copies jdk-1.7.0_79 to our e:\java folder. If you want the new java version in a different location, you can change the path in the file.

  3. Apply both WebLogic patches The patches we are applying resolve the December 2015 CVE with WebLogic. If you are using these scripts for future patches, you’ll want to update the patch ID’s in the script.

    applyWebLogicPatches.ps1

    Both patches are applied to WebLogic using the bsu command. The script assumes your patches are in the folder e:\patches\cve-2015-4852. NOTE: On one of our servers, the second patch stalled during the “Checking for Conflicts” step. If the script stalls for more than a few minutes, hit Cntl-C.

  4. Update the JAVA_HOME values

    updateJavaVersion.ps1

    The JAVA_HOME value in the setEnv.cmd script will be updated to the new path. You must update this script for each server. The paths in the script are hard-coded. (The hard coding is an obvious candidate to fix next. Should be able to use the Get-ChildItem cmdlet to find all the setEnv.cmd files.)

  5. Update Registry value for JAVA_HOME

    updateRegistryJavaVersion.ps1

    The JAVA_HOME value in the Registry for each web service will be updated. You must update this script for each server. The paths in the script are hard-coded. (Again, another place for improvement. Need to find a search cmdlet for the Registry. Could look for -PIA in the service name.)

  6. Start all web servers running on the server.

    startPIAServices.ps1

    Again, this looks for all Windows services that have *-PIA in the name and starts them. That’s it.

The scripts are pretty simple, and you can write a wrapper script to run all the sub-scripts. That way you’d have one script to kick off. Or, you could add these into a tool like Rundeck to execute from a centralized place. Once you start down the path of scripting, many opportunities open up to speed up everyday tasks.

#8 – PeopleTools 8.55 Hands-on

Merry Christmas and Happy Holidays! We have an extra long and fun episode this week but you may want to listen to this it twice 🙂

In Episode 8, we dive into PeopleTools 8.55 and get hands-on! We talk about our first experiences with the PeopleTools DPK’s, changes to Change Assistant in 8.55 and Dan’s experiment with the Let’s Encrypt project. We also ask a big question about a new Related Content feature, and then answer it after the break. This feature has the potential to substantially reduce your customizations!

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Let’s Encrypt with PeopleSoft

Let’s Encrypt is a service provided by the Internet Security Research Group to provide free SSL certificates to anyone. The goal of the project is get the entire web encrypted. I mentioned the project in Episode 7 of The PeopleSoft Administrator Podcast and thought it would be a great exercise to try it with PeopleSoft. Let’s Encrypt uses a client on the server to automate the certificate request process. The client will:
* Validate that you own the web server
* Generate a CSR
* Download the certificate
* Apply the certificate to the web server (limited support)
* Automatically renew the certificate

There are a few requirements to use the Let’s Encrypt clients though:

  • The web server needs to accessible by the internet. The Let’s Encrypt site will validate that you own the server by checking for a specific file on the web server.
  • Not all operating systems are supported, yet.
  • Some web server’s have built-in support (IIS, Apache), but others do not (e.g, WebLogic).

We can still generate certificates though, the automatic renewal won’t update the web server though.

Install Let’s Encrypt Client for Windows We’ll use the

letsencrypt-win-simple command line client for Windows. Download the latest release from GitHub and extract the folder to a permanent location.

Generate a new certificate

  1. Run .letsencrypt.exe --accepttos

    Let's Encrypt (Simple Windows ACME Client) ACME Server: https://acme-v01.api.letsencrypt.org/ 
    Config Folder: C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org 
    Loading Signer from C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orgSigner 
    Getting AcmeServerDirectory Loading Registration from C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.org
    Registration Scanning IIS 7 Site Bindings for Hosts 
    No IIS bindings with host names were found. Please add one using IIS Manager. A host name and site path are required to verify domain ownership. 
    No targets found. 
    M: Generate a certificate manually. 
    A: Get certificates for all hosts 
    Q: Quit 
    Which host do you want to get a certificate for: 
    
  2. Since we are not running IIS, we’ll generate a certificate manually.

    Which host do you want to get a certificate for: M Enter a host name: 
    
  3. Enter the DNS name for your web server.

    Enter a host name: hr.psadmin.io 
    Enter a site path (the web root of the host for http authentication): 
    
  4. Next, enter the root path for your web server. If you are running WebLogic, that will be PORTAL.war directory on your web server.

    Enter a site path (the web root of the host for http authentication): W:\pt8.55\webserv\peoplesoft\applications\peoplesoft\PORTAL.war
    
  5. Then, the Let’s Encrypt client will create a new file under PORTAL.war.well-knownacme-challenge. That file will be used to validate that you own the web server.

    Authorizing Identifier ps92t855.psadmin.io 
    Using Challenge Type http-01 Writing challenge answer to W:\pt8.55\webserv\peoplesoft\applications\peoplesoftPORTAL.war.well-known/acme-challenge /1c2yN7Y93sJwRUmRGaoG4kT-QynrIcGr4szre-3nTsQ 
    Answer should now be browsable at http://ps92t855.psadmin.io/.well-known/acme-challenge/1c2yN7Y93sJwRUmRGaoG4kT-QynrIcG r4szre-3nTsQ 
    Submitting answer Refreshing authorization Authorization Result: valid 
    Deleting answer 
    
  6. After the web server ownership is verfied, new certificates will generated and copied to your system. The certificates are copied to your %USERPROFILE%AppDataRoamingletsencrypt-win-simple folder in a few formats:

    • .der
    • .pem
    • .pfx

The client will also add the certificates to the Windows Certificate Store for you. To add the certificates to WebLogic, we’ll use the .pem

    Requesting Certificate Request Status: Created 
    Saving Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orghr.psadmin.io-crt.der 
    Saving Issuer Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orgca-009813F47513E5750B43E7431E971E44BD-crt.pem 
    Saving Certificate to C:\Users\Administrator\AppData\Roaming\letsencrypt-win-simple\httpsacme-v01.api.letsencrypt.orghr.psadmin.io-all.pfx (with no password set) 
    Opened Certificate Store "WebHosting" 
    Adding Certificate to Store 
    Closing Certificate Store 
    WARNING: Unable to configure server software. 
    Creating Task letsencrypt-win-simple httpsacme-v01.api.letsencrypt.org with Windows Task Scheduler at 9am every day. 
    Renewal Scheduled Manual hr.psadmin.io (W:pt8.55webservpeoplesoftapplicationspeoplesoftPORTAL.war) 
    Renew After 2/9/2016 
    Press enter to continue. 

Create a New pskey Keystore Now that we have certificates, let’s create a new

pskey file with the certificates. We’ll use Keystore Explorer to quickly generate the file.

  1. Open Keystore Explorer. (If it’s first time you’ve used it, follow the instructions to download the Unlimited Strength files).
  2. Create a new keystore file.
  3. Select the file type of “JKS”.
  4. Select “Tools > Import Key Pair”.
  5. Select the “OpenSSL” option.
  6. Deselect “Encrypted Private Key”.
  7. For the “OpenSSL Private Key File”, select the file hr.psadmin.io-key.pem.
  8. For the “Certificate(s) File”, select hr.psadmin.io-cert.pem
  9. Click “Import”.
  10. Enter an alias name that is descriptive. I used hr.psadmin.io-2015-12.
  11. Since the prive key was delivered without a password, we’ll want to enter one. Enter a password for the key pair.

Now you have the private and public key for your DNS entry in the keystore. Next, we need to add the root (and intermediate) certificates so that a chain of trust is established.

  1. In Keystore Explorer in our new keystore file, right-click on our certificate. Select “Edit Certificate Chain > Append Certificate”.
  2. Select the file ca-GUID-crt.pem and click “Append”.
  3. Save the file, give the keystore a password, and name the file pskey-2015-12.

Load Keystore into WebLogic After importing the certificates into

pskey-2015-12, we need to copy the file to the web server and tell WebLogic to use the new file. The integrationGateway.properties file will need to know about the new keystore as well.

  1. Copy the pskey-2015-12 file to your web server directory %PS_CFG_HOME%\webserv\peoplesoft\pia\config\keystore.
  2. Log into the WebLogic console.
  3. Navigate to “Environment > Servers > PIA > Keystores”.
  4. Click the “Lock & Edit” button to allow editing.
  5. Click the “Change” button for the Keystores option.
  6. Select “Custom Identity and Custom Trust” and “Save”.
  7. In the “Custom Identity Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  8. In the “Custom Identity Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  9. In the “Custom Trust Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  10. In the “Custom Trust Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  11. Click Save. WebLogic will look at the new keystore file. Next, we need to tell WebLogic certificate it should serve to users.

  12. Click on the “SSL” tab.

  13. Change the “Private Key Alias” to hr.psadmin.io-2015-12.
  14. In the “Private Key Passphrase” boxes, enter the password you gave the hr-psadmin.io-2015-12 keypair.
  15. Click Save.
  16. Click the “Activate Changes” button.

Update integrationGateway.properties

Before we reboot the WebLogic domain, we need to update the integrationGateway.properties file.

  1. On your web server, open the integrationGateway.properties file under %PS_CFG_HOME%\webserv\peoplesoft\applications\peoplesoft\PSIGW.war\WEB-INF.
  2. Find the line secureFileKeystorePath and change file name to pskey-2015-12.
  3. If the password you gave the keystore is different than the previous file, you’ll need to update that parameter in the file.
    1. Open a command prompt and go to %PS_CFG_HOME%\webserv\peoplesoft\bin.
    2. Run the command setEnv.cmd to set the environment variables.
    3. Go to the folder piabin.
    4. Run the command PSCipher to get the encrypted text.
  4. Restart your WebLogic domain.

Test your HTTPS Connection

As WebLogic is starting up, make sure to check the logs to verify that the server started with your new certificate. Once the server has started, open a browser and go test the site. You should see a secure connection in the browser to your site.

How to Apply WebLogic Patches – Part 2

In Part 1, I showed how to use Smart Update to patch WebLogic. Starting with WebLogic 12.1.2, OPatch handles all the pacthing. Let’s walk though using OPatch to update WebLogic to fix the latest vulnerability. OPatch is included in the WebLogic install, so everything you need to apply patches is ready to go.

Windows Path Limit

If you are on Windows and applying patches 21370953 and 22250567, you may run into an error The file name(s) would be too long for the destination folder. The patch contains so may folders that they conflict with the Windows limit of 260 characters for a file name. The work around is to use the jar utility that comes with the JDK to unzip the patch. jar -xvf p21370953_121300_Generic.zip

Set OPatch Environment

OPatch needs to know what ORACLE_HOME you are applying the patch to. Depending on your server configuration, you may need to set ORACLE_HOME to the directory that contains WebLogic.

set ORACLE_HOME=e:\middleware-854

Let’s Fix CVE-2015-4852

Since we have new patches to fix CVE-2015-4852 (T3/Java Deserialization), let’s use those as our example.Go to this page to find the applicable patch (or patches if you are on 10.3.6) to apply.

Extract Patches

Download the patches you need and unzip them. I put the patch files under

e:\patches\cve-2015-4852

on the web server.

set PATCH_TOP=e:\patches\cve-2015-4852
unzip -d %PATCH_TOP p21370953_121300_Generic.zip
unzip -d %PATCH_TOP p22250567_121300_Generic.zip

Apply Patches

Make sure all of your web server instances are shut down. Then, move into the first patch folder so it is your current directory. Once you are in the patch folder, we call OPatch.

cd patches\cve-2015-485221370953
e:\middleware-854\OPatch\opatch apply

At the end of the patch, you should see a OPatch succeeded message. Let’s apply the second patch.

cd patches\cve-2015-485222250567
e:\middleware-854\OPatch\opatch apply 

At the end of the patch, you should see a OPatch succeeded message.

Verify WebLogic Version

To verify WebLogic has the new patches, we use OPatch’s lsinventory command.

e:\middleware-854\OPatch\opatch lsinventory

The output will look similar to this:

Interim patches (2) : Patch 22250567 : applied on Fri Dec 11 07:46:45 CST 2015 
Unique Patch ID: 19584835 
Patch description: "One-off" Created on 22 Nov 2015, 01:36:21 hrs PST8PDT 
Bugs fixed: 22175246, 22200449, 22247869, 21495475 
This patch overlays patches: 21370953 
This patch needs patches: 21370953 as prerequisites 
Patch 21370953 : applied on Fri Dec 11 07:46:45 CST 2015 
Unique Patch ID: 19198495 
Patch description: "WebLogic Server 12.1.3.0.5 PSU Patch for BUG21370953 October 2015" 

The output shows that we have applied the CVE patches. Now, restart all your web servers and start testing!