Apply CPU Patches with Deployment Packages

We have talked on the podcast about different ways to apply CPU patches, but with the DPK we have another tool to help us quickly apply CPU patches. This post and video demo’s will show you how to use the DPK to quickly apply CPU patches to your servers.

Deployment Workflow

When you run the DPK, it will deploy WebLogic, Java, Tuxedo (and more) on your server. The DPK uses archives (also known as “tarballs”) of prepackaged installations and extracts those archives to your server. There is one big problem, the archives included in the DPK’s do not contain the latest security patches. So, let’s make our own tarballs that include the security patches to deploy. This process is also a great exercise to better understand how the DPK deploys software.

If you are on Linux you can use the patching functionality with the DPK, but that code has not been written for Windows. I’m not covering that feature in this post, but the DPK Install Guide has a section on using that functionality (Task 6-3-1: Using the DPK Setup Script to Apply Fixes).

Movement Scripts

There are Fusion Middleware scripts the DPK uses to deploy WebLogic and Tuxedo. (Thanks to Eric Bolinger for pointing me in this direction.) The movement scripts allow you to take a current install of WebLogic, package it up, and deploy it to additional servers. This is how the DPK deploys WebLogic. The PeopleTools team packages up a WebLogic installation and we deploy that install to our servers. The movement scripts also manage the Oracle Inventory file for you.

There are many parts to the movement scripts, but we’ll be using just one part: copyBinary. This script will take a current installation and create a .jar file from that installation. We’ll use copyBinary to package our patched WebLogic installation.

If you have errors with the pasteBinary.cmd on the target system, you may need to configure the $ORACLE_HOME\oui\oraparam.ini file. This is a configuration file used by the OUI software. To make this simple, I copied the settings in the current $BASE\dpk\archives\weblogic12.1.3.0.tgz to my $ORACLE_HOME\oui\oraparam.ini using Beyond Compare. (Yes, Beyond Compare can read inside a tarball and compare against a directory!) Then I recreated my tarball with the updated oraparam.ini file.

Create a Patched WebLogic Tarball

 

Next, it’s time to install the CPU patch and run the copyBinary.cmd script. Stop all your PIA services on the server so you can remove the existing installations.

First, let’s patch Java. For demonstration, I’m using the jdk-7u141-windows-x64 installer. I’m installing

Then, we’ll use OPatch to apply the CPU to WebLogic:

cd $PATCH
$env:ORACLE_HOME=e:\psoft\pt\bea
$env:ORACLE_HOME\OPatch\OPatch napply

Once OPatch is done, we’ll use the movement scripts to package up our installation.

$env:JAVA_HOME=e:\psoft\pt\jdk1.7.0_141
. ${env:ORACLE_HOME}\oracle_common\bin\copyBinary.cmd -javaHome ${env:JAVA_HOME} -archiveLoc ${env:TEMP}\pt-weblogic-copy.jar -sourceMWHomeLoc ${env:ORACLE_HOME}

The output file from this command needs to be named pt-weblogic-copy.jar. The DPK expects that is the name of the .jar file. Next, we create a tarball of the pt-weblogic-copy.jar and two files to do the deploy portion of the movement scripts: cloningclient.jar and pasteBinary.cmd. These movement scripts are used by the DPK to deploy WebLogic. I used 7-zip to create my tarball with these three files:

$WL_VERSION="12.1.3.170418"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\jlib\cloningclient.jar"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\bin\pasteBinary.cmd"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:TEMP}\pt-weblogic-copy.jar"

Last, we gzip the archive and drop it in the $BASE\dpk\archives folder:

$env:DPK_BASE="e:\psft"
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-weblogic${env:WL_VERSION}.tgz" "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar"

One thing to note here – the DPK doesn’t handle multiple versions of software in the dpk\archives folder well. So, only have one pt-weblogic* file in there.

For Java, we don’t need to use the movement scripts. We’ll simply tarball up the new directory and include that in our $BASE\dpk\archives folder.

$JDK_VERSION="1.7.0_141"
7z a -ttar "${env:TEMP}\pt-jdk${JDK_VERSION}.tar" $env:JAVA_HOME\*
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-jdk${JDK_VERSION}.tgz" "${env:TEMP}\pt-jdk${JDK_VERSION}.tar"

Deploy CPU Patches

 

Copy your updated tarballs to a new server. You’ll want to remove the existing tarballs from the $BASE\dpks\archive to prevent the DPK from raising an error.

We have two options for telling the DPK we want to install WebLogic. The first option is to delete the existing WebLogic and Java folders. If you stop your PeopleSoft domains, you can delete both folders. When you run the DPK it will see that WebLogic and Java are missing and reinstall them from the patched tarballs in the $BASE\dpk\archives folder.

The other option is use the redeploy: true flag in psft_customizations.yaml. If you set the redeploy variable to true, the DPK will redeploy all the software in your $BASE\dpk\archives folder. This option requires less work – set a variable in psft_customizations.yaml and run the DPK – but it can take longer because you will redeploy Java, Tuxedo, WebLogic, PS_HOME and more. I think of this option as “the Puppet way”.

For this post and demo, we’ll use the redeploy: true option in our psft_customizations.yaml file. We’ll also use one other trick for testing; we will only run the part of the DPK that handles the middleware. Instead of running the entire DPK that touches the OS, middleware, and domains, the manifest we call includes only the DPK role that ensures the middleware is installed and not touch other parts of the system. This will also speed up our CPU patch deployment.

middleware.pp

Let’s create a new file under c:\programdata\puppetlabs\puppet\etc\manifests called middleware.pp. You can start by cloning the site.pp file. Change the file to look like this:

node default {
  include ::pt_role::pt_tools_deployment
}

Save the file. That’s it!

What we have done is tell Puppet to only run the DPK role pt_tools_deployment instead of running a larger role like pt_hcm_pum.

In the video demo, we are applying patches to a PeopleSoft Image, which is a Fulltier setup. The default pt_tools_deployment.pp manifest won’t run on a Fulltier system. To get around that, I created a copy of pt_tools_deployment.pp manifest called io_tools_deployment.pp and removed the check on env_type: fulltier.

cpu.ps1

We have a few tasks to do before we can run the middleware.pp manifest. We’ll wrap those tasks in a Powershell script we can run on each server.

At a high level, here are the tasks our cpu.ps1 script will do:

  1. Copy new DPK archives to server
  2. Stop PeopleSoft Services
  3. Remove current Java and WebLogic installs (if redeploy: false)
  4. Run middleware.pp to install patched Java and WebLogic
  5. Start PeopleSoft Services

Get the Sample Code

The full code is in the ps-dpk-tarballs GitHub repository. You can find all the scripts from this post and demo on GitHub.

#58 – Pagelets and Complaints

This week on the podcast, Dan and Kyle talk about a ransomware attack, load balancer health checks, applying POC patches and complain about minor annoyances in Change Assistant. Kyle shares a story about a misbehaving pagelet and how he investigated the issue.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Monitoring WebLogic and Java

Lately, I have had interest in monitoring WebLogic’s performance. Since Weblogic is built on Java, there are some standard tools we can use to look into the Java Virutal Machine (JVM). We’ll cover two of those tools in this post: JConsole and VisualVM. Both JConsole and VisualVM are included in the Java Development Kit so they are already on your server. These tools will give you information about the JVM used to run WebLogic and can help you tune you web servers.

JMX

To get monitoring data out of WebLogic’s JVM, we need to enable JMX. Java Management Extensions (JMX) is a monitoring technology built into Java. Applications that run on Java can build instrumentation into the application to provide performance data about the application. Even without additional data, the JVM will provide CPU, Memory, Thread and other stats about the heap.

To enable JMX for WebLogic, we’ll update the setEnv.cmd or setEnv.sh file. At the end of the JAVA_OPTIONS line, add these flags:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8888 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

There are 4 flags we pass to Java when the JVM is started:

  1. Enabling JMX Remote connections
  2. JMX Remote Port
  3. Requiring SSL Connections
  4. Authentication for JMX

For testing, I’ve turned off SSL and Authentication, but if you are enabling JMX across all your servers I recommend you turn both options on. For the JMX Port, pick a port value that is unique for each instance of the JVM. Each WebLogic and App Server instance will have its own JVM. For more information on configuring JMX, this is a link to the official documentation.

If your are on Windows and updated the setEnv.cmd file, you will want to re-install the Service that starts the PIA domain. The JAVA_OPTIONS parameters are stored in the registry when you create the service. If you update setEnv.cmd, you need to recreate the service (or manually update the registry).

Now that JMX is enabled on our domains, let’s look at a few tools to help us monitor our JVMs.

JConsole

JConsole is a utility included with the JDK download. Under JAVA_HOME\bin you’ll find jconsole.exe To start, we’ll run JConsole from our web server where we enabled JXM (instead of our desktop). Open JConsole and it will ask you to connect to a JMX Process. You have two options: Local Process and Remote Process. We’ll use the remote process option and use these values to connect to our web server: localhost:8888. We don’t need a username or password since we passed the flag jmxremote.authentication=false.

jconsole01

After connecting, you’ll get a message asking about your insecure connection. Click “Insecure” to continue. On the main page, we see 4 graphs related to the JVM.

jconsole03

These graphs give you a good overview of the JVM status. The CPU graph will show you how much of the CPU that JVM is using, and the Threads graph gives a good indication of workload on the JVM. The best part of JMX is the Memory graph. Getting your JVM Heap sized correctly can make a big different in performance. The graph should follow a pattern when Garbage Collection runs.

jconsole04

You don’t want Garbage Collection to run too often, or the usage too high after Garbage Collection. This graph helps with getting the right size for your web server. (You can find more tuning information here.)

VisualVM

VisualVM is another untility included with the JDK download and is also under JAVA_HOME\bin. We’ll start VisualVM on the server as well by running jvisualvm.exe --console new.

visualmv02

When VisualVM opens, we create a new connection by right-clicking on “Local” and selecting “Add JMX Connection”. Fill in the port number and select “Do not require SSL connection”.

visualvm03

VisualVM show us similar data as JConsole, but I think it looks a nicer. Under the Monitor tab, you can also force the JVM to run a Garbage Collection. For the most part, these two applications are similar.

visualvm03b

Remote JMX Connections

We have run both applications on the server to connect to JMX, but these applications are more useful if we can connect to the servers remotely. By default, JMX will only accept local connections. To enable remote connections to JMX, we have to pass this flag:

 -Dcom.sun.management.jmxremote.local.only=false

After you add that parameter to your setEnv.cmd JAVA_OPTIONS line, restart the web server. On a different computer or server, launch VisualVM or JConsole. Create a remote connection to JMX on the server. In the Connection box, enter the server name and port for the JMX instance.

visualvm05

JMX Authentication

Once you get the basic configuration in place, you want to enable authentication to connect to the JMX instance. The default JMX authentication is stored in the JDK folder. That will affect all domains using the JDK folder. Instead, we will use a JMX password file for each web server domain.

  1. Open the file JAVA_HOME\jre\lib\management\jmxremote.access.
  2. Add the line psMonitor readonly to the bottom of the file and save. This line adds a new user named psMonitor and a read-only account to any JMX instances using this JAVA_HOME.
  3. Copy the file JAVA_HOME\jre\lib\management\jmxremote.password.template to PS_CFG_HOME\webserv\jmxremote.password.
  4. Open the new jmxremote.password file.
  5. Add the line psMonitor test123 to the bottom of the file and save. This line sets the password for the psMonitor user. To give each web server domain a different password, set a unique password in this file under each PS_CFG_HOME.
  6. Open the setEnv.cmd file and add these parameters:

    -Dcom.sun.management.jmxremote.password.file=PS_CFG_HOME\webserv\jmxremote.password -Dcom.sun.management.jmxremote.authenticate=true
    
  7. Restart the web server for the new paramters to take affect.

Now that we have a web server configured to run JMX with authentication, we will create another connection in VusualVM to use the username and password.

  1. Right-click on the remote server and select “Add JMX Connection”
  2. Enter the server name and port.
  3. Enter psMonitor for the Username and test123 for the Password.
  4. Select “Do no require SSL connection”
  5. Click OK.

jmxauth01

jmxauth02

Load Balancers and Client IP Addresses

One disadvantage of running PeopleSoft behind a load balancer is that it can hide your user’s real IP address. With a load balancer, WebLogic and PeopleSoft log the load balancer’s IP address instead of the client connecting to your system. Load balancers support an HTTP header called X-Forwarded-For, and will populate that HTTP header with the client’s IP address. We can configure WebLogic to read this header and add that to our HTTP access log.

The psaccesslog table tracks logins into PeopleSoft, but a customization is required to change the logged IP address to read from X-Forwarded-For. We won’t over that here. Instead, we’ll show a configuration change to capture this information.

X-Forwarded-For

The X-Forwarded-For HTTP header is designed to pass in the HTTP originating client IP address through the load balancer. Logging a client IP address helps you identify where users are coming from (geo-ip) and can help with auditing access to your system. Since organizations run many different load balancers, we won’t cover how to enable that. Ask your network administrators to enable and populate X-Forwarded-For and pass it through to your WebLogic servers.

If you want to test the logging without getting a network admin involved, FireFox has a nice add-on, X-Forwarded-For Header that simulates the header for you.

WebLogic Access Log

Out of the box, WebLogic doesn’t capture the X-Forwarded-For HTTP header. To read that header, we can configure the HTTP Access Log to read the header and log it. To do that, we need to enable the Access Log and tell WebLogic what header to look for.

EnableXFF

  1. Open http://server:port/console and login.
  2. Go to “Environment > Servers > PIA > Logging > HTTP”.
  3. Click “Lock & Edit”.
  4. Select the checkbox for “HTTP access log file enabled”.
  5. Save the changes.
  6. Expand the “Advanced” section.
  7. Change the Format to Extended.
  8. Add cs(X-Forwarded-For) to the Extended Logging Format Fields.
  9. Set the Log File Buffer to 0. (This will write entires immediately to the log file.)
  10. Save the changes.
  11. Click the “Release Configuration” button.
  12. Restart the web server.

In the PIA_access.log file, you should see a dash - or an IP address. If the header is blank, the dash will be populated. Otherwise, WebLogic will log the client’s real IP address.

Here a sample line from the PIA_access.log file after we enabled X-Forwarded-For logging:

#Version:   1.0
#Fields:    date time cs-method cs-uri sc-status cs-username cs(user-agent) s-ip c-ip time-taken bytes cs(X-Forwarded-For)
#Software:  WebLogic

2016-11-23  20:49:16    GET /psc/ps/EMPLOYEE/ELM/c/NUI_FRAMEWORK.PT_LANDINGPAGE.GBL 200 -   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"    0.0.0.0:10020   10.0.1.174  5.484   10907   "10.0.1.222"

The last value, "10.0.1.222" is the client IP address (my workstation). The c-ip value (10.0.1.174) is the load balancer IP adress in this scenario.

Advanced WebLogic Configuration with the DPK

Continuing on our DPK series, this week we’ll explore how to do advanced web server domain configuration. The psft_configuration.yaml file gives you many of the configuration options for your web domain. While these settings will help you configure your web domain, there are more changes to WebLogic most administrators make. One of those changes is configuring SSL/Certificates and Logging.

DPK and WSLT

Before the DPK, you could script configuration changes to WebLogic using the WebLogic Scripting language WSLT. WSLT is a great interface to WebLogic, and you can do quite a bit with it.

We use the psft_customizations.yaml file to document the WLST commands in a section called config_settings:. This section is not included in psft_configuration.yaml for the PIA, so it will be a new section for your files. The config_settings: section is included for app server and process scheduler sections. If you look at the file puppet\etc\modules\pt_config\tests\test_pia_create.pp you can see the PeopleTools team is using the config_settings section when creating a test domain.

In the psft_customizations.yaml file, the config_settings: section will go below the webserver_settings: section and above the site_settings: section. Everything under config_settings: is broken down into WSLT commands. To make a change to WebLogic, use the tree structure from your web domains’ config.xml file to identify a section to change. E.g, Servers/PIA/WebServer/PIA/WebServerLog/PIA will be the HTTP access log settings for the PIA server.

Under each WSLT header, you can include a key: value for any configurable option in WebLogic. For our Servers/PIA/WebServer/PIA/WebServerLog/PIA above, we’ll enable the access log:

config_settings:
  Servers/PIA/WebServer/PIA/WebServerLog/PIA:
    LoggingEnabled:                     true

When we run puppet apply, the Puppet code take this configuration and executes the WLST to make the change to WebLogic. After the puppet apply run, you can check the config.xml file to verify the updated configuration.

Using WSLT to Identify Config Values

If you don’t know what the path or values you want to change, you can spin up a WLST session and browse your web domain. To start WLST, run setWLSEnv.cmd script under your WL_HOME\server\bin folder. That will set up the environment variables so Java can find the WLST package.

Next, start up a WLST session with java weblogic.WLST. Your web server needs to be running to connect, and you’ll need the console username and password.

connect('system', 'Passw0rd', 't3://localhost:8000')

Once you connect to your domain, you can use cd() and ls() commands to navigate around the configuration tree. For example, this command will take us to the Access Log settings for the PIA.

cd('Servers/PIA/WebServer/PIA/WebServerLog/PIA')
ls()

You can do much more with WLST, but we’ll save that for another time.

SSL and Certificate Configuration

Now that we can configure a web domain via WLST commands, let’s set up SSL and a certificate for a domain. The DPK will do all the configuration, but it won’t update the pskey file. A good idea would be to prebuild your pskey file (using KeyStore Explorer) and be ready to copy into your domains piaconfig folder after the DPK runs.

This is what the config_settings section of the psft_customizations.yaml looks like to configure SSL:

config_settings:
  Servers/PIA: 
    CustomIdentityKeyStorePassPhrase:   Passw0rd
    CustomTrustKeyStorePassPhrase:      Passw0rd
    KeyStores:                          CustomIdentityAndCustomTrust
  Servers/PIA/SSL/PIA:
    ServerPrivateKeyAlias:              psadminio-2016
    ServerPrivateKeyPassPhrase:         Passw0rd
  Servers/PIA/WebServer/PIA/WebServerLog/PIA:
    LoggingEnabled:                     true

If you have configured SSL through the WebLogic console before, these configuraiton options should look familiar. We tell WebLogic to use a custom keystore (pskey), the password to get into the keystore, the Private Key alias and the password for the Private Key.

In production systems, it’s not a good idea to have your unecrypted password in the yaml files. For Windows, support for encrypting password in the yaml files isn’t there yet. For Linux, eyaml can encrypt your passwords.

This is what a psft_customizations.yaml looks like for a web server domain:

pia_domain_list:
  hr92dmo:
    os_user:               psadm
    ps_cfg_home_dir:       e:/psoft/hr92dmo
    gateway_user:          administrator
    gateway_user_pwd:      Passw0rd
    auth_token_domain:     ".%{::domain}"

    webserver_settings:
      webserver_type:           "%{hiera('webserver_type')}"
      webserver_home:           "%{hiera('weblogic_location')}"
      webserver_admin_user:     system
      webserver_admin_user_pwd: Passw0rd
      webserver_admin_port:     "%{hiera('pia_http_port')}"
      webserver_http_port:      "%{hiera('pia_http_port')}"
      webserver_https_port:     "%{hiera('pia_https_port')}"

    config_settings:
      Servers/PIA: 
        CustomIdentityKeyStorePassPhrase:   Passw0rd
        CustomTrustKeyStorePassPhrase:      Passw0rd
        KeyStores:                          CustomIdentityAndCustomTrust
      Servers/PIA/SSL/PIA:
        ServerPrivateKeyAlias:              psadminio-2016
        ServerPrivateKeyPassPhrase:         Passw0rd
      Servers/PIA/WebServer/PIA/WebServerLog/PIA:
        LoggingEnabled:                     true

    site_list:
      hr92dmo:
        appserver_connections: "%{hiera('pia_psserver_list')}"
        domain_conn_pwd:       "%{hiera('domain_conn_pwd')}"

        webprofile_settings:
          profile_name:        DEV
          profile_user:        PTWEBSERVER
          profile_user_pwd:    Passw0rd

        report_repository_dir: "%{hiera('report_repository_dir')}"

Update Existing Domains

Unlike other parts of the DPK, the WLST commands execute every time you run puppet apply .\site.pp. So, if you want to update a config value in your web domain, the DPK will make those changes for you after the domain was created. With the app server and process schedulers, the DPK doesn’t handle changes well. Once the domain is built, configuration changes are not applied to app and batch domains.

Repeatable Processes

The goal of the DPK, Puppet and other automation software is to create a dependable and repeatable process. Taking advantage of these features in the DPK will get you closer to the ideal of complete automation. The ability to issue WLST commands against WebLogic domains gives admins another tool with the DPK to build a system to your specifications.

Using the DPK to install only WebLogic

With the Deployment Packages, you can install an entire PeopleSoft system. But what if you want to install just one component? You can install the PIA role, but you get WebLogic, Tuxedo, Java, and a PIA domian. In this post, we’ll show how to leverage the DPK with custom Puppet manifests and install only WebLogic (and Java) on a server.

You will need to the DPK Puppet files on the server. Run the bootstrap script and stop at the “Default Initialization” question so nothing gets installed. You can use any parameters you want for the bootstrap script. We’ll be ignoring the site.pp file and using a new manifest file.

I’ll document the custom manifests in this post, but if you want to download the code you can grab it from GitHub. Copy the files in the project into the corresponding folders under the puppet\etc folder on your machine.

psft_customizations.yaml Updates

For the WebLogic install, I wanted the software to install in the e:/oracle folder. I added/updated these values in my psft_customizations.yaml file:

oracle_base:      e:/oracle

jdk_location:           "%{hiera('oracle_base')}/jdk1.7.0_95"
weblogic_location:      "%{hiera('oracle_base')}/bea"

You don’t need to use oracle_base and can ignore these changes if you want WebLogic installed to the default location.

weblogic.pp Manifest

Create a new manfiest under etc\manfiests (or copy the site.pp file) and name it weblogic.pp. In the weblogic.pp manifest, we are doing two things:

  1. Create the oracle_base folder (the DPK will fail if the base folder doesn’t exist)
  2. Call a custom DPK role (we’ll create this next)

Here is the weblogic.pp file:

# Verify the Oracle Base folder is present - DPK doesn't like it when the base folder isn't created.
file { hiera('oracle_base') :
    ensure => 'directory',
}

# Call custom profile to install WebLogic (and JDK)
node default {
  include ::pt_role::pt_tools_weblogic
}

If you’ve looked at the site.pp file you’ll recognize the second section. The first part of the file is Puppet code to create a folder if it doesn’t exist.

If you aren’t familiar with Puppet, the Puppet CookBook is a good place to get familiar with writing manifests.

WebLogic DPK Role

A new DPK role file was created under etc\modules\pt_role\manifests called pt_tools_weblogic.pp. From the post on the sites.pp file, we introduced the idea of DPK roles. The best practice is to only have one role per machine. For the WebLogic-only install, we’ll create a new role so its clear that this server is only running WebLogic.

class pt_role::pt_tools_weblogic inherits pt_role::pt_base {
  notify { "Applying pt_role::pt_tools_weblogic": }
  $ensure   = hiera('ensure')
  contain ::pt_profile::pt_weblogic
  Class['::pt_profile::pt_weblogic'] 
}

The role file uses the ensure attribute from the defaults.yaml file to determine if Puppet should install or remove WebLogic. The default value is to install (ensure = present).

Next, the role calls a new profile called pt_profile::pt_weblogic. Let’s create that file net.

WebLogic DPK Profile

A DPK profile handles the configuration lookup from Hiera and preparing Puppet for the installation (or removal). Our custom profile grabs the installation directory (oracle_base and weblogic_location) from the psft_customizations.yaml file. WebLogic requires Java too, so we configure that in the profile as well.

Create a new profile under etc\modules\pt_profile\manifests and call it pt_weblogic.pp.

# Custom Profile to prepare for WebLogic installation
class pt_profile::pt_weblogic {
  notify { "Applying pt_profile::pt_weblogic": }

  ## Hiera lookups
  $ensure                    = hiera('ensure')
  $env_type                  = hiera('env_type')

  $tools_archive_location      = hiera('archive_location')

  $jdk_hiera             = hiera('jdk')
  $jdk_location          = $jdk_hiera['location']
  $jdk_remove_value      = $jdk_hiera['remove']
  if $jdk_remove_value == false {
    $jdk_remove = false
  }
  else {
    $jdk_remove = true
  }
  notice ("JDK remove is ${jdk_remove}")

  $weblogic_hiera        = hiera('weblogic')
  $weblogic_location     = $weblogic_hiera['location']
  $weblogic_remove_value = $weblogic_hiera['remove']
  if $weblogic_remove_value == false {
    $weblogic_remove = false
  }
  else {
    $weblogic_remove = true
  }
  notice ("Weblogic remove is ${weblogic_remove}")

  $redeploy = hiera('redeploy', false)
  class { '::pt_setup::weblogic_deployment':
    ensure                 => $ensure,
    tools_archive_location => $tools_archive_location,
    inventory_location     => $inventory_location,
    jdk_location           => $jdk_location,
    jdk_remove             => $jdk_remove,
    weblogic_location      => $weblogic_location,
    weblogic_remove        => $weblogic_remove,
    redeploy               => $redeploy,
  }
  contain ::pt_setup::weblogic_deployment
}

WebLogic DPK Deployment

Finally, we’ll call the Puppet code to install Java and WebLogic! We are using two DPK types in this manifest:

  • pt_deploy_jdk
  • pt_deploy_weblogic

This is why the DPK needs to be installed before using these manifests. If the DPK is missing, these calls will fail (and so will some of the hiera lookups). The deployment manifest is under the folder etc\modules\pt_setup\manifests. Create the file weblogic_deployment.pp from the code below or from the GitHub project.

# Custom Puppet Class to deploy WebLogic using the DPK
class pt_setup::weblogic_deployment (
  $ensure                 = present,
  $deploy_pshome_only     = false,
  $tools_archive_location = undef,
  $tools_install_user     = undef,
  $tools_install_group    = undef,
  $oracle_install_user    = undef,
  $oracle_install_group   = undef,
  $db_type                = undef,
  $pshome_location        = undef,
  $pshome_remove          = true,
  $inventory_location     = undef,
  $oracleclient_location  = undef,
  $oracleclient_remove    = true,
  $jdk_location           = undef,
  $jdk_remove             = true,
  $weblogic_location      = undef,
  $weblogic_remove        = true,
  $tuxedo_location        = undef,
  $tuxedo_remove          = true,
  $ohs_location           = undef,
  $ohs_remove             = true,
  $redeploy               = false,
) {

  notice ("Applying pt_setup::weblogic_deployment")

  $jdk_tag          = 'jdk'
  $weblogic_tag     = 'weblogic'

  $jdk_archive_file      = get_matched_file($tools_archive_location, $jdk_tag)
  if $jdk_archive_file == '' {
    fail("Unable to locate archive (tgz) file for JDK in ${tools_archive_location}")
  }
  $weblogic_archive_file = get_matched_file($tools_archive_location, $weblogic_tag)
  if $weblogic_archive_file == '' {
    fail("Unable to locate archive (tgz) file for Weblogic in ${tools_archive_location}")
  }

  $jdk_patches = hiera('jdk_patches', '')
  if ($jdk_patches) and ($jdk_patches != '') {
    notice ("JDK patches exists")
    $jdk_patches_list = values($jdk_patches)
  }
  else {
    notice ("JDK  patches does not exists")
    $jdk_patches_list = undef
  }

  pt_deploy_jdk { $jdk_tag:
    ensure            => $ensure,
    deploy_user       => $tools_install_user,
    deploy_user_group => $tools_install_group,
    archive_file      => $jdk_archive_file,
    deploy_location   => $jdk_location,
    redeploy          => $redeploy,
    remove            => $jdk_remove,
    patch_list        => $jdk_patches_list,
  }

  $weblogic_patches = hiera('weblogic_patches', '')
  if ($weblogic_patches) and ($weblogic_patches != '') {
    notice ("Weblogic patches exists")
    $weblogic_patches_list = values($weblogic_patches)
  }

  pt_deploy_weblogic { $weblogic_tag:
    ensure                    => $ensure,
    deploy_user               => $tools_install_user,
    deploy_user_group         => $tools_install_group,
    archive_file              => $weblogic_archive_file,
    deploy_location           => $weblogic_location,
    oracle_inventory_location => $inventory_location,
    oracle_inventory_user     => $oracle_install_user,
    oracle_inventory_group    => $oracle_install_group,
    jdk_location              => $jdk_location,
    redeploy                  => $redeploy,
    remove                    => $weblogic_remove,
    patch_list                => $weblogic_patches_list,
    require                   => Pt_deploy_jdk['jdk'],
  }

}

The manifests under pt_setup will do the heavy work of installing new software. This custom manifest will deploy Java and WebLogic on our servers. If your bootstrap script didn’t create a dpk\archives folder, the installation will fail. The DPK manifests (and our custom one) will look for the archives before it attempts to install.

Installation

Once your custom manifests are built, go to the puppet\etc\manifests folder. Run the command

puppet apply .\weblogic.pp --trace --debug

to install WebLogic.

Future Improvements

When I did the test, I tried to follow the Roles and Profiles pattern used by the DPK. It might seem complex for a smaller installation. You could combine all this into one manifest, but then it gets ugly to maintain. When we add more software components, the abstraction between configuration and implementation allows us to re-use Puppet code.

This was a test for myself to see if I could break down some of the abstraction in the DPK and write (or modify) manifests to control more of the process. Plus, we have DPK role that we can use for WebLogic-only deployments 🙂

#26 – WebLogic Filters

This week we talk about HR Image 17, the new Security Automation tool, and share some comments from David Kurtz about PeopleSoft on 12c. Then, Kyle dives into WebLogic Servlet Filters and shares how filters can be used with PeopleSoft.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Script WebLogic and Java Patches

In December, we talked quite a bit about patching Java and WebLogic on the blog and podcast. There was a WebLogic CVE, and then a patch, to apply. If you want a recap on the CVE and patching process, here are the posts:

While applying the patches, I wanted to script the process so patching would be consistent across all our servers. I pulled the scripts into a GitHub project for sharing and reuse. If you haven’t scripted a WebLogic patch, this would be a place to start. The scripts use PowerShell and built for WebLogic 10.3.6. So, they use SmartUpdate instead of OPatch. I also added in a Java patch to the process too. You could pull out the Java patch script to use by itself. One more note: all the patches, Java, and scripts were set to run from the folder e:\installers\weblogic1036-2015-CVE-Patches. If you use these for your environment, or just use them as a template, you’ll want to update those paths for your specific configuration.

There is nothing ground-breaking about these scripts 🙂 I can write scripts, but I’m not the best script developer out there. If you see places where the scripts need improvement, file an issue with the project or submit a pull request! The main goal with this project and post is to get others started with scripting. Scripting, even if the scripts are basic, can benefit administrators. I hope that this quick overview might help someone get started.

Scripts Overview

These scripts are writtin in PowerShell. If PowerShell scripts are not enabled on the server, run this command to allow PowerShell scripts to run:

set-executionpolicy unrestricted

  1. Install new SmartUpdate version (3.3.0)

    installSmartUpdate.ps1

    The silent.xml file is used for a silent install (no prompts). The installation directory is set to e:\oracle. If you want a different directory, change the value for “BEAHOME”. 1. Stop all web servers running on the server .stopPIAServices.ps1 The script looks for any Windows service that containts “*-PIA” in the name. If you have any WebLogic domains were not created by the

    installNTService script, you may need to shut them down by hand.

  2. Prepare and copy files from the weblogic1036-2015-CVE-Patches folder

    prepareFiles.ps1

    This script performs tasks to prepare different files for patching: On our servers, two files needed updates to run the Smart Update utility. registry.xml needed to remove a reference to Tuxedo; bsu.cmd needed an increase in memory to the Java Heap. The registry.xml file also contains a reference to the server where it was installed. The script will change that value based on the new server’s name. The original files are backed up first and a .bkp extension is added to the file name. The script also copies jdk-1.7.0_79 to our e:\java folder. If you want the new java version in a different location, you can change the path in the file.

  3. Apply both WebLogic patches The patches we are applying resolve the December 2015 CVE with WebLogic. If you are using these scripts for future patches, you’ll want to update the patch ID’s in the script.

    applyWebLogicPatches.ps1

    Both patches are applied to WebLogic using the bsu command. The script assumes your patches are in the folder e:\patches\cve-2015-4852. NOTE: On one of our servers, the second patch stalled during the “Checking for Conflicts” step. If the script stalls for more than a few minutes, hit Cntl-C.

  4. Update the JAVA_HOME values

    updateJavaVersion.ps1

    The JAVA_HOME value in the setEnv.cmd script will be updated to the new path. You must update this script for each server. The paths in the script are hard-coded. (The hard coding is an obvious candidate to fix next. Should be able to use the Get-ChildItem cmdlet to find all the setEnv.cmd files.)

  5. Update Registry value for JAVA_HOME

    updateRegistryJavaVersion.ps1

    The JAVA_HOME value in the Registry for each web service will be updated. You must update this script for each server. The paths in the script are hard-coded. (Again, another place for improvement. Need to find a search cmdlet for the Registry. Could look for -PIA in the service name.)

  6. Start all web servers running on the server.

    startPIAServices.ps1

    Again, this looks for all Windows services that have *-PIA in the name and starts them. That’s it.

The scripts are pretty simple, and you can write a wrapper script to run all the sub-scripts. That way you’d have one script to kick off. Or, you could add these into a tool like Rundeck to execute from a centralized place. Once you start down the path of scripting, many opportunities open up to speed up everyday tasks.

#8 – PeopleTools 8.55 Hands-on

Merry Christmas and Happy Holidays! We have an extra long and fun episode this week but you may want to listen to this it twice 🙂

In Episode 8, we dive into PeopleTools 8.55 and get hands-on! We talk about our first experiences with the PeopleTools DPK’s, changes to Change Assistant in 8.55 and Dan’s experiment with the Let’s Encrypt project. We also ask a big question about a new Related Content feature, and then answer it after the break. This feature has the potential to substantially reduce your customizations!

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Let’s Encrypt with PeopleSoft

Let’s Encrypt is a service provided by the Internet Security Research Group to provide free SSL certificates to anyone. The goal of the project is get the entire web encrypted. I mentioned the project in Episode 7 of The PeopleSoft Administrator Podcast and thought it would be a great exercise to try it with PeopleSoft.

Let’s Encrypt uses a client on the server to automate the certificate request process. The client will:

  • Validate that you own the web server
  • Generate a CSR
  • Download the certificate
  • Apply the certificate to the web server (limited support)
  • Automatically renew the certificate

There are a few requirements to use the Let’s Encrypt clients though:

  • The web server needs to accessible by the internet. The Let’s Encrypt site will validate that you own the server by checking for a specific file on the web server.
  • Not all operating systems are supported, yet.
  • Some web server’s have built-in support (IIS, Apache), but others do not (e.g, WebLogic). We can still generate certificates though, the automatic renewal won’t update the webserver though.

Install Let’s Encrypt Client for Windows

We’ll use the letsencrypt-win-simple command line client for Windows. Download the latest release from GitHub and extract the folder to a permanent location.

Generate a new certificate

  1. Run .letsencrypt.exe --accepttos[code lang=text]
    Let’s Encrypt (Simple Windows ACME Client)

    ACME Server: https://acme-v01.api.letsencrypt.org/
    Config Folder: C:UsersAdministratorAppDataRoamingletsencrypt-win-simplehttpsacme-v01.api.letsencrypt.org
    Loading Signer from C:UsersAdministratorAppDataRoamingletsencrypt-win-simplehttpsacme-v01.api.letsencrypt.orgSign
    er

    Getting AcmeServerDirectory
    Loading Registration from C:UsersAdministratorAppDataRoamingletsencrypt-win-simplehttpsacme-v01.api.letsencrypt.orgRegistration

    Scanning IIS 7 Site Bindings for Hosts
    No IIS bindings with host names were found. Please add one using IIS Manager. A host name and site path are required to
    verify domain ownership.
    No targets found.

    M: Generate a certificate manually.
    A: Get certificates for all hosts
    Q: Quit
    Which host do you want to get a certificate for:
    [/code]

  2. Since we are not running IIS, we’ll generate a certificate manually.[code lang=text]
    Which host do you want to get a certificate for: M
    Enter a host name:
    [/code]
  3. Enter the DNS name for your web server.[code lang=text]
    Enter a host name: hr.psadmin.io
    Enter a site path (the web root of the host for http authentication):
    [/code]
  4. Next, enter the root path for your web server. If you are running WebLogic, that will be PORTAL.war directory on your web server.[code lang=text]
    Enter a site path (the web root of the host for http authentication): W:pt8.55webservpeoplesoftapplicationspeoplesoftPORTAL.war
    [/code]
  5. Then, the Let’s Encrypt client will create a new file under PORTAL.war.well-knownacme-challenge. That file will be used to validate that you own the web server.[code lang=text]
    Authorizing Identifier ps92t855.psadmin.io Using Challenge Type http-01
    Writing challenge answer to W:pt8.55webservpeoplesoftapplicationspeoplesoftPORTAL.war.well-known/acme-challenge
    /1c2yN7Y93sJwRUmRGaoG4kT-QynrIcGr4szre-3nTsQ
    Answer should now be browsable at http://ps92t855.psadmin.io/.well-known/acme-challenge/1c2yN7Y93sJwRUmRGaoG4kT-QynrIcG
    r4szre-3nTsQ
    Submitting answer
    Refreshing authorization
    Authorization Result: valid
    Deleting answer
    [/code]
  6. After the web server ownership is verfied, new certificates will generated and copied to your system. The certificates are copied to your %USERPROFILE%AppDataRoamingletsencrypt-win-simple folder in a few formats:
    • .der
    • .pem
    • .pfx

    The client will also add the certificates to the Windows Certificate Store for you. To add the certificates to WebLogic, we’ll use the .pem

    [code lang=text]
    Requesting Certificate
    Request Status: Created
    Saving Certificate to C:UsersAdministratorAppDataRoamingletsencrypt-win-simplehttpsacme-v01.api.letsencrypt.orghr.psadmin.io-crt.der
    Saving Issuer Certificate to C:UsersAdministratorAppDataRoamingletsencrypt-win-simplehttpsacme-v01.api.letsencrypt.orgca-009813F47513E5750B43E7431E971E44BD-crt.pem
    Saving Certificate to C:UsersAdministratorAppDataRoamingletsencrypt-win-simplehttpsacme-v01.api.letsencrypt.orghr.psadmin.io-all.pfx (with no password set)
    Opened Certificate Store “WebHosting”
    Adding Certificate to Store
    Closing Certificate Store
    WARNING: Unable to configure server software.
    Creating Task letsencrypt-win-simple httpsacme-v01.api.letsencrypt.org with Windows Task Scheduler at 9am every day.
    Renewal Scheduled Manual hr.psadmin.io (W:pt8.55webservpeoplesoftapplicationspeoplesoftPORTAL.war) Renew A
    fter 2/9/2016
    Press enter to continue.
    [/code]

Create a New pskey Keystore

Now that we have certificates, let’s create a new pskey file with the certificates. We’ll use Keystore Explorer to quickly generate the file.

  1. Open Keystore Explorer. (If it’s first time you’ve used it, follow the instructions to download the Unlimited Strength files).
  2. Create a new keystore file.
  3. Select the file type of “JKS”.
  4. Select “Tools > Import Key Pair”.
  5. Select the “OpenSSL” option.
  6. Deselect “Encrypted Private Key”.
  7. For the “OpenSSL Private Key File”, select the file hr.psadmin.io-key.pem.
  8. For the “Certificate(s) File”, select hr.psadmin.io-cert.pem
  9. Click “Import”.
  10. Enter an alias name that is descriptive. I used hr.psadmin.io-2015-12.
  11. Since the prive key was delivered without a password, we’ll want to enter one. Enter a password for the key pair.

Now you have the private and public key for your DNS entry in the keystore. Next, we need to add the root (and intermediate) certificates so that a chain of trust is established.

  1. In Keystore Explorer in our new keystore file, right-click on our certificate. Select “Edit Certificate Chain > Append Certificate”.
  2. Select the file ca-GUID-crt.pem and click “Append”.
  3. Save the file, give the keystore a password, and name the file pskey-2015-12.

Load Keystore into WebLogic

After importing the certificates into pskey-2015-12, we need to copy the file to the web server and tell WebLogic to use the new file. The integrationGateway.properties file will need to know about the new keystore as well.

  1. Copy the pskey-2015-12 file to your web server directory %PS_CFG_HOME%webservpeoplesoftpiaconfigkeystore.
  2. Log into the WebLogic console.
  3. Navigate to “Environment > Servers > PIA > Keystores”.
  4. Click the “Lock & Edit” button to allow editing.
  5. Click the “Change” button for the Keystores option.
  6. Select “Custom Identity and Custom Trust” and “Save”.
  7. In the “Custom Identity Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  8. In the “Custom Identity Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  9. In the “Custom Trust Keystore” box, change the file name to piaconfig/keystore/pskey-2015-12.
  10. In the “Custom Trust Keystore Passphrase” boxes, enter the keystore password you entered when saving the file in Keystore Explorer.
  11. Click Save.

WebLogic will look at the new keystore file. Next, we need to tell WebLogic certificate it should serve to users.

  1. Click on the “SSL” tab.
  2. Change the “Private Key Alias” to hr.psadmin.io-2015-12.
  3. In the “Private Key Passphrase” boxes, enter the password you gave the hr-psadmin.io-2015-12 keypair.
  4. Click Save.
  5. Click the “Activate Changes” button.

Update integrationGateway.properties

Before we reboot the WebLogic domain, we need to update the integrationGateway.properties file.

  1. On your web server, open the integrationGateway.properties file under %PS_CFG_HOME%webservpeoplesoftapplicationspeoplesoftPSIGW.warWEB-INF.
  2. Find the line secureFileKeystorePath and change file name to pskey-2015-12.
  3. If the password you gave the keystore is different than the previous file, you’ll need to update that parameter in the file.
    1. Open a command prompt and go to %PS_CFG_HOME%webservpeoplesoftbin.
    2. Run the command setEnv.cmd to set the environment variables.
    3. Go to the folder piabin.
    4. Run the command PSCipher to get the encrypted text.
  4. Restart your WebLogic domain.

Test your HTTPS Connection

As WebLogic is starting up, make sure to check the logs to verify that the server started with your new certificate. Once the server has started, open a browser and go test the site. You should see a secure connection in the browser to your site.