#88 – Fast and Furious w/ Brad Carlson

This week Brad Carlson from the University of Minnesota joins us to talk about some recent projects he worked on. Brad explains why and how they consolidated their PeopleSoft database, an IB Monitoring tool that was recently open-sourced, and how the U of M can deploy CPU patches in 21 days.

Show Notes

  • Introducing Brad @ 1:00
  • Database Consolidation @ 5:30
  • Benefits of a Single Database @ 11:00
  • IB MonitorService @ 34:30
  • Distributing Notifications @ 53:30
  • Fast CPU Patching (21 Days!) @ 58:30

#80 – Apply Your CPU Patches!

This week, Kyle and Dan talk about using Vagrant snapshots with Vagabond, strategies for managing psft_customizations.yaml files and demonstrating Fluid to end users. Dan shares a story of how an unpatched WebLogic server can leave your PeopleSoft application vulnerable to hackers.

Show Notes

  • Securing the Oracle Listener @ 1:30
  • Vagabond and Vagrant Snapshots @ 2:00
  • Vault and Hiera @ 7:45
  • Remote Desktop Spanning @ 14:00
  • Why You Apply CPU Patches – A Story @ 16:30
  • CPU Patching Wishlist @ 23:30
  • DPK and Middleware-only @ 33:15
  • psft_customizations.yaml strategies @ 39:00
  • Upgrading Interaction Hub @ 50:00
  • Demoing Fluid @ 57:00

Apply CPU Patches with Deployment Packages

We have talked on the podcast about different ways to apply CPU patches, but with the DPK we have another tool to help us quickly apply CPU patches. This post and video demo’s will show you how to use the DPK to quickly apply CPU patches to your servers.

Deployment Workflow

When you run the DPK, it will deploy WebLogic, Java, Tuxedo (and more) on your server. The DPK uses archives (also known as “tarballs”) of prepackaged installations and extracts those archives to your server. There is one big problem, the archives included in the DPK’s do not contain the latest security patches. So, let’s make our own tarballs that include the security patches to deploy. This process is also a great exercise to better understand how the DPK deploys software.

If you are on Linux you can use the patching functionality with the DPK, but that code has not been written for Windows. I’m not covering that feature in this post, but the DPK Install Guide has a section on using that functionality (Task 6-3-1: Using the DPK Setup Script to Apply Fixes).

Movement Scripts

There are Fusion Middleware scripts the DPK uses to deploy WebLogic and Tuxedo. (Thanks to Eric Bolinger for pointing me in this direction.) The movement scripts allow you to take a current install of WebLogic, package it up, and deploy it to additional servers. This is how the DPK deploys WebLogic. The PeopleTools team packages up a WebLogic installation and we deploy that install to our servers. The movement scripts also manage the Oracle Inventory file for you.

There are many parts to the movement scripts, but we’ll be using just one part: copyBinary. This script will take a current installation and create a .jar file from that installation. We’ll use copyBinary to package our patched WebLogic installation.

If you have errors with the pasteBinary.cmd on the target system, you may need to configure the $ORACLE_HOME\oui\oraparam.ini file. This is a configuration file used by the OUI software. To make this simple, I copied the settings in the current $BASE\dpk\archives\weblogic12.1.3.0.tgz to my $ORACLE_HOME\oui\oraparam.ini using Beyond Compare. (Yes, Beyond Compare can read inside a tarball and compare against a directory!) Then I recreated my tarball with the updated oraparam.ini file.

Create a Patched WebLogic Tarball

 

Next, it’s time to install the CPU patch and run the copyBinary.cmd script. Stop all your PIA services on the server so you can remove the existing installations.

First, let’s patch Java. For demonstration, I’m using the jdk-7u141-windows-x64 installer. I’m installing

Then, we’ll use OPatch to apply the CPU to WebLogic:

cd $PATCH
$env:ORACLE_HOME=e:\psoft\pt\bea
$env:ORACLE_HOME\OPatch\OPatch napply

Once OPatch is done, we’ll use the movement scripts to package up our installation.

$env:JAVA_HOME=e:\psoft\pt\jdk1.7.0_141
. ${env:ORACLE_HOME}\oracle_common\bin\copyBinary.cmd -javaHome ${env:JAVA_HOME} -archiveLoc ${env:TEMP}\pt-weblogic-copy.jar -sourceMWHomeLoc ${env:ORACLE_HOME}

The output file from this command needs to be named pt-weblogic-copy.jar. The DPK expects that is the name of the .jar file. Next, we create a tarball of the pt-weblogic-copy.jar and two files to do the deploy portion of the movement scripts: cloningclient.jar and pasteBinary.cmd. These movement scripts are used by the DPK to deploy WebLogic. I used 7-zip to create my tarball with these three files:

$WL_VERSION="12.1.3.170418"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\jlib\cloningclient.jar"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\bin\pasteBinary.cmd"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:TEMP}\pt-weblogic-copy.jar"

Last, we gzip the archive and drop it in the $BASE\dpk\archives folder:

$env:DPK_BASE="e:\psft"
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-weblogic${env:WL_VERSION}.tgz" "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar"

One thing to note here – the DPK doesn’t handle multiple versions of software in the dpk\archives folder well. So, only have one pt-weblogic* file in there.

For Java, we don’t need to use the movement scripts. We’ll simply tarball up the new directory and include that in our $BASE\dpk\archives folder.

$JDK_VERSION="1.7.0_141"
7z a -ttar "${env:TEMP}\pt-jdk${JDK_VERSION}.tar" $env:JAVA_HOME\*
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-jdk${JDK_VERSION}.tgz" "${env:TEMP}\pt-jdk${JDK_VERSION}.tar"

Deploy CPU Patches

 

Copy your updated tarballs to a new server. You’ll want to remove the existing tarballs from the $BASE\dpks\archive to prevent the DPK from raising an error.

We have two options for telling the DPK we want to install WebLogic. The first option is to delete the existing WebLogic and Java folders. If you stop your PeopleSoft domains, you can delete both folders. When you run the DPK it will see that WebLogic and Java are missing and reinstall them from the patched tarballs in the $BASE\dpk\archives folder.

The other option is use the redeploy: true flag in psft_customizations.yaml. If you set the redeploy variable to true, the DPK will redeploy all the software in your $BASE\dpk\archives folder. This option requires less work – set a variable in psft_customizations.yaml and run the DPK – but it can take longer because you will redeploy Java, Tuxedo, WebLogic, PS_HOME and more. I think of this option as “the Puppet way”.

For this post and demo, we’ll use the redeploy: true option in our psft_customizations.yaml file. We’ll also use one other trick for testing; we will only run the part of the DPK that handles the middleware. Instead of running the entire DPK that touches the OS, middleware, and domains, the manifest we call includes only the DPK role that ensures the middleware is installed and not touch other parts of the system. This will also speed up our CPU patch deployment.

middleware.pp

Let’s create a new file under c:\programdata\puppetlabs\puppet\etc\manifests called middleware.pp. You can start by cloning the site.pp file. Change the file to look like this:

node default {
  include ::pt_role::pt_tools_deployment
}

Save the file. That’s it!

What we have done is tell Puppet to only run the DPK role pt_tools_deployment instead of running a larger role like pt_hcm_pum.

In the video demo, we are applying patches to a PeopleSoft Image, which is a Fulltier setup. The default pt_tools_deployment.pp manifest won’t run on a Fulltier system. To get around that, I created a copy of pt_tools_deployment.pp manifest called io_tools_deployment.pp and removed the check on env_type: fulltier.

cpu.ps1

We have a few tasks to do before we can run the middleware.pp manifest. We’ll wrap those tasks in a Powershell script we can run on each server.

At a high level, here are the tasks our cpu.ps1 script will do:

  1. Copy new DPK archives to server
  2. Stop PeopleSoft Services
  3. Remove current Java and WebLogic installs (if redeploy: false)
  4. Run middleware.pp to install patched Java and WebLogic
  5. Start PeopleSoft Services

Get the Sample Code

The full code is in the ps-dpk-tarballs GitHub repository. You can find all the scripts from this post and demo on GitHub.

#76 – Hybrid Cloud w/ Jim Marion

This week on the podcast Jim Marion joins us to talk about Fluid Development, embedding PeopleSoft into cloud applications, Event Mapping, where the cloud makes sense and Jim’s new role at GreyHeller.

Show Notes

  • Jim’s Background @ 2:15
  • New Role with Grey Heller @ 3:45
  • Thoughts on the future of PeopleTools @ 6:00
  • Fluid Development @ 9:00
  • Improving App Designer @ 11:30
  • Event Mapping and Branding Objects @ 14:00
  • Using the Cloud at Grey Heller @ 25:45
  • Embedding PeopleSoft with Cloud Applications @ 32:00
  • Disadvantages of Hybrid Cloud Systems @ 43:30
  • Where the Cloud makes sense @ 47:30

#75 – Selling Yourself

This week, Dan and Kyle talk about testing different web server configurations, using the ACM for Elasticsearch, and how mobile browsers work with websites. Then, they discuss different ways to promote yourself and your position to a boss or organization.

Show Notes

#74 – Killing COBOL

This week on the podcast, Kyle and Dan talk about planning PeopleTools and Catch-up projects, BI Publisher security, and how to turn off excessive BI Publisher logging. We also talk about slowly killing COBOL with PeopleSoft (it’s not dead yet) and using multiple Change Assistant installations.

Show Notes

pscipher, psvault and Web Server Passwords

Encrypting passwords is a common activity for PeopleSoft administrators. For many of us, we take for granted that passwords are encrypted by the system. In this post, I want to look at password encryption for web server configuration files and how that works.

Encrypting with pscipher

pscipher is a PeopleTools utility that encrypts passwords using 3DES encryption. The utility is a wrapper for a Java class that handles the encryption and decryption of passwords. If you look at the passwords stored in the configuration.properties file, or produced by pscipher, they look something like this: {V1.1}IsZtCVg15Ls=

To encrypt a password with pscipher:

  1. Navigate to your web server’s [PS_CFG_HOME]\webserver\[domain]\piabin folder.
  2. Run .\pscipher.bat [password]

pscipher will return the encrypted output:

.\PSCipher.bat password

Your environment has been set.
Encrypted text: {V1.1}7m4OtVwXFNyLc1j6pZG69Q==

You can copy/paste the encrypted text into your web server config files. For example, below is the pskey configuration in the integrationGateway.properties file using an encrypted password:

secureFileKeystorePath=e:/psft/cfg/LM014-8.55.09/webserv/peoplesoft1/piaconfig/keystore/pskey
secureFileKeystorePasswd={V1.1}7m4OtVwXFNyLc1j6pZG69Q==

You can also encrypt passwords with pscipher through the PIA too. Navigate to PeopleTools > IB > Configuration > Gateways. Select your a gateway and click “Gateway Setup Properties”. After you log into the gateway, select the link “Advanced Properties Page”. This page lets you modify the integrationGateway.properties file directly, but it also has a Password Encryption section. This Password Encryption tool calls pscipher on your application server to encrypt passwords.

encryptPasswordPIA

Building New Keys

The {V1.1} at the beginning of the password denotes which key pscipher uses. 1.1 means your passwords are using the default key. I highly recommend you change the key. To create a new key, run the command pscipher -buildkey. A new key will be appended to the file psvault. The pscipher command will now generate {V1.2} passwords. Appended is important here. This means that you can still use {V1.1} encrypted passwords in your configuration files and the newer {V1.2} encrypted passwords.

psvault Locations

The psvault file is stored under [PS_CFG_HOME]\webserv\[domain]\piaconfig\properties\psvault. When you run -buildkey, that is the file pscipher updates. After you update the key to {V1.2}, you need to copy the updated psvault file to any other web and app servers that you want to decrypt the new passwords.

  • For web servers, copy the updated psvault file to [PS_CFG_HOME\webserv\[domain]\piaconfig\properties\psvault.
  • For app servers, copy file to [PS_HOME]\secvault\psvault.

You should copy the updated psvault file to your app servers. When you update your integrationGateway.properties file online (PeopleTools > IB > Configuration > Gateways), any passwords you encrypt using the online pages are encrypted with the app server’s copy of psvault.

So far, I haven’t been able to get the Tuxedo domains to recognize the psvault file under PS_CFG_HOME, and a Oracle confirmed with me that PS_CFG_HOME is not supported with psvault and the app server. If you are using decoupled homes, this breaks some of the benefits of having a separate and shared PS_HOME codeline. I created an Idea on the Oracle Community site to add support for PS_CFG_HOME and psvault; go vote for the idea if would like decopupled home support too.

#65 – CPU Patching

This week on the podcast, Dan and Kyle talk about using web traffic data to analyze user activity, new information on Jolt Failover, and how we generate and distribute compare reports. Then, they discuss Critical Patch Updates and how they affect PeopleSoft Administrators.

Show Notes