#109 – Technical Debt

This week on the podcast, Kyle and Dan share their strategies of how to deal with technical debt.

Show Notes

psadmin.conf: Security

The third set of psadmin.conf videos are available! Security in an important part of administering PeopleSoft. Brad Carlson and Ruth Dodson share how to apply CPU patches in 21 days. Nate Werner discusses advanced security tools that admins should know about. Greg Wendt explains how the ERP Firewall can protect PeopleSoft and Guru Naranaswamy shows how to encrypt the Report Repository.

We have released the videos as a free course so you can find the videos in one place. Head over to the psadmin.io courses page and sign up. If you have already signed up for the course, you can log in and the videos will be available.

#104 – Page and Field Configurator

This week on the podcast Dan and Kyle discuss an password vulnerability, learning VIM and migrating 2 Facter Auth devices. Then they review the new Page and Field Configurator utility (spoiler: it’s great).

Show Notes

Understanding the Check Token ID in PeopleTools 8.56

This is a guest post by a PeopleSoft security researcher.

PeopleSoft has introduced a new parameter on Node Definitions called the CheckTokenID. This parameter is required if you plan on using PeopleSoft Single Signon. In this post we will take a look at what exactly this Check Token feature is, how it works, and lastly some implications.

What is Check Token?

Check Token is a response from Oracle to combat vulnerabilities with PS_TOKEN and the ability to execute brute force attacks on the node password. Should the node password be recovered attackers would be able to create “authentic” PS_TOKENs. This was covered in a previous podcast

Check Token involves a set of low level changes to how the psp and psc servlets (and related Java code) handle the authentication process. These changes harden the servlets against such forged tokens.

However the process by which Check Token improves the security of PS_TOKEN is not very well detailed and for the remainder of this post we will look at the internals of how this set of changes improves security.

Check Token Pieces

The changes made in 8.56 for the Check Token feature can be split into 2 general sections:

  1. The Token Lookup
  2. The Knock Knock Request

The “Token Lookup” portion is a security precaution added to ensure that the PS_TOKEN being provided was actually issued by the PeopleSoft system.

The Knock Knock Request is a mechanism by which the target application can “phone home” to the site that generated the token and ask them if the Token is good.

We will go more into the details of how this is accomplished in the Step By Step section below.

Environment Setup

For the following sections assume that we have the following PeopleSoft Environments:

  • Interaction Hub (IH) on Tools 8.56
  • CRM on Tools 8.56
  • HCM on Tools 8.55

As part of configuring PeopleSoft SSO on 8.56, each Node that will participate (and is 8.56 or later) must have the Check Token ID field populated:

This CheckTokenID field must match for each node in each environment (for example, the IH node has the same value in IH and CRM, and the CRM node has the same value in IH and CRM).

Pressing the Create CheckTokenID button will securely generate a random 189 bytes (252 characters when encoded to base64). The page will also present you this value to copy out for use in the other environments.

You are able to specify any value you want for the CheckTokenID, but for the most security it is better to use the Create button.

There is more to the SSO setup than this, but this is the only portion relevant to the blog post.

An 8.56 SSO Event Step by Step

Now that the environments have been configured for SSO, the best way to show how Check Token works is to go Step by Step through an SSO event. That is when a user with an active session in the IH tries to access a page in CRM and triggers PeopleSoft SSO.

Initial Access

When a user hits a CRM page with a PS_TOKEN issued by IH, the psp or psc servlet initiates an authentication attempt by calling the authenticate method in the PSAuthenticator class (this is done in the servlet’s onLogin method). The PSAuthenticator class eventually calls an isAuthTokenValidFromRequest method. Inside this method it makes a couple of checks:

  1. Is the App Server a “portal” server
  2. Is this request from the “same site”

The “same site” is determined by looking at a cookie that is new to 8.56 called PS_LASTSITE.

If either 1 or 2 are true, then the “knock knock” portion of the CheckToken changes is not executed. In this scenario only the “Token Lookup” part is executed. The Token Lookup grabs the web server session (via the JSESSION cookie) and asks for the PS_TOKEN from the web server, it then compares this PS_TOKEN to the one that it has been presented and the check only succeeds if they match.

This in effect prevents forged tokens from being used, as they will be rejected since the token issued by the application itself wont match.

However, in our scenario the user just left IH and hit a CRM page. Because of this #1 isn’t true and PS_LASTSITE will be an IH url so #2 isn’t true. In this case the Authenticator starts the CheckToken “Knock Knock” portion.

CheckToken Request

Since CRM has decided that the Knock Knock should be performed, it begins the Knock Knock portion by calling doKnockKnockRequest of the PSCheckToken class.

The Knock Knock portion has the following steps:

  1. Get the Last Site (PS_LASTSITE cookie)
  2. Append ?cmd=checkToken to the URL from #1
  3. Form a POST request
  4. Send Request

The POST request is blank in terms of the content that gets posted, however it does contain a copy of all headers from the inbound request to CRM (including cookies like PS_TOKEN and JSESSION). The Request is sent off to the URL of the Last Site (in our case Interaction Hub) and PSCheckToken class waits for a response.

CheckToken Processing in IH

When IH receives the HTTP POST with ?cmd=checkToken the psp or psc servlet handles this in the process method. The process of handling the checkToken command is 2 steps, first IH will perform a “Token Lookup” action to ensure the IH really did issue this token. If IH did not, an exception is raised and CRM will not get a valid response.

If IH did issue the token it then begins to form the Knock Knock Response. At a high level the Knock Knock Response value consists of a digest and a nonce. A nonce is generated by using the generateNonce method of PSTrustAuthUtil this Nonce is generated by Base64 encoding 16 random bytes (source of random is SHA1PRNG).

The Digest itself is calculated by hashing the PS_TOKEN, the “knock knock constant”, the Nonce*, and the PS_TOKEN‘s issue date. The PSCheckToken class supports both SHA1 and SHA256, however the method createKnockKnockDigest appears to hard-code the algorithm to be SHA256.

The “knock knock constant” referenced above is simply the “hashed” version of CheckTokenID that was set on the Node Definition. Hashed is in quotes because its not truly a hash, but rather it is encrypted with the same algorithm that Node Passwords are. Try setting a node password and a CheckTokenID to the same value and then query the PSMSGNODEDEFN table to see that this is true.

Once the digest is calculated, the Nonce that was chosen by IH is then appended to the resulting hash and this value is set in the PS_CTDIGEST header. The * next to “Nonce” is because there seems to be a cryptographic oversight (at least in the 8.56 versions I’ve looked at). When using SHA1 the Nonce is correctly included in the hashing, but when SHA256 is used the Nonce is NOT included in the hash like it should be: ![Nonce Issue for SHA256][2]

CheckToken Response Processing

Once IH has generated the response and sent it back to CRM, the CRM application server continues execution of replyFromKnockKnockRequest in the PSCheckToken class. This method retrieves the PS_CTDIGEST header and splits the value into 2 parts, the digest and the nonce. This is doable because hashes have a determinate length so it is easy to grab the Nonce off the end.

Once the values are split CRM goes ahead and generates the same digest, using its copy of the CheckTokenID value for the appropriate node as well as the Nonce*. If the generated digest matches the digest that IH sent, authentication succeeds and the user is successfully SSO’d into CRM! The * next to “Nonce” is for the same reason as earlier.

Applications < 8.56

As mentioned in the environment setup we are pretending to have an HCM environment that is below 8.56. Let’s briefly discuss how Oracle kept compatibility for this mode of operation (IH on 8.56 and other apps lower than that).

The way Oracle engineered this change puts the responsibility of determining when to enforce CheckToken on the target application (the one that is being SSO’d to). Oracle deliberately made no changes to the format of PS_TOKEN itself so that tokens issued by 8.56 would still be valid on 8.55 systems.

Because our HCM environment knows nothing about Check Token, it will never determine that it needs to ask the IH to check the PS_TOKEN. And because of this the PeopleSoft SSO operates effectively the same way it has been.

What about that Nonce thing?

As mentioned above there seems to have been an oversight with the generation of the Knock Knock response when using SHA256 (not including the Nonce in the hash). Nonce’s are intended to prevent the ability to replay a request. So while technically the lack of the Nonce in the hash renders this protection moot, ultimately the attacker would not be able to get back to “Token Lookup” phase with a forged token because the Application Server will not find that token as not having been issued.

What is important to note is that should Oracle release patches to fix this oversight, they would need to be applied to all systems, if only applied to the CRM in our example, the digest that IH produces would no longer match what CRM produces.

Using Security Deployment with Custom Security

In recent PeopleSoft Image updates, a new tool for implementing security changes was introduced. The Security Deployment tool simplifies the application of new security changes to your environment. But, the Security Deployment tool can be used to automate the application of custom security to your environments.

In the video below, I cover how to use the Security Deployment for migrating custom security changes.

 

Apply CPU Patches with Deployment Packages

We have talked on the podcast about different ways to apply CPU patches, but with the DPK we have another tool to help us quickly apply CPU patches. This post and video demo’s will show you how to use the DPK to quickly apply CPU patches to your servers.

Deployment Workflow

When you run the DPK, it will deploy WebLogic, Java, Tuxedo (and more) on your server. The DPK uses archives (also known as “tarballs”) of prepackaged installations and extracts those archives to your server. There is one big problem, the archives included in the DPK’s do not contain the latest security patches. So, let’s make our own tarballs that include the security patches to deploy. This process is also a great exercise to better understand how the DPK deploys software.

If you are on Linux you can use the patching functionality with the DPK, but that code has not been written for Windows. I’m not covering that feature in this post, but the DPK Install Guide has a section on using that functionality (Task 6-3-1: Using the DPK Setup Script to Apply Fixes).

Movement Scripts

There are Fusion Middleware scripts the DPK uses to deploy WebLogic and Tuxedo. (Thanks to Eric Bolinger for pointing me in this direction.) The movement scripts allow you to take a current install of WebLogic, package it up, and deploy it to additional servers. This is how the DPK deploys WebLogic. The PeopleTools team packages up a WebLogic installation and we deploy that install to our servers. The movement scripts also manage the Oracle Inventory file for you.

There are many parts to the movement scripts, but we’ll be using just one part: copyBinary. This script will take a current installation and create a .jar file from that installation. We’ll use copyBinary to package our patched WebLogic installation.

If you have errors with the pasteBinary.cmd on the target system, you may need to configure the $ORACLE_HOME\oui\oraparam.ini file. This is a configuration file used by the OUI software. To make this simple, I copied the settings in the current $BASE\dpk\archives\weblogic12.1.3.0.tgz to my $ORACLE_HOME\oui\oraparam.ini using Beyond Compare. (Yes, Beyond Compare can read inside a tarball and compare against a directory!) Then I recreated my tarball with the updated oraparam.ini file.

Create a Patched WebLogic Tarball

 

Next, it’s time to install the CPU patch and run the copyBinary.cmd script. Stop all your PIA services on the server so you can remove the existing installations.

First, let’s patch Java. For demonstration, I’m using the jdk-7u141-windows-x64 installer. I’m installing

Then, we’ll use OPatch to apply the CPU to WebLogic:

cd $PATCH
$env:ORACLE_HOME=e:\psoft\pt\bea
$env:ORACLE_HOME\OPatch\OPatch napply

Once OPatch is done, we’ll use the movement scripts to package up our installation.

$env:JAVA_HOME=e:\psoft\pt\jdk1.7.0_141
. ${env:ORACLE_HOME}\oracle_common\bin\copyBinary.cmd -javaHome ${env:JAVA_HOME} -archiveLoc ${env:TEMP}\pt-weblogic-copy.jar -sourceMWHomeLoc ${env:ORACLE_HOME}

The output file from this command needs to be named pt-weblogic-copy.jar. The DPK expects that is the name of the .jar file. Next, we create a tarball of the pt-weblogic-copy.jar and two files to do the deploy portion of the movement scripts: cloningclient.jar and pasteBinary.cmd. These movement scripts are used by the DPK to deploy WebLogic. I used 7-zip to create my tarball with these three files:

$env:WL_VERSION="12.1.3.170418"
7z a -ttar "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\jlib\cloningclient.jar"
7z a -ttar "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\bin\pasteBinary.cmd"
7z a -ttar "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar" "${env:TEMP}\pt-weblogic-copy.jar"

Last, we gzip the archive and drop it in the $BASE\dpk\archives folder:

$env:DPK_BASE="e:\psft"
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-weblogic${env:WL_VERSION}.tgz" "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar"

One thing to note here – the DPK doesn’t handle multiple versions of software in the dpk\archives folder well. So, only have one pt-weblogic* file in there.

For Java, we don’t need to use the movement scripts. We’ll simply tarball up the new directory and include that in our $BASE\dpk\archives folder.

$env:JDK_VERSION="1.7.0_141"
7z a -ttar "${env:TEMP}\pt-jdk${env:JDK_VERSION}.tar" $env:JAVA_HOME\*
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-jdk${env:JDK_VERSION}.tgz" "${env:TEMP}\pt-jdk${env:JDK_VERSION}.tar"

Deploy CPU Patches

 

Copy your updated tarballs to a new server. You’ll want to remove the existing tarballs from the $BASE\dpks\archive to prevent the DPK from raising an error.

We have two options for telling the DPK we want to install WebLogic. The first option is to delete the existing WebLogic and Java folders. If you stop your PeopleSoft domains, you can delete both folders. When you run the DPK it will see that WebLogic and Java are missing and reinstall them from the patched tarballs in the $BASE\dpk\archives folder.

The other option is use the redeploy: true flag in psft_customizations.yaml. If you set the redeploy variable to true, the DPK will redeploy all the software in your $BASE\dpk\archives folder. This option requires less work – set a variable in psft_customizations.yaml and run the DPK – but it can take longer because you will redeploy Java, Tuxedo, WebLogic, PS_HOME and more. I think of this option as “the Puppet way”.

For this post and demo, we’ll use the redeploy: true option in our psft_customizations.yaml file. We’ll also use one other trick for testing; we will only run the part of the DPK that handles the middleware. Instead of running the entire DPK that touches the OS, middleware, and domains, the manifest we call includes only the DPK role that ensures the middleware is installed and not touch other parts of the system. This will also speed up our CPU patch deployment.

middleware.pp

Let’s create a new file under c:\programdata\puppetlabs\puppet\etc\manifests called middleware.pp. You can start by cloning the site.pp file. Change the file to look like this:

node default {
  include ::pt_role::pt_tools_deployment
}

Save the file. That’s it!

What we have done is tell Puppet to only run the DPK role pt_tools_deployment instead of running a larger role like pt_hcm_pum.

In the video demo, we are applying patches to a PeopleSoft Image, which is a Fulltier setup. The default pt_tools_deployment.pp manifest won’t run on a Fulltier system. To get around that, I created a copy of pt_tools_deployment.pp manifest called io_tools_deployment.pp and removed the check on env_type: fulltier.

cpu.ps1

We have a few tasks to do before we can run the middleware.pp manifest. We’ll wrap those tasks in a Powershell script we can run on each server.

At a high level, here are the tasks our cpu.ps1 script will do:

  1. Copy new DPK archives to server
  2. Stop PeopleSoft Services
  3. Remove current Java and WebLogic installs (if redeploy: false)
  4. Run middleware.pp to install patched Java and WebLogic
  5. Start PeopleSoft Services

Get the Sample Code

The full code is in the ps-dpk-tarballs GitHub repository. You can find all the scripts from this post and demo on GitHub.

#75 – Selling Yourself

This week, Dan and Kyle talk about testing different web server configurations, using the ACM for Elasticsearch, and how mobile browsers work with websites. Then, they discuss different ways to promote yourself and your position to a boss or organization.

Show Notes

#74 – Killing COBOL

This week on the podcast, Kyle and Dan talk about planning PeopleTools and Catch-up projects, BI Publisher security, and how to turn off excessive BI Publisher logging. We also talk about slowly killing COBOL with PeopleSoft (it’s not dead yet) and using multiple Change Assistant installations.

Show Notes

Running Change Assistant without “as Administrator”

One annoyance with Change Assistant (among a few) is that you have to start it “as Administrator”. If you don’t, you’ll get the message “Another instance of Change Assistant is already running” (even though it’s not). While running a program “as Administrator” is not hard, there is no reason why Change Assistant needs Administrative rights. (At least that I know of).

Folder Security

The fix to run Change Assistant without Administrator is to set the folder security permissions correctly. If you install Change Assistant to the default directory, C:\Program Files\PeopleSoft\Change Assistant, the Change Assistant folder security needs to be updated. Grant the user (or group) who will be running Change Assistant Full Control over the directory. In my case, I granted the group “Authenticated Users” full access to the folder.

Screen Shot 2017-03-08 at 5.17.53 PM

Now you can start Change Assistant as a normal application.

PeopleTools Idea

There is an Idea on the Oracle PeopleSoft Space for the PeopleTools team to fix this. If you want to vote for the Idea, you can do that here.

I don’t remember where I originally saw this solution, so I can’t give appropriate credit, but I want to thank whoever posted about this in the past.

pscipher, psvault and Web Server Passwords

Encrypting passwords is a common activity for PeopleSoft administrators. For many of us, we take for granted that passwords are encrypted by the system. In this post, I want to look at password encryption for web server configuration files and how that works.

Encrypting with pscipher

pscipher is a PeopleTools utility that encrypts passwords using 3DES encryption. The utility is a wrapper for a Java class that handles the encryption and decryption of passwords. If you look at the passwords stored in the configuration.properties file, or produced by pscipher, they look something like this: {V1.1}IsZtCVg15Ls=

To encrypt a password with pscipher:

  1. Navigate to your web server’s [PS_CFG_HOME]\webserver\[domain]\piabin folder.
  2. Run .\pscipher.bat [password]

pscipher will return the encrypted output:

.\PSCipher.bat password

Your environment has been set.
Encrypted text: {V1.1}7m4OtVwXFNyLc1j6pZG69Q==

You can copy/paste the encrypted text into your web server config files. For example, below is the pskey configuration in the integrationGateway.properties file using an encrypted password:

secureFileKeystorePath=e:/psft/cfg/LM014-8.55.09/webserv/peoplesoft1/piaconfig/keystore/pskey
secureFileKeystorePasswd={V1.1}7m4OtVwXFNyLc1j6pZG69Q==

You can also encrypt passwords with pscipher through the PIA too. Navigate to PeopleTools > IB > Configuration > Gateways. Select your a gateway and click “Gateway Setup Properties”. After you log into the gateway, select the link “Advanced Properties Page”. This page lets you modify the integrationGateway.properties file directly, but it also has a Password Encryption section. This Password Encryption tool calls pscipher on your application server to encrypt passwords.

encryptPasswordPIA

Building New Keys

The {V1.1} at the beginning of the password denotes which key pscipher uses. 1.1 means your passwords are using the default key. I highly recommend you change the key. To create a new key, run the command pscipher -buildkey. A new key will be appended to the file psvault. The pscipher command will now generate {V1.2} passwords. Appended is important here. This means that you can still use {V1.1} encrypted passwords in your configuration files and the newer {V1.2} encrypted passwords.

psvault Locations

The psvault file is stored under [PS_CFG_HOME]\webserv\[domain]\piaconfig\properties\psvault. When you run -buildkey, that is the file pscipher updates. After you update the key to {V1.2}, you need to copy the updated psvault file to any other web and app servers that you want to decrypt the new passwords.

  • For web servers, copy the updated psvault file to [PS_CFG_HOME\webserv\[domain]\piaconfig\properties\psvault.
  • For app servers, copy file to [PS_HOME]\secvault\psvault.

You should copy the updated psvault file to your app servers. When you update your integrationGateway.properties file online (PeopleTools > IB > Configuration > Gateways), any passwords you encrypt using the online pages are encrypted with the app server’s copy of psvault.

So far, I haven’t been able to get the Tuxedo domains to recognize the psvault file under PS_CFG_HOME, and a Oracle confirmed with me that PS_CFG_HOME is not supported with psvault and the app server. If you are using decoupled homes, this breaks some of the benefits of having a separate and shared PS_HOME codeline. I created an Idea on the Oracle Community site to add support for PS_CFG_HOME and psvault; go vote for the idea if would like decopupled home support too.