#107 – User Experience w/ Simon Chiu

This week Simon Chiu from the PeopleSoft UX Blog joins us to talk about his experiences with Interaction Hub, Unified Navigation, Push Push Notifications, Federated Search and the PeopleSoft Cloud Manager.

Show Notes

  • Introducing Simon @ 1:30
  • Using the Interaction Hub @ 4:45
  • Unified Navigation and the Interaction Hub @ 20:00
  • The good and bad of Push Notifications @ 32:45
  • Testing Federated Search @ 44:15
  • Cloud Manager @ 54:30

Understanding the Check Token ID in PeopleTools 8.56

This is a guest post by a PeopleSoft security researcher.

PeopleSoft has introduced a new parameter on Node Definitions called the CheckTokenID. This parameter is required if you plan on using PeopleSoft Single Signon. In this post we will take a look at what exactly this Check Token feature is, how it works, and lastly some implications.

What is Check Token?

Check Token is a response from Oracle to combat vulnerabilities with PS_TOKEN and the ability to execute brute force attacks on the node password. Should the node password be recovered attackers would be able to create “authentic” PS_TOKENs. This was covered in a previous podcast

Check Token involves a set of low level changes to how the psp and psc servlets (and related Java code) handle the authentication process. These changes harden the servlets against such forged tokens.

However the process by which Check Token improves the security of PS_TOKEN is not very well detailed and for the remainder of this post we will look at the internals of how this set of changes improves security.

Check Token Pieces

The changes made in 8.56 for the Check Token feature can be split into 2 general sections:

  1. The Token Lookup
  2. The Knock Knock Request

The “Token Lookup” portion is a security precaution added to ensure that the PS_TOKEN being provided was actually issued by the PeopleSoft system.

The Knock Knock Request is a mechanism by which the target application can “phone home” to the site that generated the token and ask them if the Token is good.

We will go more into the details of how this is accomplished in the Step By Step section below.

Environment Setup

For the following sections assume that we have the following PeopleSoft Environments:

  • Interaction Hub (IH) on Tools 8.56
  • CRM on Tools 8.56
  • HCM on Tools 8.55

As part of configuring PeopleSoft SSO on 8.56, each Node that will participate (and is 8.56 or later) must have the Check Token ID field populated:

This CheckTokenID field must match for each node in each environment (for example, the IH node has the same value in IH and CRM, and the CRM node has the same value in IH and CRM).

Pressing the Create CheckTokenID button will securely generate a random 189 bytes (252 characters when encoded to base64). The page will also present you this value to copy out for use in the other environments.

You are able to specify any value you want for the CheckTokenID, but for the most security it is better to use the Create button.

There is more to the SSO setup than this, but this is the only portion relevant to the blog post.

An 8.56 SSO Event Step by Step

Now that the environments have been configured for SSO, the best way to show how Check Token works is to go Step by Step through an SSO event. That is when a user with an active session in the IH tries to access a page in CRM and triggers PeopleSoft SSO.

Initial Access

When a user hits a CRM page with a PS_TOKEN issued by IH, the psp or psc servlet initiates an authentication attempt by calling the authenticate method in the PSAuthenticator class (this is done in the servlet’s onLogin method). The PSAuthenticator class eventually calls an isAuthTokenValidFromRequest method. Inside this method it makes a couple of checks:

  1. Is the App Server a “portal” server
  2. Is this request from the “same site”

The “same site” is determined by looking at a cookie that is new to 8.56 called PS_LASTSITE.

If either 1 or 2 are true, then the “knock knock” portion of the CheckToken changes is not executed. In this scenario only the “Token Lookup” part is executed. The Token Lookup grabs the web server session (via the JSESSION cookie) and asks for the PS_TOKEN from the web server, it then compares this PS_TOKEN to the one that it has been presented and the check only succeeds if they match.

This in effect prevents forged tokens from being used, as they will be rejected since the token issued by the application itself wont match.

However, in our scenario the user just left IH and hit a CRM page. Because of this #1 isn’t true and PS_LASTSITE will be an IH url so #2 isn’t true. In this case the Authenticator starts the CheckToken “Knock Knock” portion.

CheckToken Request

Since CRM has decided that the Knock Knock should be performed, it begins the Knock Knock portion by calling doKnockKnockRequest of the PSCheckToken class.

The Knock Knock portion has the following steps:

  1. Get the Last Site (PS_LASTSITE cookie)
  2. Append ?cmd=checkToken to the URL from #1
  3. Form a POST request
  4. Send Request

The POST request is blank in terms of the content that gets posted, however it does contain a copy of all headers from the inbound request to CRM (including cookies like PS_TOKEN and JSESSION). The Request is sent off to the URL of the Last Site (in our case Interaction Hub) and PSCheckToken class waits for a response.

CheckToken Processing in IH

When IH receives the HTTP POST with ?cmd=checkToken the psp or psc servlet handles this in the process method. The process of handling the checkToken command is 2 steps, first IH will perform a “Token Lookup” action to ensure the IH really did issue this token. If IH did not, an exception is raised and CRM will not get a valid response.

If IH did issue the token it then begins to form the Knock Knock Response. At a high level the Knock Knock Response value consists of a digest and a nonce. A nonce is generated by using the generateNonce method of PSTrustAuthUtil this Nonce is generated by Base64 encoding 16 random bytes (source of random is SHA1PRNG).

The Digest itself is calculated by hashing the PS_TOKEN, the “knock knock constant”, the Nonce*, and the PS_TOKEN‘s issue date. The PSCheckToken class supports both SHA1 and SHA256, however the method createKnockKnockDigest appears to hard-code the algorithm to be SHA256.

The “knock knock constant” referenced above is simply the “hashed” version of CheckTokenID that was set on the Node Definition. Hashed is in quotes because its not truly a hash, but rather it is encrypted with the same algorithm that Node Passwords are. Try setting a node password and a CheckTokenID to the same value and then query the PSMSGNODEDEFN table to see that this is true.

Once the digest is calculated, the Nonce that was chosen by IH is then appended to the resulting hash and this value is set in the PS_CTDIGEST header. The * next to “Nonce” is because there seems to be a cryptographic oversight (at least in the 8.56 versions I’ve looked at). When using SHA1 the Nonce is correctly included in the hashing, but when SHA256 is used the Nonce is NOT included in the hash like it should be: ![Nonce Issue for SHA256][2]

CheckToken Response Processing

Once IH has generated the response and sent it back to CRM, the CRM application server continues execution of replyFromKnockKnockRequest in the PSCheckToken class. This method retrieves the PS_CTDIGEST header and splits the value into 2 parts, the digest and the nonce. This is doable because hashes have a determinate length so it is easy to grab the Nonce off the end.

Once the values are split CRM goes ahead and generates the same digest, using its copy of the CheckTokenID value for the appropriate node as well as the Nonce*. If the generated digest matches the digest that IH sent, authentication succeeds and the user is successfully SSO’d into CRM! The * next to “Nonce” is for the same reason as earlier.

Applications < 8.56

As mentioned in the environment setup we are pretending to have an HCM environment that is below 8.56. Let’s briefly discuss how Oracle kept compatibility for this mode of operation (IH on 8.56 and other apps lower than that).

The way Oracle engineered this change puts the responsibility of determining when to enforce CheckToken on the target application (the one that is being SSO’d to). Oracle deliberately made no changes to the format of PS_TOKEN itself so that tokens issued by 8.56 would still be valid on 8.55 systems.

Because our HCM environment knows nothing about Check Token, it will never determine that it needs to ask the IH to check the PS_TOKEN. And because of this the PeopleSoft SSO operates effectively the same way it has been.

What about that Nonce thing?

As mentioned above there seems to have been an oversight with the generation of the Knock Knock response when using SHA256 (not including the Nonce in the hash). Nonce’s are intended to prevent the ability to replay a request. So while technically the lack of the Nonce in the hash renders this protection moot, ultimately the attacker would not be able to get back to “Token Lookup” phase with a forged token because the Application Server will not find that token as not having been issued.

What is important to note is that should Oracle release patches to fix this oversight, they would need to be applied to all systems, if only applied to the CRM in our example, the digest that IH produces would no longer match what CRM produces.

psadmin.conf: Development

The second set of psadmin.conf videos are available! These videos are two Blitz Talks on Development. Staci Ludwig shares 10 things you need to know about Nodes and Kyle Benson shares a project that helps integrate Git and App Designer.

We have released the videos as a free course so you can find the videos in one place. Head over to the psadmin.io courses page and sign up. If you have already signed up for the course, you can log in and the videos will be available.

#98 – Failover Testing

This week on the podcast, Dan has follow-up on using Hiera with Puppet environments, capturing WebLogic logs in Elasticsearch, and Kyle shares his thoughts on the Solaris “change”. Then Kyle discusses the in depth failover testing and how Unified Navigation behaves when app servers fail.

Show Notes

#94 – LOCAL_NODE

This week, Dan and Kyle talk about linting and the tnsnames.ora file, managing a Portal reimplementation proejct and follow-up on the TimesTen database. Then Kyle gives us a great overview of why you shouldn’t use LOCAL_NODE on Content References.

Show Notes

#86 – Usability Testing

This week on the podcast Kyle talks about Usability Testing with his PeopleTools upgrade and moving to Fluid Navigation. Dan and Kyle also talk about Unified Navigation and the Node Network, where the Portal fits in and beefing up the Integration Broker.

Show Notes

  • Excited about Fluid @ 1:00
  • Discovering Homepage Tabs @ 3:15
  • How to Approach Usability Testing @ 9:00
  • Usability Participants @ 13:45
  • Accessibility Testing @ 18:00
  • Comparing Usability Testing before Fluid @ 22:45
  • Personalizing Homepages @ 32:00
  • Unified Navigation @ 38:30
  • Getting Lost with Unified Navigation @ 47:00
  • Integration Monitor @ 50:30
  • Making Search a Critical Component @ 55:00
  • Beefing Up the IB with Unified Nav @ 57:30

#71 – DevOps and Ansible w/ Jason Gilfoil and Eric Bolinger

This week we podcast from the Alliance 2017 Conference in Las Vegas. Jason Gilfoil and Eric Bolinger join us to talk about DevOps and Ansible with PeopleSoft. We talk about application orchestration, mixing Ansible and Puppet, customizing the DPK and more.

Show Notes

  • Introducing Jason Gilfoil @ 1:30
  • Introducing Eric Bolinger @ 2:45
  • What is Ansible? @ 3:30
  • What is Orchestration? @ 8:00
  • Differences between Puppet and Ansible @ 12:00
  • Puppet Master, Hiera Hash and the DPK @ 15:15
  • Managing infrastructure code with Git @ 20:45
  • Adjusting to a DevOps culture @ 27:00
  • Getting automation started in your organization @ 30:30
  • Calculating Time Saving for Automation @ 35:30
  • Choosing an automation tool @ 41:30
  • Docker @ 43:00
  • Personal Development Environments @ 45:45
  • Starting to think “cloud” @ 50:00

#63 – Revisiting PS_APP_HOME

This week, Dan and Kyle talk about writing PTF tests for PeopleTools, running multiple IB domains and trying A CM again. Then, we revisit our strategies for managing PS_APP_HOME when applying selective maintenance.

Show Notes

#61 – Jolt Failover

This week on the podcast, Dan and Kyle launch a new course about Deployment Packages. Dan tests out a new text editor and discovers you can run OPatch on MOS. Kyle digs into Jolt Failover options with the IB and brainstorms some great configuration ideas.

Show Notes

Deploy and Configure Elasticsearch

Elasticsearch support is here for PeopleTools 8.55. In PeopleTools 8.55.11, Elasticsearch and SES are supported search engines for PeopleSoft applications. For the next 18 months, PeopleSoft will support both search engines in 8.55. After those 18 months and starting in 8.56, Elasticsearch will be the only search engine supported with PeopleSoft. Since the PeopleSoft team announced that Elasticsearch would replace SES in December 2015, the community has been eagerly waiting for Elasticsearch support go live.

In the video below, we’ll provide an in-depth walk though of this post.

 

In this post, we’ll cover the installation of Elasticsearch, how to use the REST API, and some tips when using Elasticsearch.

DPK Only

Elasticsearch is the first PeopleTools component to be distributed only by Deployment Packages. There is no virtual CD option to install Elasticsearch. This also means that Elasticsearch is a separate download from PeopleTools. To download the Elasticsearch DPK, visit the PeopleTools Patch Documentation Home and click on the “Additional DPKs” tab. Ther is also an Elasticsearch Documentation Home with information about installing Elasticsearch, migrating from SES, and more.

Downloading

Like the PeopleTools and PeopleSoft Image DPKs, you download the .zip files from Oracle Support and run a bootstrap script to start the installation. The Elasticsearch DPK is only one .zip file, and is substantially smaller than other DPKs. I like using the getMOSPatch utility for downloading patches from MOS. Here is the command to download the Windows version of the Elasticsearch DPK:

java -jar getMOSPatch.jar patch=24924150 platform=233P download=all

This is the LInux version of the command:

java -jar getMOSPatch.jar patch=24924136 platform=226P download=all

Next, unzip the ELASTICSEARCH-DPK-WIN-2.3.2_00.zip file. Unlike other DPKs, there is only one .zip file. The .zip file contains setup files, the Elasticsearch binaries, and documentation.

Installation

After you unzip the file, you run the bootstrap script under scripts to start the installation:

cd .\scripts
psft-dpk-setup.ps1 -env_type es

When you run the Elasticsearch DPK, make sure to pass the -env_type es parameter. Without the parameter, the bootstrap script will fail looking for a file that doesn’t exist in the Elasticsearch DPK.

The bootstrap script will ask you a series of questions:

  • Do you want to install Puppet: Yes
  • Enter the ES Base folder: e:\psft
  • Elasticsearch Admin Password: Passw0rd1
  • Proxy User Password: Passw0rd1
  • Elasticsearch Cluster Name: srch-d1
  • Elasticsearch Port: 9200
  • Elasticsearch Discovery Host: ["127.0.0.1"]
  • Enter Java Heap Size: 2

The Elasticsearch Discovery Host is used when you are building a cluster with more than 1 node. In our case, we will enter the local machine’s IP address since we’ll run our nodes on only this machine. If you were building a cluster with multiple nodes on different machines, you would list the IP addresses for each server running Elasticsearch. After you answer the questions, the bootstrap script will start building the Elasticsearch instance.

In my testing on the current Elasticsearch DPK, there is a bug in the bootstrap script. The script ends early and doesn’t complete the installation. If this happens to you, it is easy to resolve.

First, let’s make sure the psft_es.yaml file is updated with out settings. Under C:\ProgramData\Puppetlabs\puppet\etc\data\ open the psft_es.yaml file. Find the section

#es_data
es_http_port:        

Enter 9200 for the es_http_port: value and save the file.

es_http_port:        9200

If you changed the Discover Host value, update that line as well and save the file.

discovery_zen_ping_unicast_hosts:           '["10.0.1.173"]'

Next, navigate to C:\ProgramData\Puppetlabs\puppet\etc\manifests. We’ll start Puppet and have it finish the Elasticsearch deployment and configuration.

puppet apply .\site.pp

At the end of the run, let’s verify that Elasticsearch is up and listening on port 9200.

netstat -an | findstr 9200

You should see something like this:

  TCP    10.0.1.173:9200        0.0.0.0:0              LISTENING
  TCP    127.0.0.1:49200        127.0.0.1:49201        ESTABLISHED
  TCP    127.0.0.1:49201        127.0.0.1:49200        ESTABLISHED

Administration

Before we jump into configuring PeopleSoft to use our Elasticsearch instance, I want to talk some basic Elasticsearch administration. Unlike the SES, there is not web-based admin console. Elasticsearch uses a REST-based API for all administration. Let’s look at what this means. In your browser, go to your Elasticsearch URL http://servername:9200/ and login with esadmin and the Administrative password you entered in the bootstrap script. You’ll get a response similar to this:

{
  "name" : "elastic11.psadmin.io",
  "cluster_name" : "srch-d1",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

If you want to get status of your Elasticsearch cluster, you would use this URL: http://servername:9200/_cluster/health?pretty=true.

{
  "cluster_name" : "srch-d1",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Currently, my cluster srch-d1 has a status of “green”. But it also has no data…

PeopleSoft Configuration

Now that Elasticsearch is running, it is time to configuration our PeopleSoft application to use it. There are a few requirements in PeopleSoft and they are similar to the SES requirements:

  • You must be running 8.55.11
  • The Integration Broker is configured and running
  • The integrationGateway.properties file has an encrypted value for secureFileKeystorePasswd
  • IB Domains are active
  • REST Service URLs are configured (PeopleTools > IB > Configuration > Service Configuration > Setup Target Locations)
  • A callback user account with these roles:
    • Search Developer
    • Search Server
    • Search Query Administrator
    • Search Administrator
    • PeopleSoft User

Search Instance

Go to PeopleTools > Search Framework > Administration > Search Instance. Starting with 8.55.11, we can have 2 or more Search Instances defined. The first search instance is named PSFT_DEFAULT. We’ll leave that configured to use SES. Create a new Search Instance named ELASTIC.

The Search Instance page looks the same as before, but with the addition of a “Search Provider” drop-down menu.

  1. Select “Elasticsearch”
  2. Enter the server name where you installed Elasticsearch
  3. Enter the Elasticsearch port (default is 9200)
  4. Enter esadmin for the User Name
  5. Enter the administrative password you set in the Bootstrap script
  6. Enter people for the Proxy Name
  7. Enter the proxy password you set in the bootstrap script

In the Call Back Properties:

  1. Enter the URL for the REST Target Connector you defined under “Service Configuration > Setup Target Locations”
  2. Enter the Call Back User’s name and password.

Verify all the Ping, Login, and Validate tests return successfully.

Last, we can set the order of the Search Instances. Since we haven’t fully tested Elasticsearch yet, set it to a lower priortiy until we are ready to release it to all users. Under “PeopleTools > Search Framework > Administration > Search Instance Administration”, set the “ELASTIC” instance to Priority 10.

Deploy Indexes

Go to “PeopleTools > Search Framework > Administration > Deploy/Delete Objects”. On this page, you have to select the Search Instance you want to deploy indexes to.

  1. Select “ELASTIC” for the Search Instance.
  2. Select the checkboxes for “PTPORTALREGISTRY” and “PTSEARCHREPORTS”.
  3. Click Deploy.

There seems to be bug in the Report Sync Issues action. If you select deployed indexes and click Report Sync Issues, it will return some errors. Ignore those errors for now; Elasticsearch works despite the “errors”.

  1. Navigate to “PeopleTools > Search Framework > Administration > Schedule Search Index”
  2. Create a new run control called PTPORTALREGISTRY_FULL
  3. Select “ELASTIC” as the Search Instance.
  4. Select the search index “PTPORTALREGISTRY”.
  5. Save the run control and run the process.

Once the process starts, you can view the Asynchronous Services page to see the messages sent to Elasticsearch. Navigate to “PeopleTools > Integration Broker > Service Operation Monitor > Monitoring > Asynchronous Services”. Once the message are successfull (Operation Instance and Subscription Contracts), it’s time to test.

Test Elasticsearch

Currently, SES is still our primary search provider. We can set up per-user search provicers so individual users can begin testing Elasticsearch.

  1. Go to “PeopleTools > Search Framework > Administration > Search Instance/User”
  2. Enter your user name and “ELASTIC”.
  3. Save the page.
  4. Log out of the application and log back in.
  5. In the search bar, search for User.

You should see search results returned from Elasticsearch! Once you are comfortable with Elasticsearch, simply change the priority of the search instances to activate Elasticsearch for everyone.