#56 – Spontaneous Stress Test

This week on the podcast, Kyle recaps his successful HR 9.2 go-live and shares a few lessons learned. Dan and Kyle talk about encrypting password in YAML files, conditional navigation and PeopleTools CPU patches. Dan finishes the podcast by sharing his story about a spontaneous stress test.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

Load Balancers and Client IP Addresses

One disadvantage of running PeopleSoft behind a load balancer is that it can hide your user’s real IP address. With a load balancer, WebLogic and PeopleSoft log the load balancer’s IP address instead of the client connecting to your system. Load balancers support an HTTP header called X-Forwarded-For, and will populate that HTTP header with the client’s IP address. We can configure WebLogic to read this header and add that to our HTTP access log.

The psaccesslog table tracks logins into PeopleSoft, but a customization is required to change the logged IP address to read from X-Forwarded-For. We won’t over that here. Instead, we’ll show a configuration change to capture this information.

X-Forwarded-For

The X-Forwarded-For HTTP header is designed to pass in the HTTP originating client IP address through the load balancer. Logging a client IP address helps you identify where users are coming from (geo-ip) and can help with auditing access to your system. Since organizations run many different load balancers, we won’t cover how to enable that. Ask your network administrators to enable and populate X-Forwarded-For and pass it through to your WebLogic servers.

If you want to test the logging without getting a network admin involved, FireFox has a nice add-on, X-Forwarded-For Header that simulates the header for you.

WebLogic Access Log

Out of the box, WebLogic doesn’t capture the X-Forwarded-For HTTP header. To read that header, we can configure the HTTP Access Log to read the header and log it. To do that, we need to enable the Access Log and tell WebLogic what header to look for.

EnableXFF

  1. Open http://server:port/console and login.
  2. Go to “Environment > Servers > PIA > Logging > HTTP”.
  3. Click “Lock & Edit”.
  4. Select the checkbox for “HTTP access log file enabled”.
  5. Save the changes.
  6. Expand the “Advanced” section.
  7. Change the Format to Extended.
  8. Add cs(X-Forwarded-For) to the Extended Logging Format Fields.
  9. Set the Log File Buffer to 0. (This will write entires immediately to the log file.)
  10. Save the changes.
  11. Click the “Release Configuration” button.
  12. Restart the web server.

In the PIA_access.log file, you should see a dash - or an IP address. If the header is blank, the dash will be populated. Otherwise, WebLogic will log the client’s real IP address.

Here a sample line from the PIA_access.log file after we enabled X-Forwarded-For logging:

#Version:   1.0
#Fields:    date time cs-method cs-uri sc-status cs-username cs(user-agent) s-ip c-ip time-taken bytes cs(X-Forwarded-For)
#Software:  WebLogic

2016-11-23  20:49:16    GET /psc/ps/EMPLOYEE/ELM/c/NUI_FRAMEWORK.PT_LANDINGPAGE.GBL 200 -   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"    0.0.0.0:10020   10.0.1.174  5.484   10907   "10.0.1.222"

The last value, "10.0.1.222" is the client IP address (my workstation). The c-ip value (10.0.1.174) is the load balancer IP adress in this scenario.

#55 – 2FA and Event Mapping w/ Colton Fischer

This week on the podcast, Colton Fischer joins us to talk about 2 Factor Authentication (2FA) and Event Mapping. Colton explains what 2FA is, how to implement it, and how he used Event Mapping to add it to PeopleSoft. We also talk about Servlet Filters and using REST-based API’s with PeopleTools.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

#54 – Elasticsearch w/ Wayne Fuller

This week on the podcast we welcome back Wayne Fuller to talk about Elasticsearch. Wayne gives an excellent introduction into using Elasticsearch with PeopleSoft, some lesson’s learned from his testing, and highlights the benefits of switching to the new search tool.

One quick note – after recording, Wayne got clarification that you cannot cut over to Elasticsearch by index, only by user. We’ll talk with Wayne in an upcoming episode again about that change so everybody has accurate information. -dan

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

  • Testing Elasticsearch @ 1:00
  • Elasticsearch and IB Changes @ 5:00
  • Shards and Replicas @ 8:00
  • Elasticsearch Infrastructure @ 13:00
  • Elasticsearch Speed @ 17:15
  • Search Framework Architecture Changes @ 18:00
  • Integration Broker Impacts @ 25:00
  • Attachment Processing @ 28:00
  • Planning an Elasticsearch Deployment @ 31:30
  • Improving the Search Framework @ 38:30
  • Elasticsearch REST Interface @ 42:45
  • Health Center and Elasticsearch @ 46:00
  • Delete Processing @ 50:00
  • Beta Testing Experience @ 55:30

Deploy and Configure Elasticsearch

Elasticsearch support is here for PeopleTools 8.55. In PeopleTools 8.55.11, Elasticsearch and SES are supported search engines for PeopleSoft applications. For the next 18 months, PeopleSoft will support both search engines in 8.55. After those 18 months and starting in 8.56, Elasticsearch will be the only search engine supported with PeopleSoft. Since the PeopleSoft team announced that Elasticsearch would replace SES in December 2015, the community has been eagerly waiting for Elasticsearch support go live.

In the video below, we’ll provide an in-depth walk though of this post.

 

In this post, we’ll cover the installation of Elasticsearch, how to use the REST API, and some tips when using Elasticsearch.

DPK Only

Elasticsearch is the first PeopleTools component to be distributed only by Deployment Packages. There is no virtual CD option to install Elasticsearch. This also means that Elasticsearch is a separate download from PeopleTools. To download the Elasticsearch DPK, visit the PeopleTools Patch Documentation Home and click on the “Additional DPKs” tab. Ther is also an Elasticsearch Documentation Home with information about installing Elasticsearch, migrating from SES, and more.

Downloading

Like the PeopleTools and PeopleSoft Image DPKs, you download the .zip files from Oracle Support and run a bootstrap script to start the installation. The Elasticsearch DPK is only one .zip file, and is substantially smaller than other DPKs. I like using the getMOSPatch utility for downloading patches from MOS. Here is the command to download the Windows version of the Elasticsearch DPK:

java -jar getMOSPatch.jar patch=24924150 platform=233P download=all

This is the LInux version of the command:

java -jar getMOSPatch.jar patch=24924136 platform=226P download=all

Next, unzip the ELASTICSEARCH-DPK-WIN-2.3.2_00.zip file. Unlike other DPKs, there is only one .zip file. The .zip file contains setup files, the Elasticsearch binaries, and documentation.

Installation

After you unzip the file, you run the bootstrap script under scripts to start the installation:

cd .\scripts
psft-dpk-setup.ps1 -env_type es

When you run the Elasticsearch DPK, make sure to pass the -env_type es parameter. Without the parameter, the bootstrap script will fail looking for a file that doesn’t exist in the Elasticsearch DPK.

The bootstrap script will ask you a series of questions:

  • Do you want to install Puppet: Yes
  • Enter the ES Base folder: e:\psft
  • Elasticsearch Admin Password: Passw0rd1
  • Proxy User Password: Passw0rd1
  • Elasticsearch Cluster Name: srch-d1
  • Elasticsearch Port: 9200
  • Elasticsearch Discovery Host: ["127.0.0.1"]
  • Enter Java Heap Size: 2

The Elasticsearch Discovery Host is used when you are building a cluster with more than 1 node. In our case, we will enter the local machine’s IP address since we’ll run our nodes on only this machine. If you were building a cluster with multiple nodes on different machines, you would list the IP addresses for each server running Elasticsearch. After you answer the questions, the bootstrap script will start building the Elasticsearch instance.

In my testing on the current Elasticsearch DPK, there is a bug in the bootstrap script. The script ends early and doesn’t complete the installation. If this happens to you, it is easy to resolve.

First, let’s make sure the psft_es.yaml file is updated with out settings. Under C:\ProgramData\Puppetlabs\puppet\etc\data\ open the psft_es.yaml file. Find the section

#es_data
es_http_port:        

Enter 9200 for the es_http_port: value and save the file.

es_http_port:        9200

If you changed the Discover Host value, update that line as well and save the file.

discovery_zen_ping_unicast_hosts:           '["10.0.1.173"]'

Next, navigate to C:\ProgramData\Puppetlabs\puppet\etc\manifests. We’ll start Puppet and have it finish the Elasticsearch deployment and configuration.

puppet apply .\site.pp

At the end of the run, let’s verify that Elasticsearch is up and listening on port 9200.

netstat -an | findstr 9200

You should see something like this:

  TCP    10.0.1.173:9200        0.0.0.0:0              LISTENING
  TCP    127.0.0.1:49200        127.0.0.1:49201        ESTABLISHED
  TCP    127.0.0.1:49201        127.0.0.1:49200        ESTABLISHED

Administration

Before we jump into configuring PeopleSoft to use our Elasticsearch instance, I want to talk some basic Elasticsearch administration. Unlike the SES, there is not web-based admin console. Elasticsearch uses a REST-based API for all administration. Let’s look at what this means. In your browser, go to your Elasticsearch URL http://servername:9200/ and login with esadmin and the Administrative password you entered in the bootstrap script. You’ll get a response similar to this:

{
  "name" : "elastic11.psadmin.io",
  "cluster_name" : "srch-d1",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

If you want to get status of your Elasticsearch cluster, you would use this URL: http://servername:9200/_cluster/health?pretty=true.

{
  "cluster_name" : "srch-d1",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Currently, my cluster srch-d1 has a status of “green”. But it also has no data…

PeopleSoft Configuration

Now that Elasticsearch is running, it is time to configuration our PeopleSoft application to use it. There are a few requirements in PeopleSoft and they are similar to the SES requirements:

  • You must be running 8.55.11
  • The Integration Broker is configured and running
  • The integrationGateway.properties file has an encrypted value for secureFileKeystorePasswd
  • IB Domains are active
  • REST Service URLs are configured (PeopleTools > IB > Configuration > Service Configuration > Setup Target Locations)
  • A callback user account with these roles:
    • Search Developer
    • Search Server
    • Search Query Administrator
    • Search Administrator
    • PeopleSoft User

Search Instance

Go to PeopleTools > Search Framework > Administration > Search Instance. Starting with 8.55.11, we can have 2 or more Search Instances defined. The first search instance is named PSFT_DEFAULT. We’ll leave that configured to use SES. Create a new Search Instance named ELASTIC.

The Search Instance page looks the same as before, but with the addition of a “Search Provider” drop-down menu.

  1. Select “Elasticsearch”
  2. Enter the server name where you installed Elasticsearch
  3. Enter the Elasticsearch port (default is 9200)
  4. Enter esadmin for the User Name
  5. Enter the administrative password you set in the Bootstrap script
  6. Enter people for the Proxy Name
  7. Enter the proxy password you set in the bootstrap script

In the Call Back Properties:

  1. Enter the URL for the REST Target Connector you defined under “Service Configuration > Setup Target Locations”
  2. Enter the Call Back User’s name and password.

Verify all the Ping, Login, and Validate tests return successfully.

Last, we can set the order of the Search Instances. Since we haven’t fully tested Elasticsearch yet, set it to a lower priortiy until we are ready to release it to all users. Under “PeopleTools > Search Framework > Administration > Search Instance Administration”, set the “ELASTIC” instance to Priority 10.

Deploy Indexes

Go to “PeopleTools > Search Framework > Administration > Deploy/Delete Objects”. On this page, you have to select the Search Instance you want to deploy indexes to.

  1. Select “ELASTIC” for the Search Instance.
  2. Select the checkboxes for “PTPORTALREGISTRY” and “PTSEARCHREPORTS”.
  3. Click Deploy.

There seems to be bug in the Report Sync Issues action. If you select deployed indexes and click Report Sync Issues, it will return some errors. Ignore those errors for now; Elasticsearch works despite the “errors”.

  1. Navigate to “PeopleTools > Search Framework > Administration > Schedule Search Index”
  2. Create a new run control called PTPORTALREGISTRY_FULL
  3. Select “ELASTIC” as the Search Instance.
  4. Select the search index “PTPORTALREGISTRY”.
  5. Save the run control and run the process.

Once the process starts, you can view the Asynchronous Services page to see the messages sent to Elasticsearch. Navigate to “PeopleTools > Integration Broker > Service Operation Monitor > Monitoring > Asynchronous Services”. Once the message are successfull (Operation Instance and Subscription Contracts), it’s time to test.

Test Elasticsearch

Currently, SES is still our primary search provider. We can set up per-user search provicers so individual users can begin testing Elasticsearch.

  1. Go to “PeopleTools > Search Framework > Administration > Search Instance/User”
  2. Enter your user name and “ELASTIC”.
  3. Save the page.
  4. Log out of the application and log back in.
  5. In the search bar, search for User.

You should see search results returned from Elasticsearch! Once you are comfortable with Elasticsearch, simply change the priority of the search instances to activate Elasticsearch for everyone.

#53 – UMRUG 2016 Fall Recap

This week on the podcast, Charlie Sinks joins us to talk about everything we learned at the Upper Midwest Regional User Group meeting. We talk about PeopleTools 8.56, changes in Lifecycle Management, Fluid adoption, Elasticsearch, Dan’s DPK session, and much more.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

  • Benefits of the Roundtable @ 3:30
  • Roundtable Topics:
    • Extending 9.1 Support @ 5:15
    • Current Versions of PeopleTools @ 6:30
    • Fluid, Fluid, and more Fluid @ 8:45
    • Classic Plus @ 10:45
    • Classic Pages and Support @ 14:00
    • Elasticsearch in 8.55.11 @ 17:45
    • Getting started with Decoupled Homes @ 25:30
    • Conditional Navigation @ 32:00
    • 9.2 Upgrade and Tools? @ 35:45
    • PeopleTools 8.56 Details @ 39:30
      • Selective Adoption
      • ACM
      • Event Mapping
      • BI Publisher
      • The “PeopleSoft” name
  • Dan’s DPK Talk @ 52:00
  • LifeCycle Management
    • PTF, Usage Monitor and PTF Upgrade Utility @ 57:00
    • PUM Environments for Testing @ 61:30
    • How to determine the PeopleSoft Image you are on @ 67:45
    • Keeping older PeopleSoft Images @ 72:15
    • Best Practices for Lifecycle Management Red Paper @ 78:00
    • PUM Analytics @ 79:00