#149 – Anatomy of a Search


This week on the podcast, Kyle gives some suggestions on how to make dynamic links to other PeopleSoft homepages and Dan dives into what happens when you perform a search with Elasticsearch.

Show Notes

Elasticsearch: Anatomy of a Search

Search is becoming more important in PeopleSoft and is often required for applications to work properly. We’ve talked before on how to configure Elasticsearch with PeopleSoft, but what happens when we perform a search? There are a number of moving parts that handle the search query, applying security and displaying the results. Let’s take a look into the anatomy of a search to understand the process.

Overview of the Search Flow

The user creates a search request. Generally, this happens when a user opens the Search Box and enters a term or terms to search on. But this process can also start from a component. A good example is the “Find Learning” page in ELM. When you open the component, it runs sends a query to Elasticsearch for data to populate the page with.

Search Framework Builds the Query

The search request from the user needs to be parsed and formatted so that Elasticsearch understands the query. The app package PT_SEARCH has a number of methods that check for wildcards, partial words, and create a valid JSON request for Elasticsearch.

IB Builds JSON Message

In the app package PTSF_ES, the JSON search query string is converted into an IB Message using the IB_GENERIC message. The app package also builds the HTTP headers for the transaction, such as the content-type: application/json and the Authorization header.

The Integration Broker POSTs the message against the alias for the Search Category. If you deployed the EP_ASSETS search in Finance for the database PSFTDB, the alias is ep_assets_psftdb_orcl_es_alias. When you search, you search against the Search Category instead of the individual indices. The alias is the Elasticsearch mechanism to let you search against the category instead of each index.

Elasticsearch Validates Security

A PeopleSoft plugin running in Elasticsearch verifies the security of the user request before running the query. The orcl_es_auth plugin will validate the Basic Authentication value in the HTTP POST from the IB. The user it is validating is esadmin – the user and password defined on the Search Instance page. In your Elasticsearch logs, you can see the validation happening:

[authentication           ] Authentication plugin : authenticate method
[authentication           ] Authentication type : Base64
[authentication           ] Authentication successful

ES Runs the Query

Next, Elasticsearch will run the query. In the Elasticsearch logs, you can see which indexes are used to perform the search:

[orclacl                  ] Indices/ aliases : [ ep_assets_psftdb_orcl_es_alias ]
[orclacl                  ] Index types retrieved : [ ep_am_asset_psftdb ]

In this example, the EP_ASSETS Search Category has only one index associated with it.

ES Checks the ACL Cache

After Elasticsearch has the search results for the query, the results are filtered based on PeopleSoft security. Search Definitions can implement different security types to lock down search data. There is User/Role security, as well as Document security. If a Search Definition has a security type, the security data is collected when the Search Index is built with PTSF_GENFEED. The security attributes are attached to each row of data sent to Elasticsearch.

If the index has security enabled, Elasticsearch needs to know what security the user running the query has access to. The orcl_es_acl plugin is responsible for managing the user security data inside Elasticsearch. If the user has run a query in the last two hours, the orcl_es_acl plugin uses the existing security data for the user stored in Elasticsearch.

In the Elasticsearch logs, you can see this in action:

[orclacl                  ] Document is secured for the type : ep_am_asset_psftdb
[orclacl                  ] Orcl ACL Plugin method: getAttrValFromCache

ES Performs Callback (if needed)

If the user has not run a query for the index before, the plugin will perform a callback to PeopleSoft. The callback will ask PeopleSoft what security values the user has and stores them in a special index: orcl_es_acl. To perform the callback, Elasticsearch creates a new cURL request to the IB URI /RESTListeningConnector/PSFTDB/getsecurityvalues.v1/. The callback URL and security is stored in Elasticsearch under the index orcl_es_acl

The request includes three parameters as well:

  • ?type=ep_am_asset_psftdb
  • ?user=PS
  • ?attribute=BU_SECURITY_SES

The type parameter is the name of the search index requesting the security data. The user is the name of the user running the search query. And the attribute is the security value configured on the Search Definition page.

PS Runs the Callback Code

When the IB receives the callback request, it hands the request to the Application Package specified on the Search Definition. The App Package inspects the parameters to see what security attribute has been requested. The callback code then runs any appropriate code and SQL to build a list of Roles or Permission Lists assigned to the user.

PS Returns the ID and Security

The Permission Lists or Roles are assembled into an array and have P: or R: appended to the beginning of each item. Once the array is assembled, the values are returned to Elasticsearch in a response message. For example, the response might contain an array that looks like ["P:PTPT1000","P:PTPT1200","P:PTPT3100"] – Permission Lists assigned to that user.

ES Filters the Results

Elasticsearch has no concept of the security structures in PeopleSoft, so it uses the data returned from the callback to remove search results that a user has no security to. The callback array from above lists the Permission Lists assigned to a user. Elasticsearch will build a filter query to remove any rows from the search results that do not contain one of those Permission Lists.

Something to keep in mind – Elasticsearch can only filter the search results based on the Permission List stored in each searh result. This means that Permission Lists are attached to each search result when the PTSF_GENFEED process is run. If you change your security dramatically (say, re-implement Business Unit Security), you may need to fun a full build to push your security changes into Elasticsearch.

You can view the permission lists attached to each document with the Elasticsearch API. In your browser you can call the /_search REST method against any URL. If you deployed the PTPORTALREGISTRY search definition with the default settings, you can call the URL http://elastic.psadmin.io:9200/ptportalregistry_psftdb_orcl_es_alias/_search?pretty=true to view the data in Elasticsearch. In the document, you will find the list of security attributes like this: ["P:PTPT1000","S:Admin"].

ES Returns the Results

After the filter query completes, Elasticsearch will return the search results as a JSON message back to PeopleSoft. When the message is received, the results are parsed and displayed by the search results component. Depending on the application, you might see the generic search results page, or an application specific page.

#139 – Redeploy


This week on the podcast, Dan and Kyle discuss the new PeopleSoft Support timeframe, controlling how much data the Search Framework indexes, and how to use Facter to redeploy software via the DPK.

Show Notes

  • 2030 Commitment @ 1:00
  • Auto Select Follow-up @ 7:15
    • http://www.peoplesoftwiki.com/lookup-exclusion
    • http://blog.psftdba.com/2006/05/lookup-exclusion-table.html?m=1
  • PeopleSoft Test Framework and TLS @ 8:45
  • Facter and Redeploy @ 12:00
  • Search Framework – Last X days @ 19:00
  • Puppet variable warnings @ 27:00

#126 – Elasticsearch Backups


This week on the podcast, Dan and Kyle discuss options and considerations for backing up Elasticsearch. Kyle also shares a bug with the PIA startup scripts built by the DPK and Dan makes Vagrant work with a Web Filter.

Show Notes

#125 – Push Notifications and Phire



This week on the podcast, Kyle talks about Push Notifications he built for Phire and how you could extend Phire. Dan explains the pt_password DPK Puppet Type, and Kyle discusses a leak of 2 billion passwords.

Show Notes

#124 – Search Security


This week on the podcast, Dan shares a quick review on Classic Plus and tip when using App Designer. Then Kyle and Dan discuss how the Search Framework implements security on search results.

Show Notes

#117 – Invalid Views


This week on the podcast, Dan and Kyle discuss a WebLogic exploit used for currency mining, Dan revisits the Health Center in the latest PeopleSoft Images, and Kyle explains why you need to review the Invalid View project.

Show Notes

Using ElasticHQ to Monitor Elasticsearch

There are a variety of ways to monitor Elasticsearch with PeopleSoft. You can use the Health Center, use the Elasticsearch Interact page, make API calls. But my favorite tool to keep an Elasticsearch is ElasticHQ. ElasticHQ is an open source plugin that you can easily install into Elasticsearch.

There are two ways to install ElasticHQ:

  1. Install as Plugin
  2. Download .zip

If your Elasticsearch server has access to download from the internet, you can use this command to install the ElasticHQ plugin:

PS C:\psft\elastic\pt\es2.3.3\bin> .\plugin install royrusso/elasticsearch-HQ/v2.0.2

-> Installing royrusso/elasticsearch-HQ/v2.0.2...
https://oss.sonatype.org/service/local/repositories/releases/content/royrusso/elasticsearch-HQ/v2.0.2/elasticsear
ch-HQ-v2.0.2.zip 
Trying https://github.com/royrusso/elasticsearch-HQ/archive/v2.0.2.zip 
Downloading ..........................................................
..................................................................DONE
Verifying https://github.com/royrusso/elasticsearch-HQ/archive/v2.0.2.zip checksums if available ...
Installed hq into C:\psft\elastic\pt\es2.3.2\plugins\hq

You need to specify version 2.0.2 for PeopleSoft Elasticsearch. The latest version of the ElasticHQ plugin is designed for newer versions of Elasticsearch, not the version that ships with PeopleSoft.

If your server cannot download from the internet, you can download the .zip file, extract it, and copy to base/pt/es2.3.2/plugins

To access ElasticHQ,

  1. Open a web browser and the page http://server:9200/_plugin/hq
  2. Log in with the esadmin user
  3. Click the “Connect” button to get your Elasticsearch status

If you have an Elasticsearch cluster, ElasticHQ will show you statistics about every node in the cluster.

#106 – Purging Cache

This week on the podcast, Kyle finds the ProcessRestartMemoryLimit does not work as expected, Dan finds that the Purge Cache command has changed in 8.56, and Kyle recaps a successful Go-Live for PeopleTools 8.55 and Fluid Navigation.

Show Notes

  • Page and Field CONFIGURATOR Follow-up @ 1:00
  • October CVE Details @ 6:00
  • Understanding ProcessRestartMemoryLimit @ 11:30
  • Purge Cache Change in 8.56 @ 19:45
  • Elasticsearch 04 DPK, POC Patches @ 25:30
  • Unified Nav and My Favorites @ 31:30
  • 8.55 and Fluid Nav Go-Live Success and Issues @ 38:15

#105 – Agile PeopleSoft

This week on the podcast, Charlie Sinks joins us to talk about the Upper Midwest Regional User Group. We talk about using Agile with PeopleSoft, our experiences with Elasticearch, the Idea Space for PeopleTools, and using Git with PeopleSoft.

Show Notes

  • Chatbots Demo @ 3:30
  • Agile and PeopleSoft @ 11:00
    • SCRUM
    • Kanban
    • SAFE
  • Elasticsearch Experiences @ 33:45
  • PeopleTools 8.56 Updates @ 43:30
  • Testing Effort for Certifications @ 45:30
    • AIX/Solaris: pspuppet.sh is used to prepare the environment. Need to invoke if the bootstrap or puppet failed and you need to re-run
  • Idea Space @ 50:15
  • Advanced PS Admin Talk @ 56:30
  • Orchestration @ 61:00
  • Using Git @ 67:30