by Dan Iverson

Using MITMProxy to Debug Elasticsearch and IB

There are times when troubleshooting you want to see what data was transmitted between two systems. When working with HTTP calls with Elasticsearch or Integration Broker targets, it can be helpful to see the data that was in the HTTP transaction. Seeing the data can often help resolve issues. Using MITMProxy, we can inspect HTTP traffic between two systems. In this post, I’ll show you how to use MITMProxy between PeopleSoft and Elasticsearch, and with the PeopleSoft Integration Broker.

Installation

You can download the MITMProxy binaries right from their website https://mitmproxy.org, or you can install from a package manager.

Powershell

choco install mitmproxy -y

Bash

python3 -m pip install mitmproxy

Reverse Proxy Mode

The first example we will walk through using a Reverse Proxy to inspect callback requests from Elasticsearch to PeopleSoft. The callback process is where Elasticsearch asks PeopleSoft what security a user has so that it can filter out results the user shouldn’t see.

First, we need to start up MITMProxy. To make it easier to view our HTTP transactions, we use the mitmweb executable. mitmweb provides a simple GUI for viewing each HTTP request and response that is captured.

To enable Reverse Proxy mode, we pass in the the mode and our target endpoint. For the Elasticsearch callback, that would be our Integration Broker web server.

Powershell

cd 'C:\Program Files\mitmproxy\bin'
.\mitmweb --mode reverse:http://ib.psadmin.io:8000

When MITM starts, a UI available at http://localhost:8081. You can open that URL in a browser and see an empty screen waiting for transactions.

Updating the Callback URL

In our case, we are going to inspect the call back from Elasticsearch to PeopleSoft. To do that we open the Search Instance page and update our Callback URL to point to MITMProxy.

Callback URL: http://mitmproxyserver.psadmin.io:8080/PSIGW/RESTListeningConnector/PSFT_HR

This URL is stored inside the metadata for each index. After you update the Callback URL and save, you must click the “Update Deployed Definitions” button. This will send the new Callback URL to Elasticsearch.

Inspecting Callback Traffic

Next, execute a search in PeopleSoft. I’ll search for a page using the Navigation searche. As soon as we have results in the search bar, you can look at your MITMProxy UI and see that it captured traffic.

The PeopleSoft plugins for Elasticsearch caches security attributes for 2 hours to help with performance. If you don’t see a callback transaction, try executing a search on a different index to get a callback transaction.

In MITMProxy, open the transaction and click on the “Response” tab. There you will see the JSON that PeopleSoft returned to Elasticsearch. This infomation is used to pair down the search results so that users only see the data they are allowed to view.

{
    "ORCL_ES_CALLBACK_RESP": {
        "ORCL_ES_ATTRIBUTES": [
            "S:Admin"
        ]
    }
}

Transparent Proxy Mode

In the example above, we ran MITMProxy in a reverse proxy so that it only forwarded data to one endpoint. You can also run MITMProxy in transparent proxy mode. This can be useful when trying to debug Integration Broker issues.

To start MITMProxy as a transparent proxy, we simply launch the mitmweb executable.

Powershell

cd 'C:\Program Files\mitmproxy\bin'
.\mitmweb

Configure Integration Broker Proxy

To capture IB traffic with MITMProxy, we need to update the integrationGateway.properties file to route traffic through a proxy. Edit the the ig.proxy* lines to point to our MITMProxy instance.

ig.proxyHost=mitmproxyserver.psadmin.io
ig.proxyPort=8080

You need to restart the web server after making these changes.

Test IB Node

To test our proxy connection, create a new node under “PeopleTools > Integration Broker > Integration Setup > Node Definitions”.

  1. Name: GETTEST.
  2. Description: Test GET Request
  3. Node Type: External
  4. Default User ID: PS
  5. Connector ID: HTTPTARGET
  6. HTTPPROPERTY – Method: GET
  7. PRIMARYURL: http://httpbin.org/get

You can save the Node after adding those minimal settings. The node will make a GET HTTP request to a simple REST end point. We are using HTTP instead of HTTPS for now.

Click “Ping Node” and wait for a response. Once the ping is done, you can opent the MITMProxy UI and look at the transaction. There should be one row for our Ping test. While there isn’t much data for this test, we have MITMProxy hooked up to the IB and it’s ready for you to start using.

HTTPS

For our tests so far, we have been using HTTP endpoints. Your IB and Elasticsearch should have HTTPS in place, so to add MITMProxy to the mix requires a few extra steps. We need to trust the MITMProxy certificate so that our application doesn’t reject the HTTPS connection due to trust issues.

When MITMProxy is started, it also runs a simple web page that can be viewed if your browser is configured to route traffic through MITMProxy. For Firefox, you can enable the Proxy under Settings > Network Settings > Settings > Manual Proxy Configuration. Then enter your MITMProxy server and port 8080. Your browser will now route all traffic through MITMProxy.

Once your browser is using MITMProxy, you can go to the page http://mitm.it. You will be shown a webpage where you can download the MITMProxy certificates. We need to load this certificate into pskey on our web servers. Download the Linux certificate even if you are on Windows (mitmproxy-ca-cert.pem) since it is in the correct format for pskey. Open the certificate and copy/paste the content onto your web server. I saved my file to /tmp/mitm.pem (or c:\temp\mitm.pem for Windows).

On your web server, use the keytool tool included with Java to import and trust the MITMProxy cert into pskey.

Bash

export domain=WEBSERVER01
keytool -importcert -alias mitmproxy -storepass password -keystore $PS_CFG_HOME/webserv/$domain/piaconfig/keystore/pskey -trustcacerts -file /tmp/mitm.pem

Owner: O=mitmproxy, CN=mitmproxy
Issuer: O=mitmproxy, CN=mitmproxy    
...

Trust this certificate? [no]:  yes
Certificate was added to keystore

Powershell

$domain="WEBSERVER01"
keytool -importcert -alias mitmproxy -storepass password -keystore $env:PS_CFG_HOME\webserv\$domain\piaconfig\keystore\pskey -trustcacerts -file c:\temp\mitm.pem

Owner: O=mitmproxy, CN=mitmproxy
Issuer: O=mitmproxy, CN=mitmproxy    
...

Trust this certificate? [no]:  yes
Certificate was added to keystore

HTTPS Node Test

Open the GETTEST node definition and update the URL to use https instead of http. Click “Ping Node” to retest with HTTPS.

“Server TLS handshake failed. Certificate verify failed: IP address mismatch”

You may receive an IB error when testing the HTTPS URLs. This can be expected with HTTPS. Many sites are improving their HTTPS support with Certificate Pinning, which helps prevent against “man in the middle” (MITM) attacks. Depending on the integration you are debugging, you may be able to work with HTTPS, or you may have to revert to HTTP for initial integration debugging before adding HTTPS back into the mix.

Application Server

You can also add MITMProxy to the application server configuration. In the psappsrv.cfg file there are two Proxy config lines used by the Java layer that runs inside the application server.

Proxy Host=mitmserver.psadmin.io
Proxy Port=8080

You will need to reconfigure the application server after updating these values.

by Dan Iverson

Elasticsearch: Anatomy of a Search

Search is becoming more important in PeopleSoft and is often required for applications to work properly. We’ve talked before on how to configure Elasticsearch with PeopleSoft, but what happens when we perform a search? There are a number of moving parts that handle the search query, applying security and displaying the results. Let’s take a look into the anatomy of a search to understand the process.

Overview of the Search Flow

The user creates a search request. Generally, this happens when a user opens the Search Box and enters a term or terms to search on. But this process can also start from a component. A good example is the “Find Learning” page in ELM. When you open the component, it runs sends a query to Elasticsearch for data to populate the page with.

Search Framework Builds the Query

The search request from the user needs to be parsed and formatted so that Elasticsearch understands the query. The app package PT_SEARCH has a number of methods that check for wildcards, partial words, and create a valid JSON request for Elasticsearch.

IB Builds JSON Message

In the app package PTSF_ES, the JSON search query string is converted into an IB Message using the IB_GENERIC message. The app package also builds the HTTP headers for the transaction, such as the content-type: application/json and the Authorization header.

The Integration Broker POSTs the message against the alias for the Search Category. If you deployed the EP_ASSETS search in Finance for the database PSFTDB, the alias is ep_assets_psftdb_orcl_es_alias. When you search, you search against the Search Category instead of the individual indices. The alias is the Elasticsearch mechanism to let you search against the category instead of each index.

Elasticsearch Validates Security

A PeopleSoft plugin running in Elasticsearch verifies the security of the user request before running the query. The orcl_es_auth plugin will validate the Basic Authentication value in the HTTP POST from the IB. The user it is validating is esadmin – the user and password defined on the Search Instance page. In your Elasticsearch logs, you can see the validation happening:

[authentication           ] Authentication plugin : authenticate method
[authentication           ] Authentication type : Base64
[authentication           ] Authentication successful

ES Runs the Query

Next, Elasticsearch will run the query. In the Elasticsearch logs, you can see which indexes are used to perform the search:

[orclacl                  ] Indices/ aliases : [ ep_assets_psftdb_orcl_es_alias ]
[orclacl                  ] Index types retrieved : [ ep_am_asset_psftdb ]

In this example, the EP_ASSETS Search Category has only one index associated with it.

ES Checks the ACL Cache

After Elasticsearch has the search results for the query, the results are filtered based on PeopleSoft security. Search Definitions can implement different security types to lock down search data. There is User/Role security, as well as Document security. If a Search Definition has a security type, the security data is collected when the Search Index is built with PTSF_GENFEED. The security attributes are attached to each row of data sent to Elasticsearch.

If the index has security enabled, Elasticsearch needs to know what security the user running the query has access to. The orcl_es_acl plugin is responsible for managing the user security data inside Elasticsearch. If the user has run a query in the last two hours, the orcl_es_acl plugin uses the existing security data for the user stored in Elasticsearch.

In the Elasticsearch logs, you can see this in action:

[orclacl                  ] Document is secured for the type : ep_am_asset_psftdb
[orclacl                  ] Orcl ACL Plugin method: getAttrValFromCache

ES Performs Callback (if needed)

If the user has not run a query for the index before, the plugin will perform a callback to PeopleSoft. The callback will ask PeopleSoft what security values the user has and stores them in a special index: orcl_es_acl. To perform the callback, Elasticsearch creates a new cURL request to the IB URI /RESTListeningConnector/PSFTDB/getsecurityvalues.v1/. The callback URL and security is stored in Elasticsearch under the index orcl_es_acl

The request includes three parameters as well:

  • ?type=ep_am_asset_psftdb
  • ?user=PS
  • ?attribute=BU_SECURITY_SES

The type parameter is the name of the search index requesting the security data. The user is the name of the user running the search query. And the attribute is the security value configured on the Search Definition page.

PS Runs the Callback Code

When the IB receives the callback request, it hands the request to the Application Package specified on the Search Definition. The App Package inspects the parameters to see what security attribute has been requested. The callback code then runs any appropriate code and SQL to build a list of Roles or Permission Lists assigned to the user.

PS Returns the ID and Security

The Permission Lists or Roles are assembled into an array and have P: or R: appended to the beginning of each item. Once the array is assembled, the values are returned to Elasticsearch in a response message. For example, the response might contain an array that looks like ["P:PTPT1000","P:PTPT1200","P:PTPT3100"] – Permission Lists assigned to that user.

ES Filters the Results

Elasticsearch has no concept of the security structures in PeopleSoft, so it uses the data returned from the callback to remove search results that a user has no security to. The callback array from above lists the Permission Lists assigned to a user. Elasticsearch will build a filter query to remove any rows from the search results that do not contain one of those Permission Lists.

Something to keep in mind – Elasticsearch can only filter the search results based on the Permission List stored in each searh result. This means that Permission Lists are attached to each search result when the PTSF_GENFEED process is run. If you change your security dramatically (say, re-implement Business Unit Security), you may need to fun a full build to push your security changes into Elasticsearch.

You can view the permission lists attached to each document with the Elasticsearch API. In your browser you can call the /_search REST method against any URL. If you deployed the PTPORTALREGISTRY search definition with the default settings, you can call the URL http://elastic.psadmin.io:9200/ptportalregistry_psftdb_orcl_es_alias/_search?pretty=true to view the data in Elasticsearch. In the document, you will find the list of security attributes like this: ["P:PTPT1000","S:Admin"].

ES Returns the Results

After the filter query completes, Elasticsearch will return the search results as a JSON message back to PeopleSoft. When the message is received, the results are parsed and displayed by the search results component. Depending on the application, you might see the generic search results page, or an application specific page.

by Dan Iverson

Change Assistant Debug Mode

This is a short post, but hopefully helpful. I recently discovered that you can turn on Change Assistant with a debug mode using an environment variable. There is no documentation on this, but it seems to work well and might help you resolve any issues you have with Change Assistant.

To enable the debug mode, set the PSCADBG environment variable. I used “true” for the value, but any value should work.

$env:PSCADBG="true"

Then launch Change Assistant from the command line:

c:\Program Files\PeopleSoft\Change Assistant\changeassistant.bat

In the command line you should see the debug output, like this:

Signed on to 'PSFTDB'. Using c:\PT8.55.13_Client_ORA\
Attempt to signon using PS/PS@PSFTDB
Controller sending c:\PT8.55.13_Client_ORA client -com.peoplesoft.pt.changeassistant.pshome.request.db.SignOnDatabase
Controller received from c:\PT8.55.13_Client_ORA client -com.peoplesoft.pt.changeassistant.pshome.request.db.SignOnDatabase
Signed on to 'PSFTDB'. Using c:\PT8.55.13_Client_ORA\