#337 – ACM and Load Balancers

This week on the podcast, Kyle and Dan discuss securing your public user and leaking information, how to speed up change assistant upgrade projects, and new ACM plugins to work with Load Balanced gateways.

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson.

Show Notes

#336 – 8.60 Themes and How we use the DPK

This week on the podcast, Kyle and Dan talk about the new psadmin.io Themes for 8.60, and then they discuss how they are currently using the DPK to build new environments and what has changed in the DPK since it was released.

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson.

Show Notes

#335 – io_homes DPK Module

This week on the podcast, Kyle and Dan talk about the changes to the 8.60 database upgrade and the new PPTLS860 project. Dan also shares some updates to the IO_STYLE_859 project and discusses his new io_homes DPK module.

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson.

Show Notes

#334 – byop – Bring Your Own Patches – an Infra-DPK Builder

This week on the podcast, Dan discusses a new tool he built to simplify CPU patches by using the same tools as the Infrastructure DPK. byop, or Build Your Own Patches, will take a list of patches to download and store them in a format that matches the Infrastructure DPK.

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson.

Show Notes

Dan’s method to apply CPU patches

Custom Fact to trigger Infrastructure-DPK processing

  • /puppet/production/modules/pt_role/lib/facter/cpu.rb

    # Set the env var APPLY_INFRA_CPU=true and run the DPK to apply the Infra-DPK patches
    Facter.add(:apply_infra_cpu) do
      setcode do
        apply_infra_cpu = ENV["APPLY_INFRA_CPU"] || 'false'
      end
    end
    

Bash alias I use to apply CPU patches via Infrastructure-DPK

$ alias applycpu='sudo APPLY_INFRA_CPU=true && puppet apply -e "contain ::pt_profile::pt_tools_deployment" --confdir <dpk_home>/puppet -d'
$ applycpu
 

Building a PeopleSoft Image – OCI Marketplace

In this video we will build a new PeopleSoft Image to check out some of the newer features. We will use OCI to host our image because the PeopleSoft team provides us with images that are ready to build. We just need to provide some passwords and away we go.

YouTube player

OCI Marketplace Images

The OCI Marketplace is where you can find pre-packaged software ready to deploy on OCI. Each new image release, the PeopleSoft team pushes a new build for each application. For this demo, we will use Finance Image 46.

There are a few benefits to the OCI Marketplace-based PeopleSoft Images over other methods.

  • No need to download DPK files to a server
  • They come with Elasticsearch and Kibana pre-packaged
  • They are updated each release and easy to build for developers
  • Doesn’t require Cloud Manager or MOS Download rights

There are a few drawbacks to using these images though.

  • They don’t support the new VM.Standard Flex shapes
  • You need an OCI account and privileges to create a new instance (and virtual cloud network)

Boot Volume

When building a Marketplace-based Image, you must increase the boot volume to at least 200GB. This will ensure there is plenty of space to extract the DPK files and install PeopleSoft.

Generate Passwords

You can enter these by hand – it’s a JSON string – but there are different requirements for each password. You can use the sample JSON below for reference, but let’s take a quick tangent and I’ll show you how I generated my passwords.

The secret is psst

To run psst, you need Python and Git installed:

For Windows you can use Powershell and Chocolatey to install these:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; 
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

choco install python3 -y
choco install git -y

For Linux and macOS:

yum install git -y
yum install python3 -y

The psst tool is hosted on Github and you can clone the code to run it:

git clone https://github.com/psadmin-io/psst.git

cd psst
pip install .
psst secrets generate -oci

{
    "connect_pwd": "eu9P3HCj6WwI95vj498JX6Yzjk6VGS",
    "access_pwd": "hsRqmDFjyrntMEJ74fMBBwMKi",
    "admin_pwd": "0WkAoB531GXr#2AtvpNo9SZ5u-_gEh",
    "weblogic_admin_pwd": "#ma1Q4%7SrIyKmpfIT3iS!&1Q22o$x",
    "webprofile_user_pwd": "xSFb74gd2YeyvkXjh1s9tI7wDK9Dew",
    "gw_user_pwd": "xtc4IxtBkDiNpJCMT04wRXGUNHG4bQ",
    "domain_conn_pwd": "G2rxzYThC2BTKq5DfHc",
    "opr_pwd": "78rN8StJt8rvSaUwB1FAWgEMK"
}

You can copy the JSON and paste it directly into the OCI Console’s “cloud-init” section.

Host File

Our instance is created with a public IP address, but the DNS name is private to your OCI Cloud Network. To translate between the two, we will add a hosts entry to our computer. Grab both the Public IP and Internal FQDN values from the Instances page.

For Linux and macOS

echo "<ip address> <fqdn>" | sudo tee -a /etc/hosts

For Windows, add this line to the end of the file c:\Windows\System32\drivers\etc\hosts

<ip address> <fqdn>

For example, my hosts entry looks like this:

129.213.146.185 fscm046.subnet12081732.vcn12081732.oraclevcn.com

SSH Key

Linux, macOS, and WSL Users:

chmod 600 ~/Downloads/ssh-key-2022-12-08.key 
ssh -i ~/Downloads/ssh-key-2022-12-08.key opc@<IP>

PuTTY for Windows

  • Convert SSH Key to Putty Format with PuTTYGen
  • Connect with PuTTY

Ingress Rules in OCI

  1. PIA Rule

    • CIDR Block: 0.0.0.0/0
    • Destination Port: 8000
  2. Kibana Rule

    • CIDR Block: 0.0.0.0/0
    • Destination Port: 5601
  3. TNS Rule (Optional – Required for App Designer or SQL access)

    • CIDR Block: 0.0.0.0/0
    • Destination Port: 1521

(Optional) Add firewalld Rule for TNS

sudo firewall-cmd --permanent --zone=public --add-port=1521/tcp
sudo firewall-cmd --reload

#332 – PeopleTools 8.60

This week on the podcast, Kyle and Dan discuss PeopleTools 8.60 and the great features that were released. Dan also shares his thoughts on using Cloud Manager to upgrade environments to 8.60.

The PeopleSoft Administrator Podcast is hosted by Dan Iverson and Kyle Benson.

Show Notes


Using MITMProxy to Debug Elasticsearch and IB

There are times when troubleshooting you want to see what data was transmitted between two systems. When working with HTTP calls with Elasticsearch or Integration Broker targets, it can be helpful to see the data that was in the HTTP transaction. Seeing the data can often help resolve issues. Using MITMProxy, we can inspect HTTP traffic between two systems. In this post, I’ll show you how to use MITMProxy between PeopleSoft and Elasticsearch, and with the PeopleSoft Integration Broker.

Installation

You can download the MITMProxy binaries right from their website https://mitmproxy.org, or you can install from a package manager.

Powershell

choco install mitmproxy -y

Bash

python3 -m pip install mitmproxy

Reverse Proxy Mode

The first example we will walk through using a Reverse Proxy to inspect callback requests from Elasticsearch to PeopleSoft. The callback process is where Elasticsearch asks PeopleSoft what security a user has so that it can filter out results the user shouldn’t see.

First, we need to start up MITMProxy. To make it easier to view our HTTP transactions, we use the mitmweb executable. mitmweb provides a simple GUI for viewing each HTTP request and response that is captured.

To enable Reverse Proxy mode, we pass in the the mode and our target endpoint. For the Elasticsearch callback, that would be our Integration Broker web server.

Powershell

cd 'C:\Program Files\mitmproxy\bin'
.\mitmweb --mode reverse:http://ib.psadmin.io:8000

When MITM starts, a UI available at http://localhost:8081. You can open that URL in a browser and see an empty screen waiting for transactions.

Updating the Callback URL

In our case, we are going to inspect the call back from Elasticsearch to PeopleSoft. To do that we open the Search Instance page and update our Callback URL to point to MITMProxy.

Callback URL: http://mitmproxyserver.psadmin.io:8080/PSIGW/RESTListeningConnector/PSFT_HR

This URL is stored inside the metadata for each index. After you update the Callback URL and save, you must click the “Update Deployed Definitions” button. This will send the new Callback URL to Elasticsearch.

Inspecting Callback Traffic

Next, execute a search in PeopleSoft. I’ll search for a page using the Navigation searche. As soon as we have results in the search bar, you can look at your MITMProxy UI and see that it captured traffic.

The PeopleSoft plugins for Elasticsearch caches security attributes for 2 hours to help with performance. If you don’t see a callback transaction, try executing a search on a different index to get a callback transaction.

In MITMProxy, open the transaction and click on the “Response” tab. There you will see the JSON that PeopleSoft returned to Elasticsearch. This infomation is used to pair down the search results so that users only see the data they are allowed to view.

{
    "ORCL_ES_CALLBACK_RESP": {
        "ORCL_ES_ATTRIBUTES": [
            "S:Admin"
        ]
    }
}

Transparent Proxy Mode

In the example above, we ran MITMProxy in a reverse proxy so that it only forwarded data to one endpoint. You can also run MITMProxy in transparent proxy mode. This can be useful when trying to debug Integration Broker issues.

To start MITMProxy as a transparent proxy, we simply launch the mitmweb executable.

Powershell

cd 'C:\Program Files\mitmproxy\bin'
.\mitmweb

Configure Integration Broker Proxy

To capture IB traffic with MITMProxy, we need to update the integrationGateway.properties file to route traffic through a proxy. Edit the the ig.proxy* lines to point to our MITMProxy instance.

ig.proxyHost=mitmproxyserver.psadmin.io
ig.proxyPort=8080

You need to restart the web server after making these changes.

Test IB Node

To test our proxy connection, create a new node under “PeopleTools > Integration Broker > Integration Setup > Node Definitions”.

  1. Name: GETTEST.
  2. Description: Test GET Request
  3. Node Type: External
  4. Default User ID: PS
  5. Connector ID: HTTPTARGET
  6. HTTPPROPERTY – Method: GET
  7. PRIMARYURL: http://httpbin.org/get

You can save the Node after adding those minimal settings. The node will make a GET HTTP request to a simple REST end point. We are using HTTP instead of HTTPS for now.

Click “Ping Node” and wait for a response. Once the ping is done, you can opent the MITMProxy UI and look at the transaction. There should be one row for our Ping test. While there isn’t much data for this test, we have MITMProxy hooked up to the IB and it’s ready for you to start using.

HTTPS

For our tests so far, we have been using HTTP endpoints. Your IB and Elasticsearch should have HTTPS in place, so to add MITMProxy to the mix requires a few extra steps. We need to trust the MITMProxy certificate so that our application doesn’t reject the HTTPS connection due to trust issues.

When MITMProxy is started, it also runs a simple web page that can be viewed if your browser is configured to route traffic through MITMProxy. For Firefox, you can enable the Proxy under Settings > Network Settings > Settings > Manual Proxy Configuration. Then enter your MITMProxy server and port 8080. Your browser will now route all traffic through MITMProxy.

Once your browser is using MITMProxy, you can go to the page http://mitm.it. You will be shown a webpage where you can download the MITMProxy certificates. We need to load this certificate into pskey on our web servers. Download the Linux certificate even if you are on Windows (mitmproxy-ca-cert.pem) since it is in the correct format for pskey. Open the certificate and copy/paste the content onto your web server. I saved my file to /tmp/mitm.pem (or c:\temp\mitm.pem for Windows).

On your web server, use the keytool tool included with Java to import and trust the MITMProxy cert into pskey.

Bash

export domain=WEBSERVER01
keytool -importcert -alias mitmproxy -storepass password -keystore $PS_CFG_HOME/webserv/$domain/piaconfig/keystore/pskey -trustcacerts -file /tmp/mitm.pem

Owner: O=mitmproxy, CN=mitmproxy
Issuer: O=mitmproxy, CN=mitmproxy    
...

Trust this certificate? [no]:  yes
Certificate was added to keystore

Powershell

$domain="WEBSERVER01"
keytool -importcert -alias mitmproxy -storepass password -keystore $env:PS_CFG_HOME\webserv\$domain\piaconfig\keystore\pskey -trustcacerts -file c:\temp\mitm.pem

Owner: O=mitmproxy, CN=mitmproxy
Issuer: O=mitmproxy, CN=mitmproxy    
...

Trust this certificate? [no]:  yes
Certificate was added to keystore

HTTPS Node Test

Open the GETTEST node definition and update the URL to use https instead of http. Click “Ping Node” to retest with HTTPS.

“Server TLS handshake failed. Certificate verify failed: IP address mismatch”

You may receive an IB error when testing the HTTPS URLs. This can be expected with HTTPS. Many sites are improving their HTTPS support with Certificate Pinning, which helps prevent against “man in the middle” (MITM) attacks. Depending on the integration you are debugging, you may be able to work with HTTPS, or you may have to revert to HTTP for initial integration debugging before adding HTTPS back into the mix.

Application Server

You can also add MITMProxy to the application server configuration. In the psappsrv.cfg file there are two Proxy config lines used by the Java layer that runs inside the application server.

Proxy Host=mitmserver.psadmin.io
Proxy Port=8080

You will need to reconfigure the application server after updating these values.

#330 – New to 9.2

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson

This week on the podcast, Kyle and Dan talk about building a nightly refresh environment and the benefits of automating the process, and Dan talks about upgrading to PeopleSoft 9.2.

Show Notes

  • 9.2 Go-Live @ 1:45
  • Oracle MERGE command @ 13:30
  • Sandbox Refreshes @ 17:30

#329 – RTI Maintenance

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson

This week on the podcast, we talk about the PeopleTools RTI Maintenance job, fixing the systemd scripts from the DPK, and using the OCI-AutoScale project.

Show Notes

#328 – psadmin.conf 2022 Recap

The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson

This week on the podcast, Dan and Kyle recap the psadmin.conf 2022 conference.

Show Notes

  • Getting the Most Out of PeopleTools @ 2:30
  • Kibana Lab @ 10:15
  • Monday Open Lab @ 13:00
  • Automating Maintenance Windows @ 15:00
  • Journey to the Cloud @ 16:30
  • OCI Migration @ 18:45
  • Environment Validation Lab @ 21:30
  • Tuesday Open Lab @ 25:00
  • Securing PS with Apache Rules @ 27:00
  • PeopleTools Platform Overview @ 29:45
  • Lightning Talks @ 33:00
  • Real-Time Indexing @ 41:30