Apply CPU Patches with Deployment Packages

We have talked on the podcast about different ways to apply CPU patches, but with the DPK we have another tool to help us quickly apply CPU patches. This post and video demo’s will show you how to use the DPK to quickly apply CPU patches to your servers.

Deployment Workflow

When you run the DPK, it will deploy WebLogic, Java, Tuxedo (and more) on your server. The DPK uses archives (also known as “tarballs”) of prepackaged installations and extracts those archives to your server. There is one big problem, the archives included in the DPK’s do not contain the latest security patches. So, let’s make our own tarballs that include the security patches to deploy. This process is also a great exercise to better understand how the DPK deploys software.

If you are on Linux you can use the patching functionality with the DPK, but that code has not been written for Windows. I’m not covering that feature in this post, but the DPK Install Guide has a section on using that functionality (Task 6-3-1: Using the DPK Setup Script to Apply Fixes).

Movement Scripts

There are Fusion Middleware scripts the DPK uses to deploy WebLogic and Tuxedo. (Thanks to Eric Bolinger for pointing me in this direction.) The movement scripts allow you to take a current install of WebLogic, package it up, and deploy it to additional servers. This is how the DPK deploys WebLogic. The PeopleTools team packages up a WebLogic installation and we deploy that install to our servers. The movement scripts also manage the Oracle Inventory file for you.

There are many parts to the movement scripts, but we’ll be using just one part: copyBinary. This script will take a current installation and create a .jar file from that installation. We’ll use copyBinary to package our patched WebLogic installation.

If you have errors with the pasteBinary.cmd on the target system, you may need to configure the $ORACLE_HOME\oui\oraparam.ini file. This is a configuration file used by the OUI software. To make this simple, I copied the settings in the current $BASE\dpk\archives\weblogic12.1.3.0.tgz to my $ORACLE_HOME\oui\oraparam.ini using Beyond Compare. (Yes, Beyond Compare can read inside a tarball and compare against a directory!) Then I recreated my tarball with the updated oraparam.ini file.

Create a Patched WebLogic Tarball

 

Next, it’s time to install the CPU patch and run the copyBinary.cmd script. Stop all your PIA services on the server so you can remove the existing installations.

First, let’s patch Java. For demonstration, I’m using the jdk-7u141-windows-x64 installer. I’m installing

Then, we’ll use OPatch to apply the CPU to WebLogic:

cd $PATCH
$env:ORACLE_HOME=e:\psoft\pt\bea
$env:ORACLE_HOME\OPatch\OPatch napply

Once OPatch is done, we’ll use the movement scripts to package up our installation.

$env:JAVA_HOME=e:\psoft\pt\jdk1.7.0_141
. ${env:ORACLE_HOME}\oracle_common\bin\copyBinary.cmd -javaHome ${env:JAVA_HOME} -archiveLoc ${env:TEMP}\pt-weblogic-copy.jar -sourceMWHomeLoc ${env:ORACLE_HOME}

The output file from this command needs to be named pt-weblogic-copy.jar. The DPK expects that is the name of the .jar file. Next, we create a tarball of the pt-weblogic-copy.jar and two files to do the deploy portion of the movement scripts: cloningclient.jar and pasteBinary.cmd. These movement scripts are used by the DPK to deploy WebLogic. I used 7-zip to create my tarball with these three files:

$WL_VERSION="12.1.3.170418"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\jlib\cloningclient.jar"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\bin\pasteBinary.cmd"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:TEMP}\pt-weblogic-copy.jar"

Last, we gzip the archive and drop it in the $BASE\dpk\archives folder:

$env:DPK_BASE="e:\psft"
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-weblogic${env:WL_VERSION}.tgz" "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar"

One thing to note here – the DPK doesn’t handle multiple versions of software in the dpk\archives folder well. So, only have one pt-weblogic* file in there.

For Java, we don’t need to use the movement scripts. We’ll simply tarball up the new directory and include that in our $BASE\dpk\archives folder.

$JDK_VERSION="1.7.0_141"
7z a -ttar "${env:TEMP}\pt-jdk${JDK_VERSION}.tar" $env:JAVA_HOME\*
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-jdk${JDK_VERSION}.tgz" "${env:TEMP}\pt-jdk${JDK_VERSION}.tar"

Deploy CPU Patches

 

Copy your updated tarballs to a new server. You’ll want to remove the existing tarballs from the $BASE\dpks\archive to prevent the DPK from raising an error.

We have two options for telling the DPK we want to install WebLogic. The first option is to delete the existing WebLogic and Java folders. If you stop your PeopleSoft domains, you can delete both folders. When you run the DPK it will see that WebLogic and Java are missing and reinstall them from the patched tarballs in the $BASE\dpk\archives folder.

The other option is use the redeploy: true flag in psft_customizations.yaml. If you set the redeploy variable to true, the DPK will redeploy all the software in your $BASE\dpk\archives folder. This option requires less work – set a variable in psft_customizations.yaml and run the DPK – but it can take longer because you will redeploy Java, Tuxedo, WebLogic, PS_HOME and more. I think of this option as “the Puppet way”.

For this post and demo, we’ll use the redeploy: true option in our psft_customizations.yaml file. We’ll also use one other trick for testing; we will only run the part of the DPK that handles the middleware. Instead of running the entire DPK that touches the OS, middleware, and domains, the manifest we call includes only the DPK role that ensures the middleware is installed and not touch other parts of the system. This will also speed up our CPU patch deployment.

middleware.pp

Let’s create a new file under c:\programdata\puppetlabs\puppet\etc\manifests called middleware.pp. You can start by cloning the site.pp file. Change the file to look like this:

node default {
  include ::pt_role::pt_tools_deployment
}

Save the file. That’s it!

What we have done is tell Puppet to only run the DPK role pt_tools_deployment instead of running a larger role like pt_hcm_pum.

In the video demo, we are applying patches to a PeopleSoft Image, which is a Fulltier setup. The default pt_tools_deployment.pp manifest won’t run on a Fulltier system. To get around that, I created a copy of pt_tools_deployment.pp manifest called io_tools_deployment.pp and removed the check on env_type: fulltier.

cpu.ps1

We have a few tasks to do before we can run the middleware.pp manifest. We’ll wrap those tasks in a Powershell script we can run on each server.

At a high level, here are the tasks our cpu.ps1 script will do:

  1. Copy new DPK archives to server
  2. Stop PeopleSoft Services
  3. Remove current Java and WebLogic installs (if redeploy: false)
  4. Run middleware.pp to install patched Java and WebLogic
  5. Start PeopleSoft Services

Get the Sample Code

The full code is in the ps-dpk-tarballs GitHub repository. You can find all the scripts from this post and demo on GitHub.

#78 – Cloud Manager and Certifications w/ Mark Hoernemann

This week Mark Hoernemann joins us to talk about platform certifications with PeopleTools. Mark explains how the PeopleTools team determines what platforms to support and how the support policies work. Then Mark gives us more details on the PeopleSoft Cloud Manager.

Show Notes

  • Bringing App Engine to an SQR Discussion @ :45
  • Overriding Facter Facts @ 5:00
  • COBOL DPK and Unicode @ 7:45
  • Mark Hoernemann Interview @ 9:30
    • Deciding what platforms to support @ 11:45
    • SQL Server 2016 certification @ 15:30
    • Why platforms are dropped from support @ 18:00
    • Supporting frequent browser releases @ 20:00
    • 8.56 certification changes @ 24:00
    • Cloud Manager @ 24:45
    • Self Service Environment Administration @ 28:30
    • Getting Started with Cloud Manager @ 29:45
    • Building environments with Cloud Manager @ 37:00
    • Applying PeopleTools patches when provisioning environments @ 40:15
    • New PeopleTools features between releases @ 43:30

Configure Push Notifications

Thanks to Andy Dorfman for allowing us to guest-post his Push Notification setup. Andy did a lot of work to get Push Notifications working in a production environment and he’s sharing his notes. If you are looking to get Push Notifications running, this is a great place to start.

User Story

As a PeopleSoft user, when scheduling a process to run on a process scheduler, I would like to receive a Push Notification about the status of the process, as well as the link to the process output.

System Architecture:

  • 2 application servers

    • ps-apps-01
    • ps-apps-02
  • 2 process schedulers

    • ps-prcs-01
    • ps-prcs-02

Implementation:

Prerequisites:

  1. Until fixed in PeopleTools, on each server in PS_HOME\appserv\local_prcsgw.gbx, at ~ line 50, enclose ACCESSPOINTID value by "", i.e. change

    "PRCS_{$Inter-Domain Events\Process Scheduler Credentials[1]}"    ACCESSPOINTID={$Inter-Domain Events\Process Scheduler Credentials[1]}
    

    to

    "PRCS_{$Inter-Domain Events\Process Scheduler Credentials[1]}"    ACCESSPOINTID="{$Inter-Domain Events\Process Scheduler Credentials[1]}"
    
  2. Firewall needs to be configured to allow communication from ps-apps-01 and ps-apps-02 to ps-prcs-01:8988 and ps-prcs-02:8988, i.e. each application server needs to be able to reach both process schedulers on port 8988. It could potentially be possible to restrict source port range for greater security as well.

  3. Firewall needs to be configured to allow communication from ps-prcs-01 and ps-prcs-02 to ps-apps-01:7988 and ps-apps-02:7988, i.e. each proces scheduler needs to be able to reach both application servers on port 7988. It could potentially be possible to restrict source port range for greater security as well.

Configuration:

  1. In application server configuration, define distinct “Domain ID”s (aka [Domain Settings]\Domain ID), e.g.

    • AS1 on ps-apps-01
    • AS2 on ps-apps-01

    values can be any valid sting for [Domain Settings]\Domain ID, as long as they are distinct

  2. In process scheduler server configuration, define “PrcsServerName”s (aka [Process Scheduler]\PrcsServerName), e.g.

    • PS1 on ps-prcs-01
    • PS2 on ps-prcs-02

    values can be any valid sting. Developer statated that Application servers currently don’t look at ACCESSPOINTID for process scheduler domains and only care about the hostname:port, but if you want multiple Process Scheduler domains, the names have to be distinct regardless.

  3. On each application server domain and process scheduler domain, enable Inter-Domain Communication and Push Notifications

  4. Setup Application Server

    1. On each application server, examine PS_CFG_HOME\appserv\<DOMAIN>\interdom.gbb. Pay special attention to uncommented lines that start with “AS_<Domain ID>_<IPCKEY>”, e.g.

      "AS_AS1_53252"    GWGRP=GWTGROUP  ACCESSPOINTID="AS1_53252" CONNECTION_POLICY=ON_STARTUP
      "AS_AS1_53252"    NWADDR="//PS-APPS-01:7988"
      

      only NWADDR is important here, ACCESSPOINTID will be you Application Server Domain ID from (1)

    2. Assemble the following string using the known information

      "AS1|PS-APPS-01:7988,AS2|PS-APPS-02:7988"
      

      notice AS1 comes from (1), while ps-apps-01:7988 comes from NWADDR in (4.1). The multiple application server domain connection strings are separated by a “,” with no space.

    3. This string above would go into each process scheduler domain configuration under [Inter-Domain Events]\Application Server Credentials, either via psadmin custom configuration or manually

      [Inter-Domain Events]
      ;=========================================================================
      ; InterDomain Event settings
      ;=========================================================================
      Process Scheduler Port=8988
      Application Server Credentials=AS1|PS-APPS-01:7988,AS2|PS-APPS-02:7988
      
  5. Setup Process Scheduler

    1. Similarly, note the setting PS_CFG_HOME\appserv\prcs\<DOMAIN>\interdom.gbb on each process scheduler domain:

      "PRCS"    GWGRP=GWTGROUP1  ACCESSPOINTID="PRCS_PS-PRCS-01_PS1"  CONNECTION_POLICY=ON_STARTUP
      "PRCS"    NWADDR="//PS-PRCS-01:8988"
      

      Both ACCESSPOINTID and NWADDR are important in this case

    2. Assemble the following string using the known information

      "PRCS_PS-PRCS-01_PS1|PS-PRCS-01:8988,PRCS_PS-PRCS-02_PS2|PS-PRCS-02:8988"
      

      The format is ACCESSPOINTID|NWADDR,ACCESSPOINTID|NWADDR from (5.1). The multiple process scheduler connection strings are separated by ‘,’ with no space in between

    3. This string above should go into each application server configuration under [Inter-Domain Events]\Process Scheduler Credentials, either manually or through psadmin custom configuration menu, e.g.

      [Inter-Domain Events]
      ;=========================================================================
      ; InterDomain Event settings
      ;=========================================================================
      Process Scheduler Credentials=PRCS_PS-PRCS-01_PS1|PS-PRCS-01:8988,PRCS_PS-PRCS-02_PS2|PS-PRCS-02:8988
      Application Server Port=7988
      
  6. Clear IPC on each application server and process scheduler domain. Reconfigure and boot each application and process scheduler domain. The boot order does not matter, the ping happens every 60 seconds by default.

  7. Verify that the communication is successfully taking place by examining TUXLOG on each domain, e.g.

        094833.PS-APPS-01!GWTDOMAIN.3660.4520.0: LIBGWT_CAT:1128: INFO: Connection accepted from domain (domainid=<PRCS_PS-PRCS-01_PS1>)
    
  8. In PIA, navigate to “PeopleTools > Web Profile > Web Profile Configuration”. In “Custom Properties”, if not already there, set property EnablePNSubscriptions to true, save the profile and restart the Web Server domain.

  9. Optionally, in PIA, navigate to “My Preferences” and in “Pop-up Notification” section, set “Yes” for all States and for “Enable Popup Notification”. This will allow slide-in notifications

  10. Run XRFWIN and verify that slide-in notifications are occuring and alerts are populated under Notification Center.

#77 – Why the Cloud is Important w/ David Bain

This week on the podcast David Bain, Product Manager for PeopleTools, joins us to talk about Fluid, the Cloud and PeopleSoft, and his favorite PeopleTools features. Dan and Kyle also recap some great discussions they had at the Collaborate conference.

Show Notes

  • David Bain Interview
    • Why the cloud is important to PeopleSoft @ 2:00
    • Why Fluid is a better user interface @ 13:00
    • Favorite new PeopleTools feature @ 19:45
    • How PeopleSoft uses PTF internally @ 22:45
    • Favorite PeopleTools feature of all time @ 26:00
  • Recap of David’s Interview @ 28:00
  • Unsexy software features @ 38:00
  • “Hidden” PeopleTools features @ 39:30
  • Interaction Hub @ 45:45
  • Phire and a Testing Module @ 52:30

#76 – Hybrid Cloud w/ Jim Marion

This week on the podcast Jim Marion joins us to talk about Fluid Development, embedding PeopleSoft into cloud applications, Event Mapping, where the cloud makes sense and Jim’s new role at GreyHeller.

Show Notes

  • Jim’s Background @ 2:15
  • New Role with Grey Heller @ 3:45
  • Thoughts on the future of PeopleTools @ 6:00
  • Fluid Development @ 9:00
  • Improving App Designer @ 11:30
  • Event Mapping and Branding Objects @ 14:00
  • Using the Cloud at Grey Heller @ 25:45
  • Embedding PeopleSoft with Cloud Applications @ 32:00
  • Disadvantages of Hybrid Cloud Systems @ 43:30
  • Where the Cloud makes sense @ 47:30

#75 – Selling Yourself

This week, Dan and Kyle talk about testing different web server configurations, using the ACM for Elasticsearch, and how mobile browsers work with websites. Then, they discuss different ways to promote yourself and your position to a boss or organization.

Show Notes

Change Assistant Debug Mode

This is a short post, but hopefully helpful. I recently discovered that you can turn on Change Assistant with a debug mode using an environment variable. There is no documentation on this, but it seems to work well and might help you resolve any issues you have with Change Assistant.

To enable the debug mode, set the PSCADBG environment variable. I used “true” for the value, but any value should work.

$env:PSCADBG="true"

Then launch Change Assistant from the command line:

c:\Program Files\PeopleSoft\Change Assistant\changeassistant.bat

In the command line you should see the debug output, like this:

Signed on to 'PSFTDB'. Using c:\PT8.55.13_Client_ORA\
Attempt to signon using PS/PS@PSFTDB
Controller sending c:\PT8.55.13_Client_ORA client -com.peoplesoft.pt.changeassistant.pshome.request.db.SignOnDatabase
Controller received from c:\PT8.55.13_Client_ORA client -com.peoplesoft.pt.changeassistant.pshome.request.db.SignOnDatabase
Signed on to 'PSFTDB'. Using c:\PT8.55.13_Client_ORA\

#74 – Killing COBOL

This week on the podcast, Kyle and Dan talk about planning PeopleTools and Catch-up projects, BI Publisher security, and how to turn off excessive BI Publisher logging. We also talk about slowly killing COBOL with PeopleSoft (it’s not dead yet) and using multiple Change Assistant installations.

Show Notes

PeopleSoft and Touch Icons

I was looking through my web server logs and noticed that they contained a large amount of 404 errors for /apple-touch-icon*.png and /favicon.ico requests. The /apple-touch-icon*.png requests come from mobile devices (mostly iOS, but also some Android) and all browsers will look for the favicon.ico image file. Here is 24 hours of our top 404 requests:

Today's Top 404 Requests

The files are used as an icon if the a user saves the website to their device’s homepage, or it shows up on pinned tabs and bookmarks. That’s a nice feature, but I want to clean up my log files and removing the extraneous 404 responses. Let’s generate the images that mobile devices and browsers expect and clean up our log files. As an added benefit, we’ll generate some nice icons to add polish to our application and be more mobile-friendly.

Create apple-touch-icon.png

The first step is to decide what image you want as the icon. For my demo system, I’ll use “io” logo; it’s a simple and clean logo that will look good as a small icon. It’s best to choose and image that will look good as a square. Because phones and tablets use different size icons, we could go through the work of creating different sizes by hand, but there is web site that does all the work for us: RealFavIconGenerator.

The site is simple to work with – upload the image you want to use, change any configuration for the different images and site title, then download a zip file. You’ll also get some HTML elements to add to your signon page (signin.html if you are using the default file)

A preview of your icon on iOS, Android, and browser tabs:

The site also generates some HTML elements to put in the <head> section of your login page. All browsers will read these values to determine which icons files to use.

<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<link rel="icon" type="image/png" href="/favicon-32x32.png" sizes="32x32">
<link rel="icon" type="image/png" href="/favicon-16x16.png" sizes="16x16">
<link rel="manifest" href="/manifest.json">
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#04a6ff">
<meta name="apple-mobile-web-app-title" content="psadmin.io">
<meta name="application-name" content="psadmin.io">
<meta name="theme-color" content="#ffffff">

Deploy the Touch Icons

At this point, we have some HTML to add to our signin.html file, and a zip file that we need to deploy and extract. We can manually modify and copy each file to the web server, but let’s use the DPK and Puppet to deploy these files.

touch-icon.pp

This puppet manifest assumes you have things in place:

  • Your web domains and sites defined in psft_customizations.yaml
  • This is written for Windows, but it can easily be adapted for Linux.
  • Powershell 4 or higher installed (to use the Expand-Archive command)
  • The puppetlabs-powershell Puppet module is installed (puppet module install puppetlabs-powershell to install the module)
  • The updated signin.html and favicons.zip files on a network share

At a high level, this is what the touch-icon.pp manifest does:

  1. Define the network share where to grab the files
  2. Grab the list of PIA domains into pia_domain_list
  3. Deploy the favicons.zip to the PORTAL.war folder
  4. Extract the favicons.zip file so each image and file is at the web server’s root level
  5. Grab the list of PIA sites for the domain into site_list
  6. Deploy the updated signin.html to each site

To run the manifest, navigate to C:\ProgramData\PuppetLabs\puppet\etc\manifests and save the file as touch-icon.pp. Then run puppet apply .\touch-icon.pp --trace --debug.

Before and After for Mobile Devices

Here are two screen shots from my iPhone when I add PeopleSoft to the homepage. In the first screenshot you can see the icon is a tiny version of the login page and the title is generic. In the second screenshot (after deploying the files) you can see the excellent icon and the simple title.

Before

After

#73 – Oracle Configuration Manager

This week on the podcast, Kyle and Dan talk about Query Manager bugs, changing how they apply CPU Patches, and testing partial database refreshes. Then Dan gives an overview of Oracle Configuration Manager and laments it’s unsupported status.

Show Notes

  • Collaborate Sessions
  • PeopleTools Ideas Page @ 3:30
  • IDDA Logging Follow-up @ 4:15
  • Query Manager / JSON bug @ 6:00
  • Changing how you apply CPU Patches @ 10:15
  • ProcessRestartMemoryLimit Parameter @ 18:15
  • Testing localapp @ 24:45
  • PTF being slow @ 29:15
  • SRID in Access Log @ 32:30
  • Populating Correlation Fields @ 35:30
  • Partial database refresh to grab old code @ 37:30
  • [Keeping Older PI Images] (https://support.oracle.com/epmos/faces/DocumentDisplay?id=1579720.1) @ 40:30
  • How do you do it: Booting domains: Serial or Parallel? @ 45:45
  • Oracle Configuration Manager @ 48:15