#97 – Dozens of Us

This week on the podcast, Dan and Kyle discuss strategies to organize and manage Event Mapping code, writing code that other people will use, trying to work with Push Notifications and custom ACM modules. Kyle ends the podcast with a funny story about searching for images.
 

Show Notes

 

Using Puppet Environments with the DPK

Using Puppet Environments with the DPK

Since the Deployment Packages were released with PeopleTools 8.55, one of my criticisms has been that the DPK is a bit of a sledgehammer. If you define multiple PeopleSoft environments on a server and you want to configure one web server, ALL the domains that the DPK knows about are shut down.

Puppet has an Environments feature that lets you segregate your code and data. While the DPK does not support Puppet Environments out of the box, we can use them to make the DPK less of a sledgehammer when managing our domains. (There is still some sledgehammering going on, so go vote for this idea).

While environments let you separate the modules, manifests and data folder, in this post we’ll separate just the data folder. This will let us share a common set of code (the manifests and modules folders) but the configuration of each domain will be different.

If you want to extend this to the modules and manifests folder, copy those into the environment folders with the environment-specific changes. This is useful for testing new code changes or if you want an environment to use a different DPK Role in the site.pp file.

Create Environment Folders

  1. Make a new dev and tst folders under c:\programdata\puppetlabs\puppet\etc\environments

You can have multiple environments under this folder – as many as you want. A strategy that I’m testing is using the database name as the environment name. For this post, I’ll stick with dev and tst

  1. Copy your YAML files from puppet\etc\data to puppet\etc\environments\dev\data and puppet\etc\ environments\tst\data.

Configure Puppet Environment

Under the puppet\etc folder, add (or modify) the puppet.conf file to look like this:

[main]
environment=production
parser=future
environmentpath=c:\programdata\puppetlabs\puppet\etc\environments
hiera_config=c:\programdata\puppetlabs\hiera\etc\hiera.yaml
basemodulepath=c:\programdata\puppetlabs\puppet\etc\modules

This file tells Puppet where to look for your environments, your Hiera configuration, your default module location, and the default Puppet Environment.

Last, we’ll modify the hiera.yaml file in c:\programdata\puppetlabs\hiera\etc to include environments:

---
:backends:
  - yaml

:hierarchy:
  - "environments/%{::environment}/data/psft_customizations"
  - "environments/%{::environment}/data/psft_configuration"
  - "environments/%{::environment}/data/psft_deployment"
  - "environments/%{::environment}/data/psft_unix_system"
  - "environments/%{::environment}/data/defaults"

:yaml:
  :datadir: c:\programdata\puppetlabs\puppet\etc

If you want to share some of the files, like the defaults.yaml or the psft_unix_system.yaml file, you could keep those under the main puppet\etc\data folder. Your hiera.yaml file would look like this:

---
:backends:
  - yaml

:hierarchy:
  - "environments/%{::environment}/data/psft_customizations"
  - "environments/%{::environment}/data/psft_configuration"
  - "environments/%{::environment}/data/psft_deployment"
  - data/psft_unix_system
  - data/defaults

:yaml:
  :datadir: c:\programdata\puppetlabs\puppet\etc

Test the Environments

Once our Puppet changes are complete we can test some builds. When we run puppet apply, we’ll add an additional paratemer: the environment. To build my dev environment domains, I’ll use this procedure:

cd c:\programdata\puppetlabs\puppet\etc\manifests
puppet apply .\site.pp --environment=dev --debug

Once the dev domains are built and running, you can kick off the tst build with:

puppet apply .\site.pp --environment=tst --debug

As the tst environment is building, your dev domains should stay up and not be affected by the Puppet run. If they are affected, you may have some YAML changes that need to be made. Make sure your configuration’s between the environment don’t overlap (e.g, same PS_CFG_HOME and domain names).

Convert the DPK to use Hiera Hash Merging

The way PeopleSoft delivers Puppet and the Hiera backend, is that everything you define in psft_customizations.yaml overrides configuration defined elsewhere. This is a useful setup when getting started with the DPK and Puppet. But when using YAML files to manage your configuration across multiple servers, you’ll quickly find that you are re-entering the same configuration in many files.

Hiera, the tool Puppet uses to read YAML files, has multiple ways to look up data. First, let’s cover what a YAML hash is. A hash is a key-value structure used in the DPK to store configuration. For example, this is the hash for PS_HOME information:

ps_home:
  db_type:    "%{hiera('db_platform')}"
  unicode_db: "%{hiera('unicode_db')}"
  location:   "%{hiera('ps_home_location')}"
  remove:     true

The main hash key is ps_home, and its value is all the configuration below it. The next level down has 4 keys with 4 corresponding values. The appserver_domain_list hash is a large one that contains all the configuration for one or more app server domains.

Under the delivered setup, if you want to change a value for a domain you need to copy the entire appserver_domain_list hash into your psft_customizations.yaml file and make the change. With Hiera hashing, you could define your domains in a file named appservers.yaml and any specific server changes can be defined in hostname.yaml. For example, the hostname.yaml file could contain this hash to override a configuration:

appserver_domain_list:
  DEV:
    feature_settings:
      SERVER_EVENTS: "Yes"
      DOMAIN_GW:     "Yes"

This provides far more flexibility when working with YAML files, but it does introduce some complexity. If you want to give this a try, here is how you can convert the current DPK to use Hiera hasing.

Find/Replace

I used VisualStudio Code to do the find/replace. Open up the etc\modules directory and do these against the modules\pt_profile folder:

  • Find: hiera('tns_admin_list
  • Replace: hiera_hash('tns_admin_list

I repeated this step for the following lookups.

  • tns_admin_list
  • appserver_domain_list
  • prcs_domain_list
  • pia_domain_list

You don’t want to replace all the lookups – that will cause errors. But, you can replace additional lookups if you want. Anything that is a hash in YAML files can use the hiera_hash() lookup function. If you wanted to make the ps_home: key support hash merging, you could replace hiera('ps_home with hiera_hash('ps_home.

Change the Hiera Merge Behavior

By default, Hiera will look at the top-level keys of a hash and not merge the underlying settings. Hiera hashing will merge all the values inside the hash. This means you can you define a hash with default settings in a common file (e.g, default app server settings). Then you can specify server or application specific settings in a YAML file for that domain or server.

To enable the hash merging, open the hiera.yaml file under c:\programdata\puppetlabs\hiera\etc.

Add this line to the file:

:merger_behavior: deeper

Hiera Lookup Order

With Hiera hash merging, we can utilize more than the psft_customizations.yaml file to manage our configuration. We can use multiple YAML files to control our configuration. For example, we could have:

  • [hostname].yaml
  • dev.yaml
  • hr.yaml
  • common.yaml

So, this setup would let us define common configuration that is shared across all applications in the common.yaml. Next, we could define anything related to servers that run HR applications in the hr.yaml. For any settings that are specific to the Development region, we can add them into dev.yaml. Last, for anything that is specific to the server, we can add into the [hostname].yaml file. This setup would let you re-use the common, hr, and dev YAML files across multiple servers, and anything specific to the server would be defined in [hostname].yaml.

In the hiera.yaml file, we can define this setup like this:

:hierarchy:
  - "%{::hostname}"
  - dev
  - hr
  - common

Test Hiera Hashing

On the command line, you can use the hiera utility to test lookups with Hiera. To do a normal Hiera lookup, use

hiera appserver_domain_list

To test a hiera hash lookup, use

hiera --hash appserver_domain_list

If you have multiple YAML files with the appserver_domain_list hash, the first option will only show you the results from the top of the list. The second test should show you a merged appserver_domain_list hash.

#92 – Advanced Puppet with the DPK

This week on the podcast, Dan dives into some advanced Puppet configuration to use with the DPK. Dan and Kyle discuss Hiera hash merging, using Puppet environments, and using the ACM with the DPK. Kyle talks about a bug with the Interaction Hub and My Favorites and some odd finds in PeopleBooks.

Show Notes

Using Automated Configuration Management with the DPK

The PeopleSoft Cloud Architecture is built on two technologies: Deployment Packages (DPK) and Automated Configuration Management (ACM). On this site, we’ve talked about Deployment Packages quite a bit, but we haven’t discussed Automated Configuration Management. This post will introduce you to what the ACM is and how the DPK uses it.

Use Hiera to Turn ACM On/Off

In the past, I have created custom DPK roles without the DPK profiles that run the ACM. This was mostly to prevent ACM runs when we build a domain. It turns out that you can control the ACM (globally) with your psft_customizations.yaml file.

In the pt_tools_preboot_config and pt_tools_postboot_config profile, there is a Hiera lookup to see if ACM is enabled

$run_preboot_config_setup  = hiera('run_preboot_config_setup', true)

or

$run_postboot_config_setup  = hiera('run_postboot_config_setup', true)

If you haven’t defined these values, the default will be true. You can disable the ACM by adding this your psft_customizations.yaml:

run_preboot_config_setup: false
run_postboot_config_setup: false

There is a bug in the current DPK (8.55 and 8.56) if you set both values. In the modules/pt_profile/manifests/pt_tools_postboot_config.pp file, line 98 should be:

notify {"POST-Boot setup run is false":}

(Switch the message “PRE-Boot” to “POST-Boot”).

Building ACM Hiera Data

When the DPK runs Automated Configuration Management, it will take the plugins and properties you define in the psft_customizations.yaml file and build a template file on the fly. To define ACM steps to run, you’ll add hashes to one of two sections:

  • component_preboot_setup_list
  • component_postboot_setup_list

The labels are self-explanatory, but here is any easy way to know which plugs go in the post boot setup: If the setup requires IB to be running, it goes in post boot. That would include IB Configuration itself and Search Framework configuration.

Here is an example of setting up the Elasticsearch Search Instance via ACM. Unde the component_postboot_setup_list hash, we create a searching hash. (This name can be any string, but we’ll use that string later in the file). Then, we set the Run Control ID to use and the OS user who will run the App Engine.

component_postboot_setup_list:
  searching:
    run_control_id:                       searching
    os_user:                              "%{hiera('domain_user')}"`

Next, we define the database connection settings for the web_profile hash.

    db_settings:
      db_name:                            "%{hiera('db_name')}"
      db_type:                            "%{hiera('db_platform')}"
      db_opr_id:                          "%{hiera('db_user')}"
      db_opr_pwd:                         "%{hiera('db_user_pwd')}"
      db_connect_id:                      "%{hiera('db_connect_id')}"
      db_connect_pwd:                     "%{hiera('db_connect_pwd')}"`

Then we define the ACM Plugins we want to run. The acm_plugin_list is a hash of ACM Plugin names, and their repective configuration. To get a list of all the possible configuration, you can go into the PIA (PeopleTools > Automated Config Manager > ACM Templates > Template Definitions) and see the delivered ACM plugins.

Below is an example of the PTSFConfigureSrchInstance plugin.

    acm_plugin_list:
      PTSFConfigureSrchInstance:
        env.ptsf_search_instance:         PTSF_DEFAULT
        env.search_provider:              ES
        env.search_nodes:                 1
        env.node1_search_port:            9200
        env.node1_search_host:            psvagabond.psadmin.io
        env.node1_search_admin_user:      esadmin
        env.node1_search_admin_password:  esadmin
        env.node1_search_read_user:       people
        env.node1_search_read_password:   peop1e
        env.search_call_back_user:        PS
        env.search_call_back_password:    PS
        env.gateway_host:                 psvagabond.psadmin.io
        env.gateway_port:                 8000
        env.default_local_node:           PSFT_LM

    acm_plugin_order:
      - PTSFConfigureSrchInstance

Once you have filled out the plugin configuration vaules, and if you have multiple ACM hashes, you need to tell the DPK what order you want to execute the ACM plugins. You use the component_preboot_setup_list has to specify that order. Below, you’ll see we have 3 ACM sections we want to execute. These names are the hashes we created under the component_preboot_setup_list section.

    component_preboot_setup_list:
      - searching
      - push_notifications

PeopleBooks has a good list of delivered plugins and what configuration options are available.

Testing ACM via DPK

To test the ACM with the DPK, we can tell Puppet to run only the ACM preboot or postboot step. Use the command below to execute a single DPK Profile:

puppet apply -e "include ::pt_profile::pt_tools_postboot_config" --trace --debug

This will run the postboot configuration you have defined in psft_customizations.yaml without running anything else defined in Puppet.

Validating ACM

If you use the --trace --debug options when running Puppet, you can see the output from the PTEM_CONFIG app engine.

########################################################### 
    ######## AUTOMATED CONFIGURATION MANAGER ######## 
###########################################################

PTEM_CONFIG:PTSFConfigureSrchInstance: Configure Peoplesoft system to talk to Search Server and assign roles

Configuring plug in : PTEM_CONFIG:PTSFConfigureSrchInstance 
STATUS: SUCCESS 
DESCRIPTION: NONE 
SEVERITY: NONE 
Configuration completed : PTEM_CONFIG:PTSFConfigureSrchInstance

****Completed environment configuration****

You can also get information about missing or invalid configuration from your ACM definition with the output:

########################################################### 
    ######## AUTOMATED CONFIGURATION MANAGER ######## 
###########################################################

Validation of variables failed : PTEM_CONFIG:PTSFConfigureSrchInstance 
Required property env.ptsf_search_instance is missing

Sample psft_customizations.yaml for ACM

Here is a gist with the Elasticsearch ACM Plugin configured. This Hiera data will configure an Elasticsearch instance in your environment.

#91 – Testing Metrics w/ Frank Dolezal

This week on the podcast, Frank Dolezal joins us to talk about how he measure user testing and how it builds confidence when deploying updates. We also discuss self-service and fast database refreshes, and what skills to look for when hiring a PS Admin.

Show Notes

  • Introducing Frank @ 1:00
  • Creating Metrics for User Testing @ 4:00
  • Getting Users to Test @ 11:00
  • Justifying testing through metrics @ 17:00
  • Data Sources to gather testing metrics @ 23:00
  • Self Service Refreshes @ 28:30
  • Planning Catch-up Projects @ 39:00
  • Changing PT Release Schedule? @ 43:00
    • Oracle said that there was no plans to change, so this is speculation only
  • Skillsets when hiring a PS Admin @ 48:00

#87 – PeopleTools 8.56

PeopleTools 8.56 has been released. This week on the podcast we talk about new features in 8.56 that we are excited about and dive into some of the changes to Deployment Packages.

Show Notes

  • 8.56 is out! @ 2:00
  • Platform Changes @ 4:45
  • Fluid Changes @ 7:00
  • Event Mapping API? @ 11:00
  • Searching Improvements @ 11:45
  • PUM @ 14:00
  • Portal Changes @ 18:30
  • Unified Navigation @ 21:00
  • PTSYS Database @ 26:30
  • Classic Plus @ 30:15
  • PSCBO @ 34:30
  • Role/PL Aliases @ 35:15
  • Event Mapping @ 37:45
  • VCD Still Around @ 40:00
  • Change Assistant @ 41:45
  • Deployment Packages @ 48:00
  • RECONNECT @ 68:00

#83 – DPK – What We’d Do Different Next Time

This week on the podcast Dan and Kyle discuss changes they would make if they started working on the DPK for the first again. They also discuss setting up Elasticsearch clusters, monitoring Tuxedo queues with Elasticsearch and an annoying Bash on Windows bug.

Show Notes

  • Bash on Windows VPN Bug @ 3:15
  • 9.2 Upgrade Process @ 10:00
  • Monitoring Tuxedo Queueing in Elasticsearch @ 13:30
  • Tuxbeat Project @ 17:00
  • Elasticsearch DPK @ 21:45
  • Elasticsearch Clustering @ 27:00
  • Puppet on Linux @ 32:00
  • DPK Extract Only @ 34:30
  • What we’d do differently with the DPK @ 37:30
    • Co-locating Regions
    • Admin-only Test Servers
    • Config Homes
    • Use Git for managing code and YAML changes
    • Integrate all Puppet code into Classes and Modules
    • DPK Course
    • Use ACM via DPK
    • Go Vanilla

App Capacity Visualization

#82 – Embracing Fluid Navigation

This week on the podcast, Kyle and Dan revisit Fluid Navigation and why they are fans of the new navigation model. Kyle shares his experiences working with the DPK and his method for managing the YAML and Puppet files on servers. Dan shares a top-notch “Dad Joke”.

Show Notes

Apply CPU Patches with Deployment Packages

We have talked on the podcast about different ways to apply CPU patches, but with the DPK we have another tool to help us quickly apply CPU patches. This post and video demo’s will show you how to use the DPK to quickly apply CPU patches to your servers.

Deployment Workflow

When you run the DPK, it will deploy WebLogic, Java, Tuxedo (and more) on your server. The DPK uses archives (also known as “tarballs”) of prepackaged installations and extracts those archives to your server. There is one big problem, the archives included in the DPK’s do not contain the latest security patches. So, let’s make our own tarballs that include the security patches to deploy. This process is also a great exercise to better understand how the DPK deploys software.

If you are on Linux you can use the patching functionality with the DPK, but that code has not been written for Windows. I’m not covering that feature in this post, but the DPK Install Guide has a section on using that functionality (Task 6-3-1: Using the DPK Setup Script to Apply Fixes).

Movement Scripts

There are Fusion Middleware scripts the DPK uses to deploy WebLogic and Tuxedo. (Thanks to Eric Bolinger for pointing me in this direction.) The movement scripts allow you to take a current install of WebLogic, package it up, and deploy it to additional servers. This is how the DPK deploys WebLogic. The PeopleTools team packages up a WebLogic installation and we deploy that install to our servers. The movement scripts also manage the Oracle Inventory file for you.

There are many parts to the movement scripts, but we’ll be using just one part: copyBinary. This script will take a current installation and create a .jar file from that installation. We’ll use copyBinary to package our patched WebLogic installation.

If you have errors with the pasteBinary.cmd on the target system, you may need to configure the $ORACLE_HOME\oui\oraparam.ini file. This is a configuration file used by the OUI software. To make this simple, I copied the settings in the current $BASE\dpk\archives\weblogic12.1.3.0.tgz to my $ORACLE_HOME\oui\oraparam.ini using Beyond Compare. (Yes, Beyond Compare can read inside a tarball and compare against a directory!) Then I recreated my tarball with the updated oraparam.ini file.

Create a Patched WebLogic Tarball

 

Next, it’s time to install the CPU patch and run the copyBinary.cmd script. Stop all your PIA services on the server so you can remove the existing installations.

First, let’s patch Java. For demonstration, I’m using the jdk-7u141-windows-x64 installer. I’m installing

Then, we’ll use OPatch to apply the CPU to WebLogic:

cd $PATCH
$env:ORACLE_HOME=e:\psoft\pt\bea
$env:ORACLE_HOME\OPatch\OPatch napply

Once OPatch is done, we’ll use the movement scripts to package up our installation.

$env:JAVA_HOME=e:\psoft\pt\jdk1.7.0_141
. ${env:ORACLE_HOME}\oracle_common\bin\copyBinary.cmd -javaHome ${env:JAVA_HOME} -archiveLoc ${env:TEMP}\pt-weblogic-copy.jar -sourceMWHomeLoc ${env:ORACLE_HOME}

The output file from this command needs to be named pt-weblogic-copy.jar. The DPK expects that is the name of the .jar file. Next, we create a tarball of the pt-weblogic-copy.jar and two files to do the deploy portion of the movement scripts: cloningclient.jar and pasteBinary.cmd. These movement scripts are used by the DPK to deploy WebLogic. I used 7-zip to create my tarball with these three files:

$WL_VERSION="12.1.3.170418"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\jlib\cloningclient.jar"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:ORACLE_HOME}\oracle_common\bin\pasteBinary.cmd"
7z a -ttar "${env:TEMP}\pt-weblogic${WL_VERSION}.tar" "${env:TEMP}\pt-weblogic-copy.jar"

Last, we gzip the archive and drop it in the $BASE\dpk\archives folder:

$env:DPK_BASE="e:\psft"
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-weblogic${env:WL_VERSION}.tgz" "${env:TEMP}\pt-weblogic${env:WL_VERSION}.tar"

One thing to note here – the DPK doesn’t handle multiple versions of software in the dpk\archives folder well. So, only have one pt-weblogic* file in there.

For Java, we don’t need to use the movement scripts. We’ll simply tarball up the new directory and include that in our $BASE\dpk\archives folder.

$JDK_VERSION="1.7.0_141"
7z a -ttar "${env:TEMP}\pt-jdk${JDK_VERSION}.tar" $env:JAVA_HOME\*
7z a -tgzip "${env:DPK_BASE}\dpk\archives\pt-jdk${JDK_VERSION}.tgz" "${env:TEMP}\pt-jdk${JDK_VERSION}.tar"

Deploy CPU Patches

 

Copy your updated tarballs to a new server. You’ll want to remove the existing tarballs from the $BASE\dpks\archive to prevent the DPK from raising an error.

We have two options for telling the DPK we want to install WebLogic. The first option is to delete the existing WebLogic and Java folders. If you stop your PeopleSoft domains, you can delete both folders. When you run the DPK it will see that WebLogic and Java are missing and reinstall them from the patched tarballs in the $BASE\dpk\archives folder.

The other option is use the redeploy: true flag in psft_customizations.yaml. If you set the redeploy variable to true, the DPK will redeploy all the software in your $BASE\dpk\archives folder. This option requires less work – set a variable in psft_customizations.yaml and run the DPK – but it can take longer because you will redeploy Java, Tuxedo, WebLogic, PS_HOME and more. I think of this option as “the Puppet way”.

For this post and demo, we’ll use the redeploy: true option in our psft_customizations.yaml file. We’ll also use one other trick for testing; we will only run the part of the DPK that handles the middleware. Instead of running the entire DPK that touches the OS, middleware, and domains, the manifest we call includes only the DPK role that ensures the middleware is installed and not touch other parts of the system. This will also speed up our CPU patch deployment.

middleware.pp

Let’s create a new file under c:\programdata\puppetlabs\puppet\etc\manifests called middleware.pp. You can start by cloning the site.pp file. Change the file to look like this:

node default {
  include ::pt_role::pt_tools_deployment
}

Save the file. That’s it!

What we have done is tell Puppet to only run the DPK role pt_tools_deployment instead of running a larger role like pt_hcm_pum.

In the video demo, we are applying patches to a PeopleSoft Image, which is a Fulltier setup. The default pt_tools_deployment.pp manifest won’t run on a Fulltier system. To get around that, I created a copy of pt_tools_deployment.pp manifest called io_tools_deployment.pp and removed the check on env_type: fulltier.

cpu.ps1

We have a few tasks to do before we can run the middleware.pp manifest. We’ll wrap those tasks in a Powershell script we can run on each server.

At a high level, here are the tasks our cpu.ps1 script will do:

  1. Copy new DPK archives to server
  2. Stop PeopleSoft Services
  3. Remove current Java and WebLogic installs (if redeploy: false)
  4. Run middleware.pp to install patched Java and WebLogic
  5. Start PeopleSoft Services

Get the Sample Code

The full code is in the ps-dpk-tarballs GitHub repository. You can find all the scripts from this post and demo on GitHub.