Using the DPK Redeploy Option for CPU Patching

There are lots of ways to apply CPU patching and how to automate it. We’ve covered a few methods on the blog, but I wanted to show another way with the DPK. This method combines using patched archives (discussed here) and an option in the DPK that isn’t really talked about: redeploying software.

Redeploy Option

In the DPK, there is an option to redeploy software that you have already installed. The redeploy option is supported for all the middleware, PS_HOME, and PS_APP_HOME. If you include

redeploy: true

in your psft_customizations.yaml file, the DPK will uninstall all the middleware, PS_HOME, and PS_APP_HOME (if your DPK role is an app role), and then reinstall those from your DPK archives folder. This is slick if you want fix a bad install, or in our case, apply a new version of the software. We can replace the archive file for JDK or WebLogic, trigger a redeploy by running Puppet, and the new version will be installed.

But, there are some downsides to how the DPK handles this. First, it’s all or nothing; you can redeploy all of the middleware or none of the middleware. If you redeploy, your PS_HOME (and PS_APP_HOME) will be redeployed and you’ll need to recompile your COBOL files (if you still use them). That’s too sledgehammer-y for me.

Also, we don’t always want to redeploy our software. We may only want to run a redeploy for new patches. To enable a redeploy, we have to update our configuration, run puppet, then remove our configuration change. I don’t like making transactional changes to my configuration files. Our configuration should be static.

A Better Redeploy Option

Given these downsides, I still like the redeploy option with the DPK. I made some changes to the DPK to make it work better. The changes had two goals:

  1. Don’t include transaction data in the configuration files
  2. Offer per-software redeploy options

To get the first goal, we’ll use Facter to control when we redeploy. Facter offers the ability to override facts by using environment variables. We can set a sane default for our fact (false) but when we want to run a redeploy we can set an environment variable.

For the second goal, we will modify some of the DPK code to read new variables. The global redeploy option will stay, but there will new per-software redeploy options.

My end state for applying CPU patches looks like this:

$env:FACTER_weblogic_redeploy="true"
puppet apply .\site.pp --confdir c:\psft\dpk\puppet

Puppet will run, redeploy only WebLogic from an updated archive file, and bring my system back up. The rest of the software (PS_HOME, Tuxuedo, etc) is left untouched.

DPK Modifications

The next section is a bit code heavy, but hang tight. This all fits together really well.

Custom Facts

We can include custom facts in the DPK to report whatever we want. In this case, we’ll create facts that report false for different software to redeploy. These go in the file modules/pt_role/lib/facter/redeploy.rb.

Facter.add(:jdk_redeploy) do
  setcode do
    'false'
  end
end

Facter.add(:weblogic_redeploy) do
  setcode do
    'false'
  end
end

Facter.add(:tuxedo_redeploy) do
  setcode do
    'false'
  end
end

We now have 3 custom facts:

  • ::jdk_redeploy
  • ::weblogic_redeploy
  • ::tuxedo_redeploy

If we want to set these values to true, we can set these environment variables:

  • FACTER_jdk_redeploy=true
  • FACTER_weblogic_redeploy=true
  • FACTER_tuxedo_redeploy=true

That’s all it takes to override these facts. (This isn’t true of all facts provided by Facter, but for our custom facts this works.)

YAML Configuration

Next, we will read these facts into our configuration. Include this section at the root of your psft_customizations.yaml file.

jdk_redeploy:           "%{::jdk_redeploy}"
weblogic_redeploy:      "%{::weblogic_redeploy}"
tuxedo_redeploy:        "%{::tuxedo_redeploy}"

Puppet Changes

Now the changes get a little more involved. First, we need to read our new variables into the pt_tools_deployment.pp manifest. We’ll set false as the default value in case we forget to add the values to our psft_customizations.yaml. Then, we need to update the call to ::pt_setup::tools_deployment so our new configuration options are passed to the class. (It’s a good practice to only do the hiera lookups in the pt_profile manifests and not in the pt_setup manifests.) These changes are made in modules/pt_profile/manifests/pt_tools_deployment.pp.

  $jdk_redeploy           = hiera('jdk_redeploy', false)
  $weblogic_redeploy      = hiera('weblogic_redeploy', false)
  $tuxedo_redeploy        = hiera('tuxedo_redeploy', false)


  class { '::pt_setup::tools_deployment':
    ensure                 => $ensure,
    deploy_pshome_only     => $deploy_pshome_only,
    db_type                => $db_type,
    pshome_location        => $pshome_location,
    pshome_remove          => $pshome_remove,
    inventory_location     => $inventory_location,
    oracleclient_location  => $oracleclient_location,
    oracleclient_remove    => $oracleclient_remove,
    jdk_location           => $jdk_location,
    jdk_remove             => $jdk_remove,
    jdk_redeploy           => $jdk_redeploy,
    weblogic_location      => $weblogic_location,
    weblogic_remove        => $weblogic_remove,
    weblogic_redeploy      => $weblogic_redeploy,
    tuxedo_location        => $tuxedo_location,
    tuxedo_remove          => $tuxedo_remove,
    tuxedo_redeploy        => $tuxedo_redeploy,
    ohs_location           => $ohs_location,
    ohs_remove             => $ohs_remove,
    redeploy               => $redeploy,
  }

Next, we have to update the API for ::ps_setup::tools_deployment so it knows what values we are passing. These changes are made in modules/pt_setup/manifests/tools_deployment.pp.

class pt_setup::tools_deployment (
  $db_type                = undef,
  $pshome_location        = undef,
  $pshome_remove          = true,
  $inventory_location     = undef,
  $oracleclient_location  = undef,
  $oracleclient_remove    = true,
  $jdk_location           = undef,
  $jdk_remove             = true,
  $jdk_redeploy           = false,
  $weblogic_location      = undef,
  $weblogic_remove        = true,
  $weblogic_redeploy      = false,
  $tuxedo_location        = undef,
  $tuxedo_remove          = true,
  $tuxedo_redeploy        = false,
  $ohs_location           = undef,
  $ohs_remove             = true,
  $redeploy               = false,
)

Last, we have to add some logic to handle our new redeploy options. We’ll keep the main redeploy option, but for each software we’ll evaluate the software-specific redeploy option.

    if ($redeploy == true) or ($jdk_redeploy == true) {
      $java_redeploy = true
    } else {
      $java_redeploy = false
    }
    pt_deploy_jdk { $jdk_tag:
      ensure            => $ensure,
      deploy_user       => $tools_install_user,
      deploy_user_group => $tools_install_group,
      archive_file      => $jdk_archive_file,
      deploy_location   => $jdk_location,
      redeploy          => $java_redeploy,
      remove            => $jdk_remove,
      patch_list        => $jdk_patches_list,
    }

That’s the end of the changes. It’s a little more invasive than I’d like, but the end result is worth it. We can now tell Puppet to do software-specific redeploys. This let’s us deploy new versions of the software by updating the archive file, setting an environment variable, and then running puppet apply.

Improving Windows Services from the DPK

A common theme we write about on the blog is how to make the DPK work with multiple environments on the same machine. It’s common to run a DEV and TST on the same server. The DPK can build those environments, but there are a few changes to make the setup run well. On Windows, the services the DPK creates makes an assumption that breaks when we run multiple environments.

When starting a domain via Windows services, the service assumes that the environment variables are set for that environment. If you create your DEV environment via the DPK, that’s a good assumption. But, if you create a TST environment next, the environment variables are set to TST. When you attempt to start the DEV domain via Windows services, the domain start will fail.

To resolve this, we can improve the Ruby script that starts our domains. Under the ps_cfg_home\appserv\DOMAIN folder, there are Ruby scripts that are called by the Windows service. For the app server, it’s appserver_win_service.rb. These scripts will look for the PS_CFG_HOME environment variable and start the domains it finds under that home. We can add a line in the file to point to the correct PS_CFG_HOME location like this:

ENV["PS_CFG_HOME"]=c:\psft\cfg\DEV

While we can modify the file directly, the DPK way of handling this is to update the template in the DPK. Then, whenever we rebuild our domains the code change is automatically included.

The Ruby scripts to start/stop domains are templates in the DPK. The templates are stored under peoplesoft_base\dpk\puppet\modules\pt_config\files\pt_appserver\appserver_win_service.erb (replace pt_appserver with pt_prcs or pt_pia for the batch and PIA services.)

To make the environment variables we add dynamic, we can reference variables that exist in the Ruby environment that calls the ERB template. In the program appserver_domain_boot.rb, the variables ps_home and ps_cfg_home are set. We will use those variables to build our environment variables.

ENV["PS_HOME"] = "<%= ps_home %>"
ENV["PS_CFG_HOME"] = "<%= ps_cfg_home %>"
system("<%= ps_home %>/appserv/psadmin -c start -d <%= domain_name %>")

The <%= %> tags will output the value of that command or variable. So in our case, we are outputting the string value of ps_cfg_home.

The result of this file will look like this:

ENV["PS_HOME"] = "c:\\psft\\pt\ps_home8.56.08"
ENV["PS_CFG_HOME"] = "c:\\psft\\cfg\\DEV"
system("c:\\psft\\pt\ps_home8.56.08\\appserv\\psadmin -c start -d DEV")

When we run Puppet the next time, our Windows service will have it’s environment variables set before starting or stopping a domain.

#123 – ps_patch


This week on the podcast Dan and Kyle discuss Colton Fischer’s Web Profile discovery, Kyle shares his CPU Patching process and how he automated the process then Dan discusses how he resolved a PIA domain bug.

Show Notes

#121 – Under Review


That’s right, its another Change Assistant episode. Dan shares some updates to Change Assistant in 8.56, some 8.57 speculation, and applying PeopleTools Upgrades headlessly. Kyle shares an ACM discovery, and the PeopleTools Idea Space updates.

Show Notes

#116 – ps-terraform


This week on the podcast, Dan shares a new project called ps-terraform to help you build PeopleSoft Images on AWS. Kyle tries to resolve a Friday Night Ghost issue and Dan follows up on his love for Continuous Integration and the DPK.

Show Notes

  • Elasticsearch DPK Install Methods @ 1:00
  • GitLab Continuous Integration Follow-up @ 5:30
  • Friday Night Ghost @ 19:00
  • Ps-terraform @ 31:00

#113 – Rundeck and Bolt

This week on the podcast, Kyle and Dan discuss Bolt, a new tool from Puppet to run commands and scripts remotely on servers. Dan also talks about setting up a new Rundeck server and using Bolt with Rundeck.

Show Notes

#105 – Agile PeopleSoft

This week on the podcast, Charlie Sinks joins us to talk about the Upper Midwest Regional User Group. We talk about using Agile with PeopleSoft, our experiences with Elasticearch, the Idea Space for PeopleTools, and using Git with PeopleSoft.

Show Notes

  • Chatbots Demo @ 3:30
  • Agile and PeopleSoft @ 11:00
    • SCRUM
    • Kanban
    • SAFE
  • Elasticsearch Experiences @ 33:45
  • PeopleTools 8.56 Updates @ 43:30
  • Testing Effort for Certifications @ 45:30
    • AIX/Solaris: pspuppet.sh is used to prepare the environment. Need to invoke if the bootstrap or puppet failed and you need to re-run
  • Idea Space @ 50:15
  • Advanced PS Admin Talk @ 56:30
  • Orchestration @ 61:00
  • Using Git @ 67:30

#97 – Dozens of Us

This week on the podcast, Dan and Kyle discuss strategies to organize and manage Event Mapping code, writing code that other people will use, trying to work with Push Notifications and custom ACM modules. Kyle ends the podcast with a funny story about searching for images.
 

Show Notes

 

Using Puppet Environments with the DPK

Since the Deployment Packages were released with PeopleTools 8.55, one of my criticisms has been that the DPK is a bit of a sledgehammer. If you define multiple PeopleSoft environments on a server and you want to configure one web server, ALL the domains that the DPK knows about are shut down.

Puppet has an Environments feature that lets you segregate your code and data. While the DPK does not support Puppet Environments out of the box, we can use them to make the DPK less of a sledgehammer when managing our domains. (There is still some sledgehammering going on, so go vote for this idea).

While environments let you separate the modules, manifests and data folder, in this post we’ll separate just the data folder. This will let us share a common set of code (the manifests and modules folders) but the configuration of each domain will be different.

If you want to extend this to the modules and manifests folder, copy those into the environment folders with the environment-specific changes. This is useful for testing new code changes or if you want an environment to use a different DPK Role in the site.pp file.

Create Environment Folders

  1. Make a new dev and tst folders under c:\programdata\puppetlabs\puppet\etc\environments

You can have multiple environments under this folder – as many as you want. A strategy that I’m testing is using the database name as the environment name. For this post, I’ll stick with dev and tst

  1. Copy your YAML files from puppet\etc\data to puppet\etc\environments\dev\data and puppet\etc\ environments\tst\data.

Configure Puppet Environment

Under the puppet\etc folder, add (or modify) the puppet.conf file to look like this:

[main]
environment=production
parser=future
environmentpath=c:\programdata\puppetlabs\puppet\etc\environments
hiera_config=c:\programdata\puppetlabs\hiera\etc\hiera.yaml
basemodulepath=c:\programdata\puppetlabs\puppet\etc\modules

This file tells Puppet where to look for your environments, your Hiera configuration, your default module location, and the default Puppet Environment.

Last, we’ll modify the hiera.yaml file in c:\programdata\puppetlabs\hiera\etc to include environments:

---
:backends:
  - yaml

:hierarchy:
  - "environments/%{::environment}/data/psft_customizations"
  - "environments/%{::environment}/data/psft_configuration"
  - "environments/%{::environment}/data/psft_deployment"
  - "environments/%{::environment}/data/psft_unix_system"
  - "environments/%{::environment}/data/defaults"

:yaml:
  :datadir: c:\programdata\puppetlabs\puppet\etc

If you want to share some of the files, like the defaults.yaml or the psft_unix_system.yaml file, you could keep those under the main puppet\etc\data folder. Your hiera.yaml file would look like this:

---
:backends:
  - yaml

:hierarchy:
  - "environments/%{::environment}/data/psft_customizations"
  - "environments/%{::environment}/data/psft_configuration"
  - "environments/%{::environment}/data/psft_deployment"
  - data/psft_unix_system
  - data/defaults

:yaml:
  :datadir: c:\programdata\puppetlabs\puppet\etc

Test the Environments

Once our Puppet changes are complete we can test some builds. When we run puppet apply, we’ll add an additional paratemer: the environment. To build my dev environment domains, I’ll use this procedure:

cd c:\programdata\puppetlabs\puppet\etc\manifests
puppet apply .\site.pp --environment=dev --debug

Once the dev domains are built and running, you can kick off the tst build with:

puppet apply .\site.pp --environment=tst --debug

As the tst environment is building, your dev domains should stay up and not be affected by the Puppet run. If they are affected, you may have some YAML changes that need to be made. Make sure your configuration’s between the environment don’t overlap (e.g, same PS_CFG_HOME and domain names).