Convert the DPK to use Hiera Hash Merging

The way PeopleSoft delivers Puppet and the Hiera backend, is that everything you define in psft_customizations.yaml overrides configuration defined elsewhere. This is a useful setup when getting started with the DPK and Puppet. But when using YAML files to manage your configuration across multiple servers, you’ll quickly find that you are re-entering the same configuration in many files.

Hiera, the tool Puppet uses to read YAML files, has multiple ways to look up data. First, let’s cover what a YAML hash is. A hash is a key-value structure used in the DPK to store configuration. For example, this is the hash for PS_HOME information:

  db_type:    "%{hiera('db_platform')}"
  unicode_db: "%{hiera('unicode_db')}"
  location:   "%{hiera('ps_home_location')}"
  remove:     true

The main hash key is ps_home, and its value is all the configuration below it. The next level down has 4 keys with 4 corresponding values. The appserver_domain_list hash is a large one that contains all the configuration for one or more app server domains.

Under the delivered setup, if you want to change a value for a domain you need to copy the entire appserver_domain_list hash into your psft_customizations.yaml file and make the change. With Hiera hashing, you could define your domains in a file named appservers.yaml and any specific server changes can be defined in hostname.yaml. For example, the hostname.yaml file could contain this hash to override a configuration:

      SERVER_EVENTS: "Yes"
      DOMAIN_GW:     "Yes"

This provides far more flexibility when working with YAML files, but it does introduce some complexity. If you want to give this a try, here is how you can convert the current DPK to use Hiera hasing.


I used VisualStudio Code to do the find/replace. Open up the etc\modules directory and do these against the modules\pt_profile folder:

  • Find: hiera('tns_admin_list
  • Replace: hiera_hash('tns_admin_list

I repeated this step for the following lookups.

  • tns_admin_list
  • appserver_domain_list
  • prcs_domain_list
  • pia_domain_list

You don’t want to replace all the lookups – that will cause errors. But, you can replace additional lookups if you want. Anything that is a hash in YAML files can use the hiera_hash() lookup function. If you wanted to make the ps_home: key support hash merging, you could replace hiera('ps_home with hiera_hash('ps_home.

Change the Hiera Merge Behavior

By default, Hiera will look at the top-level keys of a hash and not merge the underlying settings. Hiera hashing will merge all the values inside the hash. This means you can you define a hash with default settings in a common file (e.g, default app server settings). Then you can specify server or application specific settings in a YAML file for that domain or server.

To enable the hash merging, open the hiera.yaml file under c:\programdata\puppetlabs\hiera\etc.

Add this line to the file:

:merger_behavior: deeper

Hiera Lookup Order

With Hiera hash merging, we can utilize more than the psft_customizations.yaml file to manage our configuration. We can use multiple YAML files to control our configuration. For example, we could have:

  • [hostname].yaml
  • dev.yaml
  • hr.yaml
  • common.yaml

So, this setup would let us define common configuration that is shared across all applications in the common.yaml. Next, we could define anything related to servers that run HR applications in the hr.yaml. For any settings that are specific to the Development region, we can add them into dev.yaml. Last, for anything that is specific to the server, we can add into the [hostname].yaml file. This setup would let you re-use the common, hr, and dev YAML files across multiple servers, and anything specific to the server would be defined in [hostname].yaml.

In the hiera.yaml file, we can define this setup like this:

  - "%{::hostname}"
  - dev
  - hr
  - common

Test Hiera Hashing

On the command line, you can use the hiera utility to test lookups with Hiera. To do a normal Hiera lookup, use

hiera appserver_domain_list

To test a hiera hash lookup, use

hiera --hash appserver_domain_list

If you have multiple YAML files with the appserver_domain_list hash, the first option will only show you the results from the top of the list. The second test should show you a merged appserver_domain_list hash.

#92 – Advanced Puppet with the DPK

This week on the podcast, Dan dives into some advanced Puppet configuration to use with the DPK. Dan and Kyle discuss Hiera hash merging, using Puppet environments, and using the ACM with the DPK. Kyle talks about a bug with the Interaction Hub and My Favorites and some odd finds in PeopleBooks.

Show Notes

#89 – Gotchas

This week Kyle and Dan discuss UI improvements from Sasank and Dan’s new Fluid Stylesheets, using Git and GitLab to manage DPK files, managing Favorites with Unified Navigation and some “Gotchas” Kyle found during his 8.55 upgrade project.

Show Notes

#84 – No Agenda

This week on the podcast Dan and Kyle talk with no agenda. We discuss our week and talk about Portal, PeopleTools Bugs, Fluid, Git, Deployment Packages, Puppet, OneNote, Kanban Boards and more.

Show Notes

  • Portal Projects @ 1:45
  • PeopleTools Bugs and Puppet @ 10:45
  • Community @ 21:00
  • Job changes @ 23:30
  • Spreading the Fluid Excitement @ 25:00
  • Selecting Fluid Nav Images @ 28:00
  • The Git “Trojan Horse” @ 31:30
  • Git Hooks, Puppet Server and getting crazy @ 39:30
  • Changes to the DPK? @ 47:00
  • Kanban Boards @ 57:00
  • Svchost.exe @ 58:30
  • OneNote Templates @ 60:30
  • Kanban Boards and Task Management @ 71:00

#83 – DPK – What We’d Do Different Next Time

This week on the podcast Dan and Kyle discuss changes they would make if they started working on the DPK for the first again. They also discuss setting up Elasticsearch clusters, monitoring Tuxedo queues with Elasticsearch and an annoying Bash on Windows bug.

Show Notes

  • Bash on Windows VPN Bug @ 3:15
  • 9.2 Upgrade Process @ 10:00
  • Monitoring Tuxedo Queueing in Elasticsearch @ 13:30
  • Tuxbeat Project @ 17:00
  • Elasticsearch DPK @ 21:45
  • Elasticsearch Clustering @ 27:00
  • Puppet on Linux @ 32:00
  • DPK Extract Only @ 34:30
  • What we’d do differently with the DPK @ 37:30
    • Co-locating Regions
    • Admin-only Test Servers
    • Config Homes
    • Use Git for managing code and YAML changes
    • Integrate all Puppet code into Classes and Modules
    • DPK Course
    • Use ACM via DPK
    • Go Vanilla

App Capacity Visualization

#71 – DevOps and Ansible w/ Jason Gilfoil and Eric Bolinger

This week we podcast from the Alliance 2017 Conference in Las Vegas. Jason Gilfoil and Eric Bolinger join us to talk about DevOps and Ansible with PeopleSoft. We talk about application orchestration, mixing Ansible and Puppet, customizing the DPK and more.

Show Notes

  • Introducing Jason Gilfoil @ 1:30
  • Introducing Eric Bolinger @ 2:45
  • What is Ansible? @ 3:30
  • What is Orchestration? @ 8:00
  • Differences between Puppet and Ansible @ 12:00
  • Puppet Master, Hiera Hash and the DPK @ 15:15
  • Managing infrastructure code with Git @ 20:45
  • Adjusting to a DevOps culture @ 27:00
  • Getting automation started in your organization @ 30:30
  • Calculating Time Saving for Automation @ 35:30
  • Choosing an automation tool @ 41:30
  • Docker @ 43:00
  • Personal Development Environments @ 45:45
  • Starting to think “cloud” @ 50:00

#61 – Jolt Failover

This week on the podcast, Dan and Kyle launch a new course about Deployment Packages. Dan tests out a new text editor and discovers you can run OPatch on MOS. Kyle digs into Jolt Failover options with the IB and brainstorms some great configuration ideas.

Show Notes

#59 – Security Deployment

This week on the podcast Dan and Kyle talk about the new CFO tool, applying CPU patches with the DPK, and how the DPK could improve with newer versions of Puppet. Then Dan digs into the new Security Deployment tool and how you can use it improve security migrations.

Show Notes

Encrypt psft_customizations.yaml Passwords

In the psft_customizations.yaml file we store configuration information for a server, including passwords. There is a project, hiera-eyaml, that supports encrypting and decrypting sensitive data in Hiera YAML files. Out of the box, the Windows-based DPK doesn’t work with hiera-eyaml. For Linux DPK, check out 2188771.1 – there is better support in the Linux DPK for hiera-eyaml.

In this post, we’ll walk through the steps to get hiera-eyaml working on Windows and how to encrypt data in the psft_customizations.yaml file.

Update RubyGems

The version of Ruby, and RubyGems, that ships with the DPK can’t install new Gems. The RubyGems version doesn’t support trust the site’s SSL certificate. To fix that, download the root certificate and tell RubyGems to trust it.

  1. Download the newer SSL certificate.
  2. Save the file as RubyGemsRootCA.pem
  3. Copy the new certificate to C:\Program Files\Puppet Labs\Puppet\sys\ruby\lib\ruby\2.0.0\rubygems\ssl_certs

Copying the new certificate to ssl_certs will tell RubyGems to trust any certificate signed by it. Now we can use RubyGems to install hiera-eyaml on the server.

Install hiera-eyaml

When Puppet is installed, it includes Ruby and RubyGems binaries because Puppet is written in Ruby. We’ll use the gem utility to install the hiera-eyaml RubyGem. First, we should update PATH to include Puppet’s Ruby binaries:

  1. $env:PATH += ";C:\Program Files\Puppet Labs\Puppet\sys\ruby\bin"
  2. gem install hiera-eyaml

RubyGems will install any dependencies and report the progress.

Fetching: trollop-2.1.2.gem (100%)
Successfully installed trollop-2.1.2
Fetching: highline-1.6.21.gem (100%)
Successfully installed highline-1.6.21
Fetching: hiera-eyaml-2.1.0.gem (100%)
Successfully installed hiera-eyaml-2.1.0
Parsing documentation for trollop-2.1.2
Installing ri documentation for trollop-2.1.2
Parsing documentation for highline-1.6.21
Installing ri documentation for highline-1.6.21
Parsing documentation for hiera-eyaml-2.1.0
Installing ri documentation for hiera-eyaml-2.1.0
3 gems installed


Hiera-eyaml uses it’s own Public and Private keys to encrypt and decrypt data. If you have inspected the puppet\ssl directory, you will see folders for public and private keys. These keys are used by Puppet to communicate with a Puppet Server. We use different keys for encrypting data in psft_customizations.yaml.

The keys should be created in the folder C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\. To ensure the keys are created in the correct location, Hiera-eyaml and Hiera know where they are, we’ll create a configuration file for Hiera-eyaml.

  1. Create eyaml.yaml under C:\ProgramData\PuppetLabs\hiera\etc and add these values:

    pkcs7_private_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\private_key.pkcs7.pem  
    pkcs7_public_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\public_key.pkcs7.pem
  2. Set the EYAML_CONFIG environment variable:

  3. Create new encryption keys for Hiera-eyaml to use:

    eyaml createkeys

Keep these new keys safe and locked down; they decrypt your passwords!

Encrypt Passwords

Now that we have installed Hiera-eyaml and created keys, let’s do a quick test to make sure we can encrypt passwords. This test will encrypt the text “VP1”:

eyaml encrypt -s VP1

The output will look similar to this:

string: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEALsKtTfAXyHyE/k5r2U2ZZU98SqaQ5/ukfNR/FkOt9bNhoZ1EomqmqIc/06l7Tk5W4BYJA0mXV6ykLgOHYTAJbVPM8gXBuHsw1jh+/VC0er7evlzqtf7UjIvu3rTo+0LUm2X3imjbWHGhyrs2bxm0L1qpC2atlTSzEYrSc6OxkTpZA19Y8iEJxFb+F0fGwsQ3SRVJD1J3Jwf0hAsHN/SXX/p2ywn5qz2BnlJl4wa7ragYv4aVBGbGF3ThvYMCTzNiFHtyHdCFvPX9i/t0fpDUJY76ndAl/T4q/Stopnq6Gm9vLJH5EC6KMUQZzb0ssDHriojQgUH7uFt8/Wn9vFeTQTA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBFcUOesdHoJgYi5PnXGmkAgBDbEJsr/tDXbDpJu7+xz9uL]


block: >

Hiera-eyaml gives two options for output: string and block. For psft_customizations.yaml I’m using the string output. It’s cleaner and easier to insert into the file. We can request string output only and assign a label to the encrypted password:

eyaml encrypt -s VP1 -o string -q -l db_user_pwd

The output should look like this:

db_user_pwd: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAV2y+yriBfuFlXspBIzZ8eBEOow7FU7mcwYL1HCpHd+XrwIliMTgDj+4X47XXQ3bce4WRvaezHUNahJQF4OZrwlGdCgXYeFG4dYvMEg/75T0704I2+y/XmLpI3Y5swd3L9LnHfxpAm6x8AJpf2yybSP4rsD1IxZgrpjy1CjFe3GuRW9ZcNFkNq5WofRweoX4C9QgNp1bmXQnJym+ZnVe1y7vQ9iEY336vF2FH3wJNgqRIy+74RWj9F+OaAg78meSxM0eM7jm4fLa32cMmOLzfU/FGFhLFcQJ2FaAa5/SWmBSgtwDUXsGaLcSa0R2nfQZrbRWmlP+s1WYL9MzkLFTEoDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBDMAfrBMvJ+HRA+iL4zyQppgBBcHlLe8hUl6JD1jFXH/N22]

We can pass the output of this command into the clipboard and then paste the line directly into psft_customizations.yaml:

eyaml encrypt -s VP1 -o string -q -l db_user_pwd | set-clipboard

set-clipboard requires WMF 5.0

Edit YAML Files

Encrypting passwords on the command line is great, but what if you want to edit all of the passwords in your psft_customizations.yaml file at once? Hiera-eyaml has an edit command that will decrypt the passwords in psft_customizations.yaml and open the file in a text editor for you. First, we need to set the EDITOR environment variable:

  1. $env:EDITOR="notepad.exe"
  2. eyaml edit .\psft_customizations.yaml

Notepad will open the psft_customizations.yaml file. At the top of the file, you will see a large comment block explaining how to add and edit passwords. (The comment block will go away when you close Notepad.)

Add New Passwords

To add a new password, you wrap the plain text password inside the brackets in this syntax: DEC::PKCS7[plaintextpassword]! For example,

db_user_pwd:     DEC::PKCS7[VP1]!

If you save and close the file, and open psft_customizations.yaml directly in Notepad, you will see the db_user_pwd: password is encrypted.

db_user_pwd:  ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAdGWx7WeuGw3lULsdDANpYotX66B1wzO3U9H47RLJA+s4cIVg5z2JtzTp+uHOp9L9SdcNyzsvo6+uPY29DxMsaIUv9Dfa5LWKv+GZypH4myJYxbNfhtRE5TcLWxxwTSji9WYxDyFu8FFJGIkdNcEzN4svG6CknDhmA/od/NPanQg+xWbjP2qJkiOMi2fDwPJd11dev7Qm4NcwkZzdcsMBpkSgL3eL2dZ/BzdJndWrsGlYfUAy0TLxJD9a4aBCiwYoWWmmS4smnmtmti0R1DPEs8BpAl5L76JItMUwzRsnmu5IZ8odxn8rQZQNJaOVk/oScp4SRIgCh5+tYp7FMvgM/jA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCqrQ+GokeF23Of2odDkv5JgBBulnH4XkLOrQBEy+fa7cMr]

Excellent – we have encrypted passwords!

Edit Passwords

The next step is to edit existing encrypted passwords in the psft_customizations.yaml file. The eyaml edit command will open the file and decrypt passwords. The password syntax will be slightly different – it will have a number assigned to the password: db_user_pwd: DEC(1)::PKCS7[VP1]!

The (1) is used internally by Hiera-eyaml, so don’t change it. But you can change the password inside the square brackets. After changing the password, save and close the file and your updated passwords will be encrypted.

Enable eyaml with the DPK

When we push psft_customizations.yaml out to servers, we also need to ensure each server has the keys used to encrypt the passwords, and also knows about Hiera-eyaml. First, if you are using the encrypted passwords on more than one server, copy the puppet\etc\secure\keys folder to each server.

Next, Hiera needs to know that we are using Hiera-eyaml. In C:\ProgramData\PuppetLabs\hiera\etc\hiera.yaml, enable eyaml as a back-end format by adding - eyaml to the ;backends: section:

    - yaml
    - eyaml

Verify that the :eyaml: section is at the bottom of hiera.yaml. Change the paths to the Public and Private keys. If you followed the steps above and created them in puppet\etc\secure\keys, the paths will look like this:

    :datadir: C:\ProgramData\PuppetLabs\puppet\etc\data
    :extension: yaml

    :pkcs7_private_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\private_key.pkcs7.pem
    :pkcs7_public_key:  C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\public_key.pkcs7.pem

Save hiera.yaml and let’s test our configuration.

Testing Hiera-eyaml

To test Hiera-eyaml and Puppet working together, we’ll encrypt the a password in psft_customizations.yaml and update UserPswd= value in psappsrv.cfg.

  1. Open psft_customizations.yaml with eyaml edit and add the line:

    db_user_pwd: DEC::PKCS7[VP1]!
  2. Save and close psft_customizations.yaml.

  3. Save this code below as pwd.pp and save it to puppet\etc\manifests. Change the $configFile path to point to your psappsrv.cfg file.

    $configFile = 'C:/Users/vagrant/psft/pt/8.55/appserv/APPDOM/psappsrv.cfg' 
    ini_setting { "eyaml_test": 
        ensure => present, 
        path => $configFile, 
        section => 'Startup', 
        setting => 'UserPswd', 
        value => hiera('db_user_pwd'), 
  4. Change directories to puppet\etc\manifests.

  5. Run puppet apply .\pwd.pp --trace --debug

  6. After the run is done, open your psappsrv.cfg file. You should see UserPswd=VP1 in the file.

If the test above worked, you’re all set to use Hiera-eyaml with the DPK and Puppet. Once Hiera knows about Hiera-eyaml, any data in Hiera can be encrypted. Happy encrypting!

Enable Tuxedo Domain Features with the DPK

Since the DPK was released, there has a been a bug (for Windows) that is quite annoying. In the psft_customizations.yaml file, the feature_settings: section is supposed to turn Tuxedo domain features on or off.

  PUBSUB:           "Yes"
  QUICKSRV:         "No"
  QUERYSRV:         "Yes"
  JOLT:             "Yes"

On Windows, these settings were ignored and the default values were used when a domain was created or reconfigured. Thanks to Dale Haman from the Community, we have a fix for the bug. The issue is this:

The DPK (Puppet) takes the feature_settings: section in the psft_customizations.yaml file and combines it into a string. The string is passed into psadmin with the -u parameter. The code that creates this string used an incorrect separator for each feature, so psadmin would ignore the entire string.

To fix this, you can edit one file under the puppet/etc/modules folder:


if Facter.value(:osfamily) == 'windows'
  # feature_settings_separator = '#'
  # For Windows, we need to use the slash
  feature_settings_separator = '/'
  feature_settings_separator = '%'

If you re-run puppet apply .\site.pp, your app server and process scheduler domains features should match what you defined in psft_customizations.yaml.

Process Scheduler Features

After testing this on a process scheduler, I noticed my scheduler had 5 features listed in psadmin but only two were configured in my psft_customizations.yaml file: MSTRSRV and APPENG. I wanted to configure the other features in the domain, so I added them to my psft_customizations.yaml file:

  MSTRSRV:          "Yes"
  APPENG:           "No"
  PPM:              "No"
  DOMAIN_GW:        "No"

If you run puppet apply .\site.pp with these parameters in your psft_customizations.yaml file, you’ll get an error saying the Feature List is not valid. The feature_settings: section is compared to an array in the Process Scheduler Puppet Type to make sure you don’t mis-type anything, or try to add a non-existent feature. But in this case, the validation array was incorrect. It was missing the "PPM" feature.


prcs_features =
  # Add PPM because it is a scheduler feature...KIOSK is not.

Comment out the bad line and change "KIOSK" to "PPM" in the array. Now, you can configure all 5 process scheduler features from the psft_customizations.yaml file.

Check out the other bug fixes for the DPK too: