The PeopleSoft Administrator Podcast hosted by Dan Iverson and Kyle Benson
This week on the podcast, Dan gives an update on his Hyper-V work with Vagabond and shares a funny log line he found, and Kyle shares some thoughts on making Hiera and eYAML more secure for PeopleTools DPK.
This week on the podcast, Kyle and Dan announce psadmin.conf 2021, they discuss environment naming conventions, and Dan follows up on the Puppet lookup options from last week.
This week on the podcast, Dan learns about a new way to enable Hiera hashing in the DPK and laments about a DPK bug that still exists, and Kyle and Dan discuss the new Prompt Table Override feature in 8.58.
This week on the podcast, Kyle adds WLST commands to his DPK builds, Dan creates custom Facts with Ruby code and shares his Hiera Hashing configuration for building 8.56 environments.
This week on the podcast, Dan has follow-up on using Hiera with Puppet environments, capturing WebLogic logs in Elasticsearch, and Kyle shares his thoughts on the Solaris “change”. Then Kyle discusses the in depth failover testing and how Unified Navigation behaves when app servers fail.
This week on the podcast, Dan and Kyle discuss strategies to organize and manage Event Mapping code, writing code that other people will use, trying to work with Push Notifications and custom ACM modules. Kyle ends the podcast with a funny story about searching for images.
Since the Deployment Packages were released with PeopleTools 8.55, one of my criticisms has been that the DPK is a bit of a sledgehammer. If you define multiple PeopleSoft environments on a server and you want to configure one web server, ALL the domains that the DPK knows about are shut down.
Puppet has an Environments feature that lets you segregate your code and data. While the DPK does not support Puppet Environments out of the box, we can use them to make the DPK less of a sledgehammer when managing our domains. (There is still some sledgehammering going on, so go vote for this idea).
While environments let you separate the modules, manifests and data folder, in this post we’ll separate just the data folder. This will let us share a common set of code (the manifests and modules folders) but the configuration of each domain will be different.
If you want to extend this to the modules and manifests folder, copy those into the environment folders with the environment-specific changes. This is useful for testing new code changes or if you want an environment to use a different DPK Role in the site.pp file.
dev and tst folders under c:\programdata\puppetlabs\puppet\etc\environmentsYou can have multiple environments under this folder – as many as you want. A strategy that I’m testing is using the database name as the environment name. For this post, I’ll stick with dev and tst
puppet\etc\data to puppet\etc\environments\dev\data and puppet\etc\ environments\tst\data.Under the puppet\etc folder, add (or modify) the puppet.conf file to look like this:
[main]
environment=production
parser=future
environmentpath=c:\programdata\puppetlabs\puppet\etc\environments
hiera_config=c:\programdata\puppetlabs\hiera\etc\hiera.yaml
basemodulepath=c:\programdata\puppetlabs\puppet\etc\modules
This file tells Puppet where to look for your environments, your Hiera configuration, your default module location, and the default Puppet Environment.
Last, we’ll modify the hiera.yaml file in c:\programdata\puppetlabs\hiera\etc to include environments:
---
:backends:
- yaml
:hierarchy:
- "environments/%{::environment}/data/psft_customizations"
- "environments/%{::environment}/data/psft_configuration"
- "environments/%{::environment}/data/psft_deployment"
- "environments/%{::environment}/data/psft_unix_system"
- "environments/%{::environment}/data/defaults"
:yaml:
:datadir: c:\programdata\puppetlabs\puppet\etc
If you want to share some of the files, like the defaults.yaml or the psft_unix_system.yaml file, you could keep those under the main puppet\etc\data folder. Your hiera.yaml file would look like this:
---
:backends:
- yaml
:hierarchy:
- "environments/%{::environment}/data/psft_customizations"
- "environments/%{::environment}/data/psft_configuration"
- "environments/%{::environment}/data/psft_deployment"
- data/psft_unix_system
- data/defaults
:yaml:
:datadir: c:\programdata\puppetlabs\puppet\etc
Once our Puppet changes are complete we can test some builds. When we run puppet apply, we’ll add an additional paratemer: the environment. To build my dev environment domains, I’ll use this procedure:
cd c:\programdata\puppetlabs\puppet\etc\manifests
puppet apply .\site.pp --environment=dev --debug
Once the dev domains are built and running, you can kick off the tst build with:
puppet apply .\site.pp --environment=tst --debug
As the tst environment is building, your dev domains should stay up and not be affected by the Puppet run. If they are affected, you may have some YAML changes that need to be made. Make sure your configuration’s between the environment don’t overlap (e.g, same PS_CFG_HOME and domain names).
The way PeopleSoft delivers Puppet and the Hiera backend, is that everything you define in psft_customizations.yaml overrides configuration defined elsewhere. This is a useful setup when getting started with the DPK and Puppet. But when using YAML files to manage your configuration across multiple servers, you’ll quickly find that you are re-entering the same configuration in many files.
Hiera, the tool Puppet uses to read YAML files, has multiple ways to look up data. First, let’s cover what a YAML hash is. A hash is a key-value structure used in the DPK to store configuration. For example, this is the hash for PS_HOME information:
ps_home:
db_type: "%{hiera('db_platform')}"
unicode_db: "%{hiera('unicode_db')}"
location: "%{hiera('ps_home_location')}"
remove: true
The main hash key is ps_home, and its value is all the configuration below it. The next level down has 4 keys with 4 corresponding values. The appserver_domain_list hash is a large one that contains all the configuration for one or more app server domains.
Under the delivered setup, if you want to change a value for a domain you need to copy the entire appserver_domain_list hash into your psft_customizations.yaml file and make the change. With Hiera hashing, you could define your domains in a file named appservers.yaml and any specific server changes can be defined in hostname.yaml. For example, the hostname.yaml file could contain this hash to override a configuration:
appserver_domain_list:
DEV:
feature_settings:
SERVER_EVENTS: "Yes"
DOMAIN_GW: "Yes"
This provides far more flexibility when working with YAML files, but it does introduce some complexity. If you want to give this a try, here is how you can convert the current DPK to use Hiera hasing.
I used VisualStudio Code to do the find/replace. Open up the etc\modules directory and do these against the modules\pt_profile folder:
hiera('tns_admin_listhiera_hash('tns_admin_listI repeated this step for the following lookups.
tns_admin_listappserver_domain_listprcs_domain_listpia_domain_listYou don’t want to replace all the lookups – that will cause errors. But, you can replace additional lookups if you want. Anything that is a hash in YAML files can use the hiera_hash() lookup function. If you wanted to make the ps_home: key support hash merging, you could replace hiera('ps_home with hiera_hash('ps_home.
By default, Hiera will look at the top-level keys of a hash and not merge the underlying settings. Hiera hashing will merge all the values inside the hash. This means you can you define a hash with default settings in a common file (e.g, default app server settings). Then you can specify server or application specific settings in a YAML file for that domain or server.
To enable the hash merging, open the hiera.yaml file under c:\programdata\puppetlabs\hiera\etc.
Add this line to the file:
:merge_behavior: deeper
With Hiera hash merging, we can utilize more than the psft_customizations.yaml file to manage our configuration. We can use multiple YAML files to control our configuration. For example, we could have:
[hostname].yamldev.yamlhr.yamlcommon.yamlSo, this setup would let us define common configuration that is shared across all applications in the common.yaml. Next, we could define anything related to servers that run HR applications in the hr.yaml. For any settings that are specific to the Development region, we can add them into dev.yaml. Last, for anything that is specific to the server, we can add into the [hostname].yaml file. This setup would let you re-use the common, hr, and dev YAML files across multiple servers, and anything specific to the server would be defined in [hostname].yaml.
In the hiera.yaml file, we can define this setup like this:
:hierarchy:
- "%{::hostname}"
- dev
- hr
- common
On the command line, you can use the hiera utility to test lookups with Hiera. To do a normal Hiera lookup, use
hiera appserver_domain_list
To test a hiera hash lookup, use
hiera --hash appserver_domain_list
If you have multiple YAML files with the appserver_domain_list hash, the first option will only show you the results from the top of the list. The second test should show you a merged appserver_domain_list hash.
In the psft_customizations.yaml file we store configuration information for a server, including passwords. There is a project, hiera-eyaml, that supports encrypting and decrypting sensitive data in Hiera YAML files.
Starting with PeopleTools 8.56, the DPK includes the the eyaml gem and has all of the configuration ready to go. These updated instructions (2/21/2020) will show you how to use eyaml with thepsft_customizations.yaml file. (Below I have left the original post with older instructions in case it helps anyone).
First, we need to set our EDITOR enviornment variable. I like VisualStudio Code, so here is how we set that up:
$env:EDITOR="code --wait"
Next, we need to move into the secure directory in the DPK folder:
cd c:\psft\dpk\puppet\secure
Now we can use eyaml to open our psft_customizations.yaml file and encrypt passwords:
eyaml edit ..\production\data\psft_customizations.yaml
To add a new password, you wrap the plain text password inside the brackets in this syntax: DEC::PKCS7[plaintextpassword]! For example,
db_user_pwd: DEC::PKCS7[VP1]!
If you save and close the file, and open psft_customizations.yaml directly in VS Code, you will see the db_user_pwd: password is encrypted.
db_user_pwd: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAdGWx7WeuGw3lULsdDANpYotX66B1wzO3U9H47RLJA+s4cIVg5z2JtzTp+uHOp9L9SdcNyzsvo6+uPY29DxMsaIUv9Dfa5LWKv+GZypH4myJYxbNfhtRE5TcLWxxwTSji9WYxDyFu8FFJGIkdNcEzN4svG6CknDhmA/od/NPanQg+xWbjP2qJkiOMi2fDwPJd11dev7Qm4NcwkZzdcsMBpkSgL3eL2dZ/BzdJndWrsGlYfUAy0TLxJD9a4aBCiwYoWWmmS4smnmtmti0R1DPEs8BpAl5L76JItMUwzRsnmu5IZ8odxn8rQZQNJaOVk/oScp4SRIgCh5+tYp7FMvgM/jA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCqrQ+GokeF23Of2odDkv5JgBBulnH4XkLOrQBEy+fa7cMr]
Excellent – we have encrypted passwords!
The next step is to edit existing encrypted passwords in the psft_customizations.yaml file. The eyaml edit command will open the file and decrypt passwords. The password syntax will be slightly different – it will have a number assigned to the password: db_user_pwd: DEC(1)::PKCS7[VP1]!
The (1) is used internally by eyaml, so don’t change it. But you can change the password inside the square brackets. After changing the password, save and close the file and your updated passwords will be encrypted.
Thats it! The DPK will know how to decrypt those passwords when you run it, and your configuration files now have encrypted passwords.
[Original Post]
Out of the box, the Windows-based DPK doesn’t work with hiera-eyaml. For Linux DPK, check out 2188771.1 – there is better support in the Linux DPK for hiera-eyaml.
In this post, we’ll walk through the steps to get hiera-eyaml working on Windows and how to encrypt data in the psft_customizations.yaml file.
The version of Ruby, and RubyGems, that ships with the DPK can’t install new Gems. The RubyGems version doesn’t support trust the site’s SSL certificate. To fix that, download the root certificate and tell RubyGems to trust it.
RubyGemsRootCA.pemC:\Program Files\Puppet Labs\Puppet\sys\ruby\lib\ruby\2.0.0\rubygems\ssl_certsCopying the new certificate to ssl_certs will tell RubyGems to trust any certificate signed by it. Now we can use RubyGems to install hiera-eyaml on the server.
When Puppet is installed, it includes Ruby and RubyGems binaries because Puppet is written in Ruby. We’ll use the gem utility to install the hiera-eyaml RubyGem. First, we should update PATH to include Puppet’s Ruby binaries:
$env:PATH += ";C:\Program Files\Puppet Labs\Puppet\sys\ruby\bin"gem install hiera-eyamlRubyGems will install any dependencies and report the progress.
Fetching: trollop-2.1.2.gem (100%)
Successfully installed trollop-2.1.2
Fetching: highline-1.6.21.gem (100%)
Successfully installed highline-1.6.21
Fetching: hiera-eyaml-2.1.0.gem (100%)
Successfully installed hiera-eyaml-2.1.0
Parsing documentation for trollop-2.1.2
Installing ri documentation for trollop-2.1.2
Parsing documentation for highline-1.6.21
Installing ri documentation for highline-1.6.21
Parsing documentation for hiera-eyaml-2.1.0
Installing ri documentation for hiera-eyaml-2.1.0
3 gems installed
Hiera-eyaml uses it’s own Public and Private keys to encrypt and decrypt data. If you have inspected the puppet\ssl directory, you will see folders for public and private keys. These keys are used by Puppet to communicate with a Puppet Server. We use different keys for encrypting data in psft_customizations.yaml.
The keys should be created in the folder C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\. To ensure the keys are created in the correct location, Hiera-eyaml and Hiera know where they are, we’ll create a configuration file for Hiera-eyaml.
Create eyaml.yaml under C:\ProgramData\PuppetLabs\hiera\etc and add these values:
---
pkcs7_private_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\private_key.pkcs7.pem
pkcs7_public_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\public_key.pkcs7.pem
Set the EYAML_CONFIG environment variable:
$env:EYAML_CONFIG="C:\ProgramData\PuppetLabs\hiera\etc\eyaml.yaml"
Create new encryption keys for Hiera-eyaml to use:
eyaml createkeys
Keep these new keys safe and locked down; they decrypt your passwords!
Now that we have installed Hiera-eyaml and created keys, let’s do a quick test to make sure we can encrypt passwords. This test will encrypt the text “VP1”:
eyaml encrypt -s VP1
The output will look similar to this:
string: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEALsKtTfAXyHyE/k5r2U2ZZU98SqaQ5/ukfNR/FkOt9bNhoZ1EomqmqIc/06l7Tk5W4BYJA0mXV6ykLgOHYTAJbVPM8gXBuHsw1jh+/VC0er7evlzqtf7UjIvu3rTo+0LUm2X3imjbWHGhyrs2bxm0L1qpC2atlTSzEYrSc6OxkTpZA19Y8iEJxFb+F0fGwsQ3SRVJD1J3Jwf0hAsHN/SXX/p2ywn5qz2BnlJl4wa7ragYv4aVBGbGF3ThvYMCTzNiFHtyHdCFvPX9i/t0fpDUJY76ndAl/T4q/Stopnq6Gm9vLJH5EC6KMUQZzb0ssDHriojQgUH7uFt8/Wn9vFeTQTA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBFcUOesdHoJgYi5PnXGmkAgBDbEJsr/tDXbDpJu7+xz9uL]
OR
block: >
ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
DQYJKoZIhvcNAQEBBQAEggEALsKtTfAXyHyE/k5r2U2ZZU98SqaQ5/ukfNR/
FkOt9bNhoZ1EomqmqIc/06l7Tk5W4BYJA0mXV6ykLgOHYTAJbVPM8gXBuHsw
1jh+/VC0er7evlzqtf7UjIvu3rTo+0LUm2X3imjbWHGhyrs2bxm0L1qpC2at
lTSzEYrSc6OxkTpZA19Y8iEJxFb+F0fGwsQ3SRVJD1J3Jwf0hAsHN/SXX/p2
ywn5qz2BnlJl4wa7ragYv4aVBGbGF3ThvYMCTzNiFHtyHdCFvPX9i/t0fpDU
JY76ndAl/T4q/Stopnq6Gm9vLJH5EC6KMUQZzb0ssDHriojQgUH7uFt8/Wn9
vFeTQTA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBFcUOesdHoJgYi5PnX
GmkAgBDbEJsr/tDXbDpJu7+xz9uL]
Hiera-eyaml gives two options for output: string and block. For psft_customizations.yaml I’m using the string output. It’s cleaner and easier to insert into the file. We can request string output only and assign a label to the encrypted password:
eyaml encrypt -s VP1 -o string -q -l db_user_pwd
The output should look like this:
db_user_pwd: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAV2y+yriBfuFlXspBIzZ8eBEOow7FU7mcwYL1HCpHd+XrwIliMTgDj+4X47XXQ3bce4WRvaezHUNahJQF4OZrwlGdCgXYeFG4dYvMEg/75T0704I2+y/XmLpI3Y5swd3L9LnHfxpAm6x8AJpf2yybSP4rsD1IxZgrpjy1CjFe3GuRW9ZcNFkNq5WofRweoX4C9QgNp1bmXQnJym+ZnVe1y7vQ9iEY336vF2FH3wJNgqRIy+74RWj9F+OaAg78meSxM0eM7jm4fLa32cMmOLzfU/FGFhLFcQJ2FaAa5/SWmBSgtwDUXsGaLcSa0R2nfQZrbRWmlP+s1WYL9MzkLFTEoDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBDMAfrBMvJ+HRA+iL4zyQppgBBcHlLe8hUl6JD1jFXH/N22]
We can pass the output of this command into the clipboard and then paste the line directly into psft_customizations.yaml:
eyaml encrypt -s VP1 -o string -q -l db_user_pwd | set-clipboard
set-clipboardrequires WMF 5.0
Encrypting passwords on the command line is great, but what if you want to edit all of the passwords in your psft_customizations.yaml file at once? Hiera-eyaml has an edit command that will decrypt the passwords in psft_customizations.yaml and open the file in a text editor for you. First, we need to set the EDITOR environment variable:
$env:EDITOR="notepad.exe"eyaml edit .\psft_customizations.yamlNotepad will open the psft_customizations.yaml file. At the top of the file, you will see a large comment block explaining how to add and edit passwords. (The comment block will go away when you close Notepad.)
To add a new password, you wrap the plain text password inside the brackets in this syntax: DEC::PKCS7[plaintextpassword]! For example,
db_user_pwd: DEC::PKCS7[VP1]!
If you save and close the file, and open psft_customizations.yaml directly in Notepad, you will see the db_user_pwd: password is encrypted.
db_user_pwd: ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAdGWx7WeuGw3lULsdDANpYotX66B1wzO3U9H47RLJA+s4cIVg5z2JtzTp+uHOp9L9SdcNyzsvo6+uPY29DxMsaIUv9Dfa5LWKv+GZypH4myJYxbNfhtRE5TcLWxxwTSji9WYxDyFu8FFJGIkdNcEzN4svG6CknDhmA/od/NPanQg+xWbjP2qJkiOMi2fDwPJd11dev7Qm4NcwkZzdcsMBpkSgL3eL2dZ/BzdJndWrsGlYfUAy0TLxJD9a4aBCiwYoWWmmS4smnmtmti0R1DPEs8BpAl5L76JItMUwzRsnmu5IZ8odxn8rQZQNJaOVk/oScp4SRIgCh5+tYp7FMvgM/jA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBCqrQ+GokeF23Of2odDkv5JgBBulnH4XkLOrQBEy+fa7cMr]
Excellent – we have encrypted passwords!
The next step is to edit existing encrypted passwords in the psft_customizations.yaml file. The eyaml edit command will open the file and decrypt passwords. The password syntax will be slightly different – it will have a number assigned to the password: db_user_pwd: DEC(1)::PKCS7[VP1]!
The (1) is used internally by Hiera-eyaml, so don’t change it. But you can change the password inside the square brackets. After changing the password, save and close the file and your updated passwords will be encrypted.
When we push psft_customizations.yaml out to servers, we also need to ensure each server has the keys used to encrypt the passwords, and also knows about Hiera-eyaml. First, if you are using the encrypted passwords on more than one server, copy the puppet\etc\secure\keys folder to each server.
Next, Hiera needs to know that we are using Hiera-eyaml. In C:\ProgramData\PuppetLabs\hiera\etc\hiera.yaml, enable eyaml as a back-end format by adding - eyaml to the ;backends: section:
:backends:
- yaml
- eyaml
Verify that the :eyaml: section is at the bottom of hiera.yaml. Change the paths to the Public and Private keys. If you followed the steps above and created them in puppet\etc\secure\keys, the paths will look like this:
:eyaml:
:datadir: C:\ProgramData\PuppetLabs\puppet\etc\data
:extension: yaml
:pkcs7_private_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\private_key.pkcs7.pem
:pkcs7_public_key: C:\ProgramData\PuppetLabs\puppet\etc\secure\keys\public_key.pkcs7.pem
Save hiera.yaml and let’s test our configuration.
To test Hiera-eyaml and Puppet working together, we’ll encrypt the a password in psft_customizations.yaml and update UserPswd= value in psappsrv.cfg.
Open psft_customizations.yaml with eyaml edit and add the line:
db_user_pwd: DEC::PKCS7[VP1]!
Save and close psft_customizations.yaml.
Save this code below as pwd.pp and save it to puppet\etc\manifests. Change the $configFile path to point to your psappsrv.cfg file.
$configFile = 'C:/Users/vagrant/psft/pt/8.55/appserv/APPDOM/psappsrv.cfg'
ini_setting { "eyaml_test":
ensure => present,
path => $configFile,
section => 'Startup',
setting => 'UserPswd',
value => hiera('db_user_pwd'),
}
Change directories to puppet\etc\manifests.
Run puppet apply .\pwd.pp --trace --debug
After the run is done, open your psappsrv.cfg file. You should see UserPswd=VP1 in the file.
If the test above worked, you’re all set to use Hiera-eyaml with the DPK and Puppet. Once Hiera knows about Hiera-eyaml, any data in Hiera can be encrypted. Happy encrypting!