Taking PeopleSoft Swimming in an OCI Instance Pool

I was recently studying for some OCI Certification exams when I came across the topic of Instance Pools. I’ve known about this OCI feature for a while but realized I somehow never thought seriously about using them with PeopleSoft. So, I decided to do a proof of concept and write about it. What I found was with a few changes to a traditional PeopleSoft topology and configuration, you can get this working!

Please keep in mind that this is a proof of concept. I know I didn’t touch on all PeopleSoft configuration and features, so there is likely more work to be done to make this setup viable. I will try to call out some “gotchas” that come to mind but feel free to call out potential issues in the comments. If we can turn this POC into something usable, it might even open the door for container exploration!

Getting Started

To get started, we first need to understand what an Instance Pool is, and what is required to create one. An Instance Pool is a feature that allows you to create and manage multiple compute instances that share the same configuration. What makes them powerful is the ability to scale instance count and attach to a Load Balancer. You can scale on a schedule, with metric based rules, with Terraform or a script using OCI CLI, or manually with the OCI Console. Adjusting the instance count will create or terminate instances automatically. Also, when the pool is attached to a load balancer, the instances are automatically added or removed from the load balancer’s backend set. As you can see, this type of functionality would be great in the PeopleSoft space, where the load on a system can vary day-to-day. Think “timesheet day,” open enrollment, course registration, etc.

To complete this POC, we first need a few things.

Load Balancer

We will leverage the Load Balancing service that OCI offers. This service’s integration with our Instance Pool will give us the automation that we need. Creating a Load Balancer is straight forward, but there are a few things to keep in mind for this POC.

  1. Select a visibility type of Public if you hope to access it from the internet.
  2. Don’t worry about adding Backends at the time of Load Balancer creation. This will be handled by the Instance Pool in the future.
  3. Make sure to enable Session Persistence on the Backend Set. This is needed for PeopleSoft to work correctly.

Custom Image

An Instance Pool uses an Instance Configuration to spin up new instances on demand. An Instance Configuration contains all the configuration details like network, OS, shape, and image needed to create an instance. The image can be an Oracle provided platform image( Oracle-Linux-7.x, etc.) or a custom image that you create.

This image marks the first real decision on how to approach this POC. What we are after is a server with PeopleSoft web and application domains fully deployed and running. I see two approaches to this. One, use a standard platform image and then use cloud-init to configure and deploy the domains using DPK at instance creation. Two, create a custom image with the domains already deployed. I chose to leverage the custom image approach for this POC. I felt this was the fastest, most reliable way to create these instances. With rule-based scaling, and maybe to a lesser extent scheduled scaling, we ideally want these instances created quickly, and DPK takes time.

If startup time at creation isn’t a concern, then an approach using cloud-init probably is the way to go. One thought I had was to keep all middleware installations on FSS. The cloud-init script could just mount that and have DPK just focus on deploying PS_CFG_HOME. That would really speed things up. Maybe something to try in a future blog post!

Create Base Instance

I found a few things that were needed when prepping a PeopleSoft webapp server for Custom Image creation. I started by creating a new instance using the latest Linux 7 platform image. Next, I downloaded the latest PeopleTools 8.58 DPK to the instance and ran the psft-dpk-setup.sh setup script, passing the --env_type midtier --domain_type all options. This gave me a server with working web and application domains.

Update PeopleSoft Configuration

If we stopped here and created a Custom Image from this base instance, we might have some success. However, every future instance created from this image would have the exact same PeopleSoft domain configuration. There are some parts of a PeopleSoft domain configuration that are required to be unique. Also, some parts of the configuration will have a hostname “hardcoded,” etc. Here is a list of changes I made to address these concerns before I created a Custom Image. As mentioned before, there are likely more changes needed. However, these were enough to get this POC working.

  1. Update Cookie Configuration
    • Make sure your cookie configuration in the weblogic.xml file is set not to include a hostname.
    • <cookie-name>DBNAME-PORTAL-PSJSESSIONID</cookie-name>
  2. Update configuration.properties
    • Set the psserver= value to use localhost, since we are using a webapp deployment approach.
    • This will pin each web domain to use its local app domain, with load balancing and failover handled strictly by the OCI Load Balancer.
  3. Update setEnv.sh
    • By default, the ADMINSERVER_HOSTNAME variable is set to the hostname the domain was installed on.
    • Change this to be a dynamic value, driving off of the $HOSTNAME environment variable.
      • ADMINSERVER_HOSTNAME=${HOSTNAME}
  4. Update psappsrv.cfg
    • The [Domain Settings]\Domain ID value should be unique across all your PeopleSoft domains.
    • It is common for domain ids to follow a sequence number pattern. Example: APPDOM1, APPDOM2, etc.
    • Update the value to use an environment variable for the sequence number, ensuring a unique ID.
      • Domain ID=APPDOM%PS_DOMAIN_NBR%
      • We will discuss ideas on how to set this %PS_DOMAIN_NBR% variable later.
  5. Update psft-appserver service script
    • To ensure our domain configurations are correct and domains start properly, we should enforce a domain configure during instance creation and boot.
    • For this POC, I simply added a psadmin configure command to the domain service script.
    • For each application domain installed by the DPK, there is a service setup using a script found here:
      • $PS_CFG_HOME/appserv/APPDOM/psft-appserver-APPDOM-domain-appdom.sh
    • Update this script in the start_application_server function and add the following configure command before the start command.
      • su - $APPSRV_ADMIN -c "$APPSRV_PS_HOME/bin/psadmin -c configure -d $APPSRVDOM" 1>>$LOG_FILE 2>&1
  6. Create $PS_DOMAIN_NBR
    • In the psadm2 ~/.bashrc file, export a $PS_DOMAIN_NBR variable.
    • The goal for this is to generate a unique number that can be used in your domain configuration.
    • When an Instance Pool creates an instance, a random number is appended to the hostname.
      • Example: webapp-634836, webapp-973369, etc.
    • To leverage this appended number for $PS_DOMAIN_NBR, you can use something like this:
      • export PS_DOMAIN_NBR=$(hostname | rev | cut -d- -f1 | rev)

Create a Custom Image

Before creating the custom image, there are a few more things to clean up. First, remove any unwanted domains or software. Depending on your DPK deployment, a process scheduler domain may have been created. We won’t be needing this, so it can be manually removed using psadmin. Next, remove any unwanted entries in /etc/hosts file. For this POC, I stripped down to just the “localhost” entries. New entries will be added when instances are created with new hostnames. Last, stop all running PeopleSoft domains. This step is just to ensure a clean configuration.

Now we are ready to create the Custom Image. In the OCI Console, find your base instance. Under More Actions, select Create Custom Image.

Instance Configuration

After the Custom Image is done provisioning, we are ready to create our Instance Configuration. To start, we need to create an instance based on our new Custom Image. In the OCI Console, navigate to Compute > Custom Image and find the new image. Click on Create Instance. Enter a name, shape, networking, and other details we want to be used in our future Instance Configuration. Select Create and wait for the instance to provision.

Once provisioned, go ahead and validate that the instance looks good. Login to the PIA, make sure the domains started correctly, etc. If all looks well, go back to your newly created instance in the OCI Console. Under More Actions, select Create Instance Configuration. Select a compartment, name, and optional tags, then click Create Instance Configuration.

Instance Pool

Now that we have our Instance Configuration, we can finally create our Instance Pool. In OCI Console, navigate to the newly created Instance Configuration and click on Create Instance Pool. Select a name, the instance configuration, and the number of instances you would like in the pool. Next, we will configure the pool placement with Availability Domain details. Also, make sure to select Attach a Load Balancer. Select the Load Balancer details we created earlier. Lastly, review the details and click Create. You can now follow the provisioning steps under Work Requests.

Once the provisioning work requests are completed, you can validate everything worked as expected. Under Instance Pool Details, you can navigate to Created Instances. You should see the newly created instances listed here. Also, you can navigate to Load Balancers. You will see the attached load balancer. Click into this to review that the Backend Set was updated with the newly created instances.

After validating the initial deployment, you can play around with the instance count. Navigate back to your Instance Pool and click on Edit. Try adding or subtracting to the Number of Instances. Monitor the work requests then validate instances were added or subtracted correctly.

Conclusion

This blog post was a quick, high-level walkthrough of using Instance Pools with PeopleSoft on OCI. My goal was mainly to prove that this was possible in a POC. However, I also wanted to help start a conversation about how this might look in a Production setting. I listed out a few things that could be done to get a custom image approach to work. What other configuration values did I not mention? What ideas do you have for a more dynamic approach, using DPK installs at creation time? As the feature set of OCI grows relating to Load Balancers and Instance Pools, I think we will be more and more motivated to get a deployment like this working outside of the lab!

Linux DPK: Dealing with Missing Required OS Packages

For those of you using the NativeOS Linux install for Update Images, you have probably come across this scenario. You start the DPK install and once you get to the Puppet installation section, the script comes to an abrupt end. What the heck! Looking in the log file, you quickly figure out your OS is missing some required packages. So now what?

In the PeopleSoft Deployment Packages for Update Images Installation document, task 2-3-3 walks you through how to get the required OS packages needed for Puppet. They make it clear that it is your job to obtain these packages and install them – you’re on your own. They then list a few steps on how to accomplish this. The steps pretty much come down to this:

  1. Install DPK
  2. DPK will fail on missing packages
  3. Find missing OS packages by reviewing the log
    • $DPK_INSTALL/setup/psft-dpk-setup.log
  4. Run DPK cleanup
  5. Install missing OS packages
  6. Install DPK again

Following the steps is pretty straight forward, but I don’t like having to manually dig through a log file and pick out the missing OS Packages. So, what I did is write a little shell script to extract them for me. This script will generate the list of missing OS packages and write it to a file. After reviewing the list, you can then use this file for installing the packages.

Here are the steps I follow to ensure I have all the needed OS packages when installing NativeOS Linux DPKs. These steps assume your current directory is $DPK_INSTALL/setup.

  1. Install DPK
  2. DPK will fail on missing packages
  3. Generate missing packages list
    • grep "is needed by" psft-dpk-setup.log | awk '{print $1;}' >> os-packages.list
  4. Run DPK cleanup
  5. Review list, edit if needed
    • vi os-packages.list
  6. Install missing OS packages
    • sudo yum install $(cat os-packages.list)
  7. Install DPK again

Unfortunately, you may have to repeat this process a few times to identify all the missing packages. Once I have gotten through a DPK install on a particular OS, I save off the os-packages.list file for safe keeping. I then make sure I install this list of packages to any new VM that I am doing a fresh DPK install on. Doing this before DPK installs will ensure we don’t see any missing OS package errors. I’m sure this list will need to be updated as time goes on and we see different versions of Puppet, etc in our DPKs.

Hopefully you found this post helpful! This little tidbit was pulled out of the PeopleSoft Deployment Package QuickStart course. Feel free to enroll in this FREE course today for more DPK goodness.

os-packages

Extending psadmin with psadmin-plus

I have created a helper menu script to extend the delivered psadmin program. The script is called psadmin-plus and I have created a repository for it on psadmin.io’s GitHub account. This was built as a self-study side project while I was on paternity leave this summer. I wanted to learn a little more about bash scripting and how to use git, and at the same time try to make a useful tool for myself and others to use. As of this writing, the tool is usable but far from complete. At the moment it only has support for Linux. I hope to make improvements over time and would invite others to summit issues on GitHub for questions, bugs or enhancement ideas. If anyone wants to contribute themselves, that would be great too!

There are two main uses for psadmin-plus. The first is actually calling the delivered psadmin program. The value add here is that it will auto-discover all your PS_CFG_HOME directories for you and source environment variables as needed. This all assumes you follow a few conventions, which should be documented in the GitHub readme or wiki pages. As mentioned in a previous blog post, this is useful if you use a single user to run your PeopleSoft environments. If you have a different user for each environment and source at login, then this feature doesn’t really help.

The second use is executing actions against multiple PS_CFG_HOMEs and domains at once. An example would be to stop all Process Scheduler domains on a machine. With this tool, you can do this with a few key strokes. You also have the option to execute now or later. If you select later, a script will generate to file. This allows you to run at a future time, maybe during a maintenance window. Again, there are a few assumed conventions that must be followed.

If you want to try it out for yourself, I have created a setup script to run against a PeopleSoft Image(VBox or Linux DPK install only). This will create a few extra PS_CFG_HOMEs and domains for you to play with in the menu. You can find instructions in the GitHub readme.

Below is a quick demo of psadmin-plus in use. For more information please see GitHub.

psadmin-plus-demo

Managing Environment Variables When Using Decoupled Homes

As a reader of this blog or a listener of the podcast, you know I am a user of both Linux and decoupled homes. Traditionally with a Linux PeopleSoft installation you need to source the delivered psconfig.sh to set your environment variables. When an entire environment was contained under its own PS_HOME, you could tweak this psconfig.sh file if customizations were needed without fear of impacting other environments. Now with decoupled homes, the PS_HOME directory will likely be shared, so changing the psconfig.sh file located there is a bad idea.

When switching to decoupled homes, I was looking for a good way to manage sourcing the psconfig.sh file and the different environment variables. While attending Alliance 2015, I saw a presentation given by Eric Bolinger from the University of Colorado. He was talking about their approach to decoupled homes and he had some really good ideas. The approach I currently use is mostly based on these ideas, with a few tweaks. The main difference is that he has a different Linux user account specific to each environment. With this approach, he is able to store the environment specific configuration file in the users home directory and source it at login time. This is similar to the approach Oracle suggests and uses with their PIs(see user psadm2). My organization didn’t go down the road of multiple users to run PeopleSoft. Instead, we have a single user that owns all the environments and we source our environment specific configuration file before we start psadmin. We use a psadmin wrapper script to help with this sourcing(which I will discuss and share in a future post). The main thing to keep in mind is regardless of how these files are sourced, the same basic approach can still be used.

The idea here is to keep as much delivered and common configuration in psconfig.sh as possible and keep environment specific customization in their own separate files. I like to keep these config files in a centralized location, that each server has access to via a NFS mount. I usually refer to this directory as $PSCONFIGS_DIR. What I do is copy the delivered psconfig.sh file to $PSCONFIGS_DIR and rename it psconfig.common.sh. I then remove any configurations that I know I will always want to set in our custom environment specific file, mainly PS_HOME. I then add any needed configuration that I know will be common across all environments (Another approach would be to create a new psconfig.common.sh file from scratch, set a few variables and then just source the delivered file cd $PS_HOME && . psconfig.sh. Either way works, but I like the cloning approach). This common file will be called at the end of every environment specific file. Remember to take care when making any changes to this file, as it will impact any environment calling it. It is also a good idea to review this file when patching or upgrading your tools.

Next for the environment specific files, I create a new file called psconfig.[env].sh. The environment name is listed in its filename. An example would be psconfig.fdev.sh. You could really choose any name for this, but I found this approach to be handy. In this file you will set the environment specific variables as needed, then end with calling psconfig.common.sh. Here is an example file:

This approach allows you to be a little more nimble when patching or upgrading. You can install new homes or middleware, then update the psconfig.[env].sh file and build new domains. When you get to go-live for Production, you can have the domains all built ahead of time. When ready, just update the config file, upgrade the database, and you are good to go!

One final note, regarding directory naming conventions. My organization tends to always have our PS_CFG_HOME directory match the environment or database name exactly, ie. fdev. I’m considering changing this, however. During our last Tools patching project, I found it a little awkward to prebuild the domains and still end up with same directory name. It seems to make much more sense to include the PeopleTools version in the directory name. That way you can prebuild the domains in a new PS_CFG_HOME, and when you go-live just blow the old home away. Another great idea I took away from Eric’s presentation was how to dynamically generate a PS_CFG_HOME directory name:

export PS_CFG_HOME=/opt/pscfg/$ENV-`$PS_HOME/bin/psadmin -v | awk '{print $2}'`

If you use this technique, you will want this to be the last line in your config file – after sourcing the common file. What this does is concatenate your environment name with the PeopleTools version, using the psadmin version command – ie. fdev-8.55.03. This will give you more clarity on what tools version the domains under this PS_CFG_HOME were built with and it will make it easier to prebuild your domains.

Switching File Attachment Storage

In previous posts I have talked about the File Attachment storage options in PeopleSoft. The three basic options are Database, FTP or HTTP. My organization initially went with HTTP, but now we are looking to move to Database storage. The requirement is to move not only future file attachments, but attachments from the past as well. At first I thought this would require a lot of work, including custom conversion AE programs, etc. However, there is a delivered process called Copy File Attachements that does all the heavy lifting for you. There is an online and batch method to run this, but I highly recommend the batch mode. There are a few other steps needed to fully convert all attachments, but it is fairly straight forward. Below are the steps I used in FSCM 9.2.17, PeopleTools 8.54.18.

Create new File Attachment Server

  1. Navigate to Main Menu > Set Up Financials/Supply Chain > Common Defictions > File Attachments > Administer File Attachments
  2. Click Add Database Server
  3. Record Name will default to PV_ATT_DB_SRV. The default is fine, but change if you would like.
  4. Set Pick Active Server to match the ID of your new server.
  5. Save. Future File Attachments will now be stored in your database.

Copy File Attachments

  1. If needed, create new URL for Database storage – PeopleTools > Utilities > Administration > URLs. Values URL Identifier: PSA_ATT_DB, URLID:record://PV_ATT_DB_SRV
  2. Navigate to run control page PeopleTools > Utilities > Administration > Administer File Processing > Copy File Attachments (Batch)
  3. Values Source: URL.PSA_ATT_HTTP, Destination:URL.PSA_ATT_DB
  4. Run the App Engine COPYATTS through Process Scheduler.
  5. This will take a long time to run, possibly hours depending on your attachment count.
  6. After completion, you will want to review the trace file AE_COPYATTS_[PIN].trc file. The AE should produce this automatically.
  7. I used this trick to generate a log of ONLY errors from the trace:

    grep "EvalCopyAttachments: failed getting file" AE_COPYATTS_*.trc > AE_COPYATTS.errors

  8. Take action on any errors that occurred.

Update previous File Attachment URL

  1. These steps point the old HTTP server setup to point to the new DB server.
  2. Navigate to PeopleTools > Utilities > Administration > URLs. Values URL Identifier
  3. Open the URL used for the previous File Attachment Server setup. Example: PSA_ATT_HTTP
  4. Change the URLID value to the new Database server URL values. Example: record://PV_ATT_DB_SRV
  5. I recommend adding comments describing this conversion process – especially if HTTP or FTP is in the URL ID and it now points to the DB. This can help avoid confusion in the future.

After we completed these steps, we decided to keep our old storage location around just in case we found any issues after the fact. We ended up renaming the directory path, just to make sure nothing was still referencing the old location. After a few weeks of no issues, we went ahead and destroyed this old storage location. As mentioned above, this was done in the FSCM application which has its own File Attachment framework built on top of the one Tools delivers. You should be able to take a similar approach with other applications, but the Create new File Attachment Server section above won’t be relevant. Instead, you can simply complete the Update previous File Attachment URL steps after your copy is complete.

Disabling the App Engine Server

I was having a discussion with another admin recently, and he questioned why we were running our process schedulers with Application Engine Server(PSAESRV) enabled. He pointed out that there is very little gained by running PSAESRV, so why have it turned on?

Now, you aren’t completely without loss in turning this off. Depending on what applications you are running and what your system’s needs are, you will have to make that decision for yourself. Check out Doc ID 651970.1 in MOS for a run down on the pros and cons for each method of running App Engines on your process schedulers.

 

Maintenance Page with Backdoor Login

A standard requirement when doing PeopleSoft maintenance is to change the normal sign in page. When the system is offline, we often like to display a maintenance page that informs users that the system will be down and prevents them from logging in. This is done easily enough by changing the signin.html page, adding a message and removing the login form.

However, there is one more requirement that is a little tricky. Once the maintenance is complete, there are often tasks that need to be completed before handing the system back to the end users. These can be configuration changes, running batch processes or simply completing validation that everything was applied correctly and is in working order. How can the core team sign on to the system and complete these tasks, all while preventing end user access?

I recently came up with a bit of JavaScript that gave us a backdoor to the system during our last PeopleTools upgrade. What the script does is hide the login form on the sign in page, preventing login.  The trick was that our core team knew the key to the backdoor. After the sign in page would load, when they pressed Ctrl+Space the login form would be reveled. The combination of keys was only known to the team, so they were the only ones able to get in. Keep in mind if you had some crafty end users this could obviously be worked around, but it did the trick for us.

The JavaScript and HTML to accomplish this is listed below.  Adding the script and HTML changes to your signin.html page is pretty straight forward. First include the file containing the JavaScript, or write it inline. You then need to wrap your login form elements in a <div> with an id=loginbox. I would suggest starting before the <div> containing ptLabelUserid and ending after the <div> containing the submit button. Lastly, you need to add hideLogin(true); to the body onload attribute.

Keep in mind the key doesn’t have to be Ctrl+Space, it can be any key combination really.You will see in the comments of the script a link to information about other keycodes that can be used.

Updated: 03/20/2023 There is an updated version of the JavaScript and HTML here that works with PeopleTools 8.60 in this Github Gist

Updated: 10/11/2016 Instead of hard coding a true value for your hideLogin() parameter, why not use a Custom Property set in the Web Profile? You can create any Custom Property you would like, for example: login.isLoginHidden and set to true. Then reference the property in your signin.hmtl page like this: hideLogin("<%=login.isLoginHidden%>"). This will allow you to toggle the hide login functionality by updating the web profile and bouncing the server.

Click here to see a working demo. Enjoy!

#1 – Decoupled Homes

I’m really happy to announce a new project we are launching today:

The PeopleSoft Administrator Podcast

Kyle Benson and I sit down and talk about all things PeopleSoft Admin. Kyle and I would meet for lunch every few weeks to talk about our latest issues at work and the latest features in PeopleTools. After a year of this, we thought it would be fun to record our conversations and share them.


In episode 1 of the PeopleSoft Administrator Podcast, we talk about decoupled homes and how to use them. They discuss the advantages of using them, how they manage each home and why you should look at using decoupled homes.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Links from this episode:

Rebuilding Tuxedo Domains

When using the command line to rebuild a Tuxedo domain, there is a new flag to keep your settings:

The –keepfeatures switch in the following command was introduced in PeopleTools 8.54.

For example:

psadmin –c import [PS_CFG_HOME] –d [old domain name] –n [new domain name] -keepfeatures

This came from the Best Practices guide for Mid-tier Deployment White Paper. At the end of the document, there is a good overview for the steps needed with a PeopleTools upgrade and the commands to help you script the upgrade.