About 2 years ago, Kyle released psadmin-plus, a bash utility to help automate and simplify domain management. We happy to announce psadmin-plus is now at 2.0!

psadmin-plus makes is easy to start, stop, reconfigure, purge cache, and more for your PeopleSoft domains:

psa stop app HDEV
psa start web HDEV
psa bounce all HDEV


Version 2.0 is all command-line driven and has a number of changes and improvements from the original release.

## Ruby Gem

First, psadmin-plus was re-written as a Ruby Gem. The original version was a bash script for Linux. A little while later, a powershell version was created for Windows. We chose to re-write psadmin-plus in Ruby so it could be a cross-platform tool. One code line makes is much easier to implement features for both platforms.

The second benefit of re-writing as a Ruby Gem is installation is much easier. This command is all you need to install psadmin-plus:

gem install psadmin_plus


When updates to psadmin-plus are released, you can upgrade with:

gem update psadmin_plus


You will need Ruby installed on your system for gem and psadmin-plus command to work. If you used the DPK to build you system Ruby is already available. (For Windows, if you use the DPK’s version of Ruby, the RubyGems SSL trusted certificate is out of date. Run these commands to uppdate the RubyGems trusted certificate.)

## Configuration File

psadmin-plus has a number of configuration options to customize the behavior of you system. You can set these options with environment variables, but version 2.0 supports the use of a configuration file. You use the file ~/.psa.conf to set your configuration options. You can also change the file by setting PS_PSA_CONF=/path/to/conf.file. Here is a sample of what your .psa.conf file would look like:

PS_POOL_MGMT=on
PS_MULTI_HOMES=c:\psft\cfg
PS_WIN_SERVICES=web
PS_HEALTH_FILE=host.html


For a full list of configuration options for psadmin-plus, check out the README on the GitHub repository.

## Windows Service Support

The first version of psadmin-plus for Windows started and stopped domains via psadmin. Many Windows admins prefer to start their domains with the Windows Services. Setting the configuration options PS_WIN_SERVICES will tell psadmin-plus to start and stop domains via the start-service and stop-service command-let. The PS_WIN_SERVICES setting has a number of options. You can specify all to start all domains with services, tux to start the Tuxedo domains with services, or specify one of web, app, prcs if you want to call out a specific type.

If you start domains via the command line, but have Windows services defined, we want to keep the service status in sync. The PS_TRAIL_SERVICE option will do just that. If you enable that setting, psadmin-plus will start or stop the domain via psadmin and then start or stop the respective Windows service. This ensures that the Windows service status matches the status of your domain.

psadmin-plus assumes you are using the DPK service names, but you can override that if you used a different method for creating Windows services. The defaults are:

"Psft*Pia*#{domain}*"
"Psft*App*#{domain}*"
"Psft*Prcs*#{domain}*"


If you want to change those names, you can add these settings in your .psa.conf file:

WEB_SERVICE_NAME="#{domain}-pia"
APP_SERVICE_NAME="#{domain}-AppServer"
PRCS_SERVICE_NAME"#{domain}-Batch"


Fill out the value to match your naming convention, but you must include #{domain}. That is how psadmin-plus finds your service.

## Multi-PS_CFG_HOME Support

One last feature we added to psadmin-plus is support for multiple PS_CFG_HOME folders. By default psadmin-plus assumes you have one config home folder on the server for all your domains. If you have more than one config home, you can set the environment variable PS_MULTI_HOMES to the base folder where your config homes live. For example, if c:\psft\cfg stored your config homes, you can set PS_MULTI_HOMES=c:\psft\cfg, and psadmin-plus will look under there for your config homes.

There is one assumption though with multi-config home support: the domain name matches the config home name. If your domain name is HDEV and your base config home is c:\psft\cfg, psadmin-plus will look in the path c:\psft\cfg\HDEV\ for your domains.

## Shell Versions

If you prefer the older shell script version of psadmin-plus, don’t worry. We saved those versions into branched on the on GitHub repository. You still use the shell scripts but we’ll only be adding new features to the RubyGems version of psadmin-plus.

# Improving Windows Services from the DPK

A common theme we write about on the blog is how to make the DPK work with multiple environments on the same machine. It’s common to run a DEV and TST on the same server. The DPK can build those environments, but there are a few changes to make the setup run well. On Windows, the services the DPK creates makes an assumption that breaks when we run multiple environments.

When starting a domain via Windows services, the service assumes that the environment variables are set for that environment. If you create your DEV environment via the DPK, that’s a good assumption. But, if you create a TST environment next, the environment variables are set to TST. When you attempt to start the DEV domain via Windows services, the domain start will fail.

To resolve this, we can improve the Ruby script that starts our domains. Under the ps_cfg_home\appserv\DOMAIN folder, there are Ruby scripts that are called by the Windows service. For the app server, it’s appserver_win_service.rb. These scripts will look for the PS_CFG_HOME environment variable and start the domains it finds under that home. We can add a line in the file to point to the correct PS_CFG_HOME location like this:

ENV["PS_CFG_HOME"]=c:\psft\cfg\DEV


While we can modify the file directly, the DPK way of handling this is to update the template in the DPK. Then, whenever we rebuild our domains the code change is automatically included.

The Ruby scripts to start/stop domains are templates in the DPK. The templates are stored under peoplesoft_base\dpk\puppet\modules\pt_config\files\pt_appserver\appserver_win_service.erb (replace pt_appserver with pt_prcs or pt_pia for the batch and PIA services.)

To make the environment variables we add dynamic, we can reference variables that exist in the Ruby environment that calls the ERB template. In the program appserver_domain_boot.rb, the variables ps_home and ps_cfg_home are set. We will use those variables to build our environment variables.

ENV["PS_HOME"] = "<%= ps_home %>"
ENV["PS_CFG_HOME"] = "<%= ps_cfg_home %>"
system("<%= ps_home %>/appserv/psadmin -c start -d <%= domain_name %>")


The <%= %> tags will output the value of that command or variable. So in our case, we are outputting the string value of ps_cfg_home.

The result of this file will look like this:

ENV["PS_HOME"] = "c:\\psft\\pt\ps_home8.56.08"
ENV["PS_CFG_HOME"] = "c:\\psft\\cfg\\DEV"


When we run Puppet the next time, our Windows service will have it’s environment variables set before starting or stopping a domain.

# ps-availability Version 2.0

About a year ago, I posted a project to check on the status of all our PeopleSoft environments. Our status page has become an important part of monitoring our environments. I check that page every morning and the email alerts let me be proactive in addressing environment issues. We also embedded the status page into our support teams homepage, so any team member can quickly see the status of environments.

I made some improvements to the script over the last year and am releasing them as version 2.0. The improvements for version 2.0 are:

• Checking for Stale process schedulers. In version 1, I simply grabbed the process scheduler status and reported on that. But, if your scheduler crashes it won’t update the server status table. So, you can get situations where the page says your scheduler is running but it’s not. In version 2.0, you can configure a Stale Interval to compare the last updated time. If the last update is greater than the interval, it will report the scheduler as “Stale”.
• I removed the interim Markdown tables that were used to create the HTML table, which let me remove the Redcarpet gem dependency. In version 2.0, the HTML tables are built as the data is collected. This let’s the script dynamically add classes for formatting, but it also lets us build more complex tables.
• IB Domain Status reporting is in version 2.0. The status of the IB domains doesn’t impact the notifications (partly because we have some domains Inactive on purpose), but you can click on an environment row to see a report from the IB Domains page. The row embeds a table with your IB domain status so you can quickly check the status.
• You can specify a homepage check for both Classic and Fluid so you don’t have to use the same title for both homepages. This is also useful if you are starting to roll out Fluid in some environments, but have Classic in others.

Here is a screenshot of the new status page:

There are three new configuration options (in the psavailability.rb script – on the list for future improvements) with version 2.0:

• Fluid Homepage Title Check (default: Homepage)
• Time Zone (default: US Central)
• Stale Interval (default: 10 minutes)

Go visit the GitHub repository to download the project and get started.

Do you get notified if an app server, web server or process scheduler go down? If you do, good for you! If you don’t receive notifications, here is a script that will send you notifications.

If you are an admin who doesn’t have access to monitoring tools (and there are lots of you), this post will go over a script I wrote. The script builds a System Status page and sends email notifications. You can run the script on any machine and it will send an email if anything has gone down. A status.html page is also generated so that users can go check on the status for their system.

This page and script is something I put together over a few days as a side project. We wanted a page that would give us the status of each component in an environment. This isn’t the most robust script (I’ll explain the limitations in the post), but I wanted to share it so other people could use it. If you rely on end users to tell if you an environment is down, this script can help you out.

All of the code is hosted on GitHub, so go grab it here.

## Install Prerequisites

The script is written in Ruby, uses tnsping, uses Mechanize gem for interacting with Peoplesoft, Markdown for formatting, the Redcarpet gem for generating HTML documents, and the Mail gem for emailing status updates. So we’ll need to install all those parts. It sounds like a lot, but it’s pretty simple.

We’ll walk through the installation process. I’m writing the instructions for Windows, but the Linux steps will be similar.

### Oracle Client

The script uses tnsping to check the database status. So, we need the Oracle Client installed on the machine. If you don’t have the Oracle Client software, you can download it here

You also need a tnsnames.ora file with entries for all the databases you want to check. You can place the tnsnames.ora file anywhere on the server. The status.bat script sets the TNS_ADMIN environment variable to point to your tnsnames.ora file.

### Ruby Dev Kit

We’ll install Ruby and the Ruby Dev Kit (2.2.4 is what I’m using) on the machine where our scripts will run. Download the Ruby installer from here:

http://rubyinstaller.org/downloads/


I installed Ruby to e:\ruby22-x64 and selected the option to “add executables to the PATH variable”.

Next, download the Ruby DevKit from the same site. The DevKit includes tools to build Gems from source code. We need to the extra tools included with the DevKit. I installed the Ruby DevKit to e:\ruby22-x64-devkit.

Open a new command prompt as an Administrator.

1. e:
2. cd ruby22-x64-devkit
3. ruby dk.rb init
4. notepad config.yml
5. Add - e:/ruby22-x64 to the end of the file (notice the dash and forward slash)
6. Save and close the config.yml file
7. ruby dk.rb install

Follow the instructions here if you have issues with the DevKit installation.

### Gems

Ruby has a powerful package manager called “Gems”. The gem command is part of Ruby, so we can install the extra packages we’ll use for our status page.

Open a new command prompt as an Administrator and we’ll install the Gems.

1. where ruby

Make sure this command returns the e:\ruby22-x64 folder first.

If it’s not listed first, change the PATH envronment variable so e:\ruby22-x64\bin\ is first.

1. gem install mechanize
2. gem install redcarpet
3. gem install mail

That’s it for the Gems.

## Scripts

There are two scripts in the project:

1. psavailability.rb
2. status.bat

The first script, psavailability.rb, is the main script. This is where all the processing happens. The second script, status.bat, is a wrapper to set environment variables like ORACLE_HOME, TNS_ADMIN and PATH.

### psavailability.rb

Let’s dive into the psavailability.rb script since that is the main part. As I mentioned before, the script is written in Ruby. I’ll be first person to tell you that I’m not an expert in Ruby. If there places where the code could be improved, let me know.

Status Check Flow

I chose to do all my status checking through the application. This is where Mechanize comes into play. I replicate the actions of an user who opens the login page, logs in, and navigates to the Process Monitor page. The main reason for this method was simplicty. I can do all my checks with one library.

+--------------+            +--------------+            +-------------+
|  Web Server  |  +----->   |  App Server  |  +------>  |  Scheduler  |
|    Check     |            |    Check     |            |    Check    |
+--------------+            +--------------+            +-------------+


The main disadvantage is: if the web server is down, the app server and process scheduler checks will fail. That may not be accurate from a technical perspective, but from a user perspective it would be true. If you can’t log in, you can do anything!

#### Setup

I’ve moved any variables that will vary from my install to the top. They are:

# ---------------------------
# Change these variables
# ---------------------------
smtpServer              = '<smtp server>'
homepageTitleCheck      = '<Homepage Title>'
deployPath              = '<e:\\path\\to\\PORTAL.war\\>'
# ---------------------------


The script assumes that you will use the same account to access all environments. I created a new account called STATUS and gave it limited permissions. STATUS can log in and open the Process Monitor Server List page. This way I can track logins from the status script, and we give the service account the least amount of security needed.

Another assumption in the script is that your Homepage Title will have similar text. In our case, we use titles like HR 9.2 Demo, FS 9.2 Test, or ELM 9.2 QA for our environments. I check for 9.2 in the Homepage Title to know if the login was successful.

#### Initialization

The next section is contains some initialization steps. I set the User-Agent to IE 9, but you can change that if you want.

Mail.defaults do
delivery_method :smtp, address: smtpServer, port: 25
end

affectedEnvironments = Array.new
notify = false

agent.user_agent_alias = 'Windows IE 9'


Then, I create the Markdown table headers for our table. I found it much easier to create the table in Markdown and then convert the table to HTML at the end.

table =          "| Environment | Database | Web Status | App Status | Scheduler | Batch Server | Update Time | Batch Status |\n"
table = table +  "| ----------- | -------- | ---------- | ---------- | --------- | ------------ | ----------- | ------------ |\n"


Last, I read in the URLs.txt file to get the list of environments and URLs to use. The project on GitHub has a sample URLs.txt file to follow.

# Get the list of environments
# the URLs.txt file is a CSV file with the format "DBNAME,baseURL,processMonitorURI"
agent = Mechanize.new
URLs = CSV.read('URLs.txt', {:col_sep => ','})


#### Checking Status

URLs.each { |environment, loginURL, prcsURI|

web_status = 'Running'
app_status = 'Running'
database   = 'Running'


We’ll loop through each environment for the status checks. In our environment, we have 25 “environemnts” (prod and non-prod) that we check. I say “environments” because production has 2 web servers and I check each one.

        begin
t = tnsping #{environment}

if t.lines.last.include? "OK"
database = 'Running'
else
database = 'Down'
end
rescue
database = 'Down'
end


To test the database, we run a tnsping command. If the response contains “OK” in the last line the ping was a success. If you are thinking to yourself, “that’s not the best way to test the database”, I agree. But this is a the quicket way to get an Up or Down response. (See the Future Improvements section at the end.)

        # Check web server by opening login page
begin
rescue
web_status = 'Down'
end


Next, we attempt to load the login page. If the page responds, we know our web server is up.

        begin
signin_form.userid = statusUser
signin_form.pwd = statusUserPwd
homepage = agent.submit(signin_form)

# We updated PeopleTools > Portal > General Settings to include '9.2' in the title (e.g, "HR 9.2 Test").
# If we see '9.2' in the title, we know the login was successful
if homepage.title.include? homepageTitleCheck
app_status = 'Runnning'
else
app_status = 'Down'
end
rescue
app_status = 'Down'
end


To check the app server status, the script logs into the application. We grab the form named login and pass in the PeopleSoft user and password. The page returned from the login attempt is stored in homepage. In our case, every environment has “9.2” in the homepage title. If “9.2” is in the title, I know we have logged in and the app server is up.

The field that holds the homepage title is psprdmdefn.descr254.

        begin
# Build URL for Process Monitor and access the component page directly
procMonURL.sub! '/psp/', '/psc/'

server_list = agent.get(procMonURL)
scheduler_status = ''

scheduler_status = ['', '', '', 'Down'].join(' | ')
schedulers = server_list.search(".PSLEVEL1GRID").collect do |html|
# Iterate through the Server List grid (but skip the first row - the header)
html.search("tr").collect.drop(1).each do |row|
server      = row.search("td[1]/div/span/a").text.strip
hostname    = row.search("td[2]/div/span").text.strip
last_update = row.search("td[3]/div/span").text.strip
status      = row.search("td[9]/div/span").text.strip

scheduler_status = [server, hostname, last_update, status].join(' | ')
end
end
rescue
scheduler_status = ['', '', '', 'Down'].join(' | ')
end


For the process scheduler, we take the Process Monitor URI and append it to the login URL. In the URL, we pass in ?Page=PMN_SRVRLIST to access to the Server List page. We also substitue /psp/ for the /psc/ servlet. That makes the screen scraping easier since we remove the header frame.

On the Server List page, we grab the grid for the batch servers, drop the header row, and capture the status for each server in the list.

        begin
agent.get(logoutURL)
rescue
end


Don’t forget to log out!

        table = table + "| #{environment} | #{database} | #{web_status} | #{app_status} | #{scheduler_status} |\n"

# If a component is down, add the environment to the affectedEnvironments list
if web_status.include?("Down") || app_status.include?("Down") || scheduler_status.include?("Down")
affectedEnvironments.push(environment)
end
}


Last, we append the status for each component into a string and add it to our Markdown table. If any components for the environment are down, we add that environment to affectedEnvironments.

#### Formatting Output

# Format Markdown table into an HTML table
options = {
filter_html:     true,
link_attributes: { rel: 'nofollow', target: "_blank" },
}

renderer = Redcarpet::Render::HTML.new(options)
markdown = Redcarpet::Markdown.new(renderer, extensions = {tables: true})
tableHTML = markdown.render(table)


This section takes our Markdown table and creates an HTML table.

# Add a style to the "Down" fields
if affectedEnvironments.empty?
tableStyleHTML = tableHTML
else
tableStyleHTML = tableHTML.gsub! '<td>Down</td>', '<td class="down">Down</td>'
end


The HTML table has no styles, so it looks plain. I want to highlight any component that is “Down”. We find any <td> with “Down” as the value and add a the class .down to it.

File.write('table.html', tableStyleHTML)

# Combine the header, table, and footer HTML files into one status HTML file
statusPage = copy /a header.html+table.html+foother.html status.html

deployFile = xcopy status.html #{deployPath} /y


At this point, we write our HTML table to the file table.html. Next, we combine the prebuilt header.html and footer.html files with the updated table.html.

Last, we copy the file to the web location where the status.html file can be viewed.

We have a page with all the links to our envitonments. I added an <iframe> at the bottom of the links page to show the status.html. Anyone who wants to check on an environment can see the status on the links page.

#### Notify

Now for the fun part – sending a notification. We scheduled the script to run every 10 minutes. But, if an environment is down for maintenace or it’s taking us a while to get it back up, I don’t want to get emails every time the script runs. I want the email to go out once each time an environment is down.

# If the environment is newly down, send the email
# If the environment was already down (exists in 'down.txt'), don't resend the email
if affectedEnvironments.empty?

# if no environments are down, delete the 'down.txt' file
if File.exist?('down.txt')
delete = del down.txt
end
else
if File.exist?('down.txt')

affectedEnvironments.each do |env|
if !(downFile.include?(env))
# If both conditions (component down, environment not stored in 'down.txt'), send an email
notify = true
end
end
else # if the file 'down.txt doesn't exist,  the component is newly down
notify = true
end

# Write down environments to file for next status check (will overwrite the existing file)
File.open("down.txt", "w") do |f|
f.puts(affectedEnvironments)
end
end


If an environment is down, the script writes that environment name to a text file, down.txt. Next time the script runs, it compares the environments marked down (in the affectedEnvironments array) to the file contents. If the environment exists in down.txt, we skip notification. If a new environment is down, we send the email.

if notify
mail = Mail.deliver do
subject  'PeopleSoft System Status: ' + affectedEnvironments.join(", ") + ' Down'

# use the markdown table as the text version
text_part do
body = table
end

# use the status.html file as the HTML version
html_part do
content_type 'text/html; charset=UTF-8'
end
end
end # end Notify


Last, create the email and send. I add status.html as the HTML content of the email, and the Markdown table as the plain text version.

### status.bat

The status.bat script is a wrapper to set environment variables and invoke psavailability.rb. The status.bat script is what our Scheduled Task calls.

This is the list of environment variables I set:

set ORACLE_HOME=e:\oracle\product\12.1.0\client_1
set PATH=%ORACLE_HOME%\bin;%PATH%
set PATH=e:\ruby22-x64\bin;%PATH%


After the environment variables are set, we invoke psavailability.rb

e:
cd \
cd psoft\status
ruby ps92availability.rb
`

## Limitations

There are some (many?) limitations in the script. That is because I wrote the script for current situation (Windows/Oracle) and this was something we built as a side project. Here is a list of the limitations that I know of:

1. Windows-only
2. Oracle-only
3. Does not support HTTPS for websites
4. Does not support TLS for SMTP
5. Requires the homepage title to be consistent
6. Doesn’t check the app server or batch server directly
7. If an app server is down, and you have Jolt failover set, you may get a false “Running” status.

The Windows and Oracle limitation wouldn’t be hard to fix. If you want to make any changes, I’d be happy to integrate them into the main project.

## Future Improvements

Here are a list of future improvements that I’ve thought of:

1. Slack integration with slack-ruby-client.
2. Better checks (or second test if web check fails) for database and batch. This could be done via database connections to run SQL statements.
3. Better checks (or second test if web check fails) for app server domains.
4. Add Integration Broker checks (messages in Error or Timeout) via SQL.