PeopleSoft System Status Notifications

Do you get notified if an app server, web server or process scheduler go down? If you do, good for you! If you don’t receive notifications, here is a script that will send you notifications.

If you are an admin who doesn’t have access to monitoring tools (and there are lots of you), this post will go over a script I wrote. The script builds a System Status page and sends email notifications. You can run the script on any machine and it will send an email if anything has gone down. A status.html page is also generated so that users can go check on the status for their system.

Status HTML Page

This page and script is something I put together over a few days as a side project. We wanted a page that would give us the status of each component in an environment. This isn’t the most robust script (I’ll explain the limitations in the post), but I wanted to share it so other people could use it. If you rely on end users to tell if you an environment is down, this script can help you out.

All of the code is hosted on GitHub, so go grab it here.

Install Prerequisites

The script is written in Ruby, uses tnsping, uses Mechanize gem for interacting with Peoplesoft, Markdown for formatting, the Redcarpet gem for generating HTML documents, and the Mail gem for emailing status updates. So we’ll need to install all those parts. It sounds like a lot, but it’s pretty simple.

We’ll walk through the installation process. I’m writing the instructions for Windows, but the Linux steps will be similar.

Oracle Client

The script uses tnsping to check the database status. So, we need the Oracle Client installed on the machine. If you don’t have the Oracle Client software, you can download it here

You also need a tnsnames.ora file with entries for all the databases you want to check. You can place the tnsnames.ora file anywhere on the server. The status.bat script sets the TNS_ADMIN environment variable to point to your tnsnames.ora file.

Ruby Dev Kit

We’ll install Ruby and the Ruby Dev Kit (2.2.4 is what I’m using) on the machine where our scripts will run. Download the Ruby installer from here:

http://rubyinstaller.org/downloads/

I installed Ruby to e:\ruby22-x64 and selected the option to “add executables to the PATH variable”.

Next, download the Ruby DevKit from the same site. The DevKit includes tools to build Gems from source code. We need to the extra tools included with the DevKit. I installed the Ruby DevKit to e:\ruby22-x64-devkit.

Open a new command prompt as an Administrator.

  1. e:
  2. cd ruby22-x64-devkit
  3. ruby dk.rb init
  4. notepad config.yml
  5. Add - e:/ruby22-x64 to the end of the file (notice the dash and forward slash)
  6. Save and close the config.yml file
  7. ruby dk.rb install

Follow the instructions here if you have issues with the DevKit installation.

Gems

Ruby has a powerful package manager called “Gems”. The gem command is part of Ruby, so we can install the extra packages we’ll use for our status page.

Open a new command prompt as an Administrator and we’ll install the Gems.

  1. where ruby

Make sure this command returns the e:\ruby22-x64 folder first.

If it’s not listed first, change the PATH envronment variable so `e:\ruby22-x64\bin\ is first.

  1. gem install mechanize
  2. gem install redcarpet
  3. gem install mail

That’s it for the Gems.

Scripts

There are two scripts in the project:

  1. psavailability.rb
  2. status.bat

The first script, psavailability.rb, is the main script. This is where all the processing happens. The second script, status.bat, is a wrapper to set environment variables like ORACLE_HOME, TNS_ADMIN and PATH.

psavailability.rb

Let’s dive into the psavailability.rb script since that is the main part. As I mentioned before, the script is written in Ruby. I’ll be first person to tell you that I’m not an expert in Ruby. If there places where the code could be improved, let me know.

Status Check Flow

I chose to do all my status checking through the application. This is where Mechanize comes into play. I replicate the actions of an user who opens the login page, logs in, and navigates to the Process Monitor page. The main reason for this method was simplicty. I can do all my checks with one library.

+--------------+            +--------------+            +-------------+
|  Web Server  |  +----->   |  App Server  |  +------>  |  Scheduler  |
|    Check     |            |    Check     |            |    Check    | 
+--------------+            +--------------+            +-------------+

The main disadvantage is: if the web server is down, the app server and process scheduler checks will fail. That may not be accurate from a technical perspective, but from a user perspective it would be true. If you can’t log in, you can do anything!

Setup

I’ve moved any variables that will vary from my install to the top. They are:

# ---------------------------
# Change these variables
# ---------------------------
smtpServer              = '<smtp server>'
statusUser              = '<PeopleSoft Username>'
statusUserPwd           = '<PeopleSoft Password>'
homepageTitleCheck      = '<Homepage Title>'
fromEmailAddress        = '<From email address>'
toEmailAddress          = '<To email address>'
deployPath              = '<e:\\path\\to\\PORTAL.war\\>'
# ---------------------------

The script assumes that you will use the same account to access all environments. I created a new account called STATUS and gave it limited permissions. STATUS can log in and open the Process Monitor Server List page. This way I can track logins from the status script, and we give the service account the least amount of security needed.

Another assumption in the script is that your Homepage Title will have similar text. In our case, we use titles like HR 9.2 Demo, FS 9.2 Test, or ELM 9.2 QA for our environments. I check for 9.2 in the Homepage Title to know if the login was successful.

Initialization

The next section is contains some initialization steps. I set the User-Agent to IE 9, but you can change that if you want.

Mail.defaults do
  delivery_method :smtp, address: smtpServer, port: 25
end

affectedEnvironments = Array.new
notify = false

agent.user_agent_alias = 'Windows IE 9'

Then, I create the Markdown table headers for our table. I found it much easier to create the table in Markdown and then convert the table to HTML at the end.

table =          "| Environment | Database | Web Status | App Status | Scheduler | Batch Server | Update Time | Batch Status |\n"
table = table +  "| ----------- | -------- | ---------- | ---------- | --------- | ------------ | ----------- | ------------ |\n"

Last, I read in the URLs.txt file to get the list of environments and URLs to use. The project on GitHub has a sample URLs.txt file to follow.

# Get the list of environments
# the URLs.txt file is a CSV file with the format "DBNAME,baseURL,processMonitorURI"
agent = Mechanize.new
URLs = CSV.read('URLs.txt', {:col_sep => ','})
URLs.shift # Remove Header Row

Checking Status

URLs.each { |environment, loginURL, prcsURI|  

        web_status = 'Running'
        app_status = 'Running'
        database   = 'Running'

We’ll loop through each environment for the status checks. In our environment, we have 25 “environemnts” (prod and non-prod) that we check. I say “environments” because production has 2 web servers and I check each one.

        begin
            t = `tnsping #{environment}`

            if t.lines.last.include? "OK"
                database = 'Running'
            else
                database = 'Down'
            end
        rescue
            database = 'Down'
        end

To test the database, we run a tnsping command. If the response contains “OK” in the last line the ping was a success. If you are thinking to yourself, “that’s not the best way to test the database”, I agree. But this is a the quicket way to get an Up or Down response. (See the Future Improvements section at the end.)

        # Check web server by opening login page
        begin
            signon_page = agent.get(loginURL + '?cmd=login')
        rescue
            web_status = 'Down'
        end

Next, we attempt to load the login page. If the page responds, we know our web server is up.

        begin 
            signin_form = signon_page.form('login')
            signin_form.userid = statusUser
            signin_form.pwd = statusUserPwd
            homepage = agent.submit(signin_form)

            # We updated PeopleTools > Portal > General Settings to include '9.2' in the title (e.g, "HR 9.2 Test"). 
            # If we see '9.2' in the title, we know the login was successful
            if homepage.title.include? homepageTitleCheck
                app_status = 'Runnning'
            else
                app_status = 'Down'
            end
        rescue
            app_status = 'Down'
        end

To check the app server status, the script logs into the application. We grab the form named login and pass in the PeopleSoft user and password. The page returned from the login attempt is stored in homepage. In our case, every environment has “9.2” in the homepage title. If “9.2” is in the title, I know we have logged in and the app server is up.

The field that holds the homepage title is psprdmdefn.descr254.

        begin
            # Build URL for Process Monitor and access the component page directly
            procMonURL = loginURL + prcsURI
            procMonURL.sub! '/psp/', '/psc/'

            server_list = agent.get(procMonURL)
            scheduler_status = ''

            scheduler_status = ['', '', '', 'Down'].join(' | ')
            schedulers = server_list.search(".PSLEVEL1GRID").collect do |html|
                # Iterate through the Server List grid (but skip the first row - the header)
                html.search("tr").collect.drop(1).each do |row|
                    server      = row.search("td[1]/div/span/a").text.strip
                    hostname    = row.search("td[2]/div/span").text.strip
                    last_update = row.search("td[3]/div/span").text.strip
                    status      = row.search("td[9]/div/span").text.strip

                    scheduler_status = [server, hostname, last_update, status].join(' | ')
                end
            end
        rescue
            scheduler_status = ['', '', '', 'Down'].join(' | ')
        end

For the process scheduler, we take the Process Monitor URI and append it to the login URL. In the URL, we pass in ?Page=PMN_SRVRLIST to access to the Server List page. We also substitue /psp/ for the /psc/ servlet. That makes the screen scraping easier since we remove the header frame.

On the Server List page, we grab the grid for the batch servers, drop the header row, and capture the status for each server in the list.

        begin
            logoutURL = loginURL + '?cmd=logout'
            agent.get(logoutURL)
        rescue
        end

Don’t forget to log out!

        table = table + "| #{environment} | #{database} | #{web_status} | #{app_status} | #{scheduler_status} |\n"

        # If a component is down, add the environment to the affectedEnvironments list
        if web_status.include?("Down") || app_status.include?("Down") || scheduler_status.include?("Down")
            affectedEnvironments.push(environment)
        end
}

Last, we append the status for each component into a string and add it to our Markdown table. If any components for the environment are down, we add that environment to affectedEnvironments.

Formatting Output

# Format Markdown table into an HTML table
options = {
  filter_html:     true,
  link_attributes: { rel: 'nofollow', target: "_blank" },
  space_after_headers: true
}

renderer = Redcarpet::Render::HTML.new(options)
markdown = Redcarpet::Markdown.new(renderer, extensions = {tables: true})
tableHTML = markdown.render(table)

This section takes our Markdown table and creates an HTML table.

# Add a style to the "Down" fields
if affectedEnvironments.empty?
    tableStyleHTML = tableHTML
else
    tableStyleHTML = tableHTML.gsub! '<td>Down</td>', '<td class="down">Down</td>'
end

The HTML table has no styles, so it looks plain. I want to highlight any component that is “Down”. We find any <td> with “Down” as the value and add a the class .down to it.

File.write('table.html', tableStyleHTML)

# Combine the header, table, and footer HTML files into one status HTML file
statusPage = `copy /a header.html+table.html+foother.html status.html`

deployFile = `xcopy status.html #{deployPath} /y`

At this point, we write our HTML table to the file table.html. Next, we combine the prebuilt header.html and footer.html files with the updated table.html.

Last, we copy the file to the web location where the status.html file can be viewed.

We have a page with all the links to our envitonments. I added an <iframe> at the bottom of the links page to show the status.html. Anyone who wants to check on an environment can see the status on the links page.

Notify

Now for the fun part – sending a notification. We scheduled the script to run every 10 minutes. But, if an environment is down for maintenace or it’s taking us a while to get it back up, I don’t want to get emails every time the script runs. I want the email to go out once each time an environment is down.

# If the environment is newly down, send the email
# If the environment was already down (exists in 'down.txt'), don't resend the email
if affectedEnvironments.empty?

    # if no environments are down, delete the 'down.txt' file
    if File.exist?('down.txt')
        delete = `del down.txt`
    end
else
    if File.exist?('down.txt')
        downFile = File.read("down.txt")

        affectedEnvironments.each do |env|
            if !(downFile.include?(env))
                # If both conditions (component down, environment not stored in 'down.txt'), send an email
                notify = true
            end
        end
    else # if the file 'down.txt doesn't exist,  the component is newly down
        notify = true
    end

    # Write down environments to file for next status check (will overwrite the existing file)
    File.open("down.txt", "w") do |f|
      f.puts(affectedEnvironments)
    end
end 

If an environment is down, the script writes that environment name to a text file, down.txt. Next time the script runs, it compares the environments marked down (in the affectedEnvironments array) to the file contents. If the environment exists in down.txt, we skip notification. If a new environment is down, we send the email.

if notify
    mail = Mail.deliver do
      from     fromEmailAddress
      to       toEmailAddress
      subject  'PeopleSoft System Status: ' + affectedEnvironments.join(", ") + ' Down'

      # use the markdown table as the text version
      text_part do 
        body = table
      end

      # use the status.html file as the HTML version
      html_part do
        content_type 'text/html; charset=UTF-8'
        body File.read('status.html')
      end
    end 
end # end Notify

Last, create the email and send. I add status.html as the HTML content of the email, and the Markdown table as the plain text version.

status.bat

The status.bat script is a wrapper to set environment variables and invoke psavailability.rb. The status.bat script is what our Scheduled Task calls.

This is the list of environment variables I set:

set ORACLE_HOME=e:\oracle\product\12.1.0\client_1
set TNS_ADMIN=%ORACLE_HOME%\network\admin
set PATH=%ORACLE_HOME%\bin;%PATH%
set PATH=e:\ruby22-x64\bin;%PATH%

After the environment variables are set, we invoke psavailability.rb

e:
cd \
cd psoft\status
ruby ps92availability.rb 

Limitations

There are some (many?) limitations in the script. That is because I wrote the script for current situation (Windows/Oracle) and this was something we built as a side project. Here is a list of the limitations that I know of:

  1. Windows-only
  2. Oracle-only
  3. Does not support HTTPS for websites
  4. Does not support TLS for SMTP
  5. Requires the homepage title to be consistent
  6. Doesn’t check the app server or batch server directly
  7. If an app server is down, and you have Jolt failover set, you may get a false “Running” status.

The Windows and Oracle limitation wouldn’t be hard to fix. If you want to make any changes, I’d be happy to integrate them into the main project.

Future Improvements

Here are a list of future improvements that I’ve thought of:

  1. Slack integration with slack-ruby-client.
  2. Better checks (or second test if web check fails) for database and batch. This could be done via database connections to run SQL statements.
  3. Better checks (or second test if web check fails) for app server domains.
  4. Add Integration Broker checks (messages in Error or Timeout) via SQL.
  5. Add HTTPS support to test Load Balancer URLs.
  6. Add TLS support to SMTP

The project is hosted on GitHub. I’ll merge any pull requests that people want to send.

As I mentioned earlier, this script started out as a way to get notifications. This was a “scratch your itch” type of project. But, if you want to use the script or improve it, you can get all the code over on GitHub.

19 thoughts on “PeopleSoft System Status Notifications

  1. Thanks for sharing this! I am not familiar with Ruby so I was wondering if you could tell me the best way to enable logging? I have it getting about halfway through my list of servers and then it shows everything as down.

    1. Hi Dale

      Ruby doesn’t the same tracing functionality as PeopleCode (which actually quite nice). But, there are a few tools we can use to get Ruby trace and debug logs. Try this first: irb psavailability.rb > trace.log

      This will create a trace.log file where you can see the values used in the code. One issue with this, the main loop URLs.each { |environment, loginURL, prcsURI| is only printed once. But, above that line you can find the array with all the URLs from the text file.

      Another option is ruby -rtracer psavailability.rb > tracer.log. This will give you LOTS of data (I think too much), but it will show the full execution of the script in the file tracer.log.

      One more option is to modify the script to enable tracing at specific lines. To add in tracing,

      1. Add require 'tracer' at the top of the file (with the other require statements).
      2. Add Tracer.on before the code you want to trace.
      3. Add Tracer.off after the code you want to trace.

      When we built the script we had this issue too. My initial guess is that the URLs aren’t quite right for a set of environments. That’s what happened to us – we copy/pasted the URLs and missed some Node changes 😉 If you want to paste a line or two of your URLs.txt file (where it works and then doesn’t work) we might be able to find something.

  2. Thanks for the suggestions so far no luck. You are right in that -rtracer is to much information! What I have figured out is that it fails on the 11th one always. I can rearrange the URLs in the list and it will complete previously failed ones but 11+ will fail every time.

    1. Yes, this should work with 8.54. If your database name is HRDMP, then the line in URLs.txt would be:

      HRDMP,https://apps.oii.oceaneering.com/psp/HRDMP/,EMPLOYEE/HRMS/c/PROCESSMONITOR.PROCESSMONITOR.GBL?Page=PMN_SRVRLIST
      
  3. I was able to get some nice logs with the following added to the psavailability.rb file.

    require 'logger'

    log = Logger.new('logger.txt')
    log.level = Logger::INFO

    agent.log = log

    Here is the relevant line in the URLs.txt file:

    STRQ,https://www.cnd.nd.gov/psp/strq/,EMPLOYEE/EMPL/c/PROCESSMONITOR.PROCESSMONITOR.GBL?Page=PMN_SRVRLIST

    I found that I was getting 400 errors. I haven’t found out if they are valid yet but this is better than what I was finding. I am not sure if it’s the /EMPL/s/ in the redirect that is blame or not.

    I, [2016-03-15T16:31:19.256368 #8916] INFO -- : Net::HTTP::Get: /psp/strq/?cmd=login
    I, [2016-03-15T16:31:19.288766 #8916] INFO -- : status: Net::HTTPOK 1.1 200 OK
    I, [2016-03-15T16:31:19.292745 #8916] INFO -- : form encoding: utf-8
    I, [2016-03-15T16:31:19.293747 #8916] INFO -- : Net::HTTP::Post: /psp/strq/?&cmd=login&languageCd=ENG
    I, [2016-03-15T16:31:19.400730 #8916] INFO -- : status: Net::HTTPFound 1.1 302 Moved Temporarily
    I, [2016-03-15T16:31:19.402681 #8916] INFO -- : follow redirect to: https://www.cnd.nd.gov/psc/strq/EMPLOYEE/EMPL/s/WEBLIB_PTBR.ISCRIPT1.FieldFormula.IScript_StartPage
    I, [2016-03-15T16:31:19.403681 #8916] INFO -- : Net::HTTP::Get: /psc/strq/EMPLOYEE/EMPL/s/WEBLIB_PTBR.ISCRIPT1.FieldFormula.IScript_StartPage
    I, [2016-03-15T16:31:19.415724 #8916] INFO -- : status: Net::HTTPBadRequest 1.1 400 Bad Request
    I, [2016-03-15T16:31:19.417032 #8916] INFO -- : Net::HTTP::Get: /psc/strq/EMPLOYEE/EMPL/c/PROCESSMONITOR.PROCESSMONITOR.GBL?Page=PMN_SRVRLIST
    I, [2016-03-15T16:31:19.455260 #8916] INFO -- : status: Net::HTTPBadRequest 1.1 400 Bad Request
    I, [2016-03-15T16:31:19.457210 #8916] INFO -- : Net::HTTP::Get: /psp/strq/?cmd=logout

    1. Dale, the logger module looks really nice. A couple of questions:

      1. Is this the first (or only) HTTPS environment?
      2. The node name in your URL is EMPL. Not sure what application your hitting, but the default is usually HRMS for HR, ERP for Finance, etc. I noticed your node was different.
      3. If you log in to the environment and then paste this URL in the browser, do you go to the page? https://www.cnd.nd.gov/psc/strq/EMPLOYEE/EMPL/s/WEBLIB_PTBR.ISCRIPT1.FieldFormula.IScript_StartPage
  4. Dan,

    No, all of my environments are https. I added agent.verify_mode = OpenSSL::SSL::VERIFY_NONE to the script.
    EMPL is Portal (Interaction Hub)
    Pasting that URL in after logging in takes me to this URL which seems to indicate that the URL is valid.
    https://www.cnd.nd.gov/psp/strq/EMPLOYEE/EMPL/h/?tab=NDS_EMPLOYEE_HUB

    1. Ah, forgot about the Portal.

      Try this – let’s write the homepage variable to file so we can see what HTML the page returns when the script attempts to log in. I added in log.info(homepage) for app servers that are down.

              if homepage.title.include? homepageTitleCheck
                  app_status = 'Runnning'
              else
                  app_status = 'Down'
                  log.info(homepage)
              end
      

      I’m curious what the server returns for HTML when the script thinks it’s down.

  5. Looks like it’s not seeing anything at the time of the errors.
    I, [2016-03-17T10:14:27.879620 #6216] INFO -- : nil

    1. Well, that’s not useful!

      Is this still happening with the 11th environment, or this specific environment?

  6. So we had the issue of all environments were showing down after a certain one. If your default local node has the same Username and password in all environments we didn’t have an issue. Our demo and production environments do. I resolved my issue by clearing cookies.

    agent.get(logoutURL)
    agent.cookie_jar.clear!

      1. Dan,
        Thank you for the script. I’m currently using the script for our environments with a few minor mods. It has proven to be very helpful.
        Mod #1
        Our environments have two process schedulers per instance one on Linux and one on Windows and I wanted to show that all on one line
        Mod #2
        In the footer I included a date time stamp so you had an idea of when the status.html file was last generated.

        I schedule the script to run on our Windows Test Process Scheduler box and I copy the status.html to another Windows server running a simple web server via IIS. So now our team can just click on a link and get this information and of course I get the emails when something comes down.
        Thank you again!

        Eric

        1. Hi Eric,

          Can you share how you added the second line of process schedulers within the same script?

          We have the same scenario with two process schedulers NT and UNIX running but the report only shows one of them.

          Hamza

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax