Do you get notified if an app server, web server or process scheduler go down? If you do, good for you! If you don’t receive notifications, here is a script that will send you notifications.

If you are an admin who doesn’t have access to monitoring tools (and there are lots of you), this post will go over a script I wrote. The script builds a System Status page and sends email notifications. You can run the script on any machine and it will send an email if anything has gone down. A status.html page is also generated so that users can go check on the status for their system.

This page and script is something I put together over a few days as a side project. We wanted a page that would give us the status of each component in an environment. This isn’t the most robust script (I’ll explain the limitations in the post), but I wanted to share it so other people could use it. If you rely on end users to tell if you an environment is down, this script can help you out.

All of the code is hosted on GitHub, so go grab it here.

Install Prerequisites

The script is written in Ruby, uses tnsping, uses Mechanize gem for interacting with Peoplesoft, Markdown for formatting, the Redcarpet gem for generating HTML documents, and the Mail gem for emailing status updates. So we’ll need to install all those parts. It sounds like a lot, but it’s pretty simple.

We’ll walk through the installation process. I’m writing the instructions for Windows, but the Linux steps will be similar.

Oracle Client

The script uses tnsping to check the database status. So, we need the Oracle Client installed on the machine. If you don’t have the Oracle Client software, you can download it here

You also need a tnsnames.ora file with entries for all the databases you want to check. You can place the tnsnames.ora file anywhere on the server. The status.bat script sets the TNS_ADMIN environment variable to point to your tnsnames.ora file.

Ruby Dev Kit

We’ll install Ruby and the Ruby Dev Kit (2.2.4 is what I’m using) on the machine where our scripts will run. Download the Ruby installer from here:

http://rubyinstaller.org/downloads/

I installed Ruby to e:\ruby22-x64 and selected the option to “add executables to the PATH variable”.

Next, download the Ruby DevKit from the same site. The DevKit includes tools to build Gems from source code. We need to the extra tools included with the DevKit. I installed the Ruby DevKit to e:\ruby22-x64-devkit.

Open a new command prompt as an Administrator.

1. e:
2. cd ruby22-x64-devkit
3. ruby dk.rb init
4. notepad config.yml
5. Add - e:/ruby22-x64 to the end of the file (notice the dash and forward slash)
6. Save and close the config.yml file
7. ruby dk.rb install

Follow the instructions here if you have issues with the DevKit installation.

Gems

Ruby has a powerful package manager called “Gems”. The gem command is part of Ruby, so we can install the extra packages we’ll use for our status page.

Open a new command prompt as an Administrator and we’ll install the Gems.

1. where ruby

Make sure this command returns the e:\ruby22-x64 folder first.

If it’s not listed first, change the PATH envronment variable so `e:\ruby22-x64\bin\ is first.

1. gem install mechanize
2. gem install redcarpet
3. gem install mail

That’s it for the Gems.

Scripts

There are two scripts in the project:

1. psavailability.rb
2. status.bat

The first script, psavailability.rb, is the main script. This is where all the processing happens. The second script, status.bat, is a wrapper to set environment variables like ORACLE_HOME, TNS_ADMIN and PATH.

psavailability.rb

Let’s dive into the psavailability.rb script since that is the main part. As I mentioned before, the script is written in Ruby. I’ll be first person to tell you that I’m not an expert in Ruby. If there places where the code could be improved, let me know.

Status Check Flow

I chose to do all my status checking through the application. This is where Mechanize comes into play. I replicate the actions of an user who opens the login page, logs in, and navigates to the Process Monitor page. The main reason for this method was simplicty. I can do all my checks with one library.

+--------------+            +--------------+            +-------------+
|  Web Server  |  +----->   |  App Server  |  +------>  |  Scheduler  |
|    Check     |            |    Check     |            |    Check    | 
+--------------+            +--------------+            +-------------+

The main disadvantage is: if the web server is down, the app server and process scheduler checks will fail. That may not be accurate from a technical perspective, but from a user perspective it would be true. If you can’t log in, you can do anything!

Setup

I’ve moved any variables that will vary from my install to the top. They are:

# ---------------------------
# Change these variables
# ---------------------------
smtpServer              = '<smtp server>'
statusUser              = '<PeopleSoft Username>'
statusUserPwd           = '<PeopleSoft Password>'
homepageTitleCheck      = '<Homepage Title>'
fromEmailAddress        = '<From email address>'
toEmailAddress          = '<To email address>'
deployPath              = '<e:\\path\\to\\PORTAL.war\\>'
# ---------------------------

The script assumes that you will use the same account to access all environments. I created a new account called STATUS and gave it limited permissions. STATUS can log in and open the Process Monitor Server List page. This way I can track logins from the status script, and we give the service account the least amount of security needed.

Another assumption in the script is that your Homepage Title will have similar text. In our case, we use titles like HR 9.2 Demo, FS 9.2 Test, or ELM 9.2 QA for our environments. I check for 9.2 in the Homepage Title to know if the login was successful.

Initialization

The next section is contains some initialization steps. I set the User-Agent to IE 9, but you can change that if you want.

Mail.defaults do
  delivery_method :smtp, address: smtpServer, port: 25
end

affectedEnvironments = Array.new
notify = false

agent.user_agent_alias = 'Windows IE 9'

Then, I create the Markdown table headers for our table. I found it much easier to create the table in Markdown and then convert the table to HTML at the end.

table =          "| Environment | Database | Web Status | App Status | Scheduler | Batch Server | Update Time | Batch Status |\n"
table = table +  "| ----------- | -------- | ---------- | ---------- | --------- | ------------ | ----------- | ------------ |\n"

Last, I read in the URLs.txt file to get the list of environments and URLs to use. The project on GitHub has a sample URLs.txt file to follow.

# Get the list of environments
# the URLs.txt file is a CSV file with the format "DBNAME,baseURL,processMonitorURI"
agent = Mechanize.new
URLs = CSV.read('URLs.txt', {:col_sep => ','})
URLs.shift # Remove Header Row

Checking Status

URLs.each { |environment, loginURL, prcsURI|  

        web_status = 'Running'
        app_status = 'Running'
        database   = 'Running'

We’ll loop through each environment for the status checks. In our environment, we have 25 “environemnts” (prod and non-prod) that we check. I say “environments” because production has 2 web servers and I check each one.

        begin
            t = `tnsping #{environment}`

            if t.lines.last.include? "OK"
                database = 'Running'
            else
                database = 'Down'
            end
        rescue
            database = 'Down'
        end

To test the database, we run a tnsping command. If the response contains “OK” in the last line the ping was a success. If you are thinking to yourself, “that’s not the best way to test the database”, I agree. But this is a the quicket way to get an Up or Down response. (See the Future Improvements section at the end.)

        # Check web server by opening login page
        begin
            signon_page = agent.get(loginURL + '?cmd=login')
        rescue
            web_status = 'Down'
        end

Next, we attempt to load the login page. If the page responds, we know our web server is up.

        begin 
            signin_form = signon_page.form('login')
            signin_form.userid = statusUser
            signin_form.pwd = statusUserPwd
            homepage = agent.submit(signin_form)

            # We updated PeopleTools > Portal > General Settings to include '9.2' in the title (e.g, "HR 9.2 Test"). 
            # If we see '9.2' in the title, we know the login was successful
            if homepage.title.include? homepageTitleCheck
                app_status = 'Runnning'
            else
                app_status = 'Down'
            end
        rescue
            app_status = 'Down'
        end

To check the app server status, the script logs into the application. We grab the form named login and pass in the PeopleSoft user and password. The page returned from the login attempt is stored in homepage. In our case, every environment has “9.2” in the homepage title. If “9.2” is in the title, I know we have logged in and the app server is up.

The field that holds the homepage title is psprdmdefn.descr254.

        begin
            # Build URL for Process Monitor and access the component page directly
            procMonURL = loginURL + prcsURI
            procMonURL.sub! '/psp/', '/psc/'

            server_list = agent.get(procMonURL)
            scheduler_status = ''

            scheduler_status = ['', '', '', 'Down'].join(' | ')
            schedulers = server_list.search(".PSLEVEL1GRID").collect do |html|
                # Iterate through the Server List grid (but skip the first row - the header)
                html.search("tr").collect.drop(1).each do |row|
                    server      = row.search("td[1]/div/span/a").text.strip
                    hostname    = row.search("td[2]/div/span").text.strip
                    last_update = row.search("td[3]/div/span").text.strip
                    status      = row.search("td[9]/div/span").text.strip

                    scheduler_status = [server, hostname, last_update, status].join(' | ')
                end
            end
        rescue
            scheduler_status = ['', '', '', 'Down'].join(' | ')
        end

For the process scheduler, we take the Process Monitor URI and append it to the login URL. In the URL, we pass in ?Page=PMN_SRVRLIST to access to the Server List page. We also substitue /psp/ for the /psc/ servlet. That makes the screen scraping easier since we remove the header frame.

On the Server List page, we grab the grid for the batch servers, drop the header row, and capture the status for each server in the list.

        begin
            logoutURL = loginURL + '?cmd=logout'
            agent.get(logoutURL)
        rescue
        end

Don’t forget to log out!

        table = table + "| #{environment} | #{database} | #{web_status} | #{app_status} | #{scheduler_status} |\n"

        # If a component is down, add the environment to the affectedEnvironments list
        if web_status.include?("Down") || app_status.include?("Down") || scheduler_status.include?("Down")
            affectedEnvironments.push(environment)
        end
}

Last, we append the status for each component into a string and add it to our Markdown table. If any components for the environment are down, we add that environment to affectedEnvironments.

Formatting Output

# Format Markdown table into an HTML table
options = {
  filter_html:     true,
  link_attributes: { rel: 'nofollow', target: "_blank" },
  space_after_headers: true
}

renderer = Redcarpet::Render::HTML.new(options)
markdown = Redcarpet::Markdown.new(renderer, extensions = {tables: true})
tableHTML = markdown.render(table)

This section takes our Markdown table and creates an HTML table.

# Add a style to the "Down" fields
if affectedEnvironments.empty?
    tableStyleHTML = tableHTML
else
    tableStyleHTML = tableHTML.gsub! '<td>Down</td>', '<td class="down">Down</td>'
end

The HTML table has no styles, so it looks plain. I want to highlight any component that is “Down”. We find any <td> with “Down” as the value and add a the class .down to it.

File.write('table.html', tableStyleHTML)

# Combine the header, table, and footer HTML files into one status HTML file
statusPage = `copy /a header.html+table.html+foother.html status.html`

deployFile = `xcopy status.html #{deployPath} /y`

At this point, we write our HTML table to the file table.html. Next, we combine the prebuilt header.html and footer.html files with the updated table.html.

Last, we copy the file to the web location where the status.html file can be viewed.

We have a page with all the links to our envitonments. I added an <iframe> at the bottom of the links page to show the status.html. Anyone who wants to check on an environment can see the status on the links page.

Notify

Now for the fun part – sending a notification. We scheduled the script to run every 10 minutes. But, if an environment is down for maintenace or it’s taking us a while to get it back up, I don’t want to get emails every time the script runs. I want the email to go out once each time an environment is down.

# If the environment is newly down, send the email
# If the environment was already down (exists in 'down.txt'), don't resend the email
if affectedEnvironments.empty?

    # if no environments are down, delete the 'down.txt' file
    if File.exist?('down.txt')
        delete = `del down.txt`
    end
else
    if File.exist?('down.txt')
        downFile = File.read("down.txt")

        affectedEnvironments.each do |env|
            if !(downFile.include?(env))
                # If both conditions (component down, environment not stored in 'down.txt'), send an email
                notify = true
            end
        end
    else # if the file 'down.txt doesn't exist,  the component is newly down
        notify = true
    end

    # Write down environments to file for next status check (will overwrite the existing file)
    File.open("down.txt", "w") do |f|
      f.puts(affectedEnvironments)
    end
end 

If an environment is down, the script writes that environment name to a text file, down.txt. Next time the script runs, it compares the environments marked down (in the affectedEnvironments array) to the file contents. If the environment exists in down.txt, we skip notification. If a new environment is down, we send the email.

if notify
    mail = Mail.deliver do
      from     fromEmailAddress
      to       toEmailAddress
      subject  'PeopleSoft System Status: ' + affectedEnvironments.join(", ") + ' Down'

      # use the markdown table as the text version
      text_part do 
        body = table
      end

      # use the status.html file as the HTML version
      html_part do
        content_type 'text/html; charset=UTF-8'
        body File.read('status.html')
      end
    end 
end # end Notify

Last, create the email and send. I add status.html as the HTML content of the email, and the Markdown table as the plain text version.

status.bat

The status.bat script is a wrapper to set environment variables and invoke psavailability.rb. The status.bat script is what our Scheduled Task calls.

This is the list of environment variables I set:

set ORACLE_HOME=e:\oracle\product\12.1.0\client_1
set TNS_ADMIN=%ORACLE_HOME%\network\admin
set PATH=%ORACLE_HOME%\bin;%PATH%
set PATH=e:\ruby22-x64\bin;%PATH%

After the environment variables are set, we invoke psavailability.rb

e:
cd \
cd psoft\status
ruby ps92availability.rb 

Limitations

There are some (many?) limitations in the script. That is because I wrote the script for current situation (Windows/Oracle) and this was something we built as a side project. Here is a list of the limitations that I know of:

1. Windows-only
2. Oracle-only
3. Does not support HTTPS for websites
4. Does not support TLS for SMTP
5. Requires the homepage title to be consistent
6. Doesn’t check the app server or batch server directly
7. If an app server is down, and you have Jolt failover set, you may get a false “Running” status.

The Windows and Oracle limitation wouldn’t be hard to fix. If you want to make any changes, I’d be happy to integrate them into the main project.

Future Improvements

Here are a list of future improvements that I’ve thought of:

1. Slack integration with slack-ruby-client.
2. Better checks (or second test if web check fails) for database and batch. This could be done via database connections to run SQL statements.
3. Better checks (or second test if web check fails) for app server domains.
4. Add Integration Broker checks (messages in Error or Timeout) via SQL.
5. Add HTTPS support to test Load Balancer URLs.
6. Add TLS support to SMTP

The project is hosted on GitHub. I’ll merge any pull requests that people want to send.

As I mentioned earlier, this script started out as a way to get notifications. This was a “scratch your itch” type of project. But, if you want to use the script or improve it, you can get all the code over on GitHub.