#173 – Trace Wizard


This week on the podcast, Kyle and Dan discuss Integration Broker Queue Partitioning and why you might want to use it. Dan shares an edge case with Change Assistants new modes, and then Dan and Kyle talk about the wonderful tool Trace Wizard.

Show Notes

8.55 – Log Analyzer

The Log Analyzer is a new PeopleTools 8.55 tool that helps you get more information from App Engine trace files. Log Analyzer is specific to analyzing App Engine traces. It feels like a simplified version of TraceMagic (a good thing!). If you have used TraceMagic for PeopleCode, or the Trace2SQL utility to parse .tracesql files, Log Analyzer feels like the two tools combined.

Trace Settings

You need to set the App Engine trace settings to at least these values in the psprcs.cfg file for the Log Analyzer to work:

  • Trace SQL: 3
  • Trace PC: 64

If you are using Config Manager, that translates to these options:

PeopleCode Trace

  • Trace Start of Programs

App Engine Trace

  • Step
  • SQL

Last, you can set them at the process level:

  • -TRACE 3 -TOOLSTRACEPC 64

You can use higher trace values, but these are the minimum for the Log Analyzer to correctly parse your files. Make sure to specify the App Engine SQL trace. If you select the PeopleTools SQL trace, you won’t get an *.AET file when the app engine runs. You can generate a SQL trace (.tracesql) and an App Engine trace (.AET). But at a minimum, you need an *.AET file. Log Analyzer also uses the .TRC PeopleCode trace file.

Log Analyzer GUI

The Log Analyzer tool is a separate java application located under PS_HOME\bin\client\winx86\AELogAnalyzerTool. To launch the tool, run the runTraceAnalyzer.bat batch script. After launching the tool, you’ll see an empty screen. You need to load in an *.AET file.

Screen Shot 2015-12-30 at 1.38.48 PM

Once you load an *.AET trace file, the Log Analyzer will display the heirarchy of the trace program in the lower window. You can expand each section and click on the SQL steps. To view the associated SQL and parameters, click on the “View Details” button. You can also look at the trace in execution order (without the parent/child nesting) by clicking on the “Linearize” button.

Load PeopleCode

If you generate a PeopleCode .TRC trace file, you can import the .TRC file into Log Analyzer. If the trace file has the “Trace Start of Programs” lines (value 64), Log Analyzer will correlate the PeopleCode statements to the app engine step. In the lower window, you can click on the PeopleCode step and view the PeopleCode much like the SQL.

Apply Filters

Log Analyzer has different ways to filter and view your trace file. There are four options to view the different SQL operations: insert, select, update, delete. You can also filter the log output based on a text value, the duration of the step (useful for tuning), or by the number of rows updated.


Log Analyzer feels like a new tool. It functions are still limited (you can filter, but not search), but I think it will be a great tool to add for admins and developers. If you have used TraceMagic, Log Analyzer will be easy to pick up.

SES Troubleshooting

The SES, Secure Enterprise Search, is used by PeopleSoft 9.2 as it’s search engine. There are lots of great resources for setting up the SES, but I want to share one troubleshooting tip that has resolved many of our SES issues.

Behind the scene, SES relies on the Feed Publishing Framework to get data from PeopleSoft. The Feed Publishing Framework takes a query and generates an RSS feed. That RSS feed is what the SES reads to get new data.

Check the Feed!

When you run into issues setting up the SES, one “trick” I use is to view the feed directly. I say “trick” because this might be unknown or forgotten by people, but we are only the RSS feed with the SES user account. By looking at the feed in a browser, we can immediately identify security issues or bad configuration.

Viewing the Feed

To view the feed, we need to log into the SES console and get the URL for the search index, and the User ID that SES will use to access the feed. For the search index URL,
click on the “Sources” tab and select the “Edit” option for the search index you want to test. In the “User Defined Source” page, you will see a URL in the “Configuration URL” box. Copy this URL into a new browser session (I use a Chrome Private Browsing session; want to log in as a new user and don’t want to mix our browser cookies with any you have stored).

After you paste the URL into a new browser window and hit Enter, you are prompted for a username and password. Enter the username from the “User Defined Source” page, and it’s appropriate password. You do not want to log in with your personal account; the purpose of this exercise is to log with the account the SES uses.

After you authenticate as the SES user, you will see an RSS feed (an XML file with special tags). In the RSS file, you should see a feedLocation tag with a URL inside. Copy this new URL and paste it into a new tab in your browser (I use a new tab because I want the new URL to use our SES user cookie).

After you paste the new URL and hit Enter, you should see another RSS file. Depending on the index you picked, and how many times it has been crawled, you may see one or more entries in the feed. For our testing, find the first link tag and copy the URL inside that entry. Copy the URL and paste it into a new tab in your browser.

After you paste the URL and hit Enter, you should see a final RSS feed. During my testing, this last page is where you will most likely encounter errors. The last file is actual data PeopleSoft publishes to the SES with the Feed Publishing Framework. If you can’t access the last RSS URL, you mostly likely have a security issue for the SES user account.

You can repeat this process using your personal account, or the account of a super user to help identify what security is missing. If you can view the RSS file using the SES account, then your SES account has proper security. The steps we followed to open the URLs is what the SES follows when it tries to crawl the feed. If you run into any errors opening feeds, the SES will encounter the same errors.

SMTP Telnet Test

We are standing up a new data center and users received SMTP errors while testing. We were able to ping the server so DNS lookup and network traffic was working. But we only had a generic SMTP error message. So, I fired up telnet and tested our SMTP connection using these commands:

telnet
o smtp.psadmin.io
ehlo
mail from:ps@psadmin.io
rcpt to:user@psadmin.io
data
Test Email
.

When we first tested send emails through the SMTP server, we received an “Unable to relay for user@psadmin.io” error. That’s much more descriptive and a useful message to send to the Exchange administrators.

PeopleSoft Instrumented Development Diagnostic Aid (IDDA)

I stumbled across this page about the PeopleSoft IDDA logger in PeopleBooks today. I haven’t seen this tool before, so I wanted to give it a quick test to see how it works. I wanted to know what type of data the IDDA logged and if the data would be useful for troubleshooting issues. After setting it up in a Development environment, I found that the PIA, Portal and Authentication traces would be helpful for sign on or cookie issues.

Enabling IDDA

You enabled the IDDA in a Web Profile Custom Property and by changing the IDDA log settings in logging.properties on your web server.

Web Profile

Go to your active Profile and add this custom value:

  • Property Name: IDDA
  • Validation Type: Number
  • Property Value: The bit value, or sum of bit values, for the area you want to trace.

If you want to trace the PIA and Report Repository, enter a value of 5.

Bit Value Functional Category
1 PeopleSoft Internet Architecture
2 Integration Broker
4 Report repository
8 Portal
16 Web server caching
32 File processing (attachments)
64 Authentication
128 Performance Monitor
256 Web Services for Remote Portlets (WSRP)
512 Jolt

Save the Web Profile changes.

Logging Properties

Open the logging.properties file under webserv\domain\piaconfig\properties and change the .LEVEL value. These are the possibles values:

  • OFF: Disables all IDDA logging.
  • SEVERE: Displays server issues, such as premature termination and failure.
  • WARNING: Displays less severe issues, such as configuration issues.
  • INFO: Displays basic operational information, such as starting and stopping.
  • FINE, FINER, FINEST: Displays internal non-critical informational messages.
  • ALL: Enables all logging levels. The default value is INFO.

I tried ALL, FINEST and INFO during my testing. I ended up sticking with FINER since I felt that gave the best detail without too much information.

PeopleBooks has the wrong path documented if you look there. You can find the location of the logging.properties file by opening setEnv.cmd and looking at the PSLOGGING variable path. Save the changes and reboot your web server.

Viewing IDDA Data

IDDA output will write to webserv\domain\servers\PIA\logs\PIA_servlets.%u.log. The output is plain text so you view it in any text editor. The output is also tab-delimited, so importing the log files into Excel or another log view is easy.

TraceMagic

TraceMagic is a utility that gives PeopleSoft system administrators, programmers and support engineers the ability to quickly isolate performance bottlenecks in SQL Statements and/or PeopleCode functions. It accomplishes this by turning the text-based, time-ordered tracesql file into a sortable-grid display, allowing the user to quickly locate system performance issues.

TraceMagic is a great tool for reading .tracesql files. The ability to group SQL by SELECT statements and see all the tables queried in the trace are great for debugging issues.