by Dan Iverson

#58 – Pagelets and Complaints

This week on the podcast, Dan and Kyle talk about a ransomware attack, load balancer health checks, applying POC patches and complain about minor annoyances in Change Assistant. Kyle shares a story about a misbehaving pagelet and how he investigated the issue.

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

by Dan Iverson

Load Balancers and Client IP Addresses

One disadvantage of running PeopleSoft behind a load balancer is that it can hide your user’s real IP address. With a load balancer, WebLogic and PeopleSoft log the load balancer’s IP address instead of the client connecting to your system. Load balancers support an HTTP header called X-Forwarded-For, and will populate that HTTP header with the client’s IP address. We can configure WebLogic to read this header and add that to our HTTP access log.

The psaccesslog table tracks logins into PeopleSoft, but a customization is required to change the logged IP address to read from X-Forwarded-For. We won’t over that here. Instead, we’ll show a configuration change to capture this information.

X-Forwarded-For

The X-Forwarded-For HTTP header is designed to pass in the HTTP originating client IP address through the load balancer. Logging a client IP address helps you identify where users are coming from (geo-ip) and can help with auditing access to your system. Since organizations run many different load balancers, we won’t cover how to enable that. Ask your network administrators to enable and populate X-Forwarded-For and pass it through to your WebLogic servers.

If you want to test the logging without getting a network admin involved, FireFox has a nice add-on, X-Forwarded-For Header that simulates the header for you.

WebLogic Access Log

Out of the box, WebLogic doesn’t capture the X-Forwarded-For HTTP header. To read that header, we can configure the HTTP Access Log to read the header and log it. To do that, we need to enable the Access Log and tell WebLogic what header to look for.

EnableXFF

  1. Open http://server:port/console and login.
  2. Go to “Environment > Servers > PIA > Logging > HTTP”.
  3. Click “Lock & Edit”.
  4. Select the checkbox for “HTTP access log file enabled”.
  5. Save the changes.
  6. Expand the “Advanced” section.
  7. Change the Format to Extended.
  8. Add cs(X-Forwarded-For) to the Extended Logging Format Fields.
  9. Set the Log File Buffer to 0. (This will write entires immediately to the log file.)
  10. Save the changes.
  11. Click the “Release Configuration” button.
  12. Restart the web server.

In the PIA_access.log file, you should see a dash - or an IP address. If the header is blank, the dash will be populated. Otherwise, WebLogic will log the client’s real IP address.

Here a sample line from the PIA_access.log file after we enabled X-Forwarded-For logging:

#Version:   1.0
#Fields:    date time cs-method cs-uri sc-status cs-username cs(user-agent) s-ip c-ip time-taken bytes cs(X-Forwarded-For)
#Software:  WebLogic

2016-11-23  20:49:16    GET /psc/ps/EMPLOYEE/ELM/c/NUI_FRAMEWORK.PT_LANDINGPAGE.GBL 200 -   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"    0.0.0.0:10020   10.0.1.174  5.484   10907   "10.0.1.222"

The last value, "10.0.1.222" is the client IP address (my workstation). The c-ip value (10.0.1.174) is the load balancer IP adress in this scenario.

by Dan Iverson

#7 – Load Balancers

In Episode 7, we talk all about Load Balancers and PeopleSoft. We also clarify PeopleTools updates with Selective Adoption, why you may not need PSAESRV, a funny Chrome extension and more!

We want to make this podcast part of the community discussion on PeopleSoft administration. If you have comments, feedback, or topics you’d like us to talk about, we want to hear from you! You can email us at podcast@psadmin.io, tweet us at @psa_io, or use the Twitter hashtag #psadminpodcast.

You can listen to the podcast here on psadmin.io or subscribe with your favorite podcast player using the URL below, or subscribe in iTunes.

Podcast RSS Feed

Show Notes

by Dan Iverson

Smart v. Dumb Load Balancing

Setting up Load Balancing is common for production systems. Lately, I have seen a few organizations set up their load balancing so they end undoing all their good intentions. At the top of the infrastructure is a smart device, the load balancer, that can look at traffic and the servers underneath it to see where the new connections should be assigned. You want to the load balancer to decide where traffic goes so your web and app servers can focus on processing requests. Often times, the web server is also set up to load balance connections as well.

In the configuration.properties file, the psserver parameter will automatically use a round robin method to assign connections to any server listed. I call this dumb load balancing because there is no logic to assigning traffic. There are many reasons why multiple servers are listed in the psserver parameter, but there are new changes in PeopleTools that let us make smarter decisions with load balancing. First, let’s cover the most common load balancing algorithms to see why they are smart. Next, we’ll talk about the load balancing for the app server layer and the changes we should make. Then, we’ll put it all together and explore different options for optimal load balancing.

Algorithms

Load balancers have many algorithms to choose how traffic is divided, but these three are the most common:

  • Round Robin The Round Robin method assigned connections down a circular queue. Round Robin is naive about traffic and there are no decisions on which servers are assigned connections. If the server is next in the queue, the load balancer will assign the connection. Round robin does not look at traffic load or performance, so it can lead to unbalance server loads.
  • Least Connection The Least Connection method tracks connections to each pool member and assigns new connections to the server with the fewest connections. This algorithm will keep the connection levels on each server as close as possible. Least connections attempts to balance server load by keeping the connection counts the same. (This article from F5)[https://devcentral.f5.com/articles/back-to-basics-least-connections-is-not-least-loaded] has a great overview on Least Connections.
  • Fastest Connection The Fastest Connection method tracks server response times and assigns new connections to the pool members with the fast responses. Fasted Connection prioritizes speed and can be useful for environments where performance is a top priority. The assumption is the fastest server has the capacity to handle more connections. With Fastest Connection, you want to set a connection limit on the pool members. You don’t want to saturate one server with too many connections. More on Fastest Connection.

App Server

“Load Balancing” When people configure the web server, they want to make sure anyone who connects to that web server can reach an application server.

In the psserver parameter, they list multiple application servers. The web server will use a round robin algorithm to “balance” the load. This also gives the web server failover capabilities. If app server 1 is down, the web server will send all the connections to app server 2 and 3.

Smart v Dumb Load Balancing RoundRobin

That diagram looks busy – it must be a good design! 😉

In earlier versions of PeopleTools and WebLogic, this was the only way to offer failover at the web layer (without putting another load balancer between the web and app server). This worked to get failover from the web to app server layers, but it removes any smart load balancing. The other option was to go 1 to 1 between the web and app server. So web server 1 points only to app server 1, web server 2 point only to app server 2. You’re smart load balancing will work much better, but if app server 2 goes down, any user routed to web server 2 will get error messages. That’s not a good user experience.

Smart v Dumb Load Balancing 1to1

Smart Load Balancing

Starting with PeopleTools 8.50, you can take advantage of Jolt Failover groups. Jolt Failover groups let you keep the smart decisions at the load balancer but still keep app server failover on the web server layer. To use Jolt Failover, use this syntax in the psserver parameter in the configuration.properties file:

server1:9000{server2:9000;server3:9000}

The web server will only assign connections to server1 while it is up and responsive. When the server1 goes down, the web server will move to server2. If server2 goes down, the web server will send connections to server3. The failover group also supports round robin within the failover group. Instead of a semi-colon, use a comma. E.g,

server1:9000{server2:9000,server3:9000}

You can also you weighted connections for round-robin. This is useful if one server is more powerful than others. E.g,

server1:9000{server2:9000#5,server3:9000#3}

In this case, if server1 is down, server2 will receive 5 connections for every 3 connections assigned to server3.

Smart v Dumb Load Balancing Failover

The main advantage of using failover groups is to keep the smart decisions for assigning traffic at the load balancer. You can read more about Jolt Configuration options in PeopleBooks.