Skip to content

Deploy and Configure Elasticsearch

Elasticsearch support is here for PeopleTools 8.55. In PeopleTools 8.55.11, Elasticsearch and SES are supported search engines for PeopleSoft applications. For the next 18 months, PeopleSoft will support both search engines in 8.55. After those 18 months and starting in 8.56, Elasticsearch will be the only search engine supported with PeopleSoft. Since the PeopleSoft team announced that Elasticsearch would replace SES in December 2015, the community has been eagerly waiting for Elasticsearch support go live.

In the video below, we’ll provide an in-depth walk though of this post.

 

In this post, we’ll cover the installation of Elasticsearch, how to use the REST API, and some tips when using Elasticsearch.

DPK Only

Elasticsearch is the first PeopleTools component to be distributed only by Deployment Packages. There is no virtual CD option to install Elasticsearch. This also means that Elasticsearch is a separate download from PeopleTools. To download the Elasticsearch DPK, visit the PeopleTools Patch Documentation Home and click on the “Additional DPKs” tab. Ther is also an Elasticsearch Documentation Home with information about installing Elasticsearch, migrating from SES, and more.

Downloading

Like the PeopleTools and PeopleSoft Image DPKs, you download the .zip files from Oracle Support and run a bootstrap script to start the installation. The Elasticsearch DPK is only one .zip file, and is substantially smaller than other DPKs. I like using the getMOSPatch utility for downloading patches from MOS. Here is the command to download the Windows version of the Elasticsearch DPK:

java -jar getMOSPatch.jar patch=24924150 platform=233P download=all

This is the LInux version of the command:

java -jar getMOSPatch.jar patch=24924136 platform=226P download=all

Next, unzip the ELASTICSEARCH-DPK-WIN-2.3.2_00.zip file. Unlike other DPKs, there is only one .zip file. The .zip file contains setup files, the Elasticsearch binaries, and documentation.

Installation

After you unzip the file, you run the bootstrap script under scripts to start the installation:

cd .\scripts
psft-dpk-setup.ps1 -env_type es

When you run the Elasticsearch DPK, make sure to pass the -env_type es parameter. Without the parameter, the bootstrap script will fail looking for a file that doesn’t exist in the Elasticsearch DPK.

The bootstrap script will ask you a series of questions:

  • Do you want to install Puppet: Yes
  • Enter the ES Base folder: e:\psft
  • Elasticsearch Admin Password: Passw0rd1
  • Proxy User Password: Passw0rd1
  • Elasticsearch Cluster Name: srch-d1
  • Elasticsearch Port: 9200
  • Elasticsearch Discovery Host: ["127.0.0.1"]
  • Enter Java Heap Size: 2

The Elasticsearch Discovery Host is used when you are building a cluster with more than 1 node. In our case, we will enter the local machine’s IP address since we’ll run our nodes on only this machine. If you were building a cluster with multiple nodes on different machines, you would list the IP addresses for each server running Elasticsearch. After you answer the questions, the bootstrap script will start building the Elasticsearch instance.

In my testing on the current Elasticsearch DPK, there is a bug in the bootstrap script. The script ends early and doesn’t complete the installation. If this happens to you, it is easy to resolve.

First, let’s make sure the psft_es.yaml file is updated with out settings. Under C:\ProgramData\Puppetlabs\puppet\etc\data\ open the psft_es.yaml file. Find the section

#es_data
es_http_port:        

Enter 9200 for the es_http_port: value and save the file.

es_http_port:        9200

If you changed the Discover Host value, update that line as well and save the file.

discovery_zen_ping_unicast_hosts:           '["10.0.1.173"]'

Next, navigate to C:\ProgramData\Puppetlabs\puppet\etc\manifests. We’ll start Puppet and have it finish the Elasticsearch deployment and configuration.

puppet apply .\site.pp

At the end of the run, let’s verify that Elasticsearch is up and listening on port 9200.

netstat -an | findstr 9200

You should see something like this:

  TCP    10.0.1.173:9200        0.0.0.0:0              LISTENING
  TCP    127.0.0.1:49200        127.0.0.1:49201        ESTABLISHED
  TCP    127.0.0.1:49201        127.0.0.1:49200        ESTABLISHED

Administration

Before we jump into configuring PeopleSoft to use our Elasticsearch instance, I want to talk some basic Elasticsearch administration. Unlike the SES, there is not web-based admin console. Elasticsearch uses a REST-based API for all administration. Let’s look at what this means. In your browser, go to your Elasticsearch URL http://servername:9200/ and login with esadmin and the Administrative password you entered in the bootstrap script. You’ll get a response similar to this:

{
  "name" : "elastic11.psadmin.io",
  "cluster_name" : "srch-d1",
  "version" : {
    "number" : "2.3.2",
    "build_hash" : "b9e4a6acad4008027e4038f6abed7f7dba346f94",
    "build_timestamp" : "2016-04-21T16:03:47Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

If you want to get status of your Elasticsearch cluster, you would use this URL: http://servername:9200/_cluster/health?pretty=true.

{
  "cluster_name" : "srch-d1",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Currently, my cluster srch-d1 has a status of “green”. But it also has no data…

PeopleSoft Configuration

Now that Elasticsearch is running, it is time to configuration our PeopleSoft application to use it. There are a few requirements in PeopleSoft and they are similar to the SES requirements:

  • You must be running 8.55.11
  • The Integration Broker is configured and running
  • The integrationGateway.properties file has an encrypted value for secureFileKeystorePasswd
  • IB Domains are active
  • REST Service URLs are configured (PeopleTools > IB > Configuration > Service Configuration > Setup Target Locations)
  • A callback user account with these roles:
    • Search Developer
    • Search Server
    • Search Query Administrator
    • Search Administrator
    • PeopleSoft User

Search Instance

Go to PeopleTools > Search Framework > Administration > Search Instance. Starting with 8.55.11, we can have 2 or more Search Instances defined. The first search instance is named PSFT_DEFAULT. We’ll leave that configured to use SES. Create a new Search Instance named ELASTIC.

The Search Instance page looks the same as before, but with the addition of a “Search Provider” drop-down menu.

  1. Select “Elasticsearch”
  2. Enter the server name where you installed Elasticsearch
  3. Enter the Elasticsearch port (default is 9200)
  4. Enter esadmin for the User Name
  5. Enter the administrative password you set in the Bootstrap script
  6. Enter people for the Proxy Name
  7. Enter the proxy password you set in the bootstrap script

In the Call Back Properties:

  1. Enter the URL for the REST Target Connector you defined under “Service Configuration > Setup Target Locations”
  2. Enter the Call Back User’s name and password.

Verify all the Ping, Login, and Validate tests return successfully.

Last, we can set the order of the Search Instances. Since we haven’t fully tested Elasticsearch yet, set it to a lower priortiy until we are ready to release it to all users. Under “PeopleTools > Search Framework > Administration > Search Instance Administration”, set the “ELASTIC” instance to Priority 10.

Deploy Indexes

Go to “PeopleTools > Search Framework > Administration > Deploy/Delete Objects”. On this page, you have to select the Search Instance you want to deploy indexes to.

  1. Select “ELASTIC” for the Search Instance.
  2. Select the checkboxes for “PTPORTALREGISTRY” and “PTSEARCHREPORTS”.
  3. Click Deploy.

There seems to be bug in the Report Sync Issues action. If you select deployed indexes and click Report Sync Issues, it will return some errors. Ignore those errors for now; Elasticsearch works despite the “errors”.

  1. Navigate to “PeopleTools > Search Framework > Administration > Schedule Search Index”
  2. Create a new run control called PTPORTALREGISTRY_FULL
  3. Select “ELASTIC” as the Search Instance.
  4. Select the search index “PTPORTALREGISTRY”.
  5. Save the run control and run the process.

Once the process starts, you can view the Asynchronous Services page to see the messages sent to Elasticsearch. Navigate to “PeopleTools > Integration Broker > Service Operation Monitor > Monitoring > Asynchronous Services”. Once the message are successfull (Operation Instance and Subscription Contracts), it’s time to test.

Test Elasticsearch

Currently, SES is still our primary search provider. We can set up per-user search provicers so individual users can begin testing Elasticsearch.

  1. Go to “PeopleTools > Search Framework > Administration > Search Instance/User”
  2. Enter your user name and “ELASTIC”.
  3. Save the page.
  4. Log out of the application and log back in.
  5. In the search bar, search for User.

You should see search results returned from Elasticsearch! Once you are comfortable with Elasticsearch, simply change the priority of the search instances to activate Elasticsearch for everyone.

35 thoughts on “Deploy and Configure Elasticsearch”

    1. That page is not required. It’s there so you can have some users test Elasticsearch before you move everyone.

  1. Hey Dan…
    Thanks for the Post….

    Just a query..Can we use the above method for VMs too or is it applicable to native DPK only ?

    Cheers..

  2. thanks for the video.

    I am getting below error while pupply apply command :

    The service ‘elasticsearch-service-x64’ has been started
    Debug: Service status: stopped
    Error: Unable to start ES Server: Error: Unknown
    C:/ProgramData/PuppetLabs/puppet/etc/modules/pt_config/lib/puppet/provider/pt_es_domain/es_domain.rb:377:in start_es_s
    rver'
    C:/ProgramData/PuppetLabs/puppet/etc/modules/pt_config/lib/puppet/provider/pt_es_domain/es_domain.rb:171:in
    post_creat

    C:/ProgramData/PuppetLabs/puppet/etc/modules/pt_config/lib/puppet/provider/pt_es_domain/es_domain.rb:95:in create'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/property/ensure.rb:16:in
    block in defaultvalues’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/property.rb:197:in call_valuemethod'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/property.rb:498:in
    set’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/property.rb:581:in sync'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction/resource_harness.rb:204:in
    sync’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction/resource_harness.rb:128:in sync_if_needed'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction/resource_harness.rb:81:in
    perform_changes’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction/resource_harness.rb:20:in evaluate'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction.rb:204:in
    apply’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction.rb:217:in eval_resource'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction.rb:147:in
    call’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction.rb:147:in block (2 levels) in evaluate'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util.rb:327:in
    block in thinmark’
    C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/2.0.0/benchmark.rb:296:in realtime'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util.rb:326:in
    thinmark’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction.rb:147:in block in evaluate'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/graph/relationship_graph.rb:118:in
    traverse’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction.rb:138:in evaluate'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/resource/catalog.rb:169:in
    block in apply’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util/log.rb:149:in with_destination'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/transaction/report.rb:112:in
    as_logging_destination’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/resource/catalog.rb:168:in apply'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/configurer.rb:120:in
    block in apply_catalog’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util.rb:161:in block in benchmark'
    C:/Program Files/Puppet Labs/Puppet/sys/ruby/lib/ruby/2.0.0/benchmark.rb:296:in
    realtime’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util.rb:160:in benchmark'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/configurer.rb:119:in
    apply_catalog’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/configurer.rb:227:in run_internal'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/configurer.rb:134:in
    block in run’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/context.rb:64:in override'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet.rb:244:in
    override’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/configurer.rb:133:in run'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application/apply.rb:297:in
    apply_catalog’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application/apply.rb:231:in block in main'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/context.rb:64:in
    override’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet.rb:244:in override'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application/apply.rb:193:in
    main’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application/apply.rb:154:in run_command'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application.rb:381:in
    block (2 levels) in run’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application.rb:507:in plugin_hook'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application.rb:381:in
    block in run’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util.rb:488:in exit_on_fail'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/application.rb:381:in
    run’
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util/command_line.rb:146:in run'
    C:/Program Files/Puppet Labs/Puppet/puppet/lib/puppet/util/command_line.rb:92:in
    execute’
    C:/Program Files/Puppet Labs/Puppet/puppet/bin/puppet:8:in `’
    Error: /Stage[main]/Pt_profile::Pt_es/Pt_es_domain[es_server]/ensure: change from absent to present failed: Unable to s
    art ES Server: Error: Unknown

    1. I faced similar issue.. Run cleanup and then ensure the elasticsearch service is removed, if not delete it manually. Just proceed with the default dpk installation process instead of puppet apply.

  3. psadmin.io is a great site/blog to learn new things as soon as it gets released by Oracle. Kudos to the team.

    I have a basic question here, how many databases we need typically to use Elastic Search?

    Do we need two different databases and two different App/Web servers like Non-Prod Elastic Search and Prod Elastic Search? Or we just need one environment only serving all PeopleSoft instances?

    1. ES doesn’t actually use a database like SES did. I’d suggest separate ES environments for Prod and Non Prod. Your server count will very depending on your data size and needs, but the suggested default setup for a Prod deployment is 3 nodes.

  4. Hello,
    The video was great.
    I have some query regarding the minimum hardware requirement which is mentioned in Oracle Support site for Elasticsearch installation .
    Which is 32 GB RAM , 100 GB Hard Drive and 4 CPU .
    Whether with 8 GB RAM and 50 GB Hard Drive with 2 CPU we can install the Elasticsearch .

    In this Demo what hardware configuration you have used .
    And I can see the Elastic search software is around 280 MB zip file

  5. Is there any details on how to restore ES configuration data after Database refresh, if so could you please share the details.

  6. Hi,

    We installed Elastic Search in same server where we have our app server running . In this case what are the files i need to edit to set JAVA Path for Elastic search as the java path set for app server is different .

  7. I have installed 3 node (server) cluster, but when I shutdown node1 (server #1) the search does not work.
    Anyone having 3 node for elasticsearch, i love to know how did you setup configuration.yml

    Thanks
    QH

  8. Hi Dan,
    we have set up elastic search as in the video but when we tested , we keep getting .

    No results were returned. Refine your search by entering a different keyword.

    do you have any ideas?

    1. We are getting same we donot have SSL, all pings are good.
      ERROR: No results were returned. Refine your search by entering a different keyword.

  9. Hi,

    When I try to deploy search definition EP_AP_VOUCHERS in elastic search, deployed process is successfull. But PTSF_GENFEED process to build index getting failed. I have already installed Portalregistry.

    Below is the error message,
    Query field referred in %QUERYFIELD:AP_SRCH_VOUCHERS:A.BU_SECURITY_SES% is not in the Query (241,218 ) PTSF_ES.ESIMPL.IngestService.OnExecute Name:indexCQSource PCPC:53179 Statement:711 Called from:PTSF_ES.ESIMPL.IngestService.OnExecute Name:initIndexingProcess Statement:17 Called from:PTSF_GENFEED.INDXDATA.GBL.default.1900-01-01.Step01.OnExecute Statement:25.
    Please help me to resolve this concern

    Thanks,
    Krishna

    1. Hi Krishna

      Have you applied the application patches for Elasticsearch? In the latest PeopleSoft Image, the PUM has a search criteria to apply fixes for Elasticsearch. You should look at those patches first since there are some changes to the search definitions for Elasticsearch.

  10. Great info as usual! Thanks a million, finally getting to 8.56 and able to play with Elasticsearch!

    Kelly

  11. Hi Dan

    Excellent video, I followed and sucesfully installed ES on PT8.56.03.

    We have a installation with a distributed architecture and when we tested the ES no data was returned.

    I found the Doc ID 2241004.1, it says there is a network issue and the ES cannot returned the data to PS.

    Do you know which port the ES is trying to connect to return the data? Im guessing the ES is trying to connect to the IB.

    Thank you in advance.

  12. Elasticsearch should use the Integration Broker port to send the data back to PeopleSoft. The search is handled via a Synchronous IB transaction.

    That error message can be misleading. Do you have HTTPS configured for Elasticsearch? If yes, does your search work when you remove the HTTPS config and go over HTTP?

  13. Hi Dan ,

    Hope you are doing well,

    we are trying to implement Elastic search on our QA system which uses https link in IB
    like this https://peoplesoft-qa1.xxxx.com/PSIGW/PeopleSoftListeningConnector

    so when we deploy Elastic search in normal manner and try to ping from diagonastics

    it shows below error:

    Ping Test Result Failure. Exception caught Cannot establish HTTP connection(158,2842)(262,612)

    can you kindly help us on this .

    1. Are you able to get results from Elasticsearch without using HTTPS? Try using on-HTTPs URLs first to verify the connections are working before you add SSL to the connection.

  14. We are getting same we donot have SSL, all pings are good.
    ERROR: No results were returned. Refine your search by entering a different keyword.

  15. Dan,
    We are implementing ElasticSearch on PT8.56.07 and everything pings correctly, call back validates correctly, but when we go to the Diagnostics page, it ping successfully, but on Step 2 Deploy we get the following error ‘Exception Caught while adding peoplesoft source’

    1. Hi Wayne, did you ever get your issue resolved? We’ve been encountering the same scenario and have not been able to find a fix for a couple of weeks now.

  16. Great video!

    I have a question here, we are on HCM 9.2, Tools 8.56.07 , and Elastic Search was working as expected, however all of sudden Search and Compare does not produce anyResults

    Problem Description :
    When Search and Compare Profiles (Workforce Development > Profile Management > Profiles > Search and Compare Profiles) is used to search person profiles – No data is retrieved.
    Below Indexes were successfully deployed the
    HC_JPM_PERSON_PROFILE
    HC_JPM_NONPERSON_PROFILE
    Tested Search results successfully using Search Test Page (Tools>Search Framework>Utilites)
    Any help will be appreciated.
    Thx

  17. We have just upgraded our FSCM production environment to Tools 8.56.09 with Elastic search. We are seeing an intermittent error message while searching for items while creating a requisition. The error message is – “Search Exception: Exception caught while applying ACL filters”

    This does not occur all the time, and if the search is performed again it will usually work without an issue.

  18. We have just upgraded our FSCM production environment to Tools 8.56.09 with Elastic search. We are seeing an intermittent error message while searching for items while creating a requisition. The error message is – “Search Exception: Exception caught while applying ACL filters”

    This does not occur all the time, and if the search is performed again it will usually work without an issue..

  19. how to restrict elastic search results for user? right now it is behaving as global search and fetching everything

  20. Hi everyone, by any chance someone have the DPK used in the example?. I need it but is deprecated in oracle support.

    Thanks in advance.

  21. Has anyone tried configuring Multiple Environments to the same cluster? Like, DEV, UAT and TST? Can we have multiple environments using the same Elastic Cluster?

    1. Yes, you can point multiple environments to the same Elasticsearch Cluster. PeopleSoft appends the database name to the indexes so there is no conflict when deploying the same index from multiple environments.

  22. Naresh Vyenktesh Maheshwar

    I am getting search exception error from PeopleSoft Main Menu>Careers page . I tried to undeploy and redeploy the search definition HC_HRS_JOB_POSTING and also cleared Cache from instance but even after that I am getting below error

    “Search Exception: (262,2018) PTSF_ES.ESIMPL.QueryService.OnExecute Name:ExecuteQuery PCPC:15128 Statement:255
    Called from:PTSF_ES.ESIMPL.Query.OnExecute Name:Execute Statement:67
    Called from:HRS_SEARCH.SearchFramework.CandidateGatewayJobSearch.OnExecute Name:BeginSearch Statement:78
    Called from:HRS_COMMON.SEARCH.BUS.JobSearchProvider.OnExecute Name:search Statement:158
    Called from:HRS_COMMON.SEARCH.BUS.AppJobSearch.OnExecute Name:execute Statement:102
    Called from:HRS_COMMON.CMP_HRS_APP_SCHJOB.BUS.SearchController.OnExec”
    .
    Could you please help on the issue.

Leave a Reply to Kevin Cancel reply

Your email address will not be published. Required fields are marked *