Digital Hive Load Balancing

Digital Hive Load Balancing

Question

Due to the number of users and the critical nature of the BI reports, it is desirable to introduce more capacity into the Digital Hive environment. Can Digital Hive be installed across multiple servers? 

Answer

Digital Hive can be installed across multiple servers to allow for additional scalability and throughput.

The use of a third-party load balancer, with persistent sessions configured, would be required to distribute the requests across the multiple Digital Hive servers.

To enable multiple Digital Hive servers, modifications will need to occur on the original Digital Hive installation, which will be referred to as the Primary server.

Changes to the Primary server

  1. Take a backup or snapshot of Digital Hive. This could be backup of the Digital Hive Content Store or a backup of the entire server
  2. Using the Services panel in Windows, stop the DigitalHive - node 1 service
  3. Stop the DigitalHive Content Store service
  4. Stop the Elasticsearch 5.5.1 (DigitalHive_9201) service
  5. Stop the Elasticsearch 5.5.1 (DigitalHive_9200) service
  6. Stop the DigitalHive-IQ-Server service

PostgreSQL changes

  1. Create this folder <Digital Hive install>/PostgreSQL/data/archive/
  2. Edit the DigitalHive/PostgreSQL/data/postgresql.conf file and make these changes:
    1. #wal_level = minimal change to wal_level = replica
    2. #synchronous_commit = on change to synchronous_commit = local
    3. #archive_mode = off change to archive_mode = on
    4. #archive_command = '' change to archive_command = 'copy "%p" "<Digital Hive root>/PostgreSQL/data/archive/%f"'
    5. #max_wal_senders = 0 change to  max_wal_senders = 2
    6. #wal_keep_segments = 0 change to  wal_keep_segments = 10
    7. #synchronous_standby_names = '' change to synchronous_standby_names = 'pgfailover001'
  3. Add the following line to the end of the <Digital Hive install>/PostgreSQL/data/pg_hba.conf file
    host       replication          replica                  all           md5
  4. Using the Services panel in Windows, start the DigitalHive Content Store service
  5. Launch pgAdmin or whatever query tool you use to query the Digital Hive Content Store
  6. Run the following SQL against the primary PostgreSQL database, called digitalhive (replace <replica_user_password> with the password you want to use):
    CREATE USER replica REPLICATION LOGIN ENCRYPTED PASSWORD ‘<replica_user_password>'

Primary Digital Hive configuration changes

Edit the <Digital Hive install>/app/node1/tomcat/conf/theia.properties file and make these changes (replace <primary> and <secondary> with the appropriate server names or IP addresses):

  1. motio.theia.db.url=jdbc:postgresql://<primary>:5432,<secondary>:5432/digitalhive
  2. motio.adfcore.db.url=jdbc:postgresql://<primary>:5432,<secondary>:5432/digitalhive
  3. motio.theia.activemq.networkConnector=static:(tcp://<primary>:61616,tcp://<secondary>:61616)
  4. motio.adfcore.activemq.networkConnector=static:(tcp://<primary>:61616,tcp://<secondary>:61616)
  5. Ensure that the value motio.adfcore.adfDataSource.autostart=true is true
  6. Ensure that the value motio.adfcore.db.backup.node=true is true
  7. Add motio.adfcore.indexer=true to the file if the setting doesn't already exist
  8. Save and close the file
Edit the <Digital Hive install>/app/node1/tomcat/conf/theia-node.properties file
  1. Look for the presence of motio.theia.activemq.networkConnector= and if it is there comment out the line by putting a # at the start of the line so the line reads: #motio.theia.activemq.networkConnector=
  2. Save and close the file

Primary Hive IQ configuration changes

Edit the <Digital Hive install>/IQ/NewIQ/app/iqConfig.ini file

  1. In the Database section, locate the connection property
  2. Edit the connection value to be jdbc:postgresql://primary_server_ip:5432/digitalhive where primary_server_ip is replaced with the physical IP address of the Primary server
  3. Save and close the file

Primary Elasticsearch configuration changes

Edit the <Digital Hive install>/Elasticsearch/config1/elasticsearch.yml file so that these settings are consistent
  1. network.bind_host: "0.0.0.0"
  2. network.publish_host: "primary server name or IP address"
  3. discovery.zen.ping.unicast.hosts: primary server name or IP address:9400
  4. Save and close the file
Edit the <Digital Hive install>/Elasticsearch/config2/elasticsearch.yml file so that these settings are consistent
  1. network.bind_host: "0.0.0.0"
  2. network.publish_host: "primary server name or IP address"
  3. discovery.zen.ping.unicast.hosts: primary server name or IP address:9400
  4. Save and close the file

Start the Primary server Digital Hive services

  1. Using the Services panel in Windows, start the Elasticsearch 5.5.1 (DigitalHive_9201) service
  2. Start the Elasticsearch 5.5.1 (DigitalHive_9200) service, if it didn't start automatically after the previous step
  3. Start the DigitalHive - node 1 service
  4. Start the DigitalHive-IQ-Server service

Creating the Secondary Digital Hive environment

  1. Copy the DigitalHive/app/node1/tomcat/conf/theia.properties and DigitalHive/app/node1/tomcat/conf/keystore.jks files from the Primary server and paste them in a folder on the Secondary server
  2. Install Digital Hive by launching the Setup.cmd file located in the Digital Hive installation files
    1. At step 5, select the Express installation mode, choose to install the Individual components and click Next


    2. Select the Digital Hive failover node and Digital Hive content store failover node options and click Next


    3. Provide an installation path for Digital Hive, in this example c:\DigitalHive


    4. Leave the default credentials, unless they were changed during the installation of the Primary Digital Hive, and click Next (the credentials MUST be identical in both installations)


    5. Browse to where the Primary server theia.properties files was copied to in Step 1. In this example c:\DH primary files\theia.properties and click Next


    6. Leave the Use custom keystore option unchecked. Browse to where the Primary keystore file was copied to in Step 1. In this example c:\DH primary files\keystore.jks and click Next


    7. Enter the IP address of this server (Secondary) and click Next


    8. Enter the replica name/password that was created on the Primary server PostgreSQL database (step 6 of PostgreSQL Changes) and enter the Primary IP address and click Next


    9. Leave default, unless the ports were changed as part of the Primary installation, and click Next


    10. Click Done

Digital Hive is now installed on the Secondary server. As part of the installation process, the Digital Hive services would have been started.

  1. Using the Services panel in Windows, stop the DigitalHive - node 1 service
  2. Stop the Elasticsearch 5.5.1 (DigitalHive_9201) service
  3. Stop the Elasticsearch 5.5.1 (DigitalHive_9200) service

Secondary Digital Hive configuration changes

Edit the <Digital Hive install>/app/node1/tomcat/conf/theia-node.properties file
  1. Look for the presence of motio.theia.activemq.networkConnector= and if it is there comment out the line by putting a # at the start of the line so the line reads: #motio.theia.activemq.networkConnector=
  2. Save and close the file

Secondary Elasticsearch configuration changes

Edit/validate the <Digital Hive install>/Elasticsearch/config1/elasticsearch.yml file so that these settings are consistent
  1. network.bind_host: "0.0.0.0"
  2. network.publish_host: "secondary server name or IP address"
  3. discovery.zen.ping.unicast.hosts: ["primary server name or IP address:9400","primary server name or IP address:9401"] 
  4. Save the changes to the file
Edit/validate the <Digital Hive install>/Elasticsearch/config2/elasticsearch.yml file so that these settings are consistent
  1. network.bind_host: "0.0.0.0"
  2. network.publish_host: "secondary server name or IP address"
  3. discovery.zen.ping.unicast.hosts: ["primary server name or IP address:9400","primary server name or IP address:9401"] 
  4. Save the changes to the file
  5. Start the DigitalHive - node 1 service
  6. Start the Elasticsearch 5.5.1 (DigitalHive_9201) service
  7. Start the Elasticsearch 5.5.1 (DigitalHive_9200) service, if it didn't start automatically after the previous step
Digital Hive has now been configured to utilize a Primary and Secondary servers to handle user requests. In this scenario, the Primary server must always be active as the active Content Store is located on the Primary server. If a third server is desired, most of the same steps for Creating the Secondary Digital Hive environment section are applicable.

Since both the Primary and Secondary servers have been configured with an instance of the PostgreSQL Content Store, it is recommended that any additional servers be configured for load balancing purposes only.

Creating the Tertiary Digital Hive environment

  1. Copy the DigitalHive/app/node1/tomcat/conf/theia.properties and DigitalHive/app/node1/tomcat/conf/keystore.jks files from the Primary server and paste them in a folder on the Tertiary server
  2. Install Digital Hive by launching the Setup.cmd file located in the Digital Hive installation files
    1. At step 5, select the Express installation mode, choose to install the Individual components and click Next


    2. As there are already two Content Store nodes, select just the Digital Hive failover node option and click Next


    3. Provide an installation path for Digital Hive, in this example c:\DigitalHive


    4. Leave the default credentials, unless they were changed during the installation of the Primary Digital Hive, and click Next (the credentials MUST be identical in both installations)


    5. Browse to where the Primary server theia.properties files was copied to in Step 1. In this example c:\DH primary files\theia.properties and click Next


    6. Leave the Use custom keystore option unchecked. Browse to where the Primary keystore file was copied to in Step 1. In this example c:\DH primary files\keystore.jks and click Next


    7. Enter the IP address of this server (Secondary) and click Next


    8. Leave default, unless the ports were changed as part of the Primary installation, and click Next


    9. Click Done

Digital Hive is now installed on the Tertiary server. As part of the installation process, the Digital Hive services would have been started.

  1. Using the Services panel in Windows, stop the DigitalHive - node 1 service
  2. Stop the Elasticsearch 5.5.1 (DigitalHive_9201) service
  3. Stop the Elasticsearch 5.5.1 (DigitalHive_9200) service

Tertiary Digital Hive configuration changes

Edit the <Digital Hive install>/app/node1/tomcat/conf/theia-node.properties file
  1. Look for the presence of motio.theia.activemq.networkConnector= and if it is there comment out the line by putting a # at the start of the line so the line reads: #motio.theia.activemq.networkConnector=
  2. Save and close the file

Tertiary Elasticsearch configuration changes

Edit/validate the <Digital Hive install>/Elasticsearch/config1/elasticsearch.yml file so that these settings are consistent
  1. network.bind_host: "0.0.0.0"
  2. network.publish_host: "tertiary server name or IP address"
  3. discovery.zen.ping.unicast.hosts: ["primary server name or IP address:9400","primary server name or IP address:9401"] 
  4. Save the changes to the file
Edit/validate the <Digital Hive install>/Elasticsearch/config2/elasticsearch.yml file so that these settings are consistent
  1. network.bind_host: "0.0.0.0"
  2. network.publish_host: "tertiary server name or IP address"
  3. discovery.zen.ping.unicast.hosts: ["primary server name or IP address:9400","primary server name or IP address:9401"] 
  4. Save the changes to the file
  5. Start the DigitalHive - node 1 service
  6. Start the Elasticsearch 5.5.1 (DigitalHive_9201) service
  7. Start the Elasticsearch 5.5.1 (DigitalHive_9200) service, if it didn't start automatically after the previous step

    • Related Articles

    • Unable to Explore Digital Hive Content

      Question When using the Explore Content interface to browse the Digital Hive content, a message stating "Something went wrong" and a Try Again button is displayed How do I resolve this? Answer This issue is usually a result of Elasticsearch being ...
    • Digital Hive Architecture Overview

      Overview Digital Hive is a lightweight, web-based application that is both easy to install as well as to maintain and manage. Other than the Microsoft C++ redistributable (https://www.microsoft.com/en-ca/download/details.aspx?id=40784), there are no ...
    • Connecting PowerBI to Digital Hive Audit Data

      Question The Digital Hive Control Center doesn't seem to provide any reports or views around usage of the platform. Is there a way to create reports to show metrics like the amount of reports executed by platform, the number of searches being made, ...
    • Digital Hive Auditing Explained

      Question The Digital Hive Control Center doesn't seem to provide any reports or views around usage of the platform. Is there a way to create reports to show metrics like the amount of reports executed by platform, the number of searches being made, ...
    • Digital Hive Logging

      Question Is there any logging available to help troubleshoot issues? Answer Digital Hive has several types of log files available for troubleshooting issues. Here are the various Digital Hive log files and what they contain. Theia Log This is the ...