Back to Articles

The ServAssure NXT Insight Views package contains a new playbook for enabling multidatacenter on Singlestore for both primary site (Site A) and secondary, or failover, site (Site B). It can be found in the /tools/ directory in the extracted Insight-Views-$version package. You can replicate or failover to the secondary site using the inventory of the site.

The playbook to execute is
ansible-playbook -i <inventoryFile-Site> tools/htap-multi-dc-manager.yml -u remote_user -bkK
You can choose one of the following options, 1, 2, 3, 4:
  1. Enable: initiate replication
  2. Disable: stop replication and remove replicated databases on the secondary site
  3. Failover: failover to the current site
  4. Status: check replication status

Enable

This procedure can be re-run. It replicates Insight and all tenant databases when it is run for the first time. You can also run this in these scenarios:
  • When a new tenant is created on Site A with HTAP_MULTI_DC.enable set to false, an error occurs and some databases are not replicated. You can use Enable to replicate rest of the tenant databases.
  • When Site A fails to replicate the newly created tenant to Site B, You can use Enable to complete the replication process in Site B.

Disable

This procedure can be re-run. It stops replication and removes all the replicated databases if the current site is a secondary site. If not, you are prompted to confirm if you are on the secondary site and manually remove the databases to continue. You can also run this in these scenarios:
  • When an error occurs with Enable or when you do not need replication.
  • When Site A fails to stop replicating the deleted tenant in site B (Site B is offline), you can use Disable to complete this process in Site B. Do this with the parameter -e multi_dc_databases=${tenant}. Multiple tenants can be deleted with a comma-separated list like so, -e multi_dc_databases=${tenant1},${tenant2},${tenant3}
    NOTE: You must update HTAP_MULTI_DC → enabled: false under NXT_INSIGHT_INTEGRATION in the site_vars file if you do not need replication anymore.

Failover

This procedure must NOT be re-run when a pair site is online. The workflow is as follows:
  • When the pair site is offline, run this procedure for the first time. This promotes the secondary site as the primary site (there are two primary sites at this point).
  • When the pair site is back online, re-run this procedure with the parameter: -e pair_site_back_online=true. This demotes the old primary site as the secondary site (there is one primary site and one secondary site at this point and failover is considered complete).
NOTE:
  • Stop the following services on the primary site before failover: kafka, spark, zookeeper, schema-registry, maintain, and api services.
  • Do not perform install tenant, delete tenant, create user, or update user tasks until the failover is completed.

Status

This procedure can be re-run. It displays the replication status in comma-separated values and no status is displayed when the primary site is offline (unreachable).
NOTE:
  • cluster_[dash] database in the first column indicates that the current site is the secondary site.
  • A value 0 in the last two columns indicate that the replication process is completed.
To use the new playbook, the following steps are required:
  1. Edit your "Site A" and "Site B" Insight Viewshost_inventory files and add the required new groups and hosts. Verify that your inventory file contains groups and hosts for components included in your deployment. Refer to Set up Insight Views inventory files.
  2. Edit your "Site A" and "Site B" Insight Viewssite_vars files and add the required new variables. Verify that your site_vars file contains parameters and variables for components and features included in your deployment. Refer to Set up Insight Views site_vars files.