The ServAssure NXT Insight Views package contains a new playbook for enabling multidatacenter on Singlestore for both primary site (Site A) and secondary, or failover, site (Site B). It can be found in the /tools/ directory in the extracted Insight-Views-$version package. You can replicate or failover to the secondary site using the inventory of the site.
The playbook to execute is
ansible-playbook -i <inventoryFile-Site> tools/htap-multi-dc-manager.yml -u remote_user -bkKYou
can choose one of the following options, 1, 2, 3, 4:- Enable: initiate replication
- Disable: stop replication and remove replicated databases on the secondary site
- Failover: failover to the current site
- Status: check replication status
Enable
This procedure can be re-run. It
replicates Insight and all tenant databases when it is run for the first time. You
can also run this in these scenarios:
- When a new tenant is created on Site A with HTAP_MULTI_DC.enable set to false, an error occurs and some databases are not replicated. You can use Enable to replicate rest of the tenant databases.
- When Site A fails to replicate the newly created tenant to Site B, You can use Enable to complete the replication process in Site B.
Disable
This procedure can be re-run. It stops
replication and removes all the replicated databases if the current site is a
secondary site. If not, you are prompted to confirm if you are on the secondary site
and manually remove the databases to continue. You can also run this in these
scenarios:
- When an error occurs with Enable or when you do not need replication.
- When Site A fails to stop
replicating the deleted tenant in site B (Site B is offline), you can use
Disable to complete this process in Site B. Do this with
the parameter
-e multi_dc_databases=${tenant}. Multiple tenants can be deleted with a comma-separated list like so,-e multi_dc_databases=${tenant1},${tenant2},${tenant3}NOTE: You must update HTAP_MULTI_DC → enabled: false under NXT_INSIGHT_INTEGRATION in the site_vars file if you do not need replication anymore.
Failover
This procedure must NOT be re-run when a pair site is online. The workflow is as
follows:
- When the pair site is offline, run this procedure for the first time. This promotes the secondary site as the primary site (there are two primary sites at this point).
- When the pair site is back online, re-run this
procedure with the parameter:
-e pair_site_back_online=true. This demotes the old primary site as the secondary site (there is one primary site and one secondary site at this point and failover is considered complete).
NOTE:
- Stop the following services on the primary site before failover: kafka, spark, zookeeper, schema-registry, maintain, and api services.
- Do not perform
install tenant,delete tenant,create user, orupdate usertasks until the failover is completed.
Status
This procedure can be re-run. It
displays the replication status in comma-separated values and no status is displayed
when the primary site is offline (unreachable).
NOTE:
- cluster_[dash] database in the first column indicates that the current site is the secondary site.
- A value 0 in the last two columns indicate that the replication process is completed.
To use the new playbook, the
following steps are required:
- Edit your "Site A" and "Site B" Insight Viewshost_inventory files and add the required new groups and hosts. Verify that your inventory file contains groups and hosts for components included in your deployment. Refer to Set up Insight Views inventory files.
- Edit your "Site A" and "Site B" Insight Viewssite_vars files and add the required new variables. Verify that your site_vars file contains parameters and variables for components and features included in your deployment. Refer to Set up Insight Views site_vars files.
