Summary
This runbook describes how to failover to the faraway replica in EDB Cloud Service by promoting it as the new Primary node.
Any automatic failovers in EDB Cloud Service with Primary/Standby High Availability would be normal standby replicas deployed in the same region. In a Disaster recovery (DR) scenario, manual intervention is needed to switch to the faraway replica, as shown in the below steps.
Failover to faraway replica
We can promote a faraway replica to a full-fledged cluster, which makes it capable of accepting writes.
1. Go to the Clusters page. A list of previously created clusters appears.
2. Select the cluster with the replica you want to promote. The cluster's replicas are under Faraway Replicas in the Overview tab.
3. Select the Promote Replica icon next to the replica you want to promote. The Promote Faraway Replica page appears.
4. Select Promote Replica after verifying that there is minimum lag.
5. We also recommend having the same instance type (RAM/CPU etc.) on the faraway replica as in the primary cluster to avoid extra downtime to change the configuration and performance issues at the time of failover.
6. Review/Edit the PostgreSQL settings for the new primary during the promotion.
IMPORTANT NOTES:
- There will be a minimum 16 MB lag on the faraway replica because WALs only get shipped to the Faraway replica after it is ready to be archived. So current WAL will still be on the primary
- Maximum lag will depend upon the application workload, and replication lag on the faraway replica during the promotion. Promoting a Faraway replica to the EDB Cloud cluster can result in some data loss because of this lag. Refer to the link below for the faraway replica limitations and its advantages. Refer to EDB Docs - Faraway replicas
- Ensure the private connection for the new primary cluster is established.
- After promotion, the new Primary Endpoint name will have to be updated in application connect strings.
Add a Faraway Replica to the new cluster
- Once you have promoted the Faraway replica as a primary node, you will lose connectivity to the old primary nodes, as they will go out of sync.
- To have a highly available setup, we need to add a Faraway replica to this promoted cluster so that it can be used as a Disaster Recovery solution in the future.
- To create the Faraway replica, Select the Create Replica option under Quick Actions menu on the Clusters page for the new primary cluster.
- This will open a new window where you can select the New region to be used for the Faraway Replica.
- The Faraway replica will take some time to be ready; the overall time will depend on the size of the cluster.