Deploying Cross-Site Replication in Percona Operator for MySQL (PXC)

Why It Matters

Cross‑site replication with the Percona Operator provides fast, automated DR for mission‑critical databases, cutting RTO/RPO and simplifying Kubernetes operations.

Key Takeaways

• Percona Operator enables cross‑site replication with minimal configuration
• LoadBalancer services expose PXC nodes for DR cluster connectivity
• Backup stored in GCP/S3 bucket and restored on DR cluster
• Automatic asynchronous replication failover switches to available DC nodes

Pulse Analysis

Disaster recovery has become a non‑negotiable requirement for businesses that rely on continuous data availability. In a multi‑cluster Kubernetes environment, setting up a separate DR site for a MySQL‑compatible database can be complex, especially when dealing with Galera‑based clusters like Percona XtraDB Cluster (PXC). The Percona Operator for MySQL abstracts much of this complexity by allowing administrators to declare cross‑site replication, expose primary nodes through LoadBalancer services, and manage backups with native XtraBackup integration. By leveraging a cloud‑native S3 bucket—here illustrated with Google Cloud Storage—the operator provides a reliable, off‑site snapshot that can be restored on a remote cluster with a few declarative commands.

The practical steps outlined in the guide start with a three‑node PXC deployment in the data‑center (DC) region, followed by enabling the "expose" flag and defining a replication channel in the custom resource file. After taking a full backup to an S3 bucket, the same backup is restored on a second Kubernetes cluster designated as the DR site. The operator then configures the DR cluster as a replication replica, adding the external IPs of the DC nodes and activating MySQL’s automatic asynchronous connection failover. This ensures that if any primary node goes down, the replica seamlessly reconnects to another source based on predefined weights, maintaining data continuity without manual intervention.

Beyond the technical walkthrough, the solution highlights key operational benefits. Automated cross‑site replication reduces the mean time to recovery (MTTR) and aligns with stringent RPO targets, while the declarative nature of the operator fits into GitOps pipelines for repeatable deployments. However, practitioners should weigh the trade‑offs of asynchronous replication—potential lag due to network latency—against the simplicity it offers compared to synchronous DR setups, which can introduce flow‑control bottlenecks. Proper monitoring, weight tuning, and periodic failover testing are essential to ensure the DR cluster performs reliably in production environments.