Active/Passive Kubernetes deployment for high availability stateful applications
This article describes an implementation concept of Active/Passive Kubernetes deployment for highly available mission-critical stateful applications.
This is nothing new under the sun, however, based on specific non-functional requirements we had to implement an alternative version of an already known solution.
About the application
The application is a Node.js service that interfaces between the emergency call network and emergency text call takers consoles that are not able to handle SIP and MSRP protocols. Our application then takes care of the complexities of SIP calls processing and text media session handling and ‘translates’ them into a simple REST interface.
This means the service is stateful as we cannot have different instances of the application handling SIP and MSRP (text) messages for the same call.
High availability then is achieved by having one (or more) passive instances of the service on standby, ready to take over in case of active instance failure.
The application is already deployed on the Azure government cloud using AKS (Managed Azure Kubernetes Service)
Leader election
Leader election is the Kubernetes pattern of choice to achieve high availability as required by our application. Usually, the leader election mechanism is based on Kubernetes endpoints. This is very well explained on Kubernetes blog. The election is based on Endpoint resources, that hold annotation about the current leader and its leadership lease time. If the leader dies, other instances of the service are free to take over the record.
In practical terms, the leader election mechanism uses sidecar applications so the leader election is handled transparently to the service. In Kubernetes, Pods are made up of one or more containers. Oftentimes, this means that you add sidecar containers to your main application to make up a Pod.
A different leader election solution
Our team wanted to have the fastest take-over possible in case of leader failure. It also considered the sidecar application as a single point of failure.
Because of that, we implemented a leader election with the following characteristics:
- Service state is saved in persistent memory
The requirement for an application that handles telephony calls is that any established call shall remain established in the event of a service instance failure. So, all data of already established calls are stored in persistent memory. We use Redis for persistent data storage.
- No sidecar
The functionality of the sidecar is implemented by the service itself.
- The service instances compete for the leadership
Upon startup, each service instance will compete to register themselves as a leader by storing their IP address and port into common persistent storage using a predefined key (We take advantage of Redis as it is already used by the application). The instance registering first will declare itself active, while the other will be on standby.
- The resulting passive instance pings the active one
The passive instance knows the active instance IP address and port from reading the persistent storage when losing the leadership race. So, it starts sending a heartbeat to the active instance and waiting for a response. It does this every one second.
- Use Pod labels and service selector to direct the traffic
Each service instance, acting as Kubernetes client patches the Pod data where they are running with a special label, let’s call it mode . So the mode label is set to active by the leader instance, and stand-by by the passive instance.
A selector mode=active is defined to direct traffic to the active Pod only.
- Takeover upon heartbeat failure
Upon failing the heartbeat, the passive instance will become active by setting its own IP address and port in the common persistent storage and patching its own Pod data with label mode=active.
- Check for inconsistencies and fix them
Although it should never happen, a separate audit service runs in another Pod that periodically checks for Pods labels mode. If they are inconsistent (both are active or both are standby), both Pods are restarted. Depending on the nature of the application it can also be fixed by patching one of the Pods.
This solution allowed us to have a fast recovery while keeping the state of the service, meaning all established calls survive a failover.
Example
For illustration purposes, I have implemented a small application applying this concept. You can check the whole code on my Github repository.
The README.md file explains how to install the example. Follow the next steps to test a takeover.
Once the Pods are running check that one instance is active and the other is on standby by listing the Pods with the --show-labels parameter.
Notice that the Pods also have the label app, which is also used by the Kubernetes service to route traffic to a Pod matching app=app and mode=active.
The example application is a simple web service that keeps a state. Let’s open the browser to the main page using the minikube service command:
On the browser, we see a greeting message with the serving Pod name (hostname).
If we issue the /state request we will see the state kept by the app is Idle
Let’s issue the /start request to change the state.
The current state is now running. Notice the serving Pod name is always the same, that is the name of the active Pod.
To initiate a takeover we will delete the active Pod.
Notice that the previously passive instance has become the active one. So, if we issue a /state request:
we notice that the service Pod name has changed while keeping the state of the service.
To test the audit function we need to generate an inconsistent state by changing one of the Pod’s mode labels to match the other Pod.
After a few seconds, the inconsistency is detected by the audit application and fixed by restarting both Pods.
Conclusion
Although this concept matches very closely the leader election with sidecar its implementation is different. We considered this implementation allows for faster recovery without a single point of failure.
