Redis replication depends on one node being treated as the master, while the remaining Redis nodes act as replicas. If that master role is assigned incorrectly, especially when pods are starting, restarting, or recovering from a failure, the cluster can easily end up in an inconsistent state.
For example, imagine three Redis pods starting at the same time:
If all three simply started as independent Redis servers without knowing their role, we could temporarily end up with:
That is not the topology we want.
Normally we want something closer to:
And later, if redis-0 fails, Sentinel may promote one of the replicas:
At that point, redis-0 is no longer the source of truth for who the master is.
This creates an important startup problem.
A Redis pod may need to answer:
Who is the master right now?
During the very first deployment, Sentinel may not even be running yet. In that case, the Redis pods still need some predictable way to bootstrap the cluster.
But after the cluster is already running and Sentinel has performed a failover, blindly assuming that redis-0 is always the master would be wrong.
That is exactly why the Redis init container in this setup exists.
What Is an Init Container?
An init container runs first, executes in a strict step-by-step order and stops when only exited with a success code before the main application starts.
Suppose our Redis StatefulSet creates three pods:
Each pod gets its own init container.
So it's not:
Instead, it's:
For every pod, Kubernetes follows roughly this sequence:
So if we're talking about redis-2:
If the pod gets recreated later, its init container runs again.
That's important because the Redis init container can make a new decision about who the current master is every time a Redis pod starts.
In our case Redis StatefulSet has three Redis pods:
Normally we want something like:
We also have three Sentinel pods:
Their job is to monitor Redis.
So overall:
The interesting question is:
How does each Redis pod know whether it should start as a master or replica?
That's where the Redis init container comes in.
Redis Init Container
Here is the important script:
This looks intimidating, but the entire thing is asking one question:
Who should I replicate from?
If the answer is "nobody, because I am the master," it doesn't add a replicaof line.
If another Redis pod is master, it adds:
Let's go through it line by line.
First:
This means:
Take our starting Redis configuration and make a working copy.
Conceptually:
Why copy it?
Because the init container wants to dynamically add settings to the configuration before Redis starts.
The main Redis container later starts with:
So /etc/redis/redis.conf is the important final configuration.
Next:
hostname -f asks Linux for the pod's fully qualified hostname.
Suppose we're inside redis-2.
It might return something like:
So now our variable contains:
In simple terms:
If this same script runs inside redis-1, it gets redis-1's address instead.
Remember: the exact same init script runs inside every Redis pod.
The result is different because each pod has a different hostname.
Next comes the slightly ugly-looking line:
This does not rename the pod.
That's worth repeating:
It does not rename redis-2 to redis-0.
It's only manipulating a string.
Suppose we're inside:
and:
returns:
The sed command changes the redis-2. part of that string to redis-0..
So:
becomes
The pod itself is still:
We simply created a variable containing redis-0's address:
Why do we need this?
Because if Sentinel isn't available yet, the setup needs some initial rule for deciding who should be master.
The rule chosen here is:
If we don't know who the master is, use redis-0 as the starting master.
That's why the variable is called SEED_MASTER.
Next:
Suppose we're inside redis-2.
This adds something like:
to redis.conf.
The >> means:
Append this line to the end of the file.
Why do this?
Kubernetes pod IPs can change.
For example, redis-2 could currently have:
After the pod is recreated, it might get:
But its StatefulSet DNS identity remains stable:
So this setup wants Redis/Sentinel to work with that stable hostname rather than depending on an ephemeral pod IP.
Next:
That's just a log message.
Then:
This is basically the Redis pod asking:
"Hey Sentinel, are you there?"
If Sentinel is alive, it should return:
So this condition:
means:
If Sentinel does NOT answer with PONG, run the cold-start logic.
The:
just hides error output from the failed connection attempt.
Now we have two possible paths:
Suppose Redis starts before Sentinel.
The script reaches:
Now we have a problem.
We can't ask Sentinel:
"Who is the master?"
because Sentinel isn't running.
So we use our fallback rule:
redis-0 is the initial master.
The script checks:
Meaning:
Am I redis-0?
If the hostname is:
the script does:
Notice something important:
It doesn't actually run a command saying:
There isn't one.
Instead, it simply does not add a replicaof setting.
So the final Redis configuration does not contain:
Redis therefore starts without being configured as anyone's replica.
In this setup, that means it starts as the master.
Now suppose the script is running inside redis-2.
This condition:
is false.
So it goes into:
and runs:
Remember:
So the actual line added to redis.conf is roughly:
That tells Redis:
I am a replica. Connect to redis-0 and replicate from it.
Again, redis-2 does not become or rename itself to redis-0.
It remains:
It's simply configured to follow:
What If All Redis Pods Start Before Sentinel?
This was an interesting scenario we discussed.
Imagine Sentinel isn't running yet.
redis-0 runs its init container:
Then redis-1:
Then redis-2:
We end up with:
So Redis can bootstrap even when Sentinel hasn't started yet.
That's exactly why the SEED_MASTER fallback exists.
What If redis-2 Starts Before redis-0?
Conceptually, if redis-2 runs this cold-start logic while redis-0 isn't available yet, it still configures itself with:
That does not mean redis-2 becomes redis-0.
It means:
I am redis-2.
My master should be redis-0.
I'll try to connect to it.
So conceptually:
In the supplied StatefulSet, podManagementPolicy is OrderedReady, so normal StatefulSet creation is intended to bring up the pods in ordinal order rather than starting redis-2 first. But the distinction above is still useful for understanding what the script itself does.
What Changes Once Sentinel Exists?
Now imagine Sentinel is running.
This check:
returns:
So instead of using the redis-0 fallback, we go into the second half of the script.
This is where things become more interesting.
Ask Sentinel Who the Current Master Is
The script runs:
This looks complicated, but conceptually it's:
Redis → Sentinel
"Who is the current master for mymaster?"
Sentinel might respond with:
The:
takes only the first line.
So we get:
Then:
simply prints that information into the init-container logs.
What If Sentinel Doesn't Return a Master?
Next:
-z checks whether the string is empty.
So this means:
Sentinel is reachable, but it didn't give me a master address.
The script then runs:
This is just a shorter shell way of saying:
So again:
What If Sentinel Says "You Are the Master"?
Next:
Suppose this script is running inside redis-2.
We have:
Sentinel also says:
They match.
Therefore:
Sentinel says redis-2 is master.
I am redis-2.
Therefore I am the current master.
The script prints:
Again, notice what it doesn't do.
It doesn't add:
So Redis starts as the master.
What If Sentinel Says Someone Else Is Master?
Finally:
Suppose we're starting redis-1.
Our identity is:
But Sentinel says:
The script therefore writes:
into redis.conf.
Meaning:
This is what makes the script useful after a failover.
Why Ask Sentinel Instead of Always Using redis-0?
Imagine the cluster originally looks like:
Then redis-0 fails.
Sentinel performs a failover and promotes redis-2.
Now:
Later redis-1 restarts.
If its init script blindly assumed:
redis-0 is always master
it would configure itself incorrectly.
Instead it asks Sentinel:
Who is master?
Sentinel says:
So redis-1 gets:
That's the important difference between:
and:
SEED_MASTER means:
"Our fallback starting point when Sentinel can't tell us anything."
MASTER means:
"The master Sentinel currently knows about."
Now Let's Look at the Sentinel Init Container
#k8s #redis #sentinel
0 Comments