How Redis Pods Find the Master using init containers (Master Election and Failover) - Part 2

Now Let's Look at the Sentinel Init Container

Compared with Redis, the Sentinel init container is much simpler:

initContainers: - args: - | cat > /etc/redis/sentinel.conf <<EOF port 5000 sentinel resolve-hostnames yes sentinel announce-hostnames yes sentinel monitor mymaster redis-0.redis.redis.svc.cluster.local 6379 2 sentinel down-after-milliseconds mymaster 5000 sentinel failover-timeout mymaster 60000 sentinel parallel-syncs mymaster 1 sentinel auth-pass mymaster "$REDIS_PASSWORD" EOF command: - sh - -c env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: key: auth name: redis-secret

Unlike the Redis init container, this one isn't trying to decide:

Am I master or replica?

Sentinel isn't a Redis data node.

Its init container simply needs to:

Create the configuration file that the Sentinel process will use.

Let's start with:

cat > /etc/redis/sentinel.conf <<EOF

This is shell syntax for writing multiple lines into a file.

Everything until the final:

EOF

gets written into:

/etc/redis/sentinel.conf

So conceptually:

cat > file <<EOF hello world EOF

creates a file containing:

hello world

In our case, we're creating Sentinel's configuration.

Next:

port 5000

This tells Sentinel:

Listen on TCP port 5000.

That's why the Redis init container contacts Sentinel using:

redis-cli -h sentinel -p 5000

The two configurations agree:

Sentinel: "I listen on 5000" Redis: "I'll contact Sentinel on 5000"

Next:

sentinel resolve-hostnames yes

Our setup uses hostnames such as:

redis-0.redis.redis.svc.cluster.local

So this setting allows Sentinel to resolve/use those hostnames when working with Redis nodes.

This is important because the configuration isn't giving Sentinel a fixed Redis pod IP.

It's giving it a Kubernetes DNS name.

Next:

sentinel announce-hostnames yes

This tells Sentinel to announce hostnames rather than relying only on IP addresses when communicating node information.

Again, this matches the general design of this setup:

Avoid depending on changing pod IPs.

Prefer stable StatefulSet DNS identities.

The Redis side also explicitly announces its stable FQDN with:

replica-announce-ip <FQDN>

So both pieces are designed around hostname-based identities.

Now:

sentinel monitor mymaster redis-0.redis.redis.svc.cluster.local 6379 2

There are several pieces here.

Let's separate them:

sentinel monitor mymaster redis-0.redis.redis.svc.cluster.local 6379 2

mymaster

This is the logical name Sentinel gives this Redis master group.

Think:

mymaster = name of this monitored Redis setup

It's also why the Redis init container asks:

sentinel get-master-addr-by-name mymaster

The names need to match.

redis-0.redis.redis.svc.cluster.local

This is the initial Redis master address Sentinel should monitor.

Again, this does not mean:

redis-0 can never stop being master.

It means:

When Sentinel first starts, begin by looking at redis-0 as the master.

After a failover, Sentinel can track a different master.

6379

That's the Redis server port.

2

This is the quorum value.

You have three Sentinel instances:

sentinel-0 sentinel-1 sentinel-2

The configured quorum is:

2

So Sentinel uses a quorum of two when determining that the monitored master is objectively down.

At a simplified level:

sentinel-0 → "master looks down" sentinel-1 → "master looks down" ``` ↓ ``` quorum reached

Next:

sentinel down-after-milliseconds mymaster 5000

5000 milliseconds is five seconds.

This configures how long a Sentinel waits without acceptable replies before considering the monitored instance subjectively down.

In simplified terms:

Redis stops responding ↓ Sentinel waits ↓ 5 seconds ↓ "This master looks down to me."

That local judgment is part of Sentinel's failure-detection process.

Next:

sentinel failover-timeout mymaster 60000

60000 milliseconds is 60 seconds.

This configures Sentinel's failover timeout for the mymaster group.

The setting participates in Sentinel's failover timing and retry behavior.

For understanding this particular init container, the simple mental model is:

down-after-milliseconds = how quickly failure suspicion begins failover-timeout = timing control around the failover process

Next:

sentinel parallel-syncs mymaster 1

Imagine this:

redis-0 MASTER ❌ fails

Sentinel promotes:

redis-2 → NEW MASTER

Now other replicas may need to synchronize with the new master.

parallel-syncs 1 says only one replica should be reconfigured/synchronized at a time during this part of failover.

With our small three-node setup, you can think of it as controlling how aggressively replicas synchronize after promotion.

Finally:

sentinel auth-pass mymaster "$REDIS_PASSWORD"

Sentinel needs credentials to communicate with the password-protected Redis nodes.

But we don't want to hard-code the password directly into this YAML.

So the init container gets it from a Kubernetes Secret.

Where Does $REDIS_PASSWORD Come From?

This part:

env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: key: auth name: redis-secret

means:

Create an environment variable called REDIS_PASSWORD.

But don't hard-code its value.

Instead, get it from:

Kubernetes Secret: redis-secret Key: auth

So conceptually:

Kubernetes Secret redis-secret | └── auth: ******** | ↓ REDIS_PASSWORD | ↓ init container | ↓ sentinel.conf

When the shell creates sentinel.conf, $REDIS_PASSWORD is expanded to the value provided to the init container.

What Happens After the Sentinel Init Container Finishes?

The init container creates:

/etc/redis/sentinel.conf

and exits.

The main Sentinel container then starts with:

redis-sentinel /etc/redis/sentinel.conf

So:

Sentinel pod created ↓ Init container ↓ Get Redis password ↓ Create sentinel.conf ↓ Init container exits ↓ Main Sentinel container starts ↓ redis-sentinel /etc/redis/sentinel.conf ↓ Sentinel starts monitoring Redis

And remember: this happens separately in all three Sentinel pods.

sentinel-0 ├── init → creates config └── Sentinel starts sentinel-1 ├── init → creates config └── Sentinel starts sentinel-2 ├── init → creates config └── Sentinel starts

How the Two Init Containers Work Together

This is the part that finally made the architecture click for me.

The Sentinel init container says:

"I'm going to create a configuration telling Sentinel where the Redis master starts and how to monitor it."

The Redis init container says:

"I'm going to find out who the current master is and configure this Redis pod appropriately."

So:

SENTINEL INIT | ↓ Create sentinel.conf | ↓ "Start monitoring redis-0" | ↓ Sentinel starts ``` REDIS INIT | ↓ "Can I ask Sentinel?" | +-------+-------+ | | NO YES | | ↓ ↓ Assume redis-0 Ask Sentinel is master who master is | | +-------+-------+ | ↓ Configure redis.conf | ↓ Redis starts ```

First-Time Startup

Imagine the whole system is being deployed for the first time and Redis starts before Sentinel.

redis-0:

Init runs ↓ Sentinel unavailable ↓ I am redis-0 ↓ No replicaof ↓ Start as MASTER

redis-1:

Init runs ↓ Sentinel unavailable ↓ I am not redis-0 ↓ replicaof redis-0 ↓ Start as REPLICA

redis-2:

Init runs ↓ Sentinel unavailable ↓ I am not redis-0 ↓ replicaof redis-0 ↓ Start as REPLICA

We get:

redis-0 MASTER / \ redis-1 redis-2 REPLICA REPLICA

Then Sentinel starts.

Each Sentinel init container creates a config saying:

Start monitoring redis-0 as "mymaster".

The Sentinel processes start monitoring the Redis deployment.

What Happens During a Failover?

Now suppose:

redis-0 MASTER

fails.

The Sentinels detect the failure and, assuming the necessary Sentinel conditions are met, a failover can occur.

Suppose redis-2 gets promoted.

Now:

redis-0 DOWN redis-1 REPLICA | ↓ redis-2 MASTER

Sentinel now knows:

mymaster = redis-2

This is where the Redis init logic becomes particularly useful.

What Happens When a Redis Pod Restarts After Failover?

Suppose redis-1 gets restarted.

Its init container starts from scratch.

It asks:

Can I contact Sentinel?

Yes.

Then:

Who is master?

Sentinel responds:

redis-2

The init container therefore writes:

replicaof redis-2... 6379

into redis-1's config.

Then Redis starts.

So:

redis-1 | | asks Sentinel ↓ "Who is master?" ↓ "redis-2" ↓ redis-1 config: replicaof redis-2 ↓ redis-1 starts as replica

Now imagine redis-2 itself restarts.

Its init container asks Sentinel:

Who is master?

Sentinel says:

redis-2

The script compares:

MASTER == MY_FQDN

and realizes:

That's me!

So it does not add a replicaof directive.

It starts as the master.

The Mental Model Worth Remembering

If you forget every shell command in these configurations, remember these two sentences.

Redis init container

"Before Redis starts, figure out who the current master is and configure myself as either master or replica."

Sentinel init container

"Before Sentinel starts, create the configuration that tells Sentinel what Redis group to monitor and how to monitor it."

Or even shorter:

Redis init = "Who should I follow?" Sentinel init = "What should I monitor?"

Once you see it that way, all the shell scripting is really just implementation detail.

Next, we will have a look into examples of full statefulset yamls of both Redis and Sentinel

#k8s #redis #sentinel