How to use Redis Cluster for caching

In this post, we will explore how we can use Redis as a cache layer for our application and as we explore it further, we will see how a Redis Cluster can provide us more scalability and reliability.

TLDR: If you are already familiar with Redis and it’s just looking for a way to spin-up a fully configured Redis Cluster using Docker, here is the Github repo. Just clone this repo, go to your terminal, run docker-compose up and you should be good to go.

What is Redis?

Redis is a key-value store. In rough terms, it works just like a database, but it keeps its data in memory, which means that reads and writes are orders of magnitude faster compared to relational databases like PostgreSQL. It is important to mention that Redis does not replace a relational database. It has its own use-cases and we will explore some of them in this post.

For more information about Redis, have a look at their website here. There you find good documentation and how to install it on your machine. However, we will be building a demo during this post and we will use an interesting setup using Docker and docker-compose that will spin up and configure the entire Redis cluster for you. The only thing you need available is Docker.

Using Redis for caching

Whenever we need fast access to some sort of data, we need to think about ways of keeping this data as close to the application layer as possible. If the amount of data is small enough, It’s generally a good idea to keep this data in the local memory so we have instant access. But when we talk about web applications, specially the ones that are stateless and can potentially run in multiple servers, we can’t guarantee that the data we need will be present as well as making sure that other servers in your cluster have fast access to this same data.

That is where databases are handy. We can write this data to a central place and other servers can fetch this data whenever they need. The issue with some databases is that if you really need blazing fast access, some of them won’t be able to deliver that at bullet speed. Redis is generally the go-to database whenever you need fast and reliable access to specific bits of data. It also provides us with ways to set expiration policies on that data so they are deleted automatically when they expire.

Redis is usually a good choice for storing:

• User sessions
• Authentication tokens
• Rate-limit counters

Redis is by no means limited to the use-cases above, but they fit well when you need fast data access, most often on every request coming through your servers.

What is the point of using a cluster?

It is usually common to start with a single server instance, perhaps connected to a database server which can take you a long way. But once you need to scale you application across different countries and sometimes different continents, it probably means that your application needs to be available 24h a day, 7 days a week. And robustness and reliability needs to be embedded in your application.

You need to start to think about what happens when one of your database servers go down, either because of an issue in the network or because of a faulty hardware. If you have only a single instance you will be dead in the water. If you have backups, it’s going to take sometime until you can spin up a new instance, configure it all up to your standards, restore the backup and put it back in business.

If your application is mission critical, you cannot afford to be offline for a few hours. Some applications cannot even be offline for a few minutes in the entire year. This is where a Cluster with replicas can save your skin when a problem like that happens.

A Redis Cluster makes sure that your data is automatically shared across multiple Redis instances, which will give you a higher level of reliability and availability. In case one of those instances experience any kind of failure, the other nodes can still serve content normally for your application.

Spinning up a Redis cluster

I’ve recently migrated a large web application from using a single Redis instance to a cluster with multiple shards, including multiple replicas. While we are using AWS infrastructure that provides us the entire cluster configuration, I couldn’t simply trust that everything would work in production. I had to make sure that we could support a Redis cluster during development, so I’ve created a setup that spawns several Redis containers and connect with each other automatically to form a cluster.

To connect to Redis from your application, you will need a library that can perform that for you (Otherwise you have to reinvent the wheel). While I’ve been using IORedis for a nodeJS application in this demo, if you have been using a different language, you will have to look for different connectors like Lettuce for Java or perhaps go-redis for Go.

The entire setup is ready for you in this Github repository here, so you don’t have to worry about creating anything from scratch. You can clone it and give it a spin while we will be walking through the files from this repo along the rest of this blogpost.

Creating a Dockerfile

While we will be using the standard Redis image available from Dockerhub to spin up several Redis containers, we still need a way to connect them. That is where we will be building a special container that can issue commands to Redis in a way that it can form a cluster.

at redis/Dockerfile we have the following content:

1FROM redis:latest
2 
3COPY ./entrypoint.sh /entrypoint.sh
4RUN chmod 755 /entrypoint.sh
5 
6ENTRYPOINT ["/entrypoint.sh"]

We will be using this Dockerfile to build our custom Docker image based on Redis. The secret sauce here is actually in at redis/entrypoint.sh. Let’s have a look at this script:

1#!/bin/sh
2 
3# Using the redis-cli tool available as default in the Redis base image
4# we need to create the cluster so they can coordinate with each other
5# which key slots they need to hold per shard
6 
7# wait a little so we give some time for the Redis containers
8# to spin up and be available on the network
9sleep 5
10# redis-cli doesn't support hostnames, we must match the
11# container IP addresses from our docker-compose configuration.
12# `--cluster-replicas 1` Will make sure that every master
13# node will have its replica node.
14echo "yes" | redis-cli --cluster create \
15  173.18.0.2:6379 \
16  173.18.0.3:6379 \
17  173.18.0.4:6379 \
18  173.18.0.5:6379 \
19  173.18.0.6:6379 \
20  173.18.0.7:6379 \
21  --cluster-replicas 1
22 
23echo "🚀 Redis cluster ready."

Here we are using the redis-cli to issue commands. This command is creating a cluster and pointing to the specific Redis instances that will be reachable when we start this script. we are using hard-coded IP addresses here that will be provided by our docker-compose.yml file later on.

This cluster is composed by 3 shards. Each shard has a master node that is responsible for all the writes, but also a Replica node that holds a copy of the data. A Redis Cluster shard can have up to 500 replicas (at least in AWS). A Replica node has the power to take over and become the Master node if the current Master becomes unavailable.

Now notice that inside of our redis folder we also have a file called redis.conf. This file will be copied to each Redis container later on so they can instruct the Redis instance to work as part of a cluster. Let’s have a look at its contents:

1# Custom config file to enable cluster mode
2# on all Redis instances started via Docker
3port 6379
4cluster-enabled yes
5# The cluster file is created and managed by Redis
6# We just need to declare it here
7cluster-config-file nodes.conf
8cluster-node-timeout 5000
9appendonly yes

There is not much going on there. The important part is cluster-enabled yes which enables our Redis instance to act as part of the cluster. We now need a way to spin up several Redis containers and make sure that they talk to each other. At the root folder of our project we have the docker-compose.yml. Let’s have a look:

1volumes:
2  redis_1_data: {}
3  redis_2_data: {}
4  redis_3_data: {}
5  redis_4_data: {}
6  redis_5_data: {}
7  redis_6_data: {}
8  # This volume is specific for the demo Express application
9  # built in this repo. You probably won't need that on your own setup.
10  node_modules: {}
11 
12services:
13  app:
14    container_name: express_app
15    image: express_app
16    build:
17      context: .
18    environment:
19      PORT: 4000
20      NODE_ENV: production
21      REDIS_CLUSTER_URLS: "redis_1:6379,redis_2:6379,redis_3:6379,redis_4:6379,redis_5:6379,redis_6:6379"
22    volumes:
23      - .:/app
24      - node_modules:/app/node_modules
25    command: ["npm", "run", "dev"]
26    depends_on:
27      - redis_1
28      - redis_2
29      - redis_3
30      - redis_4
31      - redis_5
32      - redis_6
33      - cluster_initiator
34    ports:
35      - "4000:4000"
36    stdin_open: true
37    networks:
38      redis_cluster_net:
39        ipv4_address: 173.18.0.10
40 
41  # Here we have six Redis containers with Cluster mode enabled,
42  # three of them will work as master nodes and each one of
43  # will have a replica, so in case of failures, the replica becomes the master.
44  # They are configured by the `cluster_initiator` container.
45  redis_1:
46    image: "redis:latest"
47    container_name: redis_1
48    ports:
49      - "6379"
50    volumes:
51      - redis_1_data:/data
52      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf
53    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
54    networks:
55      redis_cluster_net:
56        ipv4_address: 173.18.0.2
57 
58  redis_2:
59    image: "redis:latest"
60    container_name: redis_2
61    ports:
62      - "6379"
63    volumes:
64      - redis_2_data:/data
65      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf
66    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
67    networks:
68      redis_cluster_net:
69        ipv4_address: 173.18.0.3
70 
71  redis_3:
72    image: "redis:latest"
73    container_name: redis_3
74    ports:
75      - "6379"
76    volumes:
77      - redis_3_data:/data
78      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf
79    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
80    networks:
81      redis_cluster_net:
82        ipv4_address: 173.18.0.4
83 
84  redis_4:
85    image: "redis:latest"
86    container_name: redis_4
87    ports:
88      - "6379"
89    volumes:
90      - redis_4_data:/data
91      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf
92    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
93    networks:
94      redis_cluster_net:
95        ipv4_address: 173.18.0.5
96 
97  redis_5:
98    image: "redis:latest"
99    container_name: redis_5
100    ports:
101      - "6379"
102    volumes:
103      - redis_5_data:/data
104      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf
105    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
106    networks:
107      redis_cluster_net:
108        ipv4_address: 173.18.0.6
109 
110  redis_6:
111    image: "redis:latest"
112    container_name: redis_6
113    ports:
114      - "6379"
115    volumes:
116      - redis_6_data:/data
117      - ./redis/redis.conf:/usr/local/etc/redis/redis.conf
118    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
119    networks:
120      redis_cluster_net:
121        ipv4_address: 173.18.0.7
122 
123  # Ephemeral container to create the Redis cluster connections.
124  # Once the setup is done, this container shuts down
125  # and the cluster can be used by the service app container
126  cluster_initiator:
127    container_name: cluster_initiator
128    build:
129      context: redis
130      dockerfile: Dockerfile
131    tty: true
132    depends_on:
133      - redis_1
134      - redis_2
135      - redis_3
136      - redis_4
137      - redis_5
138      - redis_6
139    networks:
140      redis_cluster_net:
141        ipv4_address: 173.18.0.8
142 
143  # Web UI to browse through our Redis data across all nodes
144  redis_commander:
145    image: rediscommander/redis-commander:latest
146    container_name: redis_web
147    environment:
148      REDIS_HOSTS: "local:redis_1:6379,local:redis_2:6379,local:redis_3:6379"
149    ports:
150      - "5000:8081"
151    depends_on:
152      - redis_1
153      - redis_2
154      - redis_3
155      - redis_4
156      - redis_5
157      - redis_6
158      - cluster_initiator
159    networks:
160      redis_cluster_net:
161        ipv4_address: 173.18.0.9
162 
163# Rename the default network so we can easily identify it
164# Across all containers
165networks:
166  redis_cluster_net:
167    driver: bridge
168    ipam:
169      driver: default
170      config:
171        - subnet: 173.18.0.0/16

This is a long one, but here is what this docker-compose.yml does:

• Creates a container with our Express application (just for the sake of this demo)
• Creates several instances of Redis
  • Configure their IP addresses to match the ones used in our entrypoint.sh script
  • Copy the redis.conf file so they can act as a cluster
• Creates a cluster initiator container that is only necessary for executing our entrypoint.sh script and make the cluster connection
• Creates a container with the Redis Commander UI which is a nice Web UI for browsing what is stored in our Redis Cluster

Now that we went through this, let’s try this out. Go to your terminal and execute:

1docker-compose up

Once everything is ready, you should be able to open your browser and visit localhost:4000. There you have a demo web application I’ve built where you can enter a key/value pair and save it to Redis and also search for a specific key you have entered before so it can fetch it from Redis and show you the contents on the screen.

If you are wondering how the connection is setup on the JavaScript side, let’s have a look at our src/service/redisClient.js file.

1const Redis = require("ioredis");
2 
3/**
4 * Get an existing Redis client instance. Build one if necessary
5 * @return {Cluster|null} redis client
6 * */
7function buildRedisClient() {
8  try {
9    // cluster URLs should be passed in with the following format:
10    // REDIS_CLUSTER_URLS=10.0.0.1:6379,10.0.0.2:6379,10.0.0.3:6379
11    const nodes = process.env.REDIS_CLUSTER_URLS.split(",").map((url) => {
12      const [host, port] = url.split(":");
13      return { host, port };
14    });
15 
16    const client = new Redis.Cluster(nodes, {
17      redisOptions: {
18        enableAutoPipelining: true,
19      },
20    });
21 
22    client.on("error", (error) => {
23      console.error("Redis Error", error);
24    });
25 
26    // Redis emits this error when an something
27    // occurs when connecting to a node when using Redis in Cluster mode
28    client.on("node error", (error, node) => {
29      console.error(`Redis error in node ${node}`, error);
30    });
31 
32    return client;
33  } catch (error) {
34    console.error("Could not create a Redis cluster client", error);
35 
36    return null;
37  }
38}
39 
40module.exports = buildRedisClient;

This part is very simple. It reads the cluster URLs from the environment and creates an instance of Redis.Cluster using the RedisIO library. From there on we can start issue commands like redis.set, redis.get or redis.exists across our application. Here is how we do that in the demo Express app within this repo:

1const buildRedisClient = require("./service/redisClient");
2const redis = buildRedisClient();
3 
4// Have a look at src/index.js for a complete implementation
5app.post("/save-data", async (request, response) => {
6  const { key, value } = request.body;
7  await redis.set(key, value);
8  return response.status(201).render("home/index", {
9    layout: "default",
10    dataSaved: true,
11  });
12});
13 
14app.post("/search", async (request, response) => {
15  const { key } = request.body;
16  const value = await redis.get(key);
17  return response.status(200).render("home/index", {
18    layout: "default",
19    value,
20  });
21});

If you would like to explore the data stored in the cluster, go to localhost:5000 and browse through the Redis Commander UI. There you should be able see all the Master nodes and explore all keys and values.

You will notice that some keys are stored in one Master node and other keys are stored in other nodes. This is the data distribution done by Redis, which provides you load balancing across the cluster.

I hope this Docker setup can help your development workflow the same way it did for me and my team recently. Feel free to DM me via Twitter if you have any questions.