Microservices - Solve your Redis fixtures problem

Redis is a fantastic tool to handle data in an amazingly simple and rapid manner. If you’re not using it yet, here’s a way you can manage redis fixtures in your projects.

Redis and microservices

In our current web project, we use Redis to handle hot data that can be updated using microservices.

To initialize our development environment, we use scripts that populate Redis from the Postgresql databases of our microservices. Our problem: As we have quite a large set of data, it was taking time to populate Redis. This should not be the case in production, but in a development environment, this is quite common.

The quickest and most simple solution we found is to load a backup of our Redis store. Fortunately for us, Redis automatically creates a backup file (/var/lib/redis/dump.rdb) that is constantly updated. All we have to do to restore a backup is to replace the /var/lib/redis/dump.rdb file with our own.

As the file is quite large, we need to compress it. This is where we were met with a significant problem that was not covered in the article: the tar command corrupts the dump.rdb file. We simply solve it by using the bzip2 command instead.

Loading the fixtures in Vagrant and Docker

If you are using Vagrant like us for your development environment, you need to run the following script inside the vm:

cd /var/www/myapp/current
cp data/backupredis.rdb.bz2 data/dump.rdb.bz2
bunzip2 data/dump.rdb.bz2
sudo service redis-server stop
sudo mv data/dump.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo service redis-server start

Notice that you need to run it as root. If you’re not using it yet, …
- You can build a new image with the correct dump.rdb.
- You can put the dump.rdb in a shared folder and you replace it during the start.sh of your container.

Updating the fixtures

As the project goes on, we need new versions of the fixtures.
To update our backup file with the data from one environment we use the following script:

#!/usr/bin/env bash
rm -f ../data/backupredis.rdb.bz2
scp root@IP_TARGET_ENVIRONMENT:/var/lib/redis/dump.rdb ../data/backupredis.rdb
bzip2 ../data/backupredis.rdb

We then commit and push the file so that developers can quickly get the last version of the fixtures. For Docker users, if have the build a new image solution, you need to rebuild the image of your container at the end of this step.

Conclusion

Dealing with the backup of a Redis store was quite simple - another of the reasons why Redis is an awesome tool. If you liked this article and don’t want to miss the next one, you can follow me on Twitter.