We all know this. You get something to work on your own development machine, but don’t really feel confident enough to do the same on a production box. In development, you can always wipe out everything and start over. You cannot do the same in production.
Fortunately, Redis has always been a nice service to deal with. There are, however, a few things you need to take care of when your website should depend on Redis. You want to make sure everything is right, starting from the installation process to a proper backup mechanism.
Here are a few tips for setting up Redis from the very beginning to get you going. The assumption is that you have a clean machine running Ubuntu Server 12.04 or newer.
The current best practice is to use a configuration management tool and treat our setup as code. We are very happy users of Puppet ourselves (Chef is also very popular), but in this case, let’s just keep things simple and use our system’s packaging system. We can revisit our setup in a future post.
The first thing we want to do is to add a reliable source of the latest stable release of Redis (2.6.14 at the moment). To do that, we can just add a PPA from our hero Chris Lea and install the
$ sudo add-apt-repository ppa:chris-lea/redis-server $ sudo apt-get update $ sudo apt-get install redis-server
If everything went well, you should now have a running Redis server. The package also installed a simple
init.d script so we can start, restart and stop as usual. Also we don’t have to worry about starting Redis ourselves after reboot.
We can also do a quick check by running the Redis Benchmark. It will not only stress test Redis itself but also verify that there’s nothing wrong with our installation. The following will issue 100.000 requests from 50 clients sending 12 commands at the same time.
$ redis-benchmark -q -n 100000 -c 50 -P 12
Redis comes with a very reasonable and heavily commented configuration file. Take a look at the conf file at
/etc/redis/redis.conf and walk through.
One of the most important setting you need to be aware of are the persistent options. Redis provides three: RDB, AOF and both.
The RDB option represents point-in-time snapshots of the current dataset. A snapshot is represented as a regular file located at
dump.rdb (by default). Redis regularly dumps its dataset to the file, making it the perfect candidate for backups.
You can change how often Redis creates these snapshots with the
save statement in the configuration file. To dump the dataset every 15 minutes (900 seconds) if at least one key changed, you can say:
save 900 1
You can have more than one of these statements to make your snapshotting more granular. Note that there are already a few of these statements in the configuration file.
Now what happens if your machine crashes before it has a chance to save the newest dataset? Your data get lost. Sometimes is not a big deal - you might be using Redis as a cache storage and you simply don’t mind losing its data. If, however, the data is important to you and you want to make sure they are safe, you should consider AOF.
AOF (append only file) works as log of all incoming write operations. Redis instantly writes to the log file so even if your machine crashes, it can still recover and have the latest data. Similar to RDB, AOF log is represented as a regular file at
appendonly.aof (by default).
The problem with this though is that the operating system maintains an output buffer and a successful write doesn’t necessarily mean that the data gets written (flushed) to the disk immediately. To tell OS to really really write the data to the disk, Redis needs to call the
fsync() function right after the write call, which can be slow. That’s why Redis provides 3 options, from the docs:
everysec option is the default and it’s sufficient for most installations:
As mentioned in the beginning, you can safely use both RDB and AOF at the same time, I would recommend it for every non cache usage.
A few more notes:
Do not forget to restart Redis after you make any configuration changes:
$ sudo service redis-server restart
One last thing is to set the
vm.overcommit_memory variable to
1. This kernel setting affects how memory allocation is handled when a parent forks a child that starts to change the shared copy-on-write memory page. Without this setting, Redis might fail when dumping its dataset to disk. You can find more information about this at the FAQ. Edit
/etc/sysctl.conf, add the line and reboot the box:
vm.overcommit_memory = 1
Everything seems to work fine now but we want to make sure it stays that way. One way to do that is to setup a monitoring service that watches other processes and is able to react if they crash or otherwise misbehave. Our tool of choice for the job is Monit (there’s plenty of others). The installation process is again very simple:
$ sudo apt-get install monit
The configuration file (
/etc/monit/monitrc) is again very heavily commented and no significant changes are required. One thing we want to do, however, is to enable Monit’s internal web server so we can query status of services from terminal and be able to see its web interface. Add the following at the bottom of
set httpd port 8081 and use address localhost # only accept connection from localhost allow localhost # allow localhost to connect to the server and allow admin:monit # require user "admin" with password "monit"
Next, we need to tell Monit to monitor Redis. To do that, just create a file at
/etc/monit/conf.d/redis.conf with the following:
check process redis-server with pidfile "/var/run/redis/redis-server.pid" start program = "/etc/init.d/redis-server start" stop program = "/etc/init.d/redis-server stop" if failed host 127.0.0.1 port 6379 then restart if 5 restarts within 5 cycles then timeout
The file is pretty self explanatory, just note that a cycle refers to the polling interval (defaults to 120 seconds). There’s a lot you can add here such as custom alerts when CPU or RAM exceeds certain boundaries. Take a look at Monit’s documentation for all the possibilities.
To make Monit aware of these changes, reload its configuration:
$ sudo service monit reload
Now we should be able to query Monit for service statuses:
$ sudo monit status
Or we can visit
http://localhost:8081, login with
redis-server and see an output similar to this:
Note that if you are working with a remote system via SSH, you can’t directly access the remote box’s
localhost URL. A solution to this is to simply forward port from the remote machine to your local machine:
ssh -L 8081:localhost:8081 [email protected]_remote_machine -f -N
Now you can access
http://localhost:8081 and it will be as if you would issue the request from the remote machine.
Finally, let’s not forget about the log file located at
/var/log/redis/redis-server.log as it’s usually the first place to look if there’s something wrong (Redis won’t start for example).
As explained in the Configuration section, Redis provides RDB snapshots that are perfect for making backups. The RDB file contains every single piece of data you have stored and so you can safely backup just the RDB file and be able to restore if an accident happens. To make a backup, copy the RDB file:
cp /var/lib/redis/dump.rdb /somewhere/safe/dump.$(date +%Y%m%d%H%M).rdb
Simple as that. There’s no need to prepare the server, you can even do this while the server is still running because Redis dumps to a temporary file and then atomically renames to the final one, keeping the previous file available until you finish copying. As a security precaution, you should consider encrypting the file before you upload it to an external storage. A reasonable approach to automatize the process is via a cron task.
Now let’s imagine there’s been an accident (e.g. someone deleted some data) and we want to restore a previous version of our dataset from these RDB backups. The procedure depends on whether you have your Redis server configured to use AOF mode or not.
Without AOF, the restore process is really simple. First, make sure that nobody is working with or is connected to the Redis server. Next, take it down with Monit’s stop command (otherwise Monit would start the service automatically):
$ sudo monit stop redis-server
If you are not using a monitoring service, you can simply:
$ sudo service redis-server stop
Next, we want to delete the current dataset snapshot (
dump.rdb by default).
$ cd /var/lib/redis $ sudo rm -f dump.rdb
Now there’s no snapshot file, so we need to copy the backup one back in place:
$ sudo cp /backup/dump.rdb . $ sudo chown redis:redis dump.rdb
Notice the second line, we need to make sure that the snapshot is owned by the same user as the Redis server is running as. Now, we can just start the Redis server and we are done.
$ sudo monit start redis-server
With AOF, the process is a little bit more involved. When Redis starts, it looks at the AOF log as the primary source of information because it has always the newest data available. The problem is that if we only have a RDB snapshot and no AOF log (or we have a log but it has been affected by the accident so we cannot use it), Redis would still use the AOF log as the only data source. And since the log is missing (same as empty), it would not load any data at all and create a new empty snapshot file.
The following solution takes advantage of the fact that you can recreate the AOF log from the current dataset even if the AOF mode is currently disabled.
Similar to the RDB only, without AOF section, take down the server, delete any snapshots and logs and finally copy the backup
$ sudo rm -f dump.rdb appendonly.aof $ sudo cp /backup/dump.rdb . $ sudo chown redis:redis dump.rdb
Now the interesting part. Go to the configuration file (
/etc/redis/redis.conf) and disable AOF:
Next, start the Redis server and issue the
$ sudo monit start redis-server $ redis-cli BGREWRITEAOF
It might take some time, you can check the progress by looking at the
aof_rewrite_in_progress value from the info command (0 - done, 1 - not yet):
$ redis-cli info | grep aof_rewrite_in_progress
Once it’s done, you should see a brand new
appendonly.aof file. Next, stop the server, turn the AOF back on and start the server again.
Now you might be asking why we don’t just backup the AOF log. The answer is that you could but the AOF logs tend to get really large and we don’t usually want to waste disk space.
That’s all for a basic Redis setup. You should be good to go. There are plenty of things you can do to make the setup more robust, e.g. slave replication, but I would like to address that in a separate post as it’s a big topic.
We are very interested in your own tips & tricks. Please share in the comments so we can all learn something new.Written by Jiri Pospisil of sensible.io.
We're building a tool to help businesses reach out to their customers more easily. It's called SendingBee and it's going to be awesome.