Backup & Restore Your Logstash/Grafana Dashboards

I created a Chef cookbook with which you can backup and restore your Logstash/Grafana dashboards. You can find it here.

It wouldn’t be an exaggeration to say that Logstash and Grafana have changed my life. I can’t even remember how I was monitoring or investigating performance issues before having them. When I first installed those tools and started to feed them with data I was really excited by the possibilities they offered me. Building dashboards was so easy and fluid. Create a widget, select the data to display and viola – you have a neat looking graph! With time I added more and more dashboards both to Logstash and Grafana. I now have dozens in each. They show me everything I need to know and when I find they have a missing piece – I add it right away.

A sample Grafana dashboard
A sample Grafana dashboard

Everything went well, life was good and the sun was shining, until one day the worse thing happened – The servers went down and were not accessible anymore by SSH. They were hosted on EC2 so I contacted AWS support and they said the servers had some hardware failure and are lost. For good. Now there are two big problems with this:

  1. All the data which was accumulated in Graphite / ElasticSearch was lost
  2. The dashboards themselves were lost

To tell the truth, I cared less about the data. We didn’t have any crucial data which we needed to store for years. But the dashboards – I really cared about them! When I realised I am going to have to recreate all of them I was deeply disturbed, but instead of drinking a lot of whiskey to make the problems disappear, I decided to take an action so that this kind of things will never happen to me again. I started looking for a way to backup Logstash/Grafana dashboards. After I didn’t find any concrete documentation I opened the inspect element tab in chrome and went over the network logs when I save and load dashboards from files – A feature which is available both on Logstash and Grafana. I found out that both frameworks store dashboards in a dedicated index in an ElasticSearch server. Logstash (via Kibana) saves them in an index called kibana-int and Grafana stores them in an index called grafana-index. Great, now I only needed to figure out how to backup and restore indices in ElasticSearch. My backup storage of choice was S3 since I’m already inside AWS’s ecosystem.

Installing ES AWS plugin

In order to integrate ES with S3 you will need to install a plugin called elasticsearch-cloud-aws. It is a one command line installation, the command itself varies between ES versions so  I saw no reason to paste it here. Just go to the GitHub repo and pick the command which is appropriate to your ES version. You will need to provide the plugin with some AWS access key and secret so it can communicate with AWS to save data to the S3 bucket.

Creating Backup Repository

Now you need to create a backup repository. Because we are using S3 our repository will be a S3 bucket. The plugin we installed in the last section allows us to actually create such a repository. To create the repository you just send an HTTP PUT request to with the following request body: {“type”: “s3”, “settings” { “bucket”: “bucket-name”, “region”: “bucket-region”, base_path: “folder-name”}} The simplest way to do it is to use CURL. Your command should look like this:

curl -XPUT "" -d '{
  "settings": {
    "bucket": "bucket-name",
    "region": "bucket-region",
    "base_path": "folder-name"

It will create an ES backup repository named repository-name which stores backups in bucket-name/folder-name on S3.

Backup Your ES Index

Now we want to actually store the index in our newly created repository. We will need to send an HTTP PUT request which, in curl, looks like this:

curl -XPUT "" -d '{
  "indices": "kibana-int",
  "ignore_unavailable": "true",
  "include_global_state": false

This example will store the kibana-int index in the S3 bucket defined in repository-name. The snapshot-name is a name we assign to this snapshot to be able to restore it (or delete it) later. I have added wait_for_completion=true so that the requests will return only after completing the backup. It might not be a good idea if the index you’re backing up is very large, but for the dashboards index it is great because the request is synchronous and after it returns we are sure the backup was successful.

Restore From Backup

So we stored a backup of our index in S3. Now we would like to restore it. When the index is already open on ES we will need to close it for restoration, so our process will:

  1. close the index
  2. restore from snapshot
  3. open the index

To perform these tasks we will:

curl -XPOST ''
curl -XPOST ''
curl -XPOST ''

Now the index kibana-int will be restored from the last time we took a snapshot (We take a snapshot by running the request from “Backup Your ES Index” section). This works well only if we are working on an ES which already has the kibana-int index and repository-name configured. If we would like to have kibana-int restored on a clean ES installation we will need to make some configuration before we can load the data on S3 to the index:

  1. Create kibana-int index:
    curl -XPUT ''
  2. Create the backup repository on S3 – just the same HTTP request as in “Creating Backup Repository”
  3. Close, restore and open the kibana-int index as shown in the 3 steps above.

This way, we can load all our Logstash dashboards on a clean ES machine.

The Chef Cookbook

After completing this task I also created a chef cookbook so that everyone else can enjoy the safety I am enjoying when I create new Logstash/Grafana dashboards. You can find the cookbook here. It is a generic cookbook which allows you to restore and backup ES indices. You can use it on the kibana-int or grafana-index in order to have your dashboards backed up on S3. It is very easy to integrate into your existing Logstash/Grafana cookbooks so I hope you will find it useful 🙂


3 thoughts on “Backup & Restore Your Logstash/Grafana Dashboards

  1. Awesome, thanks for this!

    I removed the S3 stuff and added .kibana – all my visualizations and dashboards are now backed up in cron to a filesystem.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s