Getting the best out of Logstash for Nginx

Note: If you’re not familiar with Logstash, please watch the introduction video to get the hang of inputs, filters and outputs and how they operate within Logstash. I do assume you have the basic knowledge about Logstash in this tutorial.

Logstash was really a game changer to our monitoring capabilities at We’re using it to aggregate and analyse every system which produces log files in our infrastructure. The setup may take some time, but it is certainly worth the while.
Among all the logs we aggregate using Logstash, we also aggregate our Nginx logs, which I am sure a lot of you would like to but just don’t know exactly how. I wanted to share our configuration and how exactly we set things up to make it work. It has been working for a few good months now with incredible results.
So this is a general sketch of our system:
Logstash Architecture

The Server

The server has 4 main components which make it work:

  1. Redis: Used in order to receive and store events form the web servers
  2. Logstash Indexer: Takes events from the Redis and pushes them into ElasticSearch
  3. ElasticSearch: Gets events from the Redis and stores them
  4. Kibana Web: A very cool web application which nicely presents data stored in ElasticSearch

You will have to install Java, Redis and ElasticSearch to have it ready for action.
Afterwards, just follow this 10 minutes walkthrough to get the Logstash agents up.
Pay attention that you will have a Logstash agent to move events from the Redis to the Elasticsearch (Indexer), and another Logstash web process which holds the Kibana Web interface.

After you have everything installed, the server.conf file is really simple:

As an input, we have redis which operates on localhost. We’re reading the “logstash” key of it which holds a list of events.
As an output, we have ElasticSearch and rubydebug just to be able to watch the log files of the agent if things mess up.

The Clients

The clients (Web servers) have only a Logstash Agent which reads the NginX access log and sends it directly to the Redis on the Logstash Server.
The Logstash agent here is the same one we installed on the Server earlier. So get back to that 10-min tutorial and install the Logstash agent on your web server. Pay attention that the Logstash Web (Kibana) is not needed on the Web servers so don’t run it here.
After getting the agent installed, we need to set up the NginX logging to spit all the data we care about. We found that the default NginX log format does not supply us with all the data we’d like to have, so we altered it. This is the log format definition on our NginX servers:

The differences from the default log are:

  • We added HTTP host to the log. This is because one NginX server may serve multiple hosts and we’d like to know on which one the request operated.
  • We omitted $remote_user variable from the default log – we saw no use of it.
  • Added $request_time to have response time of static files served by NginX.
  • Added $upstream_response_time to have response time of our application server (Unicorn)

Now that we have all the needed data in the log file, all that needed is to ship it right to the Redis on the Logstash server.
This is how the agent.conf file looks on our Web Servers:

A few points worth mentioning about it:

  1. The input, as expected, is the NginX access log file.

  2. Grok

    The Grok filter is one of the most useful filters included in Logstash. It allows you to analyse each log line and fetch fields like client_ip, request and response form it, by matching the log line against regular expressions. We’re using it here to analyse the NginX log line and to extract all the fields we’d like to get out of it to ElasticSearch.
    Logstash ships with a bunch of regular expressions set up for you so you can easily start matching log lines without even writing regular expressions by yourself. You can find the list of given regular expressions here. For example, the first part of the string


    means we’re matching the pre-defined Grok pattern IPORHOST and want to have it as http_host field in ElasticSearch. So simple.
    Getting the hang of Grok is really a matter of practice. There is also a great web tool called Grok Debugger which allows you to try Grok Patterns on inputs you define.
    The reason that there are two patterns (lines 12, 13) is that Grok will try to match the first and then the second. If NginX serves a static file it won’t have $upstream_response_time. So we need one Grok pattern to “catch” requests that went to upstream (this is the first pattern) and one Grok pattern for static requests served (this is the second one).

  3. Inside the Grok pattern we explicitly mention data types on fields we would like to analyse as numbers. For example


    means that request_time field will be indexed as float on ElasticSearch and that we would be able to run numeric functions on it like mean, average etc, and to show it on the graph. This is extremely important because otherwise the response_time would have been indexed as a string and not a number and we couldn’t properly display it on a graph. NUMBER is, again, a pre-defined Grok pattern.

  4. The date filter allows us to set the date on the event sent to ElasticSearch. By default logstash puts the time the agent read the log line as the timestamp. This is not good for us because NginX writes the log line after the request has been processed, so the time the log line was written is not the time in which the request arrived to our server.

  5. The geoip filter adds geographical fields on the event. This will later allow us to ask questions like “From which country I got the most requests?”

  6. The output is pretty self explanatory: Send the events to the Logstash Server Redis and to STDOUT for debugging purposes.


Click To Enlarge
Click To Enlarge
  1. The top left graph shows us HTTP responses by time. Each response time – 200OK, redirects, not-found and errors are colored so we can distinguish between them. Dragging the mouse from left to right inside the chart will drill down on the dragged time frame on all of the graphs on the dashboard.

  2. The top right graph shows us only HTTP errors (5xx) responses by time. If we see a bump on this graph it usually means something went wrong. Again you can drag your mouse to drill on the time frame you wish to focus on.

  3. The two bottom left graphs are showing top URLs hits and top user-agents. Clicking on the magnifying glass on the right to each of them will filter all the graphs on the dashboard by this url or user-agent.

  4. The bottom right pie chart shows us the distribution of response codes. Clicking on each section on the pie will filter all graphs on the dashboard to show only requests with the chosen response code.

Click To Enlarge
Click To Enlarge
  1. The left chart simply shows us mean response time.

  2. The right map shows us geo data of requests. The more requests, the darker the color of the country of the map. Clicking on a country will filter all the graphs on the dashboard to show only requests from this country. So simple, but yet so powerful.


So what do we get out of it? Endless possibilities to analyse and drilldown our web server activity logs. We are able to instantly answer so many questions we couldn’t before. Questions like:
1. What are the top URLs in terms of hits? 404 responses?
2. What agents get the most errors?
3. From what countries are we getting most traffic?
4. What are the slowest responses and to which URLs?

The limit is your imagination and getting immediate results is in a matter of a click.

So yes, go get Logstash now. It is a great project, really well maintained and most important – will easily provide you with answers you had no way to get before.

17 thoughts on “Getting the best out of Logstash for Nginx

  1. Great post, many thanks.

    Is there any reason you didn’t opt for a logstash-forworder / filebeat on the nginx machines, as they allows raw log forwarding and then central grok parsing on the logstash machine.

    1. Hi Ian, glad you liked the post.
      When I wrote the post the forwarder was not wide spread in use as it is today. The logstash agent was the main way to deliver the logs to a central location.
      Of course I could use syslog or some other method to deliver the logs but We had many types of logs besides of the nginx logs, so it was very comfortable that every host was responsible for shipping and parsing its own logs.

  2. Hi, How do you get the pretty Kibana graphs? Do you have a guide on configuring Kibana to show those graphs and the geoip information?


    1. About running Kibana: The logstash binary has two modes – agent and web. You can run it in web mode and it will bring up a kibana server for you.
      The setup is pretty quick and basic you can have a look at the 10-min walkthrough here:
      About presenting the world map with geo-ip coloring: Kibana has a world-map widget exactly for that so once you have Kibana working it is extremely easy to set it up.

    1. Do you mean the shipper machine – the one that reads the log files and sends them to the server? Or do you mean the server machine itself?

        1. Since Kibana is a simple web server which serves some static files most of the demands are coming from the ElasticSearch component. It really depends on how much retention you would like to have on your indices. If a week or two are enough for you then a single ElasticSearch server with 7Gigs of RAM should be OK. If you would like to keep more data than two weeks then you will have to look for a cluster setup for your ElasticSearch. I can’t really tell you how many servers and what are their properties but I guess once you set up this cluster you will get the feeling of how responsive your ElasticSearch server is and adjust its capacity accordingly.

    1. Hi Daniel,

      At first we had a lot of problems with the stability of elasticsearch under high loads of write operations. We found that redis is more stable in that manner and used it as an intermediate to our ES server.
      This way, even if ES crashes for some reason, we can restore data by reading it from redis and sending it to ES.

    1. I assume that if you don’t have an upstream declaration in your nginx configuration than you won’t have $upstream_response_time available in the log format.
      In that case I’d just remove it from the log format and the logstash agent should work as expected.

  3. The Nginx configuration doesnt work :

    user nginx;
    worker_processes 1;

    error_log /var/log/nginx/error.log warn;
    pid /var/run/;

    events {
    worker_connections 1024;

    http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format ftpbro ‘$http_host ‘
    ‘$remote_addr [$time_local] ‘
    ‘”$request” $status $body_bytes_sent ‘
    ‘”$http_referer” “$http_user_agent” ‘
    ‘$request_time ‘

    access_log /var/log/nginx/access.log main;

    sendfile on;
    #tcp_nopush on;

    keepalive_timeout 65;

    #gzip on;

    include /etc/nginx/conf.d/*.conf;

    [root@localhost rsyslog.d]# journalctl -xn
    — Logs begin at Wed 2014-10-29 11:29:29 EDT, end at Thu 2014-10-30 10:08:12 EDT. —
    Oct 30 10:01:01 run-parts(/etc/cron.hourly)[19983]: finished 0anacron
    Oct 30 10:01:01 run-parts(/etc/cron.hourly)[19985]: starting 0yum-hourly.cron
    Oct 30 10:01:01 run-parts(/etc/cron.hourly)[19989]: finished 0yum-hourly.cron
    Oct 30 10:08:12 systemd[1]: Stopping nginx – high performance web server…
    — Subject: Unit nginx.service has begun shutting down
    — Defined-By: systemd
    — Support:

    — Unit nginx.service has begun shutting down.
    Oct 30 10:08:12 systemd[1]: Starting nginx – high performance web server…
    — Subject: Unit nginx.service has begun with start-up
    — Defined-By: systemd
    — Support:

    — Unit nginx.service has begun starting up.
    Oct 30 10:08:12 nginx[20088]: nginx: [emerg] unknown log format “main” in /etc/nginx/nginx.conf:26
    Oct 30 10:08:12 nginx[20088]: nginx: configuration file /etc/nginx/nginx.conf test failed
    Oct 30 10:08:12 systemd[1]: nginx.service: control process exited, code=exited status=1
    Oct 30 10:08:12 systemd[1]: Failed to start nginx – high performance web server.
    — Subject: Unit nginx.service has failed
    — Defined-By: systemd
    — Support:

    — Unit nginx.service has failed.

    — The result is failed.
    Oct 30 10:08:12 systemd[1]: Unit nginx.service entered failed state.
    [root@localhost rsyslog.d]# vim /etc/nginx/nginx.conf

    can you provide a working nginx conf file that works ?

  4. How many logstash servers do you have and are you installing all those components (redis, logstash, ES, kibana) on the same box?
    How do you plan to deal with scaling this?
    Whats your data retention policy?

    1. Good questions😉
      1. We have a single Logstash server for our production environment.
      2. We install components using Chef recipes which we crafted at FTBpro.
      3. We did not get to the problem of scaling because ElasticSearch is currently able to hold the load of our logs indexing. I guess that scaling the system is mainly scaling ElasticSearch itself and not Logstash. If you also need Logstash scaling you will need to create a layer of Logstash indexers layer that pulls all logs from clients and indices them into ES.
      4. We hold logs for 1 month. I saw that holding more data than this makes ES react slowly and throw errors. I did not have to time to deal with this yet and a month is pretty much enough for our needs right now. Also note that this time frame is very specific to your logs rate/complexity and if you log very often and have many entries in ES you may even have to lower your retention.

      Hope this answers some of your questions. I’d be happy to hear of how others deal with those issues.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s