Showing posts with label flask. Show all posts
Showing posts with label flask. Show all posts

Wednesday, January 6, 2016

Using Tufte's discrete sparklines to monitor Nginx status codes

Since I manage several of my own sites end-to-end, including configuring and running their servers, it is important for me know what the sites are up to. This includes knowing when they may be down, as well as checking out any error codes that may be generated. I went ahead and created this simple dashboard vignette:



to keep track of my `Nginx` response status codes. I can see if there are any error codes that have been generated (those in the 400s or the 500s), and act accordingly.

Why did I go ahead and make my own dashboard? First off, reading through Nginx logs in their raw form is a pain. For those who haven't worked with them before, this is what they look like:


1
2
3
180.76.15.141 - - [04/Jan/2016:07:42:26 -0500] "GET / HTTP/1.1" 200 36005 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.15.147 - - [04/Jan/2016:07:43:33 -0500] "GET / HTTP/1.1" 200 36005 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
91.65.255.241 - - [04/Jan/2016:07:50:31 -0500] "GET /portfolio/resume HTTP/1.1" 301 5 "http://rowanv.com/portfolio/about/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"


Bleh. So I decided to go out and hunt for some existing visualizations, none of which seemed too great. They seemed far too cluttered with chart junk and lacked customizability. Here is an example of an existing dashboard element that aims to visualize something similar to my final visualization:



 I found this graph fairly distracting, and for some reason the non-2xx status codes didn't show up on here. Also, any spikes in traffic would make it very hard to read -- if we had hundreds of requests that returned a 200 response, but only a handful of requests that returned a 400 or 500 response, it would be very difficult to see the 400s and 500s because they are graphed on the same axis. I didn't just want to see how many 200 responses occurred -- I wanted visual confirmation that 400s and 500s were not occurring. Besides, I didn't really care how many responses of each type were occurring -- just the presence or absence of error responses was what mattered to me, so a simple discrete chart would abstract away a lot of the distracting additional noise. It could be useful to have error rates / total requests available as well, but I think that would be best visualized in another chart.

Along the way, I also found a few completely text-based interfaces to monitor one's servers. I found these completely text-based outputs difficult to parse at a glance. In the end, creating a custom dashboard made the most sense for my needs.

The process was fairly straightforward. First, I located my Nginx access logs and wrote a Python script to read them into a Pandas Dataframe, which is more manageable than their native format. Then, I aggregated and reshaped the dataframe to get a dataset that would identify what error codes had occurred over the course of each hour. This left me with a dataframe that looked like this:



Each row consisted of an hour; each column identified how times each type of error code (one in the 200s, one in the 300s, etc.) had occurred over the course of that hour.

Then, I used Flask to serve an HTTP response within a web application which served my reshaped dataset in a JSON format. This would enable my front-end view to locate the data and read it in. Here's my basic view code that server my HTTP response:



 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import re
import pprint
import pandas as pd
import datetime

@app.route('/nginx_dash/data/status_code_hourly/')
def status_code_hourly_data():


    log_path = 'access.log'

    conf = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
    regex = ''.join(
        '(?P<' + g + '>.*?)' if g else re.escape(c)
        for g, c in re.findall(r'\$(\w+)|(.)', conf))

    conf_list = []

    f = open(log_path, 'r')
    for line in f:
        parsed_string = re.match(regex, line)
        conf_list.append(parsed_string.groupdict())
    df = pd.DataFrame(conf_list)

    df['date'] = df['time_local'].apply(lambda x: datetime.datetime.strptime(x[:11], '%d/%b/%Y'))
    df['date_time'] = df['time_local'].apply(lambda x: datetime.datetime.strptime(x[:20], '%d/%b/%Y:%H:%M:%S'))
    df['status'] = df['status'].apply(lambda x: str(x))
    df['is_100'] = df['status'].apply(lambda x: x[0] == '1')
    df['is_200'] = df['status'].apply(lambda x: x[0] =='2')
    df['is_300'] = df['status'].apply(lambda x: x[0] =='3')
    df['is_400'] = df['status'].apply(lambda x: x[0] == '4')
    df['is_500'] = df['status'].apply(lambda x: x[0] == '5')
    df['date_hour'] = df['date_time'].apply(lambda dt: dt.replace(minute=0, second=0))
    df_status_codes = df[['date_hour', 'is_100', 'is_200', 'is_300', 'is_400', 'is_500']]
    df_status_codes_grouped = df_status_codes.groupby('date_hour').sum()
    df_status_codes_grouped['date_hour'] = df_status_codes_grouped.index
    json_response = df_status_codes_grouped.to_json(orient='records')
    return Response(response=json_response,
                    status=200,
                    mimetype='application/json')


Finally, I wrote the frontend in `d3.js` which actually renders the discrete sparklines. You can check out my previous tutorial on making discrete sparklines to learn how to make this type of a visualization.

And that's it! I now have an awesome, live-updating, minimalist visualization that lets me know if any concerning error codes have occurred over the last couple of days.

If you want to learn more about sparklines, check out Tufte's Beautiful Evidence.

Friday, January 1, 2016

Serving a matplotlib plot that follows good design practices using Flask

It can be useful at times to serve images that are generated using matplotlib within a Flask application context. This enables some lightweight, non-dynamic visualization generation.

Unfortunately, the defaults for matplotlib are terrible. Ugh!


Enter prettyplotlib -- this library applies some of the work of several key influencers in the visual design space (Tufte, Brewer) to matplotlib. We eliminate chartjunk, add some better fonts, remove some redundant axes, add some better colours, and get the following:



Much better! Now how do we go about serving this type of a plot via Flask? We will have to have two views, one that creates the image, and another one that serves the template that will contain the image. Our template, contained at templates/image.html should be as follows:


<html>
  <head>
    <title>{{ title }} - image</title>
  </head>
  <body>
    <img src="/fig/" alt="Image Placeholder">
  </body>
</html>

and our main application file, titled run.py, should contain the following:


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from io import BytesIO
from flask import Flask, render_template, send_file, make_response
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
import numpy as np
import prettyplotlib as ppl
import matplotlib.pyplot as plt


app = Flask(__name__)

@app.route('/fig/')
def fig():
    fig, ax = plt.subplots(1)
    ppl.bar(ax, np.arange(10), np.abs(np.random.randn(10)))
    canvas = FigureCanvas(fig)
    img = BytesIO()
    fig.savefig(img)
    img.seek(0)
    return send_file(img, mimetype='image/png')


@app.route('/image/')
def images():
    return render_template("image.html")


if __name__ == '__main__':
    app.run(debug=True)

The key is line 14 -- rather than calling matplotlib's plt.bar, we call prettyplotlib's plt.bar . Just like that, our output is much more visually appealing and easier to read!

For further reading, check out the following Stack Overflow thread about serving matplotlib files via Flask, this Gist about plotting a PNG using matplotlib (uses Python 2 -- you'll have to adapt it to use the io library if you'd like to try something similar in Python 3) and the official prettyplotlib documentation.

Sign up to receive data viz talk tips and updates via email. We're low traffic and take privacy very seriously.