Data Viz Talk: Tufte

Showing posts with label Tufte. Show all posts

Sunday, January 17, 2016

How to make Tufte's line graph sparklines with d3.js

In this blog post, I will cover how to make a sparkline using d3.js . Our final product will look like this:

For those who are unfamiliar with them, sparklines were first discussed by Edward Tufte, who defines them as follows:

A sparkline is a small intense, simple, word-sized graphic with typographic resolution. Sparklines mean that graphics are no longer cartoonish special occasions with captions and boxes, but rather sparkline graphics can be everywhere a word or number can be: embedded in a sentence, table, headline, map, spreadsheet, graphic. Data graphics should have the resolution of typography. -- Edward Tufte, Beautiful Evidence

The first step in creating our sparkline is to create our HTML template. Our `d3.js` script will then add elements where necessary, based on our script's specifications.

<html>
<meta charset='utf-8'>




    <link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'>
    <style>
    text {
        font-family: 'Open Sans', sans-serif;
    }
    .sparkline-container {
        position:absolute;
        top:0px;
        left:0;
        float:left;
        width:300px;
        height:100px;
    }
    .sparkcircle {
      fill: #f00;
      stroke: none;
    }
    path {
        stroke: #000;
        stroke-width: 0.35px;
        fill: none;
    }
    </style>



<body>


    <div id="graph" class="aGraph sparkline-container" style=""></div>


</body>


<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
<script src="
https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.12/d3.js" charset='utf-8'></script>

<script src='https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js'></script>
<script>

$(document).ready(function() {



});
</script>

</html>

We will add our `d3.js` code at line 50, within the `$(document).ready(function() {});`. This function simply ensures that our DOM is completely ready before the `d3.js` script runs.

First, let's define some test data. I'll go ahead and use the currency fluctuations of the Brazilian Real vs. the USD over the past month, which I have lying around from the sparkline dashboard project:

        var data = { 'rates.BRL': [3.7629, 3.853, 3.853, 3.853, 3.9045,
        3.8771, 3.9369, 3.877, 3.9004, 3.9004, 3.9004, 3.976,
        3.9827, 3.9677, 3.941, 3.941, 3.941, 3.941, 3.9257,
        3.852, 3.898, 3.9604, 3.9604, 3.9604, 3.9604, 4.0395,
        4.0036, 4.0338, 4.0487, 4.0487]}

Next, we define the dimensions of the box that contains our sparkline. While this may seem trivial, it is the essential element that sets a sparkline aside from a regular line graph -- it is a small visual representation of data. We will use this variable in a bit:

Now, we create a skeleton of our sparkline object. We will have the side text which corresponds to the sparkline label, an svg element that will show the sparkline graph, and a little red circle that will highlight the sparkline's endpoint. Feel free to play around with including or not including the end circle -- I thought that it added to the visualization, but depending on where exactly you are using your sparkline, it might make sense to leave it out.

        d3.select('body')
          .selectAll('g')
          .data(d3.keys(data))
            .enter()
            .append('g')
            .classed('sparkline-container', function(d) {
                return true;
            })
            .style('top', function(d, i) { return (i + 1) * 35 + 'px';})
            .style('left', '500px')

          .append('text')
            // get last 3 characters since format is like rates.JPY
            .text(function(d) {return d.slice(-3);})
          .append('svg')
            .attr('id', function(d) { return 'graph' + d.slice(-3);})
          .append('circle')
            .attr('id', function(d) { return 'circle' + d.slice(-3);})
            .attr('r', 1.5)
            .classed('sparkcircle', true);

Let's break down that last chunk of code. In line 1, we select our body element, to which we will be appending the rest of our visualization. Then, we choose to append as many 'g' elements as we have data points. In this case, we only have one time series, but we could easily re-use this code to add numerous sparklines. In line 6, we make sure that our sparkline g elements have the `sparkline-container` class. Based on our styles that we defined, this will give them a certain width, height, and alignment. In line 9, we give it a distance from the top that depends on our i, or incrementing, value -- this will make sure that our sparklines will not be on top of each other if we have more than one.

You'll notice that so far, the svg doesn't have any line elements within it, and the circle doesn't have any `cx` nor `cy` attributes, so neither one of these elements will render a visual element. Let's make a function that will actually draw our sparklines and end circles for us!

         function drawSparkline(svgElemId, circleElemId, data) {

            // create an SVG element inside the #elemId div that fills 100% of the div
        var graph = d3.select(svgElemId)
            .append("svg:svg")
            .attr("width", "100%")
            .attr("height", "100%");


        // X scale will fit values from 0-60 within pixels 0-width
        var x = d3.scale.linear()
            .domain([0, 60])
            .range([0, boxDimensions.width]);
        // Y scale will fit values from [the extent of our currency fluctations
        // dataset] within pixels 0-100
        var y = d3.scale.linear()
            .domain(d3.extent(data, function(d) { return d; }))
            .range([0, boxDimensions.height]);

        // create a line object that represents the SVG line we're creating
        var line = d3.svg.line()
            // Makes our plots smooth
            .interpolate('monotone')
            // assign the X function to plot our line as we wish
            .x(function(d,i) {
                return x(i);
            })
            .y(function(d) {
                return y(d);
            });

            // display the line by appending an svg:path element with the data line we created above
            graph.append("svg:path").attr("d", line(data));

        var circle = d3.select(circleElemId)
            .attr('cx', function(d, i) {return x(29); })
            .attr('cy', function(d) {
                /* And this is really helpful for figuring out what is going on */
                return y(data.slice(-1)); })

        }

The key part of this chunk of code is lines 21 - 30 -- these actually create a line object that represents the SVG line that we are creating. We use the x and y scale functions that we have defined previously, and then actually display the line by appending an svg:path. Finally, lines 35-39 give our circles `cx` and `cy` attributes, which will define where the center of the circle should be. Our `cx` is fixed, since we always want for the circle to be drawn at the last data point, and our `cy` consists of the last element in our data set.

Now that we have this handy sparkline function, let's call it for each data element!

 
        $.each(data, function(country_name, country_series){

            drawSparkline('#graph' + country_name.slice(-3), '#circle' + country_name.slice(-3),data[country_name]);
        });

And we do a tiny bit of visual cleanup to ensure that the labels and the SVGs are not on top of each other:

1	$('svg').css({left: 45, position: 'absolute'});

Which gives us a working sparkline.

Check out the full example on Github. If you're interested in sparklines, you can read about creating a dashboard that shows numerous sparklines for comparative purposes here. You can learn more about sparklines by reading Tufte's Beautiful Evidence.

How to make a dashboard with Tufte's sparklines in d3.js

Thus far, we've used traditional charts to visualize our data. We will now be adding a visualization that is less common: a sparkline. What is a sparkline?

A sparkline is a small intense, simple, word-sized graphic with typographic resolution.
Sparklines mean that graphics are no longer cartoonish special occasions with captions and boxes,
but rather sparkline graphics can be everywhere a word or number can be: embedded in a sentence,
table, headline, map, spreadsheet, graphic. Data graphics should have the resolution of typography.
-- Edward Tufte, Beautiful Evidence

Sparklines can be very handy for quickly analyzing several time series at a time. In this blog post, I'll explain how you can create a dashboard using `d3.js` that visualizes sparklines of currency fluctuations. This dashboard not only visualizes currency fluctuations over the past month for major currencies, it also updates in real time.

You can view a live version of the dashboard at http://sparklines-dash.rowanv.com.

Why Visualize Currency Exchange Rates using Sparklines?

So first off, what prompted me to make this dashboard? I was looking at the exchange rates put out by the European Central Bank. The ECB publishes exchange rates for a number of major currencies each day. However, their interface misses out on a great opportunity to use sparklines:

While they show a small arrow that illustrates the derivative of each of the major currencies, their dashboard does not show the currency fluctuations over time. In order to view this type of a graph, one has to click through to the individual graphs by clicking on the 'Chart' icon. This prevents quick, side-by-side comparisons of the currency fluctuations over time. A quick peek at last month's currency fluctuations would be a great visualization for using sparklines in order to quickly compare the shapes of the time series. While I don't think that they completely replace the chart tool that the ECB has published, they'd be a nice complement to the data that they provide.

Getting the Data:

This merits its own blog post. For now, you can check out my iPython notebook here for a quick overview of the process.

Making the d3.js visualization:

After the data acquisition and cleaning process, I had a view within my web application which served a JSON response of my currency data. This data included time series for the past month for each of the major currencies. If we visualize the data in a tabular format, it would look like this:

Each column corresponded to a currency type, and each row corresponded to a day of observations. With this data in hand, I created a basic HTML document that would read the data within d3:

<html>
<meta charset='utf-8'>


    
    <link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'> <!--#1-->
    <style>
    text {
        font-family: 'Open Sans', sans-serif;
    }
    <!--#2-->
    .sparkline-container {
        position:absolute;
        top:0px;
        left:0;
        float:left;
        width:300px;
        height:100px;
    }
    <!--#3-->
    .sparkcircle {
      fill: #f00;
      stroke: none;
    }
    <!--#4-->
    path {
        stroke: #000;
        stroke-width: 0.35px;
        fill: none;
    }
    </style>


<body>
    <!--#5-->
    <div id="graph" class="aGraph sparkline-container" style=""></div>

</body>


<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script>
<script src="
https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.12/d3.js" charset='utf-8'></script>

<script src='https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js'></script>
<script>
// # 6
$(document).ready(function() {


    d3.json('/data/not_realtime/currency_rates_month/', function(error, data) {
 
    });
});

</script>


</html>

Let's highlight some key things that our template is doing:
# 1: We specify the font that we will be using for the text next to our sparklines
# 2: We specify the width and height of our sparklines container
# 3: We specify how we'd like to format the little red circles at the end of the sparklines
# 4: Our sparklines will be 0.35px thick and use a black stroke colour
# 5: This is the actual div element which we will be using as our starting point for the visualization. We'll select this element and then attach all other visual elements to it.
# 6: We write our actual `d3.js` script between these `script` tags. So far, we've used a jQuery helper function to keep our `d3.js` script from running until the DOM is ready for it, and we've requested our data set from the URL at `/data/not_realtime/currency_rates_month/', where it will be served as a json response.

Actually Drawing Our First Sparkline

Now that we have our HTML template, we can draw our first sparkline. You can read through an example of how to draw an individual sparkline in my sparkline blogpost.

Reshaping Our Data
Based on the way that we have written our sparkline visualization function, it would be useful to reshape our data. It will be easiest to work with if we have a set of json objects with each key representing our currency country name and each value representing a list of time series values. We do some reshaping:

$(document).ready(function() {


    d3.json('/data/not_realtime/currency_rates_month/', function(error, data) {
        var boxDimensions = {width: 300, height: 15}

        // cleaning data
        // Avoid iterating through all stuff in prototype chain
        $.each(data, function(country_name, country_series){

            data_list = []
            for (i in _.range(30)){

                data_list.push(country_series[i]);
            }
            data[country_name] = data_list.reverse()
            // and flip it so the end sparkline will have oldest dates on the left
            // newest dates on the right
        });

    });
});

Next, let's start adding our visual elements at the point in our code represented by line 23 in the last code snippet. We can add our container divs for our set of sparklines. This includes a general container div, a text element, an svg element, and a circle element:

        // Add container divs:

        d3.select('body')
          .selectAll('div')
          .data(d3.keys(data))
            .enter()
            .append('div')
            .classed('sparkline-container', function(d) {
                console.log(d);
                return true;
            })
            .style('top', function(d, i) { return i * 35 + 'px';})
            .style('left', '500px')

          .append('text')
            // get last 3 characters since format is like rates.JPY
            .text(function(d) {return d.slice(-3);})
          .append('svg')
            .attr('id', function(d) { return 'graph' + d.slice(-3);})
          .append('circle')
            .attr('id', function(d) { return 'circle' + d.slice(-3);})
            .attr('r', 1.5)
            .classed('sparkcircle', true);

Let's break down this code snippet. This code snippet basically creates the skeleton for our visualization -- it creates the text and a set of elements that will then be 'filled in' by the `drawSparkline` function in the following code snippet. In line 3, we select the body element, to which we will be appending our sparkline elements. We then add a set of divs in lines 4-13, with one div for each sparkline, since we have bound it to our dataset. Each div has a `sparkline-container` class, is offset from the left, and is offset from the top by a variable amount so that the sparklines are not all on top of each other.

In lines 15-17, we append the 3-character labels that represent the sparkline countries' names. Then, in lines 18-28, we add elements that will correspond with our little red circles, and the svg containers for our sparklines. .

Putting it all together:

Now that we have our visualization skeleton, we can use the `drawSparkline` function to actually append our sparklines:

         function drawSparkline(svgElemId, circleElemId, data) {

            // create an SVG element inside the #elemId div that fills 100% of the div
        var graph = d3.select(svgElemId)
            .append("svg:svg")
            .attr("width", "100%")
            .attr("height", "100%");


        // X scale will fit values from 0-60 within pixels 0-width
        var x = d3.scale.linear()
            .domain([0, 60])
            .range([0, boxDimensions.width]);
        // Y scale will fit values from [the extent of our currency fluctations
        // dataset] within pixels 0-100
        var y = d3.scale.linear()
            .domain(d3.extent(data, function(d) { return d; }))
            .range([0, boxDimensions.height]);

        // create a line object that represents the SVN line we're creating
        var line = d3.svg.line()
            // Makes our plots smooth
            .interpolate('monotone')
            // assign the X function to plot our line as we wish
            .x(function(d,i) {
                return x(i);
            })
            .y(function(d) {
                return y(d);
            });

            // display the line by appending an svg:path element with the data line we created above
            graph.append("svg:path").attr("d", line(data));

        var circle = d3.select(circleElemId)
            .attr('cx', function(d, i) {return x(29); })
            .attr('cy', function(d) {console.log('circle');
                /* And this is really helpful for figuring out what is going on */
                console.log(svgElemId);
                console.log(data);
                console.log(data.slice(-1));
                return y(data.slice(-1)); })

        }

Lines 11 to 23 cover the same process that we would follow to add a line graph, except that we are making each graph tiny since it is a sparkline. Then in lines 35 to 42, we give our circle elements `cx` and `cy` attributes -- these tell us where the center of each circle will be. Without these attributes, our circles would not show up.

Finally, we call this function in order to draw our set of sparklines:

          // Then add sparklines and dots for circles
        $.each(data, function(country_name, country_series){

            drawSparkline('#graph' + country_name.slice(-3), '#circle' + country_name.slice(-3),data[country_name]);
        });
        // And move the SVGs by a bit so they are not on top of the
        // text boxes
        $('svg').css({left: 45, position: 'absolute'});

Check out the live example, or the github repository with the complete codebase.

Wednesday, January 6, 2016

Using Tufte's discrete sparklines to monitor Nginx status codes

Since I manage several of my own sites end-to-end, including configuring and running their servers, it is important for me know what the sites are up to. This includes knowing when they may be down, as well as checking out any error codes that may be generated. I went ahead and created this simple dashboard vignette:

to keep track of my `Nginx` response status codes. I can see if there are any error codes that have been generated (those in the 400s or the 500s), and act accordingly.

Why did I go ahead and make my own dashboard? First off, reading through Nginx logs in their raw form is a pain. For those who haven't worked with them before, this is what they look like:

1
2
3

180.76.15.141 - - [04/Jan/2016:07:42:26 -0500] "GET / HTTP/1.1" 200 36005 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
180.76.15.147 - - [04/Jan/2016:07:43:33 -0500] "GET / HTTP/1.1" 200 36005 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
91.65.255.241 - - [04/Jan/2016:07:50:31 -0500] "GET /portfolio/resume HTTP/1.1" 301 5 "http://rowanv.com/portfolio/about/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"

Bleh. So I decided to go out and hunt for some existing visualizations, none of which seemed too great. They seemed far too cluttered with chart junk and lacked customizability. Here is an example of an existing dashboard element that aims to visualize something similar to my final visualization:

I found this graph fairly distracting, and for some reason the non-2xx status codes didn't show up on here. Also, any spikes in traffic would make it very hard to read -- if we had hundreds of requests that returned a 200 response, but only a handful of requests that returned a 400 or 500 response, it would be very difficult to see the 400s and 500s because they are graphed on the same axis. I didn't just want to see how many 200 responses occurred -- I wanted visual confirmation that 400s and 500s were not occurring. Besides, I didn't really care how many responses of each type were occurring -- just the presence or absence of error responses was what mattered to me, so a simple discrete chart would abstract away a lot of the distracting additional noise. It could be useful to have error rates / total requests available as well, but I think that would be best visualized in another chart.

Along the way, I also found a few completely text-based interfaces to monitor one's servers. I found these completely text-based outputs difficult to parse at a glance. In the end, creating a custom dashboard made the most sense for my needs.

The process was fairly straightforward. First, I located my Nginx access logs and wrote a Python script to read them into a Pandas Dataframe, which is more manageable than their native format. Then, I aggregated and reshaped the dataframe to get a dataset that would identify what error codes had occurred over the course of each hour. This left me with a dataframe that looked like this:

Each row consisted of an hour; each column identified how times each type of error code (one in the 200s, one in the 300s, etc.) had occurred over the course of that hour.

Then, I used Flask to serve an HTTP response within a web application which served my reshaped dataset in a JSON format. This would enable my front-end view to locate the data and read it in. Here's my basic view code that server my HTTP response:

import re
import pprint
import pandas as pd
import datetime

@app.route('/nginx_dash/data/status_code_hourly/')
def status_code_hourly_data():


    log_path = 'access.log'

    conf = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
    regex = ''.join(
        '(?P<' + g + '>.*?)' if g else re.escape(c)
        for g, c in re.findall(r'\$(\w+)|(.)', conf))

    conf_list = []

    f = open(log_path, 'r')
    for line in f:
        parsed_string = re.match(regex, line)
        conf_list.append(parsed_string.groupdict())
    df = pd.DataFrame(conf_list)

    df['date'] = df['time_local'].apply(lambda x: datetime.datetime.strptime(x[:11], '%d/%b/%Y'))
    df['date_time'] = df['time_local'].apply(lambda x: datetime.datetime.strptime(x[:20], '%d/%b/%Y:%H:%M:%S'))
    df['status'] = df['status'].apply(lambda x: str(x))
    df['is_100'] = df['status'].apply(lambda x: x[0] == '1')
    df['is_200'] = df['status'].apply(lambda x: x[0] =='2')
    df['is_300'] = df['status'].apply(lambda x: x[0] =='3')
    df['is_400'] = df['status'].apply(lambda x: x[0] == '4')
    df['is_500'] = df['status'].apply(lambda x: x[0] == '5')
    df['date_hour'] = df['date_time'].apply(lambda dt: dt.replace(minute=0, second=0))
    df_status_codes = df[['date_hour', 'is_100', 'is_200', 'is_300', 'is_400', 'is_500']]
    df_status_codes_grouped = df_status_codes.groupby('date_hour').sum()
    df_status_codes_grouped['date_hour'] = df_status_codes_grouped.index
    json_response = df_status_codes_grouped.to_json(orient='records')
    return Response(response=json_response,
                    status=200,
                    mimetype='application/json')

Finally, I wrote the frontend in `d3.js` which actually renders the discrete sparklines. You can check out my previous tutorial on making discrete sparklines to learn how to make this type of a visualization.

And that's it! I now have an awesome, live-updating, minimalist visualization that lets me know if any concerning error codes have occurred over the last couple of days.

If you want to learn more about sparklines, check out Tufte's Beautiful Evidence.

Monday, January 4, 2016

How to make Tufte's discrete sparklines using d3.js

I've been working on a dashboard and found a data set that would look great represented as a discrete sparkline. This type of visualization is great for quickly showing trends at a glance in discrete variable datasets. For those who are not familiar with them, discrete sparklines look like this:

This example is from Tufte's site, and was made in collaboration with Adam Schwartz . The presence of a horizontal line represents a home game.

I decided to use his example as a starting point, working from the HTML file that is in Adam's github, which he kindly passed along. For my purposes, I only needed to illustrate a dichotomous variable, so I went ahead and removed the horizontal bars which represented home vs. away games, as well as the colouring which represented winning streaks. After recoding it to leverage the `d3.js` library, simplifying the styles, and reworking the code to make it reusable for future dichotomous variable projects, I was able to create my own example using d3, which you can see below Adam's:

Here's the relevant d3 code:

Let's break it down further.
Lines 7 - 10: We bind each data point to a div element
Line 11: We give each div the 'drawn-line' class which will format each div element as a line (black background colour, 1px by 8 px).
Lines 12-19: We then give it an HTML5 data attribute called data-binary-var which will be a 0 if the game was lost and a 1 if the game was won.
Line 20: Finally, we use our counter variable i to correctly place the lines along the x axis.

And here's our final product!

You can check out the full example I've written up on Github.

If you want to learn more about sparklines, check out Tufte's Beautiful Evidence. Thanks as well to Adam Schwartz for creating the original example that inspired this recreation - you can check out more of his work here and follow him on twitter here.

And here's some of the work in progress for my dashboard sparkline visualization element which prompted this whole exploration: