How, when, and why to use bullet charts

Bullet charts, an invention of Stephen Few's, are a handy visualization tool. Though not often seen, the bullet chart is often a better option than other competing visualization types, such as the dashboard gauge.

Why use a bullet chart?

Why should we prefer to use the bullet chart? The dashboard gauge places all of the values to be analyzed around a circle. This then means that we must make quick angle comparisons if we want to distinguish one value from another.

Take a look at this example from Klipfolio:

How far below our target is our measurement? What is the value that is being indicated by the indicator bar? It turns out that the value is $23.6M. That is an incredibly difficult calculation to make, though -- you have to mentally calculate how many millions of dollars correspond to each portion of the arc, determine how many degrees away from 0 the indicator bar is, and determine the degree difference between the $30M target and the indicator. 

Where the dashboard gauge necessitates an angle comparison to determine the difference between two charts, a bullet chart only requires a comparsion between two common scales. This is preferable because we are making use of a stronger preattentive attribute of visual perception. In other words, we are more readily able to discern objects that make us undertake linear scale comparisons than those that make us undertake slope or angle comparisons. You can see a very useful chart that compares the relative accuracy of different attributes of visual perception here:

The visualization is from Alberto Cairo's superb The Functional Art. If you are unfamiliar with preattentive visual attributes, check out Cairo's book or Stephen Few's excellent article, Tapping the Power of Visual Perception.

We ought to prefer bullet charts because they make use of a stronger preattentive attribute. If we are looking at two bullet charts that are aligned, we even get to make a comparison along two common, aligned scales.

When to use a bullet chart?

A bullet chart is useful when you have a single, continuous variable whose value you wish to indicate in relation to several categorical regions. For example, you might wish to categorize economic growth for a country (measured in percentage points, so a continuous variable) as good, average, or bad (a categorical indicator).

How to create a bullet chart?

There are a number of ways to create bullet charts using modern visualization tools. One approach is to make them from scratch using d3. For a great overview of how to create these, you can read Mike Bostock's Bullet Chart Block.

For those wishing to use a higher-level JavaScript library which will enable you to make a bullet chart using just a few lines, check out NVD3.

How to make Tufte's line graph sparklines with d3.js

In this blog post, I will cover how to make a sparkline using d3.js . Our final product will look like this:

For those who are unfamiliar with them, sparklines were first discussed by Edward Tufte, who defines them as follows:

    A sparkline is a small intense, simple, word-sized graphic with typographic resolution. Sparklines mean that graphics are no longer cartoonish special occasions with captions and boxes, but rather sparkline graphics can be everywhere a word or number can be: embedded in a sentence, table, headline, map, spreadsheet, graphic. Data graphics should have the resolution of typography.    -- Edward Tufte, Beautiful Evidence

The first step in creating our sparkline is to create our HTML template. Our `d3.js` script will then add elements where necessary, based on our script's specifications. 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

<html> <meta charset='utf-8'> <link href='https://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css'> <style> text { font-family: 'Open Sans', sans-serif; } .sparkline-container { position:absolute; top:0px; left:0; float:left; width:300px; height:100px; } .sparkcircle { fill: #f00; stroke: none; } path { stroke: #000; stroke-width: 0.35px; fill: none; } </style> <body> <div id="graph" class="aGraph sparkline-container" style=""></div> </body> <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script> <script src=" https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.12/d3.js" charset='utf-8'></script> <script src='https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js'></script> <script> $(document).ready(function() { }); </script> </html>

We will add our `d3.js` code at line 50, within the `$(document).ready(function() {});`. This function simply ensures that our DOM is completely ready before the `d3.js` script runs. 

First, let's define some test data. I'll go ahead and use the currency fluctuations of the Brazilian Real vs. the USD over the past month, which I have lying around from the sparkline dashboard project:

1 2 3 4 5

var data = { 'rates.BRL': [3.7629, 3.853, 3.853, 3.853, 3.9045, 3.8771, 3.9369, 3.877, 3.9004, 3.9004, 3.9004, 3.976, 3.9827, 3.9677, 3.941, 3.941, 3.941, 3.941, 3.9257, 3.852, 3.898, 3.9604, 3.9604, 3.9604, 3.9604, 4.0395, 4.0036, 4.0338, 4.0487, 4.0487]}

Next, we define the dimensions of the box that contains our sparkline. While this may seem trivial, it is the essential element that sets a sparkline aside from a regular line graph -- it is a small visual representation of data. We will use this variable in a bit:


Now, we create a skeleton of our sparkline object. We will have the side text which corresponds to the sparkline label, an svg element that will show the sparkline graph, and a little red circle that will highlight the sparkline's endpoint. Feel free to play around with including or not including the end circle -- I thought that it added to the visualization, but depending on where exactly you are using your sparkline, it might make sense to leave it out. 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

d3.select('body') .selectAll('g') .data(d3.keys(data)) .enter() .append('g') .classed('sparkline-container', function(d) { return true; }) .style('top', function(d, i) { return (i + 1) * 35 + 'px';}) .style('left', '500px') .append('text') // get last 3 characters since format is like rates.JPY .text(function(d) {return d.slice(-3);}) .append('svg') .attr('id', function(d) { return 'graph' + d.slice(-3);}) .append('circle') .attr('id', function(d) { return 'circle' + d.slice(-3);}) .attr('r', 1.5) .classed('sparkcircle', true);

Let's break down that last chunk of code.  In line 1, we select our body element, to which we will be appending the rest of our visualization. Then, we choose to append as many 'g' elements as we have data points. In this case, we only have one time series, but we could easily re-use this code to add numerous sparklines. In line 6, we make sure that our sparkline g elements have the `sparkline-container` class. Based on our styles that we defined, this will give them a certain width, height, and alignment. In line 9, we give it a distance from the top that depends on our i, or incrementing, value -- this will make sure that our sparklines will not be on top of each other if we have more than one. 


You'll notice that so far, the svg doesn't have any line elements within it, and the circle doesn't have any `cx` nor `cy` attributes, so neither one of these elements will render a visual element. Let's make a function that will actually draw our sparklines and end circles for us!

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

function drawSparkline(svgElemId, circleElemId, data) { // create an SVG element inside the #elemId div that fills 100% of the div var graph = d3.select(svgElemId) .append("svg:svg") .attr("width", "100%") .attr("height", "100%"); // X scale will fit values from 0-60 within pixels 0-width var x = d3.scale.linear() .domain([0, 60]) .range([0, boxDimensions.width]); // Y scale will fit values from [the extent of our currency fluctations // dataset] within pixels 0-100 var y = d3.scale.linear() .domain(d3.extent(data, function(d) { return d; })) .range([0, boxDimensions.height]); // create a line object that represents the SVG line we're creating var line = d3.svg.line() // Makes our plots smooth .interpolate('monotone') // assign the X function to plot our line as we wish .x(function(d,i) { return x(i); }) .y(function(d) { return y(d); }); // display the line by appending an svg:path element with the data line we created above graph.append("svg:path").attr("d", line(data)); var circle = d3.select(circleElemId) .attr('cx', function(d, i) {return x(29); }) .attr('cy', function(d) { /* And this is really helpful for figuring out what is going on */ return y(data.slice(-1)); }) }

The key part of this chunk of code is lines 21 - 30 -- these actually create a line object that represents the SVG line that we are creating. We use the x and y scale functions that we have defined previously, and then actually display the line by appending an svg:path. Finally, lines 35-39 give our circles `cx` and `cy` attributes, which will define where the center of the circle should be. Our `cx` is fixed, since we always want for the circle to be drawn at the last data point, and our `cy` consists of the last element in our data set.

Now that we have this handy sparkline function, let's call it for each data element!

1 2 3 4 5

$.each(data, function(country_name, country_series){ drawSparkline('#graph' + country_name.slice(-3), '#circle' + country_name.slice(-3),data[country_name]); });

And we do a tiny bit of visual cleanup to ensure that the labels and the SVGs are not on top of each other:

1	$('svg').css({left: 45, position: 'absolute'});

Which gives us a working sparkline.

Check out the full example on Github. If you're interested in sparklines, you can read about creating a dashboard that shows numerous sparklines for comparative purposes here. You can learn more about sparklines by reading Tufte's Beautiful Evidence.

Using Tufte's discrete sparklines to monitor Nginx status codes

Since I manage several of my own sites end-to-end, including configuring and running their servers, it is important for me know what the sites are up to. This includes knowing when they may be down, as well as checking out any error codes that may be generated. I went ahead and created this simple dashboard vignette:

to keep track of my `Nginx` response status codes. I can see if there are any error codes that have been generated (those in the 400s or the 500s), and act accordingly.

Why did I go ahead and make my own dashboard? First off, reading through Nginx logs in their raw form is a pain. For those who haven't worked with them before, this is what they look like:

1 2 3

180.76.15.141 - - [04/Jan/2016:07:42:26 -0500] "GET / HTTP/1.1" 200 36005 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 180.76.15.147 - - [04/Jan/2016:07:43:33 -0500] "GET / HTTP/1.1" 200 36005 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" 91.65.255.241 - - [04/Jan/2016:07:50:31 -0500] "GET /portfolio/resume HTTP/1.1" 301 5 "http://rowanv.com/portfolio/about/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"

Bleh. So I decided to go out and hunt for some existing visualizations, none of which seemed too great. They seemed far too cluttered with chart junk and lacked customizability. Here is an example of an existing dashboard element that aims to visualize something similar to my final visualization:

 I found this graph fairly distracting, and for some reason the non-2xx status codes didn't show up on here. Also, any spikes in traffic would make it very hard to read -- if we had hundreds of requests that returned a 200 response, but only a handful of requests that returned a 400 or 500 response, it would be very difficult to see the 400s and 500s because they are graphed on the same axis. I didn't just want to see how many 200 responses occurred -- I wanted visual confirmation that 400s and 500s were not occurring. Besides, I didn't really care how many responses of each type were occurring -- just the presence or absence of error responses was what mattered to me, so a simple discrete chart would abstract away a lot of the distracting additional noise. It could be useful to have error rates / total requests available as well, but I think that would be best visualized in another chart.

Along the way, I also found a few completely text-based interfaces to monitor one's servers. I found these completely text-based outputs difficult to parse at a glance. In the end, creating a custom dashboard made the most sense for my needs.

The process was fairly straightforward. First, I located my Nginx access logs and wrote a Python script to read them into a Pandas Dataframe, which is more manageable than their native format. Then, I aggregated and reshaped the dataframe to get a dataset that would identify what error codes had occurred over the course of each hour. This left me with a dataframe that looked like this:

Each row consisted of an hour; each column identified how times each type of error code (one in the 200s, one in the 300s, etc.) had occurred over the course of that hour.

Then, I used Flask to serve an HTTP response within a web application which served my reshaped dataset in a JSON format. This would enable my front-end view to locate the data and read it in. Here's my basic view code that server my HTTP response:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

import re import pprint import pandas as pd import datetime @app.route('/nginx_dash/data/status_code_hourly/') def status_code_hourly_data(): log_path = 'access.log' conf = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"' regex = ''.join( '(?P<' + g + '>.*?)' if g else re.escape(c) for g, c in re.findall(r'\$(\w+)|(.)', conf)) conf_list = [] f = open(log_path, 'r') for line in f: parsed_string = re.match(regex, line) conf_list.append(parsed_string.groupdict()) df = pd.DataFrame(conf_list) df['date'] = df['time_local'].apply(lambda x: datetime.datetime.strptime(x[:11], '%d/%b/%Y')) df['date_time'] = df['time_local'].apply(lambda x: datetime.datetime.strptime(x[:20], '%d/%b/%Y:%H:%M:%S')) df['status'] = df['status'].apply(lambda x: str(x)) df['is_100'] = df['status'].apply(lambda x: x[0] == '1') df['is_200'] = df['status'].apply(lambda x: x[0] =='2') df['is_300'] = df['status'].apply(lambda x: x[0] =='3') df['is_400'] = df['status'].apply(lambda x: x[0] == '4') df['is_500'] = df['status'].apply(lambda x: x[0] == '5') df['date_hour'] = df['date_time'].apply(lambda dt: dt.replace(minute=0, second=0)) df_status_codes = df[['date_hour', 'is_100', 'is_200', 'is_300', 'is_400', 'is_500']] df_status_codes_grouped = df_status_codes.groupby('date_hour').sum() df_status_codes_grouped['date_hour'] = df_status_codes_grouped.index json_response = df_status_codes_grouped.to_json(orient='records') return Response(response=json_response, status=200, mimetype='application/json')

Finally, I wrote the frontend in `d3.js` which actually renders the discrete sparklines. You can check out my previous tutorial on making discrete sparklines to learn how to make this type of a visualization.

And that's it! I now have an awesome, live-updating, minimalist visualization that lets me know if any concerning error codes have occurred over the last couple of days.

If you want to learn more about sparklines, check out Tufte's Beautiful Evidence.

How to make a real-time updating map for your dashboard with JS

It is often useful to be able to quickly analyze events that have a geographic component. A real-time updating map that shows events that occur at certain locations on the map can be very handy for a dashboard. While it is possible to create these maps from scratch using `d3.js`, it can be useful to use some higher-level abstractions to quickly generate our maps. I'll take you through the process of creating an interactive map like this:

Check out the interactive version here

by using `DataMaps`, a library that uses `d3.js` to produce interactive maps (you can check out the library's documentation here). Rather than needing to source your geo data from scratch and process it using command line tools like `ogr2ogr` and `topojson`, then render your polygons and boundaries, and ultimately add labels and locations to your map, I'll to through a more agile process. In addition, we'll go ahead and query a live-updating JSON data source within our visualization script. This will allow for our data visualization to fetch the latest version of the earthquake data each time that it is rendered. 

First, let's create the HTML structure for our visualization:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

<html> <meta charset='utf-8'> <head> </head> <body> <div id="map_container" style="position: relative; width: 80%; max-height: 450px;"></div> </body> <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.12/d3.js" charset='utf-8'></script> <script src="http://d3js.org/topojson.v1.min.js"></script> <script src="http://datamaps.github.io/scripts/datamaps.world.min.js?v=1"></script> <script> </script> </html>

We've created a div element onto which our script will bind SVG objects to create our map. We've also included the dependecies that we will be needing (`jQuery`, `d3.js`, `topojson` and `DataMaps`). We'll now start filling in the script tags on lines 21-23:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

$(document).ready(function() { d3.json("http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson",function(error,data){ data = data.features; var map = new Datamap({ scope: 'world', element: document.getElementById('map_container'), projection: 'equirectangular', height: 500, fills: { defaultFill: '#F1EBF4', lt50: 'rgba(0,244,244,0.9)', gt50: '#20438A', }, geographyConfig: { highlightFillColor: '#bfdfec', highlightBorderColor: 'white', }, bubblesConfig: { highlightFillColor: '#2265aa', highlightBorderColor: '#61b3b7', } }); }); });

We wrap our entire script in a document ready function to ensure that it doesn't run until our DOM is ready.  Then, we use the `d3.json` function to read in our data from the USGS earthquake GeoJSON feed. For more information about this feed, check out their docs here. We then create a var called map which is a new `Datamap` object. This object will actually render our map based on the configurations that we pass on to it. This is great so far, but it will not include the bubbles that indicate where recent earthquakes have occurred. To do that, we'll have to add another set of functions at line 25 - 26:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

var bubble_array = []; function getBubble(lat, lon, name){ var bubble = { name: name, latitude: lat, longitude: lon, radius: 10, fillKey: 'gt50', } bubble_array.push(bubble); return bubble; } for (var i = 0; i < data.length; i++) { getBubble(data[i].geometry.coordinates[1], data[i].geometry.coordinates[0], data[i].properties.title); } //bubbles, custom popup on hover template map.bubbles(bubble_array, { popupTemplate: function(geo, data) { return "<div class='hoverinfo'>" + data.name + "</div>"; } });

Line 1: We create an array that will hold the data for the earthquake bubbles that we will add to the map.
Lines 3-13: We create a function that creats a bubble object with the attributes that we will be needing in order to place these on the map. This includes the latitude and longitude, which our `dataMap` object will use to plot the bubbles in their correct location when we pass the array to `map.bubbles()`.
Lines 15-19: We iterate through our data set, calling the `getBubble` function at each data point. This pushes a `bubble` item onto our array with the correct data for each point.
Lines 22-26: We finally pass the `bubble_array` to `map.bubbles`. We also go ahead and specify the `popupTemplate` var, which tells our map what to render when we hover over.

And we're done! You can see the full code snippet on Github here and the live example here.

How to make Tufte's discrete sparklines using d3.js

I've been working on a dashboard and found a data set that would look great represented as a discrete sparkline. This type of visualization is great for quickly showing trends at a glance in discrete variable datasets. For those who are not familiar with them, discrete sparklines look like this:

This example is from Tufte's site, and was made in collaboration with Adam Schwartz . The presence of a horizontal line represents a home game.

I decided to use his example as a starting point, working from the HTML file that is in Adam's github, which he kindly passed along. For my purposes, I only needed to illustrate a dichotomous variable, so I went ahead and removed the horizontal bars which represented home vs. away games, as well as the colouring which represented winning streaks. After recoding it to leverage the `d3.js` library, simplifying the styles,  and reworking the code to make it reusable for future dichotomous variable projects, I was able to create my own example using d3, which you can see below Adam's:

Here's the relevant d3 code:


Let's break it down further.
Lines 7 - 10: We bind each data point to a div element
Line 11: We give each div the 'drawn-line' class  which will format each div element as a line (black background colour, 1px by 8 px).
Lines 12-19: We then give it an HTML5 data attribute called data-binary-var which will be a 0 if the game was lost and a 1 if the game was won.
Line 20: Finally, we use our counter variable i to correctly place the lines along the x axis.

And here's our final product!

 You can check out the full example I've written up on Github.

 If you want to learn more about sparklines, check out Tufte's Beautiful Evidence. Thanks as well to Adam Schwartz for creating the original example that inspired this recreation - you can check out more of his work here and follow him on twitter here.

And here's some of the work in progress for my dashboard sparkline visualization element which prompted this whole exploration: