Thursday, January 7, 2016

Visualizing Big Data with d3.js

You're creating a dashboard or other visualization of Big Data in `d3.js`. You've heard that SVG-based visualizations can experience a performance hit as they render more and more elements.
Perhaps you've even hit the object threshold where you are starting to notice significant lags. Where to now? How do you build your visualization to ensure that it scales?

  • Simplify your data/dashboard by aggregating. Are all of the elements that you are rendering necessary? Make sure that each pixel counts and that an aggregate data view wouldn't more effectively convey your point. For example, Google Maps successfully manages to visualize a large quantity of data. However, they don't just throw all of their data at you -- depending on the zoom level that you choose, you will be presented with a certain amount of map detail. This allows for a better and more responsive visual experience than if they did not have aggregated data as part of their visualization. 
  • Simplify your data/dashboard by sampling. Try sampling a subset of your dataset and visualizing that -- that is often enough to perceive the general shape of your data. 
  • Try to move some of the rendering server-side. Normally, the bulk of rendering is done client-side with `d3.js`. However, you can generate an HTML document of part of your visualization, save it in your server's memory, and then inject it into your client's visualization, where the chunk that your server rendered appears together with the rest of the visualization that your client renders. For more details, check out a tutorial here.
  • Try a hybrid canvas/svg approach. This gives you some of the benefits of the SVG, while still avoiding the performance constraints of too many SVG objects.
  • Change tools. At some point, the performance hit of creating your DOM and manipulating will be significant enough that the other optimization techniques will not be sufficient. The weakest point in the `d3.js` stack is your browser -- change your rendering process by hooking up `d3.js` to Google BigQuery. You can check out this link for more info.

Do you have other tips and tricks that you've used to visualize big data with `d3.js`? Have another stack that you recommend? Tell us about it in the comments below!

1 comment:

  1. Last year, I ran into performance issues with D3.js (SVG really). I explored canvas-based solutions and WebGL too. WebGL was the most performant. However, going from D3 to WebGL there's a considerably steep learning curve. Ultimately, in my case, there was simply too much to re-implement in WebGL which you normally take for granted in html/css/D3.


Sign up to receive data viz talk tips and updates via email. We're low traffic and take privacy very seriously.