Implementing the D3 General Update Pattern into a Qlik Sense Extension




Animation in data visualization is a relatively new area that we still don't know much about. Like any visual component of a visualization, animations used in the wrong way can be unnecessarily flashy and distracting. However, there is evidence that animations can also add value to visualizations when used appropriately.

In this interactive example, Mike Bostock demonstrates the concept of object constancy:

a graphical element that represents a particular data point...can be tracked visually through the transition. This lessens the cognitive burden by using preattentive processing of motion rather than sequential scanning of labels.

Animation therefore can help users track changes in the data. When filtering, this can help the user discern what changes their filters caused, as opposed to having to reset their bearings on a newly appearing chart. When data in a chart is filtered, there are three possible events that can happen: data elements either update with new properties, exit because they were filtered out of the data set, or enter because they are filtered in to the data set. These are the main events we handle in typical Qlik Sense extensions. For more on animation possibilities, check out https://www.youtube.com/watch?v=vLk7mlAtEXI#t=54

D3 has a very useful way of animating these transitions through the general update pattern. This pattern involves appending data to a set of elements, and then defining the transitions for updating, entering, and exiting elements. Bostock provides a nice visual example here http://bl.ocks.org/mbostock/3808234.

The rest of this post will walk through an example of how to implement the general update pattern into a Qlik Sense Extension. The source code is available on GitHub.

An example of the extension can be used in this app. A preview:
Updating Bar Chart

Review of a typical extension

Let's start with an example extension that renders a simple bar chart using d3.js. This example follows the typical method of rendering extensions: every time the extension needs to render, it empties out the chart that was previously updated. The chart isn't actually being updated; rather, it is completely redrawn from scratch each time the data is filtered.

This extension is available in the repository in this folder. Walking through the JS file that renders the extension section by section:

[code lang="js"]
define(["jquery", "text!./simplebar.css","./d3.min"], function($, cssContent) {'use strict';
$("<style>").html(cssContent).appendTo("head");

return {
initialProperties : {
version: 1.0,
qHyperCubeDef : {
qDimensions : [],
qMeasures : [],
qInitialDataFetch : [{
qWidth : 2,
qHeight : 1000
}]
}
},
definition : {
type : "items",
component : "accordion",
items : {
dimensions : {
uses : "dimensions",
min : 1,
max:1
},
measures : {
uses : "measures",
min : 1,
max:1
},
sorting : {
uses : "sorting"
},
settings : {
uses : "settings"
}
}
},
snapshot : {
canTakeSnapshot : true
},
paint : function($element,layout) {
[/code]

The block above defines the files to load first. These include jQuery, our CSS file for styling, and d3. Once we've loaded these, we define our extension properties. This extension is a pretty simple one: it's a basic bar chart, so it just needs 1 dimension and 1 measure.

The next step is to define the paint method for the extension. The simple bar chart starts by extracting the data in a format that is suitable for the d3 code and setting up the element and properties that will contain the visualization:
[code]
paint : function($element,layout) {
// Create a reference to the app, which will be used later to make selections
var self = this;

// Get the data
var qMatrix = layout.qHyperCube.qDataPages[0].qMatrix;
var data = qMatrix.map(function(d) {
return {
"Dim":d[0].qText,
"Dim_key":d[0].qElemNumber,
"Value":d[1].qNum
}
});

// Get the extension container properties
var ext_height = $element.height(), // height
ext_width = $element.width(), // width
ext_id = "ext_" + layout.qInfo.qId; // chart id

// Create or empty the chart container
if(document.getElementById(ext_id)) {
// If the element already exists, empty it out so we can start from scratch
$("#" + ext_id).empty();
}
else {
// If the element doesn't exist, create it
$element.append($("<div />").attr("id",ext_id).width(ext_width).height(ext_height));
}

// Call the visualization function
viz();
[/code]
A key takeaway from the paint method above is the if-else block starting on line 61. This block checks if a container element for the visualization has been created. If it hasn't, it creates that element in the else block. This action happens the first time the extension is rendered. However, if the element does exist, then the first part of the if block empties out its contents. This is the key differentiator between a visually updating extension and the standard extension method. The current standard I see in extensions is to completely clean out the chart each time the data is updated. The extension is drawing the visual from scratch every time.

After setting up the data and the container element for the extension, the viz function to generate the chart is run:
[code]
function viz() {
// define margins
var margin = {
top:10,
left:30,
right:10,
bottom:20
};

// Plot dimensions
var plot_width = ext_width - margin.left - margin.right,
plot_height = ext_height - margin.top - margin.bottom;

var svg = d3.select("#" + ext_id).append("svg")
.attr("width",ext_width)
.attr("height",ext_height)
.attr("id",ext_id + "_svg");

var plot = svg
.append("g")
.attr("id",ext_id+"_svg_g");

// Scales
var x = d3.scale.linear()
.domain([0,d3.max(data,function(d) {return d.Value})])
.range([0,plot_width]),
y = d3.scale.ordinal()
.domain(data.map(function(d) {return d.Dim}))
.rangeRoundBands([0, plot_height], .25),
xAxis = d3.svg.axis()
.scale(x)
.tickSize(5)
.tickFormat(d3.format(",.0f")),
yAxis = d3.svg.axis()
.scale(y)
.orient("left");

// Create a temporary yAxis to get the width needed for labels and add to the margin
svg.append("g")
.attr("class","y axis temp")
.attr("transform","translate(0," + 0 + ")")
.call(yAxis);

// Get the temp axis max label width
var label_width = d3.max(svg.selectAll(".y.axis.temp text")[0], function(d) {return d.clientWidth});

// Remove the temp axis
svg.selectAll(".y.axis.temp").remove();

// Update the margins, plot width, and x scale range based on the label size
margin.left = margin.left + label_width;
plot_width = ext_width - margin.left - margin.right;
x.range([0,plot_width]);

// Adjust the plot area for the labels
plot
.attr("transform","translate(" + margin.left + "," + margin.top + ")");

// Get the bar height from the y scale range band
var bar_height = y.rangeBand();

// Add the bars to the plot area
var bars = plot.selectAll("#" + ext_id + " .simplebar")
.data(data)
.enter()
.append("rect")
.attr("class","simplebar")
.attr("x",0)
.attr("y",function(d) {return y(d.Dim)})
.attr("height",bar_height)
.attr("width",function(d) {return x(d.Value)})
.on("click",function(d) {self.backendApi.selectValues(0,[d.Dim_key],true);});

// Add the axes
plot.append("g")
.attr("class","x axis")
.attr("transform","translate(0," + plot_height + ")")
.call(xAxis);

plot.append("g")
.attr("class","y axis")
.attr("transform","translate(0," + 0 + ")")
.call(yAxis);

// Add a click function to the y axis
plot.selectAll("#" + ext_id + " .y.axis .tick")
.on("click",function(d) {self.backendApi.selectValues(0,[getProp(data,"Dim",d,"Dim_key")],true);})
}
[/code]

Because our previous code ensured that the containing element would be empty, the visualization code can just draw from scratch every time. It defines some standard margins, dimensions, creates the svg element and a plot area based on the margins, creates scales and axes, and draws the bars.

One important piece to take note of is the block from line 38 to 57. This block creates a temporary y-axis, determines the size of the labels, then removes this axis and updates the margins, plot area, and scale to account for label size. This process is necessary for dynamically sizing the axis based on the length of the labels. With Qlik Sense, a user could load in data with dimension labels of various lengths, so the extension must be able to adjust its chart margins accordingly.

Modifying the extension to animate updates

In order to animate changes in this extension, a few modifications need to be made to the process for charting the data. In the example above, the extension clears out the chart containing element each time. Therefore, the only statement defined for the bars was a d3 .enter() statement. The data coming in is always being drawn as new bars because there are no existing bars to modify.

In order to animate the updates, we need to keep track of elements that are already on the page. We can't just clean out the chart each time and start from scratch. Rather, we can categorize the data elements into three categories:

  • entering: the data coming into the extension is new; there is no bar for it currently, so the extension needs to draw a new bar for it
  • updating: the data element already exists in the extension; a bar has already been drawn, so the extension needs to change the properties of this bar based on changes in values
  • exiting: the data element exists in the extension but no longer exists in the Qlik Sense data, thanks to filtering. The extension needs to remove this bar from the graph

With those three categories in mind, we can define a sequence of actions when the data changes. For this example, the sequence is defined as so:

  1. Any exiting data is removed from the extension first. The exiting bars will fade out.
  2. Any updating data is updated. The bars will re-size and re-position themselves if necessary.
  3. Any entering data is added to the extension. The new bars will fade in.

In the updating bar chart, the bar rendering statement is split into 3 pieces to reflect this sequence:
[code]
// Transition duration for update animations
var dur = 750;

// Add the data first, with a key
var bars = plot.selectAll(".updatingbar")
.data(data,function(d) {return d.Dim});

// Update logic
var updatedBars = bars
.transition()
.duration(dur)
.delay(!bars.exit().empty() * dur) // if there are no exiting bars, update immediately. otherwise, wait the duration before updating
.attr("y",function(d) {return y(d.Dim)})
.attr("width",function(d) {return x(d.Value)})
.attr("height",bar_height)
.attr("opacity",1);

// Enter logic
bars
.enter()
.append("rect")
.attr("class","updatingbar")
.attr("x",0)
.attr("y",function(d) {return y(d.Dim)})
.attr("opacity",0)
.attr("height",bar_height)
.attr("width",function(d) {return x(d.Value)})
.on("click",function(d) {self.backendApi.selectValues(0,[d.Dim_key],true);})
.transition()
.duration(dur)
.delay((!bars.exit().empty() + !updatedBars.empty()) * dur) // if there are no exiting bars and no updating bars, enter new bars immediately. otherwise, wait for the other animations to finish
.attr("opacity",1);

// Exit logic
bars
.exit()
.transition()
.duration(dur) // exit immediately
.attr("opacity",0)
.remove();
[/code]

Animation duration
In the code above, a transition duration is defined first in milliseconds. This variable will be used to control how long each animation takes.

Appending the data with a key
The data needs to be bound to the bar elements. Previously, this was done with a statement like:
[code]
var bars = plot.selectAll(".simplebar").data(data);
[/code]
This line appends the data to the bar elements. However, the updating extension needs to keep track of how the data is changing - what elements are new or exiting? In order to do this properly with our Qlik Sense data, we need to define a key that D3 will use to track the data. This key can be defined in the second parameter of the .data() call:
[code]
var bars = plot.selectAll(".updatingbar") .data(data,function(d) {return d.Dim});
[/code]
Now, D3 will use the values in property "Dim" to keep track of each data element. It can use that dimension to see if a piece of data already exists, has exited, or is entering the graph.

Exiting the data
The code above is defined in update - enter - exit order, but let's look at the pieces in the exit - update - enter order since that is sequence order in real time. First, the exit piece:
[code]
// Exit logic
bars
.exit()
.transition()
.duration(dur) // exit immediately
.attr("opacity",0)
.remove();
[/code]
Line 3 tells D3 to select the exiting elements and apply any subsequent code to them. The subsequent code defines a transition with the duration of the variable previously defined as 750ms. Any attribute changes defined in the transition will happen over the course of 750ms. There is no delay defined, so the animation will start as soon as elements exit the graph. The attribute that changes is the opacity, which is set to 0. This will cause the bar to slowly fade out of view. At the end of the transition, I call a remove() statement which will remove the bar element from the DOM, which removes it completely from existence.

Note: An attribute cannot be transitioned without a starting and ending point. The end point of this transition is for the opacity to be 0. Therefore, when creating the bar elements in the enter statement, the opacity must be set to a starting value for this transition to work. In this extension, I set the starting opacity to "1", which corresponds with 100% visibility.

Updating the data
Once the data has exited, the extension needs to update the already existing values:
[code]
// Update logic
var updatedBars = bars
.transition()
.duration(dur)
.delay(!bars.exit().empty() * dur) // if there are no exiting bars, update immediately. otherwise, wait the duration before updating
.attr("y",function(d) {return y(d.Dim)})
.attr("width",function(d) {return x(d.Value)})
.attr("height",bar_height)
.attr("opacity",1);
[/code]
The updating logic needs to take existing bars and change their size and position based on any changes in the data. The y, width, and height attributes are transitioned over the defined duration. The width accounts for the change in bar length based on any value changes. The y accounts for the change in vertical position of a bar based on sorting or number of bars in the chart. The height accounts for the change in bar height when a different number of bars are rendered in the plot area.

There is an additional transition parameter necessary for the updating bars: a delay. The updating sequence should only begin once the exiting sequence has finished. At first glance, it would seem appropriate to make the delay equal to the transition duration. That way, when 750ms has passed for exiting, the updating function will begin. However, this doesn't account for scenarios where nothing is exiting. If there is nothing exiting that needs to be animated, the updating logic shouldn't wait 750ms before it starts. It should start immediately instead. To account for this, the delay parameter checks to see if there is anything in the exit selection that will need to animate using the .empty() function. .empty() will return true if there are exiting elements, and false otherwise. By multiplying the inverse of .empty() by the duration, the delay is toggled between 750ms or 0ms, depending on whether there are exiting elements or not.

Note: The updating logic also sets the opacity attribute to "1". This may seem redundant, but it is actually important because animations can be interrupted if a user filters data rapidly in the extension. For example, a user may apply a filter that causes a bar to exit. Half-way through the bar exiting, the user removes this filter. This interrupts the exit animation and switches it to an update animation, since the bar is now part of the data again. However, because the exit animation was half-way finished, the bar may have an opacity of ".5" because it did not finish it's transition to "0". Adding the opacity attribute to the update logic ensures that this bar will get its opacity reset to the appropriate value, regardless of what was happening when the update animation started.

Entering the data
The last step is to define the creation of new bars:
[code]
// Enter logic
bars
.enter()
.append("rect")
.attr("class","updatingbar")
.attr("x",0)
.attr("y",function(d) {return y(d.Dim)})
.attr("opacity",0)
.attr("height",bar_height)
.attr("width",function(d) {return x(d.Value)})
.on("click",function(d) {self.backendApi.selectValues(0,[d.Dim_key],true);})
.transition()
.duration(dur)
.delay((!bars.exit().empty() + !updatedBars.empty()) * dur) // if there are no exiting bars and no updating bars, enter new bars immediately. otherwise, wait for the other animations to finish
.attr("opacity",1);
[/code]
This function should look pretty familiar. The bar's positional and size attributes are set before the transition begins. The initial opacity is set to "0". The transition then sets the opacity to "1", causing the bar to fade in. The delay concept from the updating logic is repeated here. However, the enter logic is third in the sequence so it needs to check both the exiting elements and the updating elements before it enters new data. If there are elements exiting, it will wait 750ms. If there are elements updating, it will wait an additional 750ms. If nothing is exiting or updating, the bars will enter immediately. This scenario occurs when the extension is rendered for the first time. There is no existing data to exit or update, so the extension runs the enter animation immediately.

Explore the rest of the code

The rest of the code has additional modifications to incorporate the bar chart animations. There are examples of how to set up the svg once and grab it subsequently, settings up axes and plot area transitions, etc.

If you have any questions about the process for implementing the general update pattern, please leave them in the comments or ask me on Twitter.

-Speros

TAGS: Miscellany

Related News

"...Thirteen's just obnoxious."

Read More

d3vl I've uploaded a new extension to GitHub called the d3 visualization library, or d3vl...

Read More