Creating Grace: A bespoke data processing platform

FigureNZ

Background:

Figure.NZ is a non-profit open-data organisation, the aim of which is to make data about New Zealand more accessible - so you don’t need to be an expert to able to find and use data. Figure.NZ’s goals are also around encouraging data literacy, using data as an input into decision making and providing context to understand the world around you. Originally, Figure.NZ was built on a simple Wordpress site, using a javascript template to display the data points we had created in a spreadsheet into a simple interactive graph or map. It would take hours to produce a single graph using this process, once you factor in the cleaning of the data, accurately entering the metadata, auditing and attaching the related csv to the resulting chart. We needed a way to scale up the process of cleaning the data, extracting the information to display as charts, in a way that meant that the related table was attached, errors were reduced, store in a database and that could be easily found on the website.

With a ton of snacks and plenty of ideas, Lillian organised our very own Hackathon day with some awesome brains from a range of different skills - to figure out how to automate the data visualisation process where we could, store the data we wanted, and link it together in ways to make it easier to find.

My role:

Communicate the pain points of the current process and the knowledge of how open data is provided by sources, while also investigating how the search engine would connect to this database to make it easier to find information.

As this product was developed, I tested and entered our first datasets to be processed, creating some of the first extractions, tables and charts that then appeared on the website.

I worked on how Elastic Search could be utilised best when pulling out and displaying the most relevant data from a user perspective, and using Google Analytics (and later Woopra and Heap) to monitor and iterate how to improve search results.

We worked through a number of exercises to prioritise the core features we wanted, including of course - PostIt notes.

After we created a basic structure for what we wanted, Opcode, comprised of 3 awesome developers, created Grace, something that is forever being iterated and improved from the original MVP version as we learn more about what we want to be able to do with the data.

The result:

Grace is a platform hosted on AWS, built using Python/Flask, where the primary data store is PostgresSQL. This includes a UI for extracting and updating the data, allowing a Data Officer to create standardised long form tables and where we could distinguish between content that was a representation of NZ Dollars, Text or Percentage for example.

After data is processed and within Grace, there is a UI for creating charts, that is then rendered using d3/vega, displayed via API onto the Figure.NZ website. We went from creating 600 charts in 12 months, to 10,000 and increasing. With a focus on reducing errors and increasing productivity, this means that the Data Officers can spend more time on the things they are best at and less on the admin side.