Archive | Data RSS feed for this section

The Demise of Government-Created Statistical Data?

By on May 11, 2012 in Data, Musings, On the Job

Washington MonumentLike most data people, I prefer order and logic. So it was a huge shock when I joined a federal budget research organization and started learning about the orderly and logical process by which the U.S. government creates an annual budget. An orderly and logical process that Congress mostly disregards.

Really, the whole politicized debacle offends my sensibilities as a citizen and as a data professional.

Furthermore, the recent zeal for budget cuts has resulted in budget cuts that affect our ability to make smart budget cuts. Specifically, I’m talking about attacks on government-created statistical data—data that’s* used by lawmakers, social service organizations, and businesses to make decisions and allocate increasingly-scarce resources.

Two examples I’ve written about recently:

  • Is Federal Spending Transparency on the Decline?: a guest post for the Sunlight Foundation’s blog about the demise of the Consolidated Federal Funds Report and why that makes it harder to understand federal spending.
  • American Community Survey Under Attack: the House recently passed a spending bill that prohibits the Department of Commerce from funding the American Community Survey (ACS). The yearly ACS replaced the decennial census long-form questionnaire, and its data helps* state and local governments determine how to distribute funds, among other things. See here, here, and here for more information about the widespread usefulness of the ACS.

Of course, order and logic sometimes need to be tempered with a dose of pragmatism. But when our governing body is governed almost entirely by short-term thinking, we should think about not electing them again.

*Language evolves!

Comments { 0 }

Strata 2012: Making Data Work

By on April 5, 2012 in Data, Programming, Visualization

It’s been over a month since Strata 2012. Lest they meet the same fate as scribbles from conferences long past, I’m putting my notes here instead of leaving them in whatever notebook I happened to be toting around that week.

The biggest a-ha moment, and one that I’ll be writing about in the future, came from Ben Goldacre’s keynote, when he compares big data practitioners to drunks looking for car keys only where the light shines. We focus on the data that’s available without asking, “what’s missing?” Plus, it’s fun to hear someone with a British accent say “blobogram.”

Continue Reading →

Comments { 0 }

Protovis Visualization for Older IE

By on March 1, 2012 in Programming, Visualization

Two days ago, I posted my Flare visualizations—based on a Flash/Actionscript library–explaining that we can’t yet use the D3 visualization library because it outputs SVG, which isn’t supported by older versions of IE.

The very next day, Hjalmar Gislasun of DataMarket gave a talk at O’Reilly’s Strata Conference. DataMarket faced the same problem back in 2010 after reviewing over 100 visualization libraries and choosing Protovis (a predecessor of D3). Not wanting to exclude the 20% of the world still using IE 7/8, they developed protovis-msie, a tool to convert Protovis SVG output to VML, a vector format understood by older browsers.

And…they open sourced it. So Protovis is now on the table for use at National Priorities Project. Thank you, DataMarket!

Like Flare, Protovis is no longer under active development. That said, it still has an active user community (unlike Flare). And the output won’t be Flash, so iOS is back on the table.

DataMarket’s strategy is to continue using Protovis until most IE users are on version 9 (which supports SVG) and then switch over to D3. It was refreshing to hear browser support strategies from people developing visualizations for commercial use; they don’t have the luxury of ignoring IE 8, which is tempting to do but not viable in the real world.

Comments { 0 }

Data Visualizations with Flare

By on February 28, 2012 in Design, On the Job, Programming, Visualization

Two weeks ago, the White House released President Obama’s FY 2013 budget request. Using the numbers scrubbed by NPP’s crack research team, I created a few visualizations using the Actionscript/Flash-based Flare data visualization library.

Flare was ideal because it includes sample code for a stacked area chart with tooltips–exactly what we wanted. I had some concerns about the Flash output, but many of our website visitors use browsers that don’t support SVG (IE8), so tools like D3 aren’t an option just yet.

Here’s a preview of what we’ll include (not the final version).  The first example is built with normalized data:

Apologies, but you need Flash to view this content.

Get Adobe Flash player

For the second example (total federal spending by category), we wanted to convey the overall size of the budget over time, so we didn’t normalize the data. As a result, the huge numbers caused some formatting issues, but it’s still an interesting story–especially the 2009 spike. Also note the rise in healthcare spending over time: 7% of the budget in 1976 and 25% in 2013.

Apologies, but you need Flash to view this content.

Get Adobe Flash player

Flare makes it easy to lay out the data and create the animated transitions, and after making a few tweaks to the Flare library and the stacked area sample code, I’m happy with the way these turned out.

That said, I’d be reluctant to use Flare again. It isn’t being actively developed, and there’s nowhere to turn for help when you get stuck. Visualizations are evolving, and the tools to create them–no matter how good they are–need to evolve too.

Comments { 1 }

No Excuses for Ugly Excel Charts

By on January 29, 2012 in Design, On the Job, Visualization

2/2/2012: Corrected the revised bar chart by setting the horizontal axis minimum to zero. Thanks to Jon Peltier for catch.

Excel remains the de-facto graphing tool at National Priorities Project. A simple chart is often the best way to convey information about federal spending and budgeting, and Excel is the common language among our researchers and IT team.

Using Excel, however, is no excuse for ignoring style and the best practices of information display. So many organizations put out amazing, well-researched publications and then tack on default Excel graphs as an afterthought. But graphs are often what people look at first, and they deserve to be first-class citizens in the editing process.

I created some Excel chart templates for NPP, drawing on two sources for inspiration and practical advice: the classic Visual Display of Quantitative Information by Edward Tufte and The Wall Street Journal Guide to Information Graphics by Dona Wong.

Tufte is big on eliminating “unnecessary ink” that distracts from the information, and Wong advocates requiring the least amount of work on the reader’s part. With their advice in mind, I modified Excel’s default bar chart from this:

Excel bar chart - default

Bar chart: Excel default

To this:

Excel bar chart - modified

Bar chart: new template

  • Smaller gap between bars
  • Don’t make readers guess the numbers; if possible, label the bars directly
  • Direct labeling means you don’t need the noisy gridlines or even the x-axis
  • Remove the y-axis tick marks for even more noise reduction
  • Get rid of those zeros by showing data in millions or billions
  • Make sure the entire length of the bars is shown (in this case, by setting the horizontal axis minimum to zero). HT Jon Peltier.

The pie chart got a similar treatment. The Excel default:

Excel pie chart - default

Pie chart: Excel default

The new template:

Excel pie chart - modified

Pie chart: new template

  • Label the pie slices directly—don’t make people use a legend to decode
  • Avoid the default Office color palette and develop your own (ours is based on colors from our website)
  • A white line between pie slices emphasizes the boundaries

Excel isn’t perfect, but it’s out there in the world, and you can’t ignore it. Luckily, a little extra effort goes a long way.

Comments { 2 }

2012 Technical Goals

By on January 4, 2012 in Database, Musings

It’s easy to get distracted by the constant barrage of new technology. Sometimes I miss my days of COBOL programming. Things changed a bit more slowly then, allowing programmers to truly master a language without worrying about missing the next big thing.

Overall, however, being a technologist is a lot more exciting in 2012 than it was in the Y2K remediation days. That said, it’s time to stop worrying about every shiny new toy that comes along and focus on a few things that appeal to my interests and career aspirations.

NoSQL
I know my way around an RDBMS, but 2012 will be the year I stop reading about NoSQL and start doing something with it. Flavor TBD.

D3.js
Data visualization is more than a fad. Not those disjointed, meaningless infographics floating around, but numbers rendered as pictures that tell a story. I’m not a graphic designer, but I am a programmer, so it makes sense to learn a tool that can programmatically create pictures of the federal data I work with every day.

Comments { 0 }

Python, Django, & MySQL on Windows 7, Part 5: Installing MySQL

By on September 14, 2011 in Database, Programming

This is the fifth and final post in a  dummies guide to getting stared with Python, Django, & MySQL on Windows 7.

By now, you should have Django installed into a virtual environment.  These tutorials aren’t meant to cover building a django app, just to point out the quirks involved with getting a project up and running on Windows.  These tutorials also assume you want to construct real applications using a real development environment.

To that end, you’ll want a heftier database than sqlite.  We use MySQL at the office, so these instructions cover installing it and using it with Django.

Install MySQL

  1. Download and install MySQL.
  2. Once MySQL is installed, proceed through the configuration wizard. Check Include Bin Directory in Windows PATH box.
  3. When prompted, set a password for the MySQL root account.
  4. Once the installation wizard is done, open a command window and log in to MySQL with the root account: mysql -uroot -p (you’ll be prompted for the password).
  5. After logging in, run the following commands to create a database, create a user for your Django project, and grant the user database access.

Install MySQL-python

You’ll need the MySQL-python package, a Python interface to MySQL.

  1. Download the windows MySQL-python distribution here.  The author has some instructions about the appropriate version; assuming a 32-bit version of Python 2.7, you’d download this package (.exe).
  2. After downloading, do not run the Windows installer. Doing so will install MySQL-python to your root python, which virtual environments created via –no-site-packages won’t be able to see.
  3. Instead, install the downloaded package to your virtual environment by using easy_install, which can install from Windows binary installers:
    easy_install file://c:/users/you/downloads/mysql-python-1.2.3.win32-py2.7.exe (modify to reflect the location of the downloaded installer and its name).installing mysql-python package via easy_install

Configure Django

Next, you’ll need to update the database-related settings of your Django project.

  1. From the directory of your Django project, open settings.py using your favorite editor.
  2. Update the default key in the DATABASES dictionary.  Set ENGINE to django.db.backends.mysql and set NAME, USER, and PASSWORD to the database name, username, and password you chose when installing MySQL.  See Part I of the Django tutorial for more information about database settings.
  3. Open a command window, activate your virtual environment, and change to the directory of your Django project.
  4. Type python manage.py syncdb. This command creates the underlying tables required for your Django project.
    syncdb output
  5. If the syncdb worked, you have Python, Django, and MySQL communicating in harmony.  Congratulations!  You can now proceed through the Django tutorial and create your first application.
Comments { 5 }

Python, Django, & MySQL on Windows 7, Part 4: Installing Django

By on September 14, 2011 in Database, Programming

This is the fourth post in a  dummies guide to getting stared with Python, Django, & MySQL on Windows 7.

We’re finally ready to install Django, a popular Web-development framework. Detailed instructions for building out a Django site are beyond the scope of this humble tutorial; try The Definitive Guide to Django or Django’s online Getting started docs for that.

These directions will simply make sure you can get up and running.

Installing Django

  1. Open a command window.
  2. Go to (or create) the virtual environment you’ll be using for your django project. For this example, I created a virtualenv called django-tutorial: virtualenv django-tutorial --no-site-packages
  3. Install django: pip install django
    install django 
  4. Start an interactive interpreter by typing python (or iPython, if you’ve made it virtual environment-aware).
  5. Test the install by importing the django module and checking its version: https://gist.github.com/1177372
  6. Create a new directory to hold your Django projects and code. Change to it.
  7. Think of a name for your first Django project and create it by running the following command: python -m django-admin startproject [projectname].
    Important: most Django docs show django-admin.py startproject [projectname] to start a new project, which can cause import errors and other trouble for Windows users. See this stackoverflow thread for details.
  8. You should now see the project’s folder in your Django directory:django project folder
  9. Change into the new project folder.
  10. Test the new project by typing python manage.py.  Manage.py is Django’s command line utility; you should see a list of its available subcommands.
  11. A further test is to start up Django’s development server: python manage.py runserver. You should see something like this:
    django runserver

If you’ve made it this far, you’ve successfully installed Django and created your first project.

Next up is Part 5: Installing MySQL.

Comments { 5 }

Two month milestone

By on May 18, 2011 in Database, Musings, Programming

flowering tree in Childs Park

Monday marked two completed months with the National Priorities Project. Though these weeks haven’t produced much writing, they’ve been a whirlwind of learning:

  • Python
  • Django
  • MySQL
  • The joy of setting up a proper Windows dev environment using the above three items
  • Piston, a tool for powering APIs through Django
  • Linux
  • Git/Github
  • The Federal Budget process
  • The Consolidated Federal Funds Report , a huge annual file of government expenditures.
  • Various other indicators about the state of our union: gas emissions by state, average teacher salaries, people in poverty, insurance enrollments, etc.
  • Finally, I’m NPP’s interim Twitterer, a fascinating distraction.

One day soon I’ll write a Dummies Guide to Setting up Python/Django/MySQL on Windows post. In the meantime, it’s great to be back in the hands-on tech saddle.

Comments { 2 }

Save the data, save on FOIA?

By on April 3, 2011 in Data, Musings

Last week I wrote my first entries on the National Priorities Project’s (NPP) blog. Friday’s piece concerned the potential $32 million cuts to the Federal government’s open data initiatives.

Alexander Howard wrote a tremendous overview of the situation, from the recent history of open government platforms to the less-than-perfect implementation of those platforms to the implications of having their funding cut from $34 million to $2 million.

He quoted some of NPP’s numbers that try to put $32 million in context. In terms of the Federal budget, it’s a tiny sum of money–.0009% of the proposed FY11 spending.

That’s an interesting figure, but even if $32 million is just a drop in the bucket, that’s not to say we should spend it carelessly. I’m new to the open government scene, but you don’t have to dig too far into Data.gov to realize it’s far from perfect. Howard’s primer provides some insight into the perverse incentives behind quirks like datasets split up by geography and agencies that don’t publish their juicy stuff.

But consider another number we published: $32 million is 7.7% of the amount that the government spent processing Freedom of Information Act (FOIA) requests in FY10.

A compelling story would be to find out what types of FOIA requests could be serviced via the Data.gov suite of sites. Even better, why not use these requests to prioritize the data that’s released online?

If we can use $32 million to take a bite out of that $416+ million FOIA bill*, why not pursue that investment?

Some of my colleagues would say because it’s not about the money—it’s about policy. As a developer, I have a hard time wrapping my brain around that. Policy? Why wouldn’t our elected officials just make decisions that are logical?

It seems I have much to learn.

*figure pulled from FOIA.gov/data.

Comments { 0 }