Civic Hacking

Today's columnist is Christopher Sean Morrison from BRL-CAD. He writes:

"The three principles of transparency, participation, and collaboration form the cornerstone of an open government."

Peter R. Orszag, Open Government Directive

A couple months ago, a friend of mine at the Oregon State University's Open Source Lab asked if I'd heard about an upcoming "Civic Hack Day" event happening in Baltimore. I'm usually pretty in tune with the tech-happenings and other events going on in the city where I live, but this was rather surprising news that affected me personally. Not only was this event being held in the same city, it was in my own neighborhood, less than a quarter mile from my house, and I was learning about it from someone more than 2500 miles away on the other side of the country. It couldn't have been arranged more conveniently.

I had been following Baltimore's Open Data Initiative that was announced just a couple weeks earlier in January, but hadn't yet put much thought into the possibilities. Not a whole lot of cities have made their information available online, including less than a dozen in the United States. What does one do with data on things like parking tickets, city drainage routes, and property tax values? Even with my ignorance -- or perhaps especially because of it -- I was more than happy to sign up for this all-day collaborative coding session that had been arranged to get the mental juices flowing. If anything, I figured that amongst the few dozen developers in attendance that there'd at least be a few interesting ideas I might be able to help with.

This wasn't the first time I'd worked with civic data, to say the least, but this was the first time with data that I had a personal connection to. With the various federal and state data sets I've poked at, (for me at least) the information very quickly becomes collections of impersonal statistics that one has no control or influence over. Even when you find trends or correlations, the information is often just not particularly useful. It can be informative, newsworthy, and fantastic for government accountability but unlikely that it will affect your day-to-day behavior unless you work in politics.

There's something alluring about city data, though. The numbers and records are no longer aggregate information, it gets personal. Not only was my house tax valuation easily found, but there was the data for my property line, my building plot, streets through my neighborhood, and so much more. It was "my" data. Sure enough, a quick search through an XML text file even revealed that parking ticket from last fall.

The publishing of civic data in conveniently consumable form has been ongoing for several years now. Tracey Lauriault introduced OSBR readership to civic data back in February 2008. A year later, Jennifer Bell wrote about how open data and government transparency are affecting our role as participatory citizens. Companies, non-governmental organizations, and private individual software developers are springing up around the data to help manage, characterize, visualize, and repurpose this information in meaningful ways. Today, you can find numerous governments at the city, state, province, and federal level digitally publishing all kinds of raw data on how they operate, where money is spent, what information they track, when transactions occur, and who they interact with. It's a new level of access to data that is only beginning to be realized.

At the federal level, the amount of data being made available and the rate at which it's increasing is staggering. The United States now has more than 300,000 data sets available with more being added on a daily basis. In January 2009 on his first day in office, President Obama issued an Open Government Initiative memo committing the U.S. federal government to "an unprecedented level of openness" in order to establish "a system of transparency, public participation, and collaboration." A little less than a year later in December 2009, that became an Open Government Directive, the http://DATA.gov website was established, and federal agencies were mandated to develop a plan specifically towards improving transparency, participation, and collaboration with the public. Now a little more than a year since, 17 of 29 government agencies fully meet the requirements set forth by the directive with others making significant progress towards openness.

Cities, however, have actually taken much longer to publish their data. Perhaps it's due to local politics or funding, but more than likely it's at least in part because the information is more personally identifiable that they have to be more careful. If someone reports a pothole that needs to be fixed, it is useful for their request to be geotagged (i.e., have the exact physical location annotated with the report). Service personnel will know which streets to focus attention on and you'll know which ones to avoid. Similarly, geotagged crime data makes it very easy to create a visual graph of the crime hot spots in a city. Certain types of crime reports, though, might give away a victim's location and make them susceptible to repeated crime offenses. The datasets have to be carefully reviewed before they can be made available to the world.

That said, there are several big and small cities alike now that publish their civic information to the Internet. Having published substantial repositories of information in 2009, San Francisco continues to be a great example on providing excellent civic data openness. They followed on the heels of the massive data repositories released through the Washington D.C. Data Catalog. For more than a year, London has also published hundreds of datasets. Toronto is yet another. The city of Ottawa recently wrapped up their Open Data App Contest where developers competed for cash prizes as they looked to find innovative and useful things to do with their city's data. New York similarly sponsored a BigApps 2.0 contest to make use of their civic data. The list goes on and on as city governments around the world increasingly open up access to their civic data.

As "Civic Hack Day" came to a close in Baltimore, I was left awestruck at the abundance of personal, professional, and commercial possibilities. In that single day of coding, we'd collectively developed numerous proof-of-concept applications that leveraged Baltimore's available civic data. One guy had a crime density map working while another had an animation of crime incidents over time. Another group created an application that listens for people tweeting 311 support requests (with geotagged pictures attached) via Twitter.

What did I end up with? I developed a working geometry importer for my favorite open source CAD system to read in shapefile data as 3D geometry. By converting the city data into 3D geometry, it becomes easy to procedurally generate an accurate model of the entire city based on any number of criteria. Perhaps it'd be useful to model each tax district with buildings scaled according to their recorded tax value. An analytic visibility analysis might show if there's a correlation between light poles, public video cameras, and crime levels or whether the crime is merely shifted around the corner out of sight. Maybe, just maybe, it'll even help avoid that next parking ticket.