Today we’re announcing the public beta of Dotspotting, a project designed to help people work with geographic data in ways that are intelligible, straightforward and useful in the real world. You can sign up for an account or just take a peek around at http://dotspotting.stamen.com.
Dotspotting is the first project Stamen is releasing as part of Citytracking, a project funded by the Knight News Challenge (more here and here). It’s being headed up by Stamen’s own Aaron Cope with input from most of the rest of the team. We’re making tools to help people gather data about cities and make that data more legible. Our hope is to do this in a way that’s simple enough for regular people to get involved, but robust enough for real research to happen along the way.
The basic idea
There’s currently a whole chain of elements involved in building digital civic infrastructure for the public, and these are represented by various Stamen projects and those of others. At the moment, the current hodgepodge of bits — including APIs and official sources, scraped websites, sometimes-reusable data formats and datasets, visualizations, embeddable widgets etc. — is fractured, overly technical and obscure, held in the knowledge base of a relatively small number of people, and requires considerable expertise to harness. That is, unless you’re willing to use generic tools like Google Maps, and agree to terms of service which allow them to share your content with other people. We want to change this. Visualizing city data shouldn’t be this hard, or this proprietary, or this generic.
So the first part of this project is to start from scratch, in a ‘clean room’ environment. We’ve started from a baseline that’s really straightforward, tackling the simplest part: getting dots on maps, without legacy code or any baggage. Just that, to start. Dots on maps. And then we’ll share that work with the public, and see what happens next.
The problem
But dots on maps implies a few other things: getting the locations, putting them on there, working with them, and — crucially — getting them out in a format that people can work with.
We’ve had several interactions with different city agencies so far, and while the situation has changed alot in the last few years, we’ve realized that, for the foreseeable future, people aren’t going to stop using Word and Excel and Pages and Numbers to work with their data, or even stop using paper. It’s made us think that if this stuff is really going to work out in the long run, we need to focus our thinking on projects that can consume as well as export things that cities and people actually use and use now, and not stick with projects that have to rely on fancy APIs or the latest database flavor.
It’s great that San Francisco and New York are releasing structured XML data, but Oakland is still uploading Excel spreadsheets (it’s actually awesome that they do), and the Tenderloin police lieutenants are printing out paper maps and hand-placing colored stickers on them. At some point, if this really is the way things are going, we’re going to need to meet the needs of actual functioning city agencies — and while APIs are great and necessary, for now that means Excel spreadsheets and Word docs. It also means being able to easily read in data that people have uploaded to google maps, interface with SMS systems like those that Ushahidi are pioneering. And it means being able to export to things like PowerPoint and Keynote, scary as that may seem.
What we’ve launched with is the baseline work that’s being done to make this stuff internet-native. There’s a login and permissions system that pretty much works. Uploading .csv files full of dots works. Each dot has an HTML page of its own, for example, like they do on Crimespotting. Collections of dots (we’re calling them sheets) work, and you can export them. And there are dots on maps.
Getting started
The first thing to do is to take a look at the site’s FAQ, which has detailed instructions on how the whole thing fits together. Where we’re at now is that you can upload comma-separated value files, or .csv files, containing geographic information, and dotspotting will try and put them on maps. You can write these files in any text editor, or export them directly from a spreadsheet program like Excel.
The first and simplest thing I could think of was to take the two locations that Stamen has had out studios, on opposite corners of the lovely 16th and Mission plaza. That csv file is here, and it looks like this:
address,description
"2017 Mission St San Francisco CA", HQ2
"3012 16th Street San Francisco California", HQ1
The top line (address,description) is the title line; it basically describes the lines that are to follow. So for every other line in the document, the first item will be considered an address, and the second item will be considered a description, separated by a comma (“comma-separated values,” remember).
I uploaded the file here, and here’s what I get as a result, on a map:
We’re calling the resulting collection of maps that results from an upload a “sheet,” partly because we think it’ll make sense later on for people to think of these as spreadsheets or other documents, and partly because Aaron wants to use this image on the site:
What I also get, in addition to the map, are two html pages, one for each dot that’s been created (there’s a dot for every line in my original file). And those dots have significantly more information now than they did when they were uploaded: where previously there was just an address and a description, there’s now a bunch of other stuff, like latitude and longitude of course, but also things like aWhere On Earth ID, or more simply “woeid,” that tells me that these dots are in zip code 94110. So I’m getting a lat/long lookup as well as the addition of some other sample geographically-relevant information. And, crucially, I can export this data, which now looks like this:
id,sheet_id,user_id,perms,created,latitude,longitude,
geohash,location,type,address,description,yahoo:woeid
77171,271,12,public,"2010-12-07 23:27:53",37.764673,-122.419621,
9q8yy6bg0j63,woeid:12797161,,"2017 Mission St San Francisco CA",HQ2,12797161
77173,271,12,public,"2010-12-07 23:27:53",37.765042,-122.419930,
9q8yy6bt2fh4,woeid:12797154,,"3012 16th Street San Francisco California",HQ1,12797154
More like that
There’s more to talk about, but that’ll do for a first post. As Aaron so nicely puts it, HEY LOOK! THIS IS THE SUPER ALPHA-BETA-DISCO-BALL VERSION ( tell me more ), and we’re very much thinking of this as a first pass, with lots of missing pieces. Matt Mullenweg says that “if you’re not embarrassed when you ship your first version you waited too long” and there are worse people to listen to.
Whoosh! Off we go. We’d love it if you visit the project and let us know what you think.