Waiting for Lunchtime

MySociety and Heritage Lottery Fund

I've been on the developers' team for MySociety.org for a few months now, but haven't really done much more than a few noddy bits of coding for them. But this evening I've managed to get my teeth into something a little bit more meaty.

One of their "bubbling under" sites is YourHistoryHere, which is hoping to grow up into a community-based site giving "odd factoids, rumours or tidbits" about places all over the country. MySociety have been given the opportunity to get in touch with the Heritage Lottery Fund to collaborate on building that site into something much bigger, more powerful, and of even more use.

What they wanted was someone to try to scrape some of the data from the HLF website about projects they've already funded, and to plonk that into Google Maps in some fairly basic but functional form. That way MySociety can present the HLF with a good working example of what can be done, and hopefully that should help things move forwards much quicker.

Well, scraping data from websites, mungeing it in various ways, and chucking it back on the web interfacing with the Google Maps API is exactly the sort of thing that Perl excels at, and the sort of thing that I've done many times. So I was happy to offer my help, and I've just finished getting the main data-scraper working, pulling down the details of several thousand projects from the Local Heritage Initiative site and writing it out as XML.

I'll be kicking that off on its full run in a couple of hours time (to try to minimise any potential load on the LHI server, although it seems a nice fast Apache box), and then the next stage is to pass the XML over to MySociety so that they can append latitude and longitude data to it, based on postcode lookups. Then it's back over to me for the Google Maps side of things. With a bit of luck and patience that should all be up and running by the end of this weekend.

Yay!

posted by Iain @ 22:37