Comments on: Scraping Google Groups http://saturnboy.com/2010/03/scraping-google-groups/ Code, Work, and Life Thu, 10 May 2012 17:13:39 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: justin http://saturnboy.com/2010/03/scraping-google-groups/comment-page-1/#comment-338 justin Fri, 21 Oct 2011 17:35:28 +0000 http://saturnboy.com/?p=1105#comment-338 @Neal: Best would be if Google made a Google Groups API that allowed you to get forum data back out. Until they do that, everything is basically a big hack (like my scraping code posted above). @Neal: Best would be if Google made a Google Groups API that allowed you to get forum data back out. Until they do that, everything is basically a big hack (like my scraping code posted above).

]]> By: Neal McBurnett http://saturnboy.com/2010/03/scraping-google-groups/comment-page-1/#comment-337 Neal McBurnett Thu, 20 Oct 2011 17:14:47 +0000 http://saturnboy.com/?p=1105#comment-337 Thanks - I love it when people help liberate data! My notion is that it would be good to scrape it into a standard data format, e.g. the ATOM syndication standard from the IETF: http://tools.ietf.org/html/rfc4287 Then people who write forum and blog software could support imports from ATOM, and this could be come a really easy thing to do. For some archives, I guess ATOM might not be able to directly represent everything, so additions to the schema might be necessary, but I haven't looked at that. Would that make sense? Do you know how what you produced differs from ATOM? Thanks – I love it when people help liberate data!

My notion is that it would be good to scrape it into a standard data format, e.g. the ATOM syndication standard from the IETF:

http://tools.ietf.org/html/rfc4287

Then people who write forum and blog software could support imports from ATOM, and this could be come a really easy thing to do.

For some archives, I guess ATOM might not be able to directly represent everything, so additions to the schema might be necessary, but I haven’t looked at that.

Would that make sense? Do you know how what you produced differs from ATOM?

]]>