RE: [backstage] News on a map

[David Tattersall]

Duncan,   You could see which feed the article is from (i.e. BBC North West) and exclude any possible place names that lie out side of the bounds of this area. You may get news items referring to other places, but its unlikely if it was specifically filed under a certain feed.   David From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Duncan Barclay Sent: 10 June 2005 7:15 To: backstage@lists.bbc.co.uk Subject: [backstage] News on a map Hi, There has been a lot of talk about adding geographical data to the BBC RSS feeds recently, with many people saying that it isn't possible to do it in a way that isn't manual.  I agree that it isn't 100% accurate doing it automatically, but I have found that it is possible to do it reasonably accurately. Yes, that's right.  I made another prototype :). It can be found at http://backstage.min-data.co.uk/geotagged/ .  There are several problems with it at the moment, such as the fact that it doesn't update ever as it takes too long to update for me to be happy to make it update regularly (I will try to optimise the code when I have the time). So far I haven't seen it get a news story completely wrong, but I will have to check it over a longer period of time than 2 days. Some of you are probably wanting to know how it works out where a news item is from, so I might as well say.  Firstly it gets the news article from the BBC News site (hope this is ok, don't want to be breaking the T&Cs), strips it down to the actual article (not replies, extra info, etc) and then works out which words could be place names, quite simply by working out which ones have a capital letter to start with.  It then removes some common words and checks the remaining ones in a database.  At the same time it retrieves the longitude and latitude of those locations. Then, if there are no locations, it basically ignores the article as there is no way to plot it on the map.  That is why there are so few articles there.  If there is one location and it is mentioned enough times in the text (number of times varies depending on the length of the article), that point is plotted.  If there are 2 or more, it works out which two points are closest to each other, and if they are less than 50 miles apart it plots the place that is mentioned first in the article. As I mentioned, there are a few problems with it.  I am not entirely clear what the rules are about scraping the BBC News site for use purely by the script (the description and title for the articles all come from the RSS feed), so some guidelines on that would be useful.  There is also a problem where it won't work with places with two words in the name, which isn't ideal. If any of you have any comments about it (and I am sure you will), please tell me.  All feedback on this would be appreciated. Thanks, Duncan