I did consider adding stuff from
other feeds, in which case I think it would be limited to that specific
area (or, say, 100 miles around the area). The main problem with that
idea is that it would mean getting lots of whole articles from the BBC
News website, which they might not like. And I don't want to be in
violation of the terms and conditions ;).
Any clarification from BBC people on downloading articles from the site
like that would be useful.
The other problem with that idea is that the script takes ages to get
just the one feed and do stuff with it. Around 2 or 3 seconds per news
story.
If I can do that though, I would like to. I would also like to
optimise the script a bit.
Duncan
David Tattersall wrote:
Duncan,
You could see which feed the
article is from (i.e. BBC North West) and exclude any possible place
names that lie out side of the bounds of this area. You may get news
items referring to other places, but its unlikely if it was
specifically filed under a certain feed.
David
Hi,
There has been a lot of talk about adding geographical data to the BBC
RSS feeds recently, with many people saying that it isn't possible to
do it in a way that isn't manual. I agree that it isn't 100% accurate
doing it automatically, but I have found that it is possible to do it
reasonably accurately.
Yes, that's right. I made another prototype :).
It can be found at http://backstage.min-data.co.uk/geotagged/
. There are several problems with it at the moment, such as the fact
that it doesn't update ever as it takes too long to update for me to be
happy to make it update regularly (I will try to optimise the code when
I have the time).
So far I haven't seen it get a news story completely wrong, but I will
have to check it over a longer period of time than 2 days.
Some of you are probably wanting to know how it works out where a news
item is from, so I might as well say. Firstly it gets the news article
from the BBC News site (hope this is ok, don't want to be breaking the
T&Cs), strips it down to the actual article (not replies, extra
info, etc) and then works out which words could be place names, quite
simply by working out which ones have a capital letter to start with.
It then removes some common words and checks the remaining ones in a
database. At the same time it retrieves the longitude and latitude of
those locations.
Then, if there are no locations, it basically ignores the article as
there is no way to plot it on the map. That is why there are so few
articles there. If there is one location and it is mentioned enough
times in the text (number of times varies depending on the length of
the article), that point is plotted. If there are 2 or more, it works
out which two points are closest to each other, and if they are less
than 50 miles apart it plots the place that is mentioned first in the
article.
As I mentioned, there are a few problems with it. I am not entirely
clear what the rules are about scraping the BBC News site for use
purely by the script (the description and title for the articles all
come from the RSS feed), so some guidelines on that would be useful.
There is also a problem where it won't work with places with two words
in the name, which isn't ideal.
If any of you have any comments about it (and I am sure you will),
please tell me. All feedback on this would be appreciated.
Thanks,
Duncan
|
