Some rss feeds provide a ttl tag (time to live) which indicates how long you can go without rescanning the feed. Would it be possible for bbc news to add this or would it be irrelevant given that the news could potentially update every minute?
Matt On 5/28/05, Ben Metcalfe <[EMAIL PROTECTED]> wrote: > > Another issue to think about on this project is what your unique ID will be > in identifying each story - and deciding whether a given RSS item is a new > story or an existing story. > > Headlines change (eg "Many feared dead in crash" -> "3 confirmed dead in > crash"), but also urls for stories change. The editorial policy in BBC News > is such that if a "major rewrite" occurs to a story, it gets a new id (and > as such a new url - the story id is the number before the .stm at the end of > the url, btw). To continue to the above example, when the journalist > looking after the crash story found out that three people had been confirmed > dead, they might update the story, update the headline and save as a major > rewrite (thus producing a new url). > > (Also you'll see that in the RSS feeds, we're explicitly stating that RSS > item urls aren't permalinks: <guid > isPermaLink="false">http://news.bbc.co.uk/1/hi/world/africa/4588937.stm</guid>) > > If you are trying to measure the 'weight' of a story, I assume the scope of > what you would want to track would effectively be both of the ids from the > above example, classed as a single 'story'. > > You might want to dump 12 or 24 hrs worth of RSS from the front page, and > have a look at how stories change before you embark on writing any > algorithms. > > > > Ben Metcalfe :: backstage.bbc.co.uk > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of L Dicken > Sent: 28 May 2005 11:04 > To: [email protected] > Subject: RE: [backstage] RSS at night. > > > This isn't necessarily a great heuristic to use since you are coding in > certain presumptions about the way the news stories are posted. This is the > case for any type of approach based on a "naieve" heuristic. A better way to > do this might involve some form of feedback neural net and an engine to > parse a story and establish the likelihood of follow-up / connected stories > based on content (although if you were going down this route you could > circumvent your entire method and establish a story's significance based on > content), then use this as a basis for a dynamically adapted poll interval. > However, the way news develops and is posted will be in such a random and > unpredictable fashion that really you might find it easier to just stick > with a single methodical system rather than trying to be tricky with it. At > the end of the day, as Ben said, BBC news is a 24 hour operation, and there > can't be a huge overhead involved in polling too frequently? > > Luke > > > -----Original Message----- > From: Chris Neale [mailto:[EMAIL PROTECTED] > Sent: 28 May 2005 09:39 > To: [email protected] > Subject: Re: [backstage] RSS at night. > > > Hi, > > One possible approach would be to have your programme alter the time between > updates dynamically. > > For example: you fetch the news at 1am .. and then again at 1:15am and find > there's been a new story. This means something is happening, so you reduce > the time until the next fetch to 10 minutes. At 1:25am you fetch again, and > theres a another update, so you reduce the time by 5 minutes again. At > 1:30am you fetch the feed and find no new stories, so you increase to 10 > minutes. > > This way the programme will (might) find it's own level. > > Actually coding the system wouldn't be terrifically difficult. The approach > I'd take would be to write the time of the next fetch to either a database > or file, and have a cron job run a script every five minutes. The script > would check the time of the next fetch, if it's passed then it would do the > fetch and update the time stored for the next check. > > Chris > > > Hi, > > Another prototype I am working on (yes, another) involves using the rss > feeds to work out which news items are the most important by how long they > are near the top of the feed. This obviously means checking the feeds > fairly often, but I am not entirely sure how often it should be. At the > moment I am thinking about every half an hour during the day, but the night > is another matter. How often would the news be updated during the night > (ie, is it worth updating every half an hour or would it be more useful to > check every few hours)? > > Thanks, > Duncan >
