Re: RSS Aggregation?
That could be. I vaguely remember that the spec wasn't final, but was very, very close. This was probably 6 months ago or so. Maybe that timeline will tell you something. And I did use the wiki several times so who knows what insanity I mixed in. :-) I'll go back and take a look at my code to see what I did or whether my memory is just bad and I'll post back to this thread in case anyone's interested. On 4/20/06, Roger Benningfield [EMAIL PROTECTED] wrote: I surely thought I remembered one from when I built my reader. Rob: Could it have been one of the interim specs you were looking at? 'Cause there was all kinds of odd stuff in there at certain points... particularly in the pre-IETF drafts. In addition, there was (and is) a lotta stuff on the wiki that is foreign to the spec as well, which can be pretty confusing to implementors. -- Roger ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238249 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
On Wednesday 19 April 2006 15:43, Neil Middleton wrote: Currently the site is aggregating ~500 RSS feeds, but checking these feeds is growing to be a pain in the butt. Having to get CF to check each of these feeds regulary (ideally every 15 minutes) is more difficult than it sounds. Why not: Give each feed a 'last check time' Once every 5 minutes, pick the oldest and retrieve it, then update last check time to now. Also: I doubt most of your users check every 15 mins, so why does your site need to ? -- Tom Chiverton Advanced ColdFusion Programmer ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238254 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
Well, what I have gone for as an interim is something similar. Every feed has a lastCheck time, and every minute, the app checks the oldest 10 feeds. Therefore each feed should get checked roughly hourly. Seems to be working well at the moment, I'll consider dropping the frequency once I know how this behaves.. Neil Why not: Give each feed a 'last check time' Once every 5 minutes, pick the oldest and retrieve it, then update last check time to now. Also: I doubt most of your users check every 15 mins, so why does your site need to ? -- Tom Chiverton Advanced ColdFusion Programmer ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238256 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
So, based on Roger's comments, I checked my code and, in fact, there is no Atom 1.0 element analogous to TTL. And I didn't make one up for inclusion in my code. :-) Thanks for the clarification. On 4/20/06, Roger Benningfield [EMAIL PROTECTED] wrote: I surely thought I remembered one from when I built my reader. Rob: Could it have been one of the interim specs you were looking at? 'Cause there was all kinds of odd stuff in there at certain points... particularly in the pre-IETF drafts. In addition, there was (and is) a lotta stuff on the wiki that is foreign to the spec as well, which can be pretty confusing to implementors. -- Roger ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238261 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
Currently the site is aggregating ~500 RSS feeds, but checking these feeds is growing to be a pain in the butt. Having to get CF to check each of these feeds regulary (ideally every 15 minutes) is more difficult than it sounds. Neil: Polling every fifteen minutes is an enormous waste of CPU and bandwidth... for both you and the source sites. For example, if you're aggregating individual blogs, once every 24 hours will cover the vast majority just fine. Ideally, you'd either opt for some middle ground (once an hour or so), or come up with adaptive code that spaces out polling based upon observed update periods. But even if you're gonna stick with over-polling (a good way to get your IP blocked), there are places to optimize: * Use Conditional GET... since 90% of feeds won't have seen an update in the last fifteen minutes, you've saved nearly 90% of your server's effort. * Make your spider compatible with RFC 3229. It won't help in most cases, but some high-flow publishers (Microsoft, etc.) will send you deltas of their sliding-window feeds. That'll cut down on parsing time. * Try CFX_HTTP5 in async mode. -- Roger Benningfield http://admin.mxblogspace.journurl.com/ ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238116 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
Neil - To prevent over-polling and, as Roger pointed out, potentially getting your IP blocked, consider Etag/If-None-Match headers as well as the Last-Modified/If-Modified-Since headers: 1. When you retrieve a feed, store the ETag and Last-Modified response headers 2. When you next poll the feed, only retrieve those feeds that have been updated cfhttp url=#variables.feedURL# method=GET useragent=feedsquirrel.com (or whatever) throwonerror=yes cfhttpparamtype=header name=If-None-Match value=#variables.storedEtagValue# / cfhttpparamtype=header name=If-Modified-Since value=#variables.storedLastModifiedValue# / /cfhttp A nice way to reduce bandwidth consumption and be respectful of the host server/feed author. A couple of additional suggestions: 1. Provide a user agent that allows a host server to know where the request is coming from and, if the feel it necessary, block that request. 2. Respect the feed authors TTL value (in the case of an RSS 2.0 feed). Don't update the feed any more often than requested in this value (if there is one). 3. Again, in the case of RSS 2.0 feeds, respect any skipDays and skipHours values. Don't poll on Sundays if the author has told you that the feed won't be updated on Sundays. I know there is a TTL equivalent in Atom 1.0/RSS 1.0, but honestly can't remember what it is. If you look at the specs, it should jump out. It's been a while since I wrote the feed aggregator that is embedded in the product I build. I don't recall there being a decent equivalent for RSS 1.0 or Atom 1.0 for skipDays and skipHours. On 4/19/06, Roger Benningfield [EMAIL PROTECTED] wrote: Currently the site is aggregating ~500 RSS feeds, but checking these feeds is growing to be a pain in the butt. Having to get CF to check each of these feeds regulary (ideally every 15 minutes) is more difficult than it sounds. Neil: Polling every fifteen minutes is an enormous waste of CPU and bandwidth... for both you and the source sites. For example, if you're aggregating individual blogs, once every 24 hours will cover the vast majority just fine. Ideally, you'd either opt for some middle ground (once an hour or so), or come up with adaptive code that spaces out polling based upon observed update periods. But even if you're gonna stick with over-polling (a good way to get your IP blocked), there are places to optimize: * Use Conditional GET... since 90% of feeds won't have seen an update in the last fifteen minutes, you've saved nearly 90% of your server's effort. * Make your spider compatible with RFC 3229. It won't help in most cases, but some high-flow publishers (Microsoft, etc.) will send you deltas of their sliding-window feeds. That'll cut down on parsing time. * Try CFX_HTTP5 in async mode. -- Roger Benningfield http://admin.mxblogspace.journurl.com/ ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238126 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
Thanks guys for this...it's been plugging some of the gaps I hadn't really considered. The site is very much in development still, even if it doesn't appear to be changing on the surface. I will definitely consider using the last modified headers. I didn't realise you could retreive them without pulling down the whole file. That in itself would alleviate things massively. One thing I have noticed with the CF community is that the RSS feeds that are published seem to be all over the place, some doing things one way, some doing it another. This doesn't help when trying to write a spider ;-) Does anyone have any possible insight into why the async gateways might by unreliable? Neil - To prevent over-polling and, as Roger pointed out, potentially getting your IP blocked, consider Etag/If-None-Match headers as well as the Last-Modified/If-Modified-Since headers: 1. When you retrieve a feed, store the ETag and Last-Modified response headers 2. When you next poll the feed, only retrieve those feeds that have been updated cfhttpurl=#variables.feedURL# method=GET useragent=feedsquirrel.com (or whatever) throwonerror=yes cfhttpparamtype=header name=If-None-Match value=#variables.storedEtagValue# / cfhttpparamtype=header name=If-Modified-Since value=#variables.storedLastModifiedValue# / /cfhttp A nice way to reduce bandwidth consumption and be respectful of the host server/feed author. A couple of additional suggestions: 1. Provide a user agent that allows a host server to know where the request is coming from and, if the feel it necessary, block that request. 2. Respect the feed authors TTL value (in the case of an RSS 2.0 feed). Don't update the feed any more often than requested in this value (if there is one). 3. Again, in the case of RSS 2.0 feeds, respect any skipDays and skipHours values. Don't poll on Sundays if the author has told you that the feed won't be updated on Sundays. I know there is a TTL equivalent in Atom 1.0/RSS 1.0, but honestly can't remember what it is. If you look at the specs, it should jump out. It's been a while since I wrote the feed aggregator that is embedded in the product I build. I don't recall there being a decent equivalent for RSS 1.0 or Atom 1.0 for skipDays and skipHours. On 4/19/06, Roger Benningfield [EMAIL PROTECTED] wrote: ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238130 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
I still can't help with the unreliable gateways, but a couple of things to note. First, the last modified header will not help you retrieve a partial feed. It will still retrieve the entire feed, but only if the feed has changed (better than nothing). You're right on your second point. The feeds that are .cfm URLs generate the feed on the fly (I assume) rather than generate a static file behind the scenes. That's why I use both the Etag and Last-Modified headers. Only static files appear to carry the ETag header, from what I've been able to tell. On 4/19/06, Neil Middleton [EMAIL PROTECTED] wrote: Thanks guys for this...it's been plugging some of the gaps I hadn't really considered. The site is very much in development still, even if it doesn't appear to be changing on the surface. I will definitely consider using the last modified headers. I didn't realise you could retreive them without pulling down the whole file. That in itself would alleviate things massively. One thing I have noticed with the CF community is that the RSS feeds that are published seem to be all over the place, some doing things one way, some doing it another. This doesn't help when trying to write a spider ;-) Does anyone have any possible insight into why the async gateways might by unreliable? Neil - To prevent over-polling and, as Roger pointed out, potentially getting your IP blocked, consider Etag/If-None-Match headers as well as the Last-Modified/If-Modified-Since headers: 1. When you retrieve a feed, store the ETag and Last-Modified response headers 2. When you next poll the feed, only retrieve those feeds that have been updated cfhttpurl=#variables.feedURL# method=GET useragent=feedsquirrel.com (or whatever) throwonerror=yes cfhttpparamtype=header name=If-None-Match value=#variables.storedEtagValue# / cfhttpparamtype=header name=If-Modified-Since value=#variables.storedLastModifiedValue# / /cfhttp A nice way to reduce bandwidth consumption and be respectful of the host server/feed author. A couple of additional suggestions: 1. Provide a user agent that allows a host server to know where the request is coming from and, if the feel it necessary, block that request. 2. Respect the feed authors TTL value (in the case of an RSS 2.0 feed). Don't update the feed any more often than requested in this value (if there is one). 3. Again, in the case of RSS 2.0 feeds, respect any skipDays and skipHours values. Don't poll on Sundays if the author has told you that the feed won't be updated on Sundays. I know there is a TTL equivalent in Atom 1.0/RSS 1.0, but honestly can't remember what it is. If you look at the specs, it should jump out. It's been a while since I wrote the feed aggregator that is embedded in the product I build. I don't recall there being a decent equivalent for RSS 1.0 or Atom 1.0 for skipDays and skipHours. On 4/19/06, Roger Benningfield [EMAIL PROTECTED] wrote: ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238134 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
RE: RSS Aggregation?
Neil: Polling every fifteen minutes is an enormous waste of CPU and bandwidth... for both you and the source sites. For example, if you're aggregating individual blogs, once every 24 hours will cover the vast majority just fine. I disagree. RSS was originally built as a solution to provide near real-time updates on a site. Fullasagoog polls every 15 minutes, not sure how often MXNA does it. RSS load is a common problem for website admins, but I don't ever hear people saying I need to stop my visitors from hitting my site so often, but rather I need to find a way to deal with the load. I know that was a problem for SlashDot when they first released RSS for their site. But again, they didn't try to limit the traffic but rather dealt with the load. If I were using an RSS reader (like the squirrel), and it didn't tell me about an update for 24 hours, I'd find a different reader. -- This transmission may contain information that is privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. A1. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238194 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
I'm going to chime in somewhere between Jacob and Roger. With the bandwidth saving features I mentioned earlier, I'd say 30 mins to an hour should be sufficient in almost every case. Of course, now we're just stating opinions, but that's mine. For what it's worth... On 4/19/06, Munson, Jacob [EMAIL PROTECTED] wrote: Neil: Polling every fifteen minutes is an enormous waste of CPU and bandwidth... for both you and the source sites. For example, if you're aggregating individual blogs, once every 24 hours will cover the vast majority just fine. I disagree. RSS was originally built as a solution to provide near real-time updates on a site. Fullasagoog polls every 15 minutes, not sure how often MXNA does it. RSS load is a common problem for website admins, but I don't ever hear people saying I need to stop my visitors from hitting my site so often, but rather I need to find a way to deal with the load. I know that was a problem for SlashDot when they first released RSS for their site. But again, they didn't try to limit the traffic but rather dealt with the load. If I were using an RSS reader (like the squirrel), and it didn't tell me about an update for 24 hours, I'd find a different reader. -- This transmission may contain information that is privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. A1. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238196 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
This is what I was thinking. Sure there may well be some optimisations that can be done for the process, but I agree that RSS is something that should be checked often. Which brings me back to my original problem... How do I go about getting this data checked, parsed and dumped into the db in a reliable way that doesn't kill the server? Neil Neil: Polling every fifteen minutes is an enormous waste of CPU and bandwidth... for both you and the source sites. For example, if you're aggregating individual blogs, once every 24 hours will cover the vast majority just fine. I disagree. RSS was originally built as a solution to provide near real-time updates on a site. Fullasagoog polls every 15 minutes, not sure how often MXNA does it. RSS load is a common problem for website admins, but I don't ever hear people saying I need to stop my visitors from hitting my site so often, but rather I need to find a way to deal with the load. I know that was a problem for SlashDot when they first released RSS for their site. But again, they didn't try to limit the traffic but rather dealt with the load. If I were using an RSS reader (like the squirrel), and it didn't tell me about an update for 24 hours, I'd find a different reader. -- This transmission may contain information that is privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. A1. ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238198 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
One thing I have noticed with the CF community is that the RSS feeds that are published seem to be all over the place, some doing things one way, some doing it another. Neil: Pete Freitag and I have both published tips for getting CF-based feeds to provide the correct headers and HTTP responses, but it's always gonna be an uphill battle. Most of us (self definitely included) have worked on the web for years without any significant understanding of how the machinery of HTTP is meant to be used, and there are tons of bad habits to unlearn. We'll get there eventually, though. :) -- Roger Benningfield http://admin.mxblogspace.journurl.com/ ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238218 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
RSS was originally built as a solution to provide near real-time updates on a site. Jacob: The blogosphere has mechanisms to handle real-time updates, and syndication feeds ain't one of 'em. Never has been. Fullasagoog polls every 15 minutes, not sure how often MXNA does it. If all they're doing is HEADing the resource, fine. If they're GETting it, then something's broken. RSS load is a common problem for website admins, but I don't ever hear people saying I need to stop my visitors from hitting my site so often... You're not paying attention to the syndication community, then. :D Complaints about overly aggressive aggregators are common. But again, they didn't try to limit the traffic but rather dealt with the load. Slashdot explicitly insists that feeds only be polled once every thirty minutes. And they've been known to block IPs. -- Roger Benningfield http://admin.mxblogspace.journurl.com/ ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238221 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
I know there is a TTL equivalent in Atom 1.0/RSS 1.0... Rob: Nope, there's no ttl equivalent in Atom 1.0. Someone brought up ttl/skipHours/skipDays in the IETF WG (or the pre-IETF group) at one point, and the consensus was that the elements are seldom used, not well understood when they *are* used, and largely unnecessary when HTTP is leveraged appropriately. RSS 1.0 is a different story. There are probably twenty different extension elements to control frequency out there. But like most RSS 1.0 extensions beyond content:encoded, there's not much support. -- Roger Benningfield http://admin.mxblogspace.journurl.com/ ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238227 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
Thanks for the clarification. I surely thought I remembered one from when I built my reader. Hardly the first time my memory has failed me. On 4/19/06, Roger Benningfield [EMAIL PROTECTED] wrote: I know there is a TTL equivalent in Atom 1.0/RSS 1.0... Rob: Nope, there's no ttl equivalent in Atom 1.0. Someone brought up ttl/skipHours/skipDays in the IETF WG (or the pre-IETF group) at one point, and the consensus was that the elements are seldom used, not well understood when they *are* used, and largely unnecessary when HTTP is leveraged appropriately. RSS 1.0 is a different story. There are probably twenty different extension elements to control frequency out there. But like most RSS 1.0 extensions beyond content:encoded, there's not much support. -- Roger Benningfield http://admin.mxblogspace.journurl.com/ ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238231 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=11502.10531.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
Re: RSS Aggregation?
I surely thought I remembered one from when I built my reader. Rob: Could it have been one of the interim specs you were looking at? 'Cause there was all kinds of odd stuff in there at certain points... particularly in the pre-IETF drafts. In addition, there was (and is) a lotta stuff on the wiki that is foreign to the spec as well, which can be pretty confusing to implementors. -- Roger ~| Message: http://www.houseoffusion.com/lists.cfm/link=i:4:238246 Archives: http://www.houseoffusion.com/cf_lists/threads.cfm/4 Subscription: http://www.houseoffusion.com/lists.cfm/link=s:4 Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4 Donations Support: http://www.houseoffusion.com/tiny.cfm/54
