Roger Benningfield wrote:
However, if I put something like:
User-agent: PubSub
Disallow: /
...in my robots.txt and you ignore it, then you very much
belong on the Bad List.
I don't think so. The reason is that I believe that robots.txt has
nothing to do with any service I provide or
On 26/8/05 3:55 PM, Bob Wyman [EMAIL PROTECTED] wrote:
Remember, PubSub never does
anything that a desktop client doesn't do.
Periodic re-fetching is a robotic behaviour, common to both desktop
aggregators and server based aggregators. Robots.txt was established to
minimise harm caused by
On Friday, August 26, 2005, at 04:39 AM, Eric Scheid wrote:
On 26/8/05 3:55 PM, Bob Wyman [EMAIL PROTECTED] wrote:
Remember, PubSub never does
anything that a desktop client doesn't do.
Periodic re-fetching is a robotic behaviour, common to both desktop
aggregators and server based
* Bob Wyman [EMAIL PROTECTED] [2005-08-26 01:00]:
My impression has always been that robots.txt was intended to
stop robots that crawl a site (i.e. they read one page, extract
the URLs from it and then read those pages). I don't believe
robots.txt is intended to stop processes that simply
There are no wildcards in /robots.txt, only path prefixes and user-agent
names. There is one special user-agent, *, which means all.
I can't think of any good reason to always ignore the disallows for *.
I guess it is OK to implement the parts of a spec that you want.
Just don't answer yes when
Antone Roundy wrote:
I'm with Bob on this. If a person publishes a feed without limiting
access to it, they either don't know what they're doing, or they're
EXPECTING it to be polled on a regular basis. As long as PubSub
doesn't poll too fast, the publisher is getting exactly what they
Ok, so this discussion has definitely been interesting... let's see if
we can turn it into something actionable.
1. Desktop aggregators and services like pubsub really do not fall into
the same category as robots/crawlers and therefore should not
necessarily be paying attention to
Hi,
I am developing a web app framework and am using a RSSesque xml format to allow
the application layer talk to a higher layer. XSL is used to combine this and
then passes it on up to the gui layer.
I would like to move to Atom instead of RSS since Atom seems to encapsulate more
things. I
* [EMAIL PROTECTED] [EMAIL PROTECTED] [2005-08-26 19:20]:
Is there an attribute to the LINK tag I can use as some sort of id that i can
assign to the feed tag and link to it via the link tag?
My first thought would be to use xml:id on the feed element and a
fragment identifier in the @href.
On 8/25/05, Roger B. [EMAIL PROTECTED] wrote:
Mhh. I have not looked into this. But is not every desktop aggregator
a robot?
Henry: Depends on who you ask. (See the Newsmonster debates from a
couple years ago.)
As I am the one who kicked off the Newsmonster debates a couple years
ago, I
--On August 26, 2005 9:51:10 AM -0700 James M Snell [EMAIL PROTECTED] wrote:
Add a new link rel=readers whose href points to a robots.txt-like file that
either allows or disallows the aggregator for specific URI's and establishes
polling rate preferences
User-agent: {aggregator-ua}
Graham wrote:
(And before you say but my aggregator is nothing but a podcast
client, and the feeds are nothing but links to enclosures, so it's
obvious that the publisher wanted me to download them -- WRONG! The
publisher might want that, or they might not ...
So you're saying browsers
Le 05-08-25 à 18:51, Bob Wyman a écrit :
At PubSub we *never* crawl to discover feed URLs. The only feeds
we know about are:
1. Feeds that have announced their presence with a ping
2. Feeds that have been announced to us via a FeedMesh message.
3. Feeds that have been manually
On 26 Aug 2005, at 7:46 pm, Mark Pilgrim wrote:
2. If a user gives a feed URL to a program *and then the program finds
all the URLs in that feed and requests them too*, the program needs to
support robots.txt exclusions for all the URLs other than the original
URL it was given.
...
(And
Mark Pilgrim wrote (among other things):
(And before you say but my aggregator is nothing but a podcast
client, and the feeds are nothing but links to enclosures, so
it's obvious that the publisher wanted me to download them -- WRONG!
I agree with just about everything that Mark wrote
Remember, PubSub never does
anything that a desktop client doesn't do.
Bob: What about FeedMesh? If I ping blo.gs, they pass that ping along
to you, and PubSub fetches my feed, then PubSub is doing something a
desktop client doesn't do. It's following a link found in one place
and
* Bob Wyman [EMAIL PROTECTED] [2005-08-26 22:50]:
It strikes me that not all URIs are created equally and not
everything that looks like crawling is really crawling.
@xlink:type?
Regards,
--
Aristotle Pagaltzis // http://plasmasturm.org/
Roger Benningfield wrote:
We've got a mechanism that allows any user with his own domain
and a text editor to tell us whether or not he wants us messing with
his stuff. I think it's foolish to ignore that.
The problem is that we have *many* such mechanisms. Robots.txt is
only one.
Karl Dubost wrote:
- How one who has previously submitted a feed URL remove it from
the index? (Change of opinions)
If you are the publisher of a feed and you don't want us to monitor
your content, complain to us and we'll filter you out. Folk do this every
once in a while. Send us an
Le 05-08-26 à 17:53, Bob Wyman a écrit :
Karl Dubost wrote:
- How one who has previously submitted a feed URL remove it from
the index? (Change of opinions)
If you are the publisher of a feed and you don't want us to
monitor
your content, complain to us and we'll filter you out. Folk
Karl Dubost points out that it is hard to figure out what email address to
send messages to if you want to de-list from PubSub...:
Karl, Please, accept my apologies for this. I could have sworn we
had the policy prominently displayed on the site. I know we used to have it
there. This must
On 27/8/05 6:40 AM, Bob Wyman [EMAIL PROTECTED] wrote:
I think crawling URI's found in link/ tags,
img/ tags and enclosures isn't crawling... Or... Is there something I'm
missing here?
crawling img tags isn't a huge problem because it doesn't lead to a
recursive situation. Same withh
I'm adding robots@mccmedia.com to this dicussion. That is the classic
list for robots.txt discussion.
Robots list: this is a discussion about the interactions of /robots.txt
and clients or robots that fetch RSS feeds. Atom is a new format in
the RSS family.
--On August 26, 2005 8:39:59 PM +1000
23 matches
Mail list logo
