I'm not trying to argue legalities, just pointing out that there's an
undercurrent out there in the community where there's some backlash
against SE's and crawlers because of the cache. Here's an example; this
guy: http://incredibill.blogspot.com/ is scraper/bot/crawler crazy.
And he actively blocks nutch. *and* that blog is widely read.
(actually, I think what he does is serve some nonsense phrase that gets
indexed. That lets him search in your SE for his nonsense phrase).
It's a good idea to keep the content providers happy. If we don't, more
of them can block our crawlers for those engines they feel don't provide
value to them. And that's bad.
I'd be curious if anyone has any good reasons for actually showing the
'cache'. I personally don't see any real use for it, other than for
someone's competitors using it to check to see if they're cloaking.
g.
Nutch Newbie wrote:
Hmmm.. How about this... The photographer who take a photo has the
copyright over the photo not the owner of the picture motive, you, me
or any other photo object. So caching is nothing but taking a picture
using another sort of camera called robot :-) Nothing more really. If
a browser maker decides to show an HTML tag lets say <H1> in 300
pixels will that be a copyright or trademark violation then?
What one can do is to prevent one to be photographed or stop the
robots visit one's website :-)
On 3/30/06, Insurance Squared Inc. <[EMAIL PROTECTED]> wrote:
FWIW, I believe all of what's been stated is the case - and I'd also
assume that since Google/MSN/Yahoo are all doing this that it's been
tested and OK.
However I know many people complain about the cache. Some people see it
as a copyright violation - technically correct or not, the cache does
basically duplicate their site and make it available online. And I've
never seen how to argue against that other than 'legally it's not'. IMO
it's cutting it pretty close.
The other issue some have with displaying cache is that it allows people
to pull down websites without ever visiting the website in questions.
If I put serious effort into blocking bots and scrapers for example, but
let the SE's in so I can get indexed, then the bots and scrapers can
completely bypass my efforts, visit the SE and pull down the cached
pages there. They can then do nasty stuff with my content, like copy it
on their site for their own purposes. Not good, and that's the reason
why I don't show the cache on my SE.
g.
Dan Morrill wrote:
If I remember it correctly, google as been sued and won a number of times on
this issue, you can cache, you can search others web sites, grocklaw has the
data on this one, but I know you can search, you can cache under fair use,
and the idea of public access, as long as you are not cracking passwords,
and honor robots.txt and they post it on the web, it is considered public in
that regard.
I am not a lawyer, check grocklaw.
r/d
-----Original Message-----
From: TDLN [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 30, 2006 3:34 AM
To: nutch-user@lucene.apache.org
Subject: Re: Legal issues
Google's and Yahoo's Terms of Service provide interesting reading regarding
such legal issues.
http://www.google.com/terms_of_service.html
http://docs.yahoo.com/info/terms/
Rgrds, Thomas
On 3/30/06, gekkokid <[EMAIL PROTECTED]> wrote:
Shouldn't be a problem if your honouring the robots.txt
Legal issues could be Stealing Copyrighted Material? thats if your
reproducing it but if your analysing the content and links and keeping to
the robots.txt rules I doubt your have a problem unless its crawling every
10 minutes,
wouldn't grabbing the RSS feed be better?
would http://diggdot.us be a good example of what your trying to do? or
have
i got the wrong idea entirely?
Any one else have any thoughts?
_gk
----- Original Message -----
From: "Berlin Brown" <[EMAIL PROTECTED]>
To: <nutch-user@lucene.apache.org>
Sent: Thursday, March 30, 2006 8:13 AM
Subject: Legal issues
What are say the legal issues of crawling a site like reddit, digg or
slashdot. Assuming that you are just collecting links that users post
through that service and then you are regathering those links. I
can't see an issue there.
The other extreme would be crawling google and requerying or something
along those lines.
