Well here's the deal .... I use the following on every HTML page:
<HTML>
<BODY>
<SERVLET CODE="MyServlet">
</SERVLET>
....
</BODY>
</HTML>
This is the servlet that gives out IDs (it yields no output). So it is not
being called directly by anything. It simply gets executed when the page is
requested. So robots.txt is not going to help, is it?
Any ideas on how to resolve this?
-----Original Message-----
From: Simon Christian [mailto:[EMAIL PROTECTED]]
Sent: Monday, October 11, 1999 12:42 PM
To: [EMAIL PROTECTED]
Subject: Re: Real People or Search Engines?
No and yes, depending on what you specify. In general indexing a servlet
isn't
likely to be useful (unless it's purely a filtering servlet). So simply
disallow robots from your servlet paths e.g.
User-agent: *
Disallow: /servlets
in your /robots.txt file
If your site is all servlets then that might not be such a good idea.
- simon
"Boemio, Neil (CAP, FGI)" wrote:
> If I do this, will this mean my site won't be indexed?
>
> -----Original Message-----
> From: A.W.F. Boer [mailto:[EMAIL PROTECTED]]
> Sent: Monday, October 11, 1999 11:21 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Real People or Search Engines?
>
> If you do not want to give out IDs to robots, tell the robots not to visit
> your servlet in your /robots.txt file. See the Web Robots FAQ at:
>
> http://info.webcrawler.com/mak/projects/robots/faq.html
>
> I does work for most well-behaved search engine robots.
>
> Alexander W. F. Boer
> Dept. of Computer Science and Law,
> University of Amsterdam, the Netherlands
> email: [EMAIL PROTECTED]
>
> ----------
> Van: Lance Lavandowska[SMTP:[EMAIL PROTECTED]]
> Antwoord naar: Lance Lavandowska
> Verzonden: Monday, October 11, 1999 3:43 PM
> Aan: [EMAIL PROTECTED]
> Onderwerp: Re: Real People or Search Engines?
>
> Search engines often/usually/sometimes (cover my bases) have a different
> UserAgent header than the standard browsers. You could set up a
> "pass-through" servlet to log the various UserAgents that are hitting your
> site, and continue from there.
>
> Lance Lavandowska
>
> -----Original Message-----
> From: Boemio, Neil (CAP, FGI) <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
> Date: Sunday, October 10, 1999 11:30 PM
> Subject: Real People or Search Engines?
>
> >Is there anyway to tell if my servlet is being called by a real person
> >hitting my site or by search engines or robots or whatever.
> >
> >I ask because my servlet gives out an ID in a cookie if the user doesn't
> >have one already. And it seems that I am giving out tons of IDs now!
> How
> >can I give out IDs to "real people" ONLY?
> >
> >
___________________________________________________________________________
To unsubscribe, send email to [EMAIL PROTECTED] and include in the body
of the message "signoff SERVLET-INTEREST".
Archives: http://archives.java.sun.com/archives/servlet-interest.html
Resources: http://java.sun.com/products/servlet/external-resources.html
LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
___________________________________________________________________________
To unsubscribe, send email to [EMAIL PROTECTED] and include in the body
of the message "signoff SERVLET-INTEREST".
Archives: http://archives.java.sun.com/archives/servlet-interest.html
Resources: http://java.sun.com/products/servlet/external-resources.html
LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
