Bob Sprenger wrote:
>
> UdmSearch version: 3.1.18/MYSQL
> Platform: Pentium II/400
> OS: suse linux
> Database: MYSQL
> Statistics: Total 0 6074
>
> When the indexer reads the robots.txt only the first path is in the database.....
> Sampel from robots.txt:
> Disallow: /images/ /karte/ /manual/ /neomail/images/
>
> In the database is only the /images/ the other paths are not stored.
Check this document:
http://www.robotstxt.org/wc/exclusion-admin.html
<...>
The "/robots.txt" file usually contains a record looking like this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
In this example, three directories are excluded.
Note that you need a separate "Disallow" line for every URL prefix you
want to exclude -- you cannot say "Disallow: /cgi-bin/ /tmp/".
<...>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
