Gregor Streng wrote: > Hi There, > > we are developping a search engine, which should index actually over 400 > sites. > The server-table is generated automatically and my problem is, > that I don't want to index all pages within an site for 2 resons: > > 1. Some sites are just very large > 2. Some sites use session ids which change, so the indexer reindexes some > pages several times. > > How can I reduce the multiple indexing? > How can I limit indexing by pages per Server? > > Thanks for your help, > > Gregor
Hello! You can remove session id's using ReverseAlias command. This feature appeared in 3.2.0. There is no possibility to limit pages number from the same site. However you can limit either a way in "mouse clicks" using MaxHops or limit indexing depth using Allow/Disallow commands combination. For example this disallows pages on 3d level and deeper: Disallow http://.*/.*/.*/.* ___________________________________________ If you want to unsubscribe send "unsubscribe general" to [EMAIL PROTECTED]
