look here, it is blocking robots: http://ulysses.wyona.org/robots.txt
User-agent: * Disallow: /foo/bar.html User-agent: lenya Disallow: /foo/bar.html Michael Wechner wrote: > Hi > > I am trying to index http://ulysses.wyona.org/ but somehow it just > indexes the homepage but doesn't seem to follow > any links. I have set "depth 3" and other sites are being crawled > deeper without a problem but not the Ulysses page. > > Has anyone made similar experiences? > > Is it possible that Nutch has problem with well-formed XHTML > (application/xhtml+xml)? > > Thanks > > Michi > ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
