> On Mar 28, 2019, at 2:10 PM, Konstantin Tokarev <annu...@yandex.ru> wrote: > > > > 28.03.2019, 23:58, "Alexey Proskuryakov" <a...@webkit.org > <mailto:a...@webkit.org>>: >> Hello, >> >> The robots.txt file that we have on bugs.webkit.org >> <http://bugs.webkit.org/> currently allows search engines access to >> individual bug pages, but not to any bug lists. As a result, search engines >> and the Internet Archive only index bugs that were filed before robots.txt >> changes a few years ago, and bugs that are directly linked from webpages >> elsewhere. These bugs are where most spam content naturally ends up on. >> >> This is quite wrong, as indexing just a subset of bugs is not beneficial to >> anyone other than spammers. So we can go in either direction: >> >> 1. Allow indexers to enumerate bugs, thus indexing all of them. >> >> Seems reasonable that people should be able to find bugs using search >> engines. > > Yes, and it may give better result even than searching bugzilla directly > >> On the other hand, we'll need to do something to ensure that indexers don't >> destroy Bugzilla performance, > > This can be solved by caching > >> and of course spammers will love having more flexibility. > > rel="nofollow" on all links in comments should be enough to make spamming > useless
Theoretically yes… but a couple google searches say it doesn’t make a difference. Here is one of many https://www.seroundtable.com/google-nofollow-link-attribute-failed-comments-26959.html <https://www.seroundtable.com/google-nofollow-link-attribute-failed-comments-26959.html> I expect that spammers don’t reply care if they get a nofollow or not, they are mostly un-manned scripts anyway. I’m not opposed to adding this, I just don’t expect it will solve the problem. We could measure and see. Lucas > >> >> 2. Block indexing completely. >> >> Seems like no one was bothered by lack of indexing on new bugs so far. > > That's survival bias - if nobody can find relevant bugs, nobody will ever > complain > >> >> Thoughts? >> >> For reference, here is the current robots.txt content: >> >> $ curl https://bugs.webkit.org/robots.txt >> User-agent: * >> Allow: /index.cgi >> Allow: /show_bug.cgi >> Disallow: / >> Crawl-delay: 20 >> >> - Alexey >> - Alexey >> >> _______________________________________________ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> https://lists.webkit.org/mailman/listinfo/webkit-dev > > -- > Regards, > Konstantin > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org <mailto:webkit-dev@lists.webkit.org> > https://lists.webkit.org/mailman/listinfo/webkit-dev > <https://lists.webkit.org/mailman/listinfo/webkit-dev>
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
