On Wednesday, January 06, 2016 10:30:52 AM Ben Cooksley wrote: > On Wed, Jan 6, 2016 at 3:17 AM, Kevin Funk <kf...@kde.org> wrote: > > On Wednesday, December 30, 2015 12:57:23 PM Ben Cooksley wrote: > >> On Tue, Dec 29, 2015 at 11:16 PM, Kevin Funk <kf...@kde.org> wrote: > >> > On Tuesday, December 29, 2015 10:39:01 PM Ben Cooksley wrote: > >> >> On Tue, Dec 29, 2015 at 7:59 AM, Lydia Pintscher <ly...@kde.org> wrote: > >> >> > On Sun, Dec 27, 2015 at 12:35 PM, Ben Cooksley <bcooks...@kde.org> > > > > wrote: > >> >> >>> Is there some place where search engines can easily index our > >> >> >>> source > >> >> >>> code or are we shooting ourselves in the foot here? > >> >> >> > >> >> >> We could probably make it available by publishing the source trees > >> >> >> used by LXR / EBN. > >> >> >> This would only have the main branches obviously rather than > >> >> >> everything > >> >> >> though. > >> >> >> > >> >> >> I haven't checked, but LXR may already make it's copy of the code > >> >> >> accessible...> > >> >> > > >> >> > I think making our sourcecode available to search engines is pretty > >> >> > important for the reasons already mentioned by others. Do you need > >> >> > help for it? If you write down what's needed I can help find someone > >> >> > to do it. > >> >> > >> >> I've now provisioned https://sources.kde.org/ > >> > > >> > I'm not sure this is super useful, to be honest (as mentioned in #kde- > >> > sysadmins already). > >> > > >> > This is really just plain file serving, with no cross-references to > >> > either > >> > LXR (or apidocs). This is basically a dead-end when you follow a result > >> > on Google. > >> > > >> > Wouldn't it be possible to let robots index https://lxr.kde.org/source/ > >> > > >> > instead? We have the infrastructure... > >> > >> We'll give it a shot. > > > > Just to stress again this would be *really* useful to have. > > ????
Ah, I see robots.txt on lxr.kde.org allows crawling source/ now. Thanks! > > I answered a post on SO: > > http://stackoverflow.com/a/34612692/592636 > > > > Tried to link kwallet's FindGpgpme.cmake into the answer; and there's *no* > > easy way quickly get a link to KDE infrastructure serving the file via > > Google (not even api.kde.org). > > > > Try googling for "kwallet findgpgme.cmake" (very specific search after all): > > https://www.google.de/search?q=kwallet+findgpgme.cmake > > > > -> First result: Github..., rest: mildly interesting > > > > > > Different issue I just noticed: There's no way to get the plain-text (raw) > > representation of a given file on LXR, is there? Would be useful as well. > > There isn't a link in our templates, but my Google fu (and subsequent > tests confirm) that adding the parameter "_raw=1" to a LXR source view > URL will return the file without any HTML around it. > > > Cheers, > > Kevin > > Regards, > Ben > > >> > Of course we need to blacklist all the pages allowing to actively > >> > *search* > >> > LXR for robots, in order to avoid abuse. > >> > >> Note that despite robots.txt, many spiders (including Google, Yahoo > >> and Bing) will actively disregard the instructions in there. > >> While they may not return the results - or omit snippets of the page > >> content - they have all been guilty (at least in the past) of > >> disregarding our restrictions, resulting in downtime (which have in > >> some cases necessitated full host reboots to fix) for numerous KDE.org > >> subsites in the past. > >> > >> This is why QuickGit and WebSVN have extremely restrictive robots.txt > >> policies, in addition to blacklist rules within our web server > >> configurations. > >> > >> > Cheers, > >> > Kevin > >> > >> Regards, > >> Ben > >> > >> >> > Cheers > >> >> > Lydia > >> >> > >> >> Regards, > >> >> Ben > >> >> > >> >> > -- > >> >> > Lydia Pintscher - http://about.me/lydia.pintscher > >> >> > KDE e.V. Board of Directors / KDE Community Working Group > >> >> > http://kde.org - http://open-advice.org > >> >> > > >> >> >>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to > >> >> >>> unsubscribe <<>> > >> >> >> > >> >> >> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to > >> >> >> unsubscribe > >> >> >> << > >> > > >> > -- > >> > Kevin Funk | kf...@kde.org | http://kfunk.org > > > > -- > > Kevin Funk | kf...@kde.org | http://kfunk.org -- Kevin Funk | kf...@kde.org | http://kfunk.org
signature.asc
Description: This is a digitally signed message part.
>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<
