robots.txt in quickgit.kde.org

[Ashish Bansal]

Hi everyone,

"quickgit.kde.org" contains robots.txt[0] which is disallowing search
engines to fetch the project repos. I just wanted to know if this is
intentional or not?

If I recall correctly, mirror of kde repositories on github was created
just because it wasn't being indexed by the search engines.

[0] https://quickgit.kde.org/robots.txt

-- 

*Regards,*
*Ashish Bansal*
*http://ashish-bansal.in *

>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<

Re: robots.txt in quickgit.kde.org

[Ben Cooksley]

On Sun, Dec 27, 2015 at 11:53 PM, Ashish Bansal
 wrote:
> Hi everyone,

Hi Ashish,

>
> "quickgit.kde.org" contains robots.txt[0] which is disallowing search
> engines to fetch the project repos. I just wanted to know if this is
> intentional or not?
>
> If I recall correctly, mirror of kde repositories on github was created just
> because it wasn't being indexed by the search engines.

This is intentional, and is done to reduce the server load created by
indexers such as Google on the system hosting quickgit.kde.org.
(Generation of the pages, including the main index is substantially
more expensive than it appears due to the disk access required by
Git/SVN to return the needed information).

>
> [0] https://quickgit.kde.org/robots.txt
>
> --
>
> Regards,
> Ashish Bansal
> http://ashish-bansal.in

Regards,
Ben Cooksley
KDE Sysadmin

>
>
>>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe
>>> <<
>

>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<

Re: robots.txt in quickgit.kde.org

[Lydia Pintscher]

On Sun, Dec 27, 2015 at 12:08 PM, Ben Cooksley  wrote:
> On Sun, Dec 27, 2015 at 11:53 PM, Ashish Bansal
>  wrote:
>> Hi everyone,
>
> Hi Ashish,
>
>>
>> "quickgit.kde.org" contains robots.txt[0] which is disallowing search
>> engines to fetch the project repos. I just wanted to know if this is
>> intentional or not?
>>
>> If I recall correctly, mirror of kde repositories on github was created just
>> because it wasn't being indexed by the search engines.
>
> This is intentional, and is done to reduce the server load created by
> indexers such as Google on the system hosting quickgit.kde.org.
> (Generation of the pages, including the main index is substantially
> more expensive than it appears due to the disk access required by
> Git/SVN to return the needed information).

Is there some place where search engines can easily index our source
code or are we shooting ourselves in the foot here?


Cheers
Lydia

-- 
Lydia Pintscher - http://about.me/lydia.pintscher
KDE e.V. Board of Directors / KDE Community Working Group
http://kde.org - http://open-advice.org

>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<

Re: robots.txt in quickgit.kde.org

[Ben Cooksley]

On Mon, Dec 28, 2015 at 12:15 AM, Lydia Pintscher  wrote:
> On Sun, Dec 27, 2015 at 12:08 PM, Ben Cooksley  wrote:
>> On Sun, Dec 27, 2015 at 11:53 PM, Ashish Bansal
>>  wrote:
>>> Hi everyone,
>>
>> Hi Ashish,
>>
>>>
>>> "quickgit.kde.org" contains robots.txt[0] which is disallowing search
>>> engines to fetch the project repos. I just wanted to know if this is
>>> intentional or not?
>>>
>>> If I recall correctly, mirror of kde repositories on github was created just
>>> because it wasn't being indexed by the search engines.
>>
>> This is intentional, and is done to reduce the server load created by
>> indexers such as Google on the system hosting quickgit.kde.org.
>> (Generation of the pages, including the main index is substantially
>> more expensive than it appears due to the disk access required by
>> Git/SVN to return the needed information).
>
> Is there some place where search engines can easily index our source
> code or are we shooting ourselves in the foot here?

We could probably make it available by publishing the source trees
used by LXR / EBN.
This would only have the main branches obviously rather than everything though.

I haven't checked, but LXR may already make it's copy of the code accessible...

>
>
> Cheers
> Lydia

Regards,
Ben

>
> --
> Lydia Pintscher - http://about.me/lydia.pintscher
> KDE e.V. Board of Directors / KDE Community Working Group
> http://kde.org - http://open-advice.org
>
>>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<

>> Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<

Re: robots.txt in quickgit.kde.org

[Thomas Lübking]

On Sonntag, 27. Dezember 2015 12:35:51 CEST, Ben Cooksley wrote:


We could probably make it available by publishing the source trees
used by LXR / EBN.



Because if it's not in google, it doesn't exist?

We've lxr which is a dedicated and *far* superior way to search our code, so what exactly 
is the purpose of finding "m_fooBar = new KFoo::Bar()" via google? (let alone 
bing ;-P )


Cheers, sorry if I sound stupid.
Thomas


Visit http://mail.kde.org/mailman/listinfo/kde-devel#unsub to unsubscribe <<