[Gluster-infra] [Bug 1367588] Improve the redirection for specific URL for RTD coming from old website

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1367588

Nigel Babu  changed:

   What|Removed |Added

 Status|NEW |CLOSED
 Resolution|--- |WONTFIX
Last Closed||2018-08-13 00:19:17



--- Comment #4 from Nigel Babu  ---
I'd like to close this bug as WONT FIX.

We should identify gaps in our current docs and file issues to fix them against
glusterdocs.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=mkParRj2lT=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1367588] Improve the redirection for specific URL for RTD coming from old website

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1367588



--- Comment #3 from M. Scherer  ---
So I did a quick verification on the whole set of logs, and we have since the
26 July around 22 000 hits. 

# grep /community/documentation/index.php www.gluster.org-access_log* |wc -l
22708

Around 90% of the traffic is bots:
# grep /community/documentation/index.php www.gluster.org-access_log*  |grep -v
g2reader-bot/ | grep -v Slurp\; |grep -vi bingbot |grep -vi googlebot |grep -v
Baiduspider/ |grep -v AhrefsBot/  |grep -v MJ12bot/ | grep -v 'Sogou web' |grep
-v SeznamBot/ |grep -v electricmonk/ | grep -v 'HaosouSpider;' |grep -v
archive.org_bot  |grep -v Feedly/1.0 |grep -v SputnikBot/ | grep -v yoozBot |wc
-l
2598

I suspect on top of that that there is lots of refresh and duplicate ips

# grep /community/documentation/index.php www.gluster.org-access_log*  |grep -v
g2reader-bot/ | grep -v Slurp\; |grep -vi bingbot |grep -vi googlebot |grep -v
Baiduspider/ |grep -v AhrefsBot/  |grep -v MJ12bot/ | grep -v 'Sogou web' |grep
-v SeznamBot/ |grep -v electricmonk/ | grep -v 'HaosouSpider;' |grep -v
archive.org_bot  |grep -v Feedly/1.0 |grep -v SputnikBot/ | grep -v yoozBot 
|awk '{print $1}' |awk -F: '{print $2}'  |sort -u |wc -l
649

Then trying to group by network just show around 600 hits. That's roughly 2 to
3 visitors per day on the wiki. 

After removing the various hacking attempt (aimed at joomla), the hit on the
redirect page itself, the tentative to login for spam, and favicon, we are down
to 1500 hits (without deduplication):

# grep /community/documentation/index.php www.gluster.org-access_log*  |grep -v
g2reader-bot/ | grep -v Slurp\; |grep -vi bingbot |grep -vi googlebot |grep -v
Baiduspider/ |grep -v AhrefsBot/  |grep -v MJ12bot/ | grep -v 'Sogou web' |grep
-v SeznamBot/ |grep -v electricmonk/ | grep -v 'HaosouSpider;' |grep -v
archive.org_bot  |grep -v Feedly/1.0 |grep -v SputnikBot/ | grep -v yoozBot
|grep -v docs-redirect   |awk '{print $7}' |grep -v 'Special:UserLogin' |grep
-v '=history'  |grep -v '%22%20h=/' |grep -v /favicon.ico |wc -l
1524


Then the 30 most popular URLs are:

[root@supercolony httpd]# grep /community/documentation/index.php
www.gluster.org-access_log*  |grep -v g2reader-bot/ | grep -v Slurp\; |grep -vi
bingbot |grep -vi googlebot |grep -v Baiduspider/ |grep -v AhrefsBot/  |grep -v
MJ12bot/ | grep -v 'Sogou web' |grep -v SeznamBot/ |grep -v electricmonk/ |
grep -v 'HaosouSpider;' |grep -v archive.org_bot  |grep -v Feedly/1.0 |grep -v
SputnikBot/ | grep -v yoozBot |grep -v docs-redirect   |awk '{print $7}' |grep
-v 'Special:UserLogin' |grep -v '=history'  |grep -v '%22%20h=/' |grep
-v /favicon.ico |sort |uniq -c  |sort -rn | head -n 30
206
/community/documentation/index.php/Gluster_3.1:_Manually_Mounting_Volumes
143 /community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options
 87 /community/documentation/index.php/QuickStart
 69
/community/documentation/index.php/Gluster_3.2:_Starting_Gluster_Geo-replication
 52 /community/documentation/index.php/Gluster_3.2:_gluster_Command
 43 /community/documentation/index.php/Main_Page
 37 /community/documentation/index.php/Translators/storage/bdb
 37
/community/documentation/index.php/Gluster_3.2:_Monitoring_your_GlusterFS_Workload
 36 /community/documentation/index.php/Gluster_3.2:_Terminology
 35
/community/documentation/index.php/Gluster_3.2:_Displaying_Volume_Information
 29 /community/documentation/index.php/Gluster_3.2:_Expanding_Volumes
 24
/community/documentation/index.php/Gluster_3.2:_Manually_Mounting_Volumes
 22 /community/documentation/index.php/GlusterFS_Concepts
 21
/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Striped_Volumes
 16 /community/documentation/index.php/User_Guide
 16 /community/documentation/index.php/Gluster_3.2:_Tuning_Volume_Options
 16 /community/documentation/index.php/Getting_started_overview
 15
/community/documentation/index.php/Gluster_3.4:_Brick_Restoration_-_Replace_Crashed_Server
 14
/community/documentation/index.php/Gluster_3.1:_Understanding_the_GlusterFS_License
 12 /community/documentation/index.php/Translators/performance
 12 /community/documentation/index.php/Gluster_Translators
 12 /community/documentation/index.php/GlusterHPC_FAQ
 12
/community/documentation/index.php/Gluster_3.2:_Manually_Mounting_Volumes_Using_NFS
 12 /community/documentation/index.php/Getting_started_test_it_out
 10 /community/documentation/index.php/About_GlusterFS_3.3
  9
/community/documentation/index.php/Gluster_3.2:_Installing_GlusterFS_on_Red_Hat_Package_Manager_(RPM)_Distributions
  9
/community/documentation/index.php/Gluster_3.2:_GlusterFS_Geo-replication_Deployment_Overview
  9 /community/documentation/index.php/Documenting_the_undocumented
  8 /community/documentation/index.php/MediaWiki:Userlogin
  8

[Gluster-infra] [Bug 1367588] Improve the redirection for specific URL for RTD coming from old website

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1367588



--- Comment #2 from M. Scherer  ---
So, looking in more details, for the url
Gluster_3.2:_Configuring_Distributed_Striped_Volumes , there is more bots I
didn't filtered, and the same ip downloading the page 10 times.

The same goes for Gluster_3.1:_Manually_Mounting_Volumes, 27 hits from the same
ip in Island, and bots. And ip from the same country ( 2 times ), and 2 indians
hits.

I suspect that we would need more data to see what should be mapped, and/or
make a editorial choice based on existing stuff.


Alternatively, someone can decide to revert the complete change and redirection
for the time being, but that trading one set of issue for another one.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=yR7cor2ChD=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-infra

[Gluster-infra] [Bug 1367588] Improve the redirection for specific URL for RTD coming from old website

[bugzilla]

https://bugzilla.redhat.com/show_bug.cgi?id=1367588

M. Scherer  changed:

   What|Removed |Added

Summary|Setup redirection from  |Improve the redirection for
   |gluster.org/community/docum |specific URL for RTD coming
   |entation to the new RTD |from old website
   |site|



-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug 
https://bugzilla.redhat.com/token.cgi?t=8XLT5uCLZ8=cc_unsubscribe
___
Gluster-infra mailing list
Gluster-infra@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-infra