JAllemandou added a comment.
Some more info on this aspect: I have done a quick analysis over September
queries today and found that my assumption that long queries were made by users
from UI is wrong.
First, total numbers of request and sum of query-time split by queries taking
more than 1s or less:
+-------+---------+-----------+
|more_1s|requests |query_time |
+-------+---------+-----------+
|false |160185762|11285161245|
|true |2757758 |22233005459|
+-------+---------+-----------+
The proportions of number of queries per time classes are the same whether a
referer is present (expected UI) or not (expected bot).
+-----------+----------------+---------+-----------+
|has_referer|query_time_class|count |query_time |
+-----------+----------------+---------+-----------+
|false |1_less_10ms |8613461 |43244699 |
|false |2_10ms_to_100ms |118036102|3382186064 |
|false |3_100ms_to_1s |28377288 |7058252741 |
|false |4_1s_to_10s |1815394 |6081683264 |
|false |5_more_10s |591957 |14313410554|
|true |1_less_10ms |24329 |133314 |
|true |2_10ms_to_100ms |3123534 |140796917 |
|true |3_100ms_to_1s |2011048 |660547510 |
|true |4_1s_to_10s |310037 |800937814 |
|true |5_more_10s |40370 |1036973827 |
+-----------+----------------+---------+-----------+
Below are some information on the top-100 user-agents/referer making most
requests with duration greatest than 1s:
+---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------+----------------+------------------+----------------+------------------+
|user_agent
|referer
|requests_more_1s|query_time_more_1s|requests_less_1s|query_time_less_1s|
+---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------+----------------+------------------+----------------+------------------+
|ChemAxon-Marvin/20.15.0
|null
|209930 |1816992218 |0 |0
|
|SAP/1.0
|null
|143198 |248447172 |5100562
|1834622022 |
|okhttp/4.0.0-alpha02
|null
|128509 |552415825 |2967
|679104 |
|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36
(KHTML\, like Gecko) Chrome/50.0.2661.102 Safari/537.36 |null
|114446 |467381034 |5899288
|441228641 |
|commonscat_copy_from_P373 Pywikibot/3.1.dev0 (g6) requests/2.22.0
Python/2.7.13.final.0
|null |99342 |404639751 |5907
|4823011 |
|sparqlwrapper 1.8.2 (rdflib.github.io/sparqlwrapper)
|null
|86830 |2843969537 |289618
|45768268 |
|Apache-HttpClient/4.5.10 (Java/1.8.0_242)
|null
|70949 |1331327131 |3127
|2242093 |
|bbw-bot
|null
|68936 |292089223 |1957742
|234481876 |
|MyCoolTool/0.1 dlworb1...@yonsei.ac.kr
|null
|52715 |149532780 |275170
|34374373 |
|python-requests/2.24.0
|null
|49917 |458021897 |364755
|33714974 |
|Drupal (+http://drupal.org/)
|null
|48584 |121073196 |1064339
|147238383 |
|@kartotherian/geoshapes/1.1.2-wmf.0 (https://mediawiki.org/Maps)
|null
|41760 |215390636 |475774
|128305673 |
|python-requests/2.21.0
|null
|34603 |387626287 |209090
|34216920 |
|Python/vita (+https://suggap.com)
|null
|32866 |49728342 |4615922
|762279738 |
|hauki (https://tools.wmflabs.org/hauki; ali...@fagerving.se)
python-requests/2.21.0
|null |28580 |189414718
|92521 |14721648 |
|ChemAxon-Marvin/20.17.0
|null
|27756 |324795611 |0 |0
|
|ChicoBot Test agent
|null
|26828 |60242977 |4288061
|550297847 |
|pers_update Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.19.1
Python/3.7.0.final.0
|null |26130 |256430860 |0
|0 |
|NOS-Wikidata-Enricher-bot/0.5.1 (alexandre.carva...@parceiros.nos.pt)
node-fetch-with-proxy/0.1.2 |null
|25315 |67531382 |1223988
|302777392 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/79.0.3945.88 Safari/537.36 |null
|23988 |68155568 |76
|6579 |
|Chrome
|https://query.wikidata.org/ |23543 |70863364
|13518 |6094006 |
|C++ WikiAPI
|null
|22286 |181332220 |61106
|16271833 |
|Toolforge - legacy code
|null
|21706 |119046262 |1093947
|78807958 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/73.0.3683.103 Safari/537.36
|https://query.wikidata.org/ |20862 |72585466
|1667806 |160719635 |
|prefer Pywikibot/3.1.dev0 (g12567) requests/2.21.0 Python/3.5.3.final.0
|null
|20158 |128562631 |15720
|5852920 |
|RestSharp/106.6.10.0
|null
|19050 |177242545 |144
|125079 |
|Sockets from free hosting account if abused please report to
respo...@ifastnet.com with the ip address of the server connections are going
TO|null |18514 |168573523 |0
|0 |
|libcurl/7.68.0 r-curl/4.3 httr/1.4.2
|null
|18240 |43325076 |25536
|854676 |
|descripciones Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.19.1
Python/3.7.0.final.0
|null |17324 |45140218
|142983 |43316480 |
|WDQS-example Python/3.6
|null
|14764 |67966642 |1498670
|48097146 |
|TaxonBot/1.0
|null
|14556 |71356033 |128785
|32021301 |
|sparqlwrapper 1.8.5 (rdflib.github.io/sparqlwrapper)
|null
|11900 |108125583 |1974605
|110042533 |
|pages_processor Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.23.0
Python/3.5.3.final.0
|null |11859 |75438705
|19265 |4686168 |
|new3 Pywikibot/3.1.dev0 (g7939) requests/2.21.0 Python/3.5.3.final.0
|null
|11750 |31170186 |162936
|23802647 |
|Ruby SPARQL::Client/3.1.0
|null
|11323 |55954076 |439713
|41453405 |
|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML\, like Gecko)
Chrome/23.0.1271.64 Safari/537.11 |null
|10962 |52026459 |290564
|52478878 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101
Firefox/80.0
|https://query.wikidata.org/ |10723 |78108472
|47406 |12266791 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101
Firefox/80.0
|https://fr.wiktionary.org/ |10559 |101901873 |1
|759 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/85.0.4183.102 Safari/537.36
|https://query.wikidata.org/ |10312 |88721330
|63329 |14632348 |
|simbadbot/0.1 (https://www.wikidata.org/wiki/User:Ghuron)
|null
|9962 |231662013 |106
|72863 |
|OpenMapTiles OSM name resolver
5.2.0(https://github.com/openmaptiles/openmaptiles)
|null |9917
|18693583 |6968 |3546943 |
|Python/3.7 aiohttp/3.6.2
|null
|9610 |134249956 |681073
|67962609 |
|Mozilla/5.0 (compatible; Google-Apps-Script; beanserver;
+https://script.google.com; id: UAEmdDd_HBTqPOXeoeIuziRB3UkXGizS0-g)
|null |9506 |327207415
|0 |0 |
|python-requests/2.22.0
|null
|9459 |79727366 |1829866
|117007183 |
|Go-http-client/2.0
|null
|9151 |34421237 |228632
|17675041 |
|ad...@justinmaltais.com
|null
|9086 |24709451 |3524048
|273894180 |
|python-requests/2.23.0
|null
|8968 |111086510 |315680
|18095184 |
|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.12 (KHTML\, like Gecko)
Chrome/40.0.1271.64 Safari/537.12 |null
|8856 |50185413 |805156
|82269062 |
|Needle/0.9.2 (Node.js v8.11.1; linux x64)
|null
|7719 |212611599 |528999
|18197496 |
|2020-04 Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.21.0
Python/3.6.8.final.0
|null |7298 |54033213
|4285012 |282547548 |
|ChemAxon-Marvin/20.20.0
|null
|6797 |45234838 |0 |0
|
|WDQS-example Python/3.7
|null
|6745 |32044158 |2353569
|74414295 |
|reqwest/0.9.17
|null
|6707 |144739624 |33
|26906 |
|Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101
Firefox/80.0
|https://query.wikidata.org/ |6477 |43525056
|45545 |9230591 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/85.0.4183.83 Safari/537.36
|https://query.wikidata.org/ |6311 |56655015
|38411 |7104118 |
|got (https://github.com/sindresorhus/got)
|null
|5824 |53693478 |266428
|13596394 |
|python
|null
|5708 |28203981 |3064
|386820 |
|node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
|null
|5698 |32477521 |69937
|6949803 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/80.0.3987.132 Safari/537.36 |null
|5610 |74486886 |2413
|248270 |
|LaylaAiBot/0.1 (https://johnpaulvaughan.github.io/layla/;
johnpaulvaug...@hotmail.com)
|null |5408 |20445320
|11601268 |328061428 |
|Mozilla/5.0 (compatible; Google-Apps-Script; beanserver;
+https://script.google.com; id: UAEmdDd8HGcZkAiLOXmpegWJ-hkdGtFyS5g)
|null |5329 |15264833
|57 |49223 |
|python-requests/2.20.1
|null
|5010 |88763315 |37
|12706 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/85.0.4183.121 Safari/537.36
|https://query.wikidata.org/ |4807 |44660702
|27389 |5917205 |
|WDQS-example Python/3.8
|null
|4725 |21343632 |1401892
|68247117 |
|Toolforge - mix-n-match
|null
|4699 |19556218 |11234347
|381320844 |
|recently_deceased Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.19.1
Python/3.7.0.final.0 |null
|4577 |28594706 |0
|0 |
|ChemAxon-Marvin/20.11.0
|null
|4434 |35330168 |0 |0
|
|User-Agent:test/1.0 (https://test.test; t...@test.org)
UsedBaseLibrary/1.0.0
|null |4420 |185805916
|31801 |1725480 |
|Inquire/1.4.4 (com.tamper.Curiosity; build:871; iOS 14.0.0)
Alamofire/4.9.1
|null |4392 |19598004
|3348 |1534146 |
|sparqlwrapper 1.8.4 (rdflib.github.io/sparqlwrapper)
|null
|4363 |134216393 |55583
|3673910 |
|Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/75.0.3770.100 Safari/537.36 |null
|4268 |5930321 |370396
|13825351 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/84.0.4147.135 Safari/537.36
|https://query.wikidata.org/ |4239 |21430712
|29809 |5907540 |
|Apache-HttpClient/4.5.12 (Java/1.8.0_202)
|null
|4228 |53038800 |235
|53087 |
|okhttp/4.2.0
|null
|4181 |9987597 |6723
|3018748 |
|Python-urllib/3.5
|null
|4011 |11163383 |13362
|7076039 |
|curl/7.65.0
|null
|3976 |63798455 |1466
|620960 |
|commonscat_move_from_P910 Pywikibot/3.1.dev0 (g6) requests/2.22.0
Python/2.7.13.final.0
|null |3976 |13628170 |2033
|1632196 |
|Inquire/1.4.3 (com.tamper.Curiosity; build:870; iOS 13.6.1)
Alamofire/4.9.1
|null |3891 |15457407
|4004 |1724857 |
|n...@news360.com OB-bot/1.0
|null
|3890 |18611665 |697
|436577 |
|Twisted PageGetter
|null
|3883 |60439026 |11924503
|185350492 |
|covid-19-open-data/0.0 (linux-gnu)
|null
|3853 |5521084 |17893489
|453825510 |
|PLbot: [[ Wikidata:Database reports/Humans with missing claims]]
|null
|3784 |25484240 |18082
|3734897 |
|MBHbot
|null
|3772 |39298498 |467
|111267 |
|Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
|null
|3680 |33812478 |3937
|218718 |
|PLnode
|null
|3646 |12793487 |178593
|13297814 |
|Wolfram HTTPClient 11.3
|null
|3574 |24986061 |52736
|8715045 |
|wle2020-wmse-map/0.1.0(karl.wettin+...@wikimedia.se)
|null
|3509 |6023848 |13603
|6657041 |
|app Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.24.0 Python/3.6.8.final.0
|null
|3508 |6733242 |199727
|24522721 |
|ChemAxon-Marvin/18.13.0
|null
|3466 |30963797 |1
|924 |
|Toolforge - wikidata-todo
|null
|3456 |22143007 |97023
|8144124 |
|Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0
|null
|3404 |8476841 |65128
|5434962 |
|WDQS Python/3.7
|null
|3404 |17762938 |29022
|7447662 |
|python-requests - http://osmose.openstreetmap.fr
|null
|3316 |8214349 |48
|42187 |
|Apache-Jena-ARQ/3.12.0
|null
|3273 |5587995 |72220
|8969698 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/85.0.4183.102 Safari/537.36
|https://fr.wiktionary.org/ |3176 |33227183 |0
|0 |
|Inkbot/0.1 (www.customink.com) Ruby/0.1
|null
|3120 |12317430 |520994
|16466122 |
|EssiellLoadData/1.0 python-requests/2.21.0
|null
|3111 |9372442 |18143
|3874138 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/74.0.3729.169 Safari/537.36 |null
|3092 |6514061 |896234
|105666490 |
|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like
Gecko) Chrome/85.0.4183.102 Safari/537.36
|https://query.wikidata.org/embed.html|3082 |16494191 |8496
|2687395 |
|Jakarta Commons-HttpClient/3.1
|null
|3082 |14465120 |204364
|24081155 |
+---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------+----------------+------------------+----------------+------------------+
It is interesting to notice that some of those user-agents/referer make a lot
of fast requests as well !
TASK DETAIL
https://phabricator.wikimedia.org/T261841
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Zbyszko, JAllemandou
Cc: CBogen, JAllemandou, Aklapper, Gehel, Alter-paule, Beast1978, Un1tY,
Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana,
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas,
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
