JAllemandou added a comment.
Some more info on this aspect: I have done a quick analysis over September queries today and found that my assumption that long queries were made by users from UI is wrong. First, total numbers of request and sum of query-time split by queries taking more than 1s or less: +-------+---------+-----------+ |more_1s|requests |query_time | +-------+---------+-----------+ |false |160185762|11285161245| |true |2757758 |22233005459| +-------+---------+-----------+ The proportions of number of queries per time classes are the same whether a referer is present (expected UI) or not (expected bot). +-----------+----------------+---------+-----------+ |has_referer|query_time_class|count |query_time | +-----------+----------------+---------+-----------+ |false |1_less_10ms |8613461 |43244699 | |false |2_10ms_to_100ms |118036102|3382186064 | |false |3_100ms_to_1s |28377288 |7058252741 | |false |4_1s_to_10s |1815394 |6081683264 | |false |5_more_10s |591957 |14313410554| |true |1_less_10ms |24329 |133314 | |true |2_10ms_to_100ms |3123534 |140796917 | |true |3_100ms_to_1s |2011048 |660547510 | |true |4_1s_to_10s |310037 |800937814 | |true |5_more_10s |40370 |1036973827 | +-----------+----------------+---------+-----------+ Below are some information on the top-100 user-agents/referer making most requests with duration greatest than 1s: +---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------+----------------+------------------+----------------+------------------+ |user_agent |referer |requests_more_1s|query_time_more_1s|requests_less_1s|query_time_less_1s| +---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------+----------------+------------------+----------------+------------------+ |ChemAxon-Marvin/20.15.0 |null |209930 |1816992218 |0 |0 | |SAP/1.0 |null |143198 |248447172 |5100562 |1834622022 | |okhttp/4.0.0-alpha02 |null |128509 |552415825 |2967 |679104 | |Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/50.0.2661.102 Safari/537.36 |null |114446 |467381034 |5899288 |441228641 | |commonscat_copy_from_P373 Pywikibot/3.1.dev0 (g6) requests/2.22.0 Python/2.7.13.final.0 |null |99342 |404639751 |5907 |4823011 | |sparqlwrapper 1.8.2 (rdflib.github.io/sparqlwrapper) |null |86830 |2843969537 |289618 |45768268 | |Apache-HttpClient/4.5.10 (Java/1.8.0_242) |null |70949 |1331327131 |3127 |2242093 | |bbw-bot |null |68936 |292089223 |1957742 |234481876 | |MyCoolTool/0.1 dlworb1...@yonsei.ac.kr |null |52715 |149532780 |275170 |34374373 | |python-requests/2.24.0 |null |49917 |458021897 |364755 |33714974 | |Drupal (+http://drupal.org/) |null |48584 |121073196 |1064339 |147238383 | |@kartotherian/geoshapes/1.1.2-wmf.0 (https://mediawiki.org/Maps) |null |41760 |215390636 |475774 |128305673 | |python-requests/2.21.0 |null |34603 |387626287 |209090 |34216920 | |Python/vita (+https://suggap.com) |null |32866 |49728342 |4615922 |762279738 | |hauki (https://tools.wmflabs.org/hauki; ali...@fagerving.se) python-requests/2.21.0 |null |28580 |189414718 |92521 |14721648 | |ChemAxon-Marvin/20.17.0 |null |27756 |324795611 |0 |0 | |ChicoBot Test agent |null |26828 |60242977 |4288061 |550297847 | |pers_update Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.19.1 Python/3.7.0.final.0 |null |26130 |256430860 |0 |0 | |NOS-Wikidata-Enricher-bot/0.5.1 (alexandre.carva...@parceiros.nos.pt) node-fetch-with-proxy/0.1.2 |null |25315 |67531382 |1223988 |302777392 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/79.0.3945.88 Safari/537.36 |null |23988 |68155568 |76 |6579 | |Chrome |https://query.wikidata.org/ |23543 |70863364 |13518 |6094006 | |C++ WikiAPI |null |22286 |181332220 |61106 |16271833 | |Toolforge - legacy code |null |21706 |119046262 |1093947 |78807958 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/73.0.3683.103 Safari/537.36 |https://query.wikidata.org/ |20862 |72585466 |1667806 |160719635 | |prefer Pywikibot/3.1.dev0 (g12567) requests/2.21.0 Python/3.5.3.final.0 |null |20158 |128562631 |15720 |5852920 | |RestSharp/106.6.10.0 |null |19050 |177242545 |144 |125079 | |Sockets from free hosting account if abused please report to respo...@ifastnet.com with the ip address of the server connections are going TO|null |18514 |168573523 |0 |0 | |libcurl/7.68.0 r-curl/4.3 httr/1.4.2 |null |18240 |43325076 |25536 |854676 | |descripciones Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.19.1 Python/3.7.0.final.0 |null |17324 |45140218 |142983 |43316480 | |WDQS-example Python/3.6 |null |14764 |67966642 |1498670 |48097146 | |TaxonBot/1.0 |null |14556 |71356033 |128785 |32021301 | |sparqlwrapper 1.8.5 (rdflib.github.io/sparqlwrapper) |null |11900 |108125583 |1974605 |110042533 | |pages_processor Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.23.0 Python/3.5.3.final.0 |null |11859 |75438705 |19265 |4686168 | |new3 Pywikibot/3.1.dev0 (g7939) requests/2.21.0 Python/3.5.3.final.0 |null |11750 |31170186 |162936 |23802647 | |Ruby SPARQL::Client/3.1.0 |null |11323 |55954076 |439713 |41453405 | |Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML\, like Gecko) Chrome/23.0.1271.64 Safari/537.11 |null |10962 |52026459 |290564 |52478878 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0 |https://query.wikidata.org/ |10723 |78108472 |47406 |12266791 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0 |https://fr.wiktionary.org/ |10559 |101901873 |1 |759 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/85.0.4183.102 Safari/537.36 |https://query.wikidata.org/ |10312 |88721330 |63329 |14632348 | |simbadbot/0.1 (https://www.wikidata.org/wiki/User:Ghuron) |null |9962 |231662013 |106 |72863 | |OpenMapTiles OSM name resolver 5.2.0(https://github.com/openmaptiles/openmaptiles) |null |9917 |18693583 |6968 |3546943 | |Python/3.7 aiohttp/3.6.2 |null |9610 |134249956 |681073 |67962609 | |Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: UAEmdDd_HBTqPOXeoeIuziRB3UkXGizS0-g) |null |9506 |327207415 |0 |0 | |python-requests/2.22.0 |null |9459 |79727366 |1829866 |117007183 | |Go-http-client/2.0 |null |9151 |34421237 |228632 |17675041 | |ad...@justinmaltais.com |null |9086 |24709451 |3524048 |273894180 | |python-requests/2.23.0 |null |8968 |111086510 |315680 |18095184 | |Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.12 (KHTML\, like Gecko) Chrome/40.0.1271.64 Safari/537.12 |null |8856 |50185413 |805156 |82269062 | |Needle/0.9.2 (Node.js v8.11.1; linux x64) |null |7719 |212611599 |528999 |18197496 | |2020-04 Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.21.0 Python/3.6.8.final.0 |null |7298 |54033213 |4285012 |282547548 | |ChemAxon-Marvin/20.20.0 |null |6797 |45234838 |0 |0 | |WDQS-example Python/3.7 |null |6745 |32044158 |2353569 |74414295 | |reqwest/0.9.17 |null |6707 |144739624 |33 |26906 | |Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0 |https://query.wikidata.org/ |6477 |43525056 |45545 |9230591 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/85.0.4183.83 Safari/537.36 |https://query.wikidata.org/ |6311 |56655015 |38411 |7104118 | |got (https://github.com/sindresorhus/got) |null |5824 |53693478 |266428 |13596394 | |python |null |5708 |28203981 |3064 |386820 | |node-fetch/1.0 (+https://github.com/bitinn/node-fetch) |null |5698 |32477521 |69937 |6949803 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/80.0.3987.132 Safari/537.36 |null |5610 |74486886 |2413 |248270 | |LaylaAiBot/0.1 (https://johnpaulvaughan.github.io/layla/; johnpaulvaug...@hotmail.com) |null |5408 |20445320 |11601268 |328061428 | |Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: UAEmdDd8HGcZkAiLOXmpegWJ-hkdGtFyS5g) |null |5329 |15264833 |57 |49223 | |python-requests/2.20.1 |null |5010 |88763315 |37 |12706 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/85.0.4183.121 Safari/537.36 |https://query.wikidata.org/ |4807 |44660702 |27389 |5917205 | |WDQS-example Python/3.8 |null |4725 |21343632 |1401892 |68247117 | |Toolforge - mix-n-match |null |4699 |19556218 |11234347 |381320844 | |recently_deceased Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.19.1 Python/3.7.0.final.0 |null |4577 |28594706 |0 |0 | |ChemAxon-Marvin/20.11.0 |null |4434 |35330168 |0 |0 | |User-Agent:test/1.0 (https://test.test; t...@test.org) UsedBaseLibrary/1.0.0 |null |4420 |185805916 |31801 |1725480 | |Inquire/1.4.4 (com.tamper.Curiosity; build:871; iOS 14.0.0) Alamofire/4.9.1 |null |4392 |19598004 |3348 |1534146 | |sparqlwrapper 1.8.4 (rdflib.github.io/sparqlwrapper) |null |4363 |134216393 |55583 |3673910 | |Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/75.0.3770.100 Safari/537.36 |null |4268 |5930321 |370396 |13825351 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/84.0.4147.135 Safari/537.36 |https://query.wikidata.org/ |4239 |21430712 |29809 |5907540 | |Apache-HttpClient/4.5.12 (Java/1.8.0_202) |null |4228 |53038800 |235 |53087 | |okhttp/4.2.0 |null |4181 |9987597 |6723 |3018748 | |Python-urllib/3.5 |null |4011 |11163383 |13362 |7076039 | |curl/7.65.0 |null |3976 |63798455 |1466 |620960 | |commonscat_move_from_P910 Pywikibot/3.1.dev0 (g6) requests/2.22.0 Python/2.7.13.final.0 |null |3976 |13628170 |2033 |1632196 | |Inquire/1.4.3 (com.tamper.Curiosity; build:870; iOS 13.6.1) Alamofire/4.9.1 |null |3891 |15457407 |4004 |1724857 | |n...@news360.com OB-bot/1.0 |null |3890 |18611665 |697 |436577 | |Twisted PageGetter |null |3883 |60439026 |11924503 |185350492 | |covid-19-open-data/0.0 (linux-gnu) |null |3853 |5521084 |17893489 |453825510 | |PLbot: [[ Wikidata:Database reports/Humans with missing claims]] |null |3784 |25484240 |18082 |3734897 | |MBHbot |null |3772 |39298498 |467 |111267 | |Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) |null |3680 |33812478 |3937 |218718 | |PLnode |null |3646 |12793487 |178593 |13297814 | |Wolfram HTTPClient 11.3 |null |3574 |24986061 |52736 |8715045 | |wle2020-wmse-map/0.1.0(karl.wettin+...@wikimedia.se) |null |3509 |6023848 |13603 |6657041 | |app Pywikibot/3.1.dev0 (-1 (unknown)) requests/2.24.0 Python/3.6.8.final.0 |null |3508 |6733242 |199727 |24522721 | |ChemAxon-Marvin/18.13.0 |null |3466 |30963797 |1 |924 | |Toolforge - wikidata-todo |null |3456 |22143007 |97023 |8144124 | |Mozilla/5.0 (Windows NT 6.1; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0 |null |3404 |8476841 |65128 |5434962 | |WDQS Python/3.7 |null |3404 |17762938 |29022 |7447662 | |python-requests - http://osmose.openstreetmap.fr |null |3316 |8214349 |48 |42187 | |Apache-Jena-ARQ/3.12.0 |null |3273 |5587995 |72220 |8969698 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/85.0.4183.102 Safari/537.36 |https://fr.wiktionary.org/ |3176 |33227183 |0 |0 | |Inkbot/0.1 (www.customink.com) Ruby/0.1 |null |3120 |12317430 |520994 |16466122 | |EssiellLoadData/1.0 python-requests/2.21.0 |null |3111 |9372442 |18143 |3874138 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/74.0.3729.169 Safari/537.36 |null |3092 |6514061 |896234 |105666490 | |Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML\, like Gecko) Chrome/85.0.4183.102 Safari/537.36 |https://query.wikidata.org/embed.html|3082 |16494191 |8496 |2687395 | |Jakarta Commons-HttpClient/3.1 |null |3082 |14465120 |204364 |24081155 | +---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------+----------------+------------------+----------------+------------------+ It is interesting to notice that some of those user-agents/referer make a lot of fast requests as well ! TASK DETAIL https://phabricator.wikimedia.org/T261841 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, JAllemandou Cc: CBogen, JAllemandou, Aklapper, Gehel, Alter-paule, Beast1978, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs