Addshore added a comment.
The only other thing I was going to add (forgot before i hit submit on the last post) Within the cluster varnish cached results for entities return much faster than the php returned results (of course) | entity | varnish result | php result | page selection | | Q1.ttl?flavour=dump | ~0.06-0.07s | ~0.6-0.7s | randomish | | Q64.ttl?flavour=dump | ~0.15-0.16s | ~2.3-2.5s | randomish | | Q100.ttl?flavour=dump | ~0.13-0.14s | ~2s | randomish | | Q55886027.ttl?flavour=dump | ~0.14s | ~7-17s? | LongPages | | Q2911127.ttl?flavour=dump | ~~0.02s | 0.06s | ShortPages | | Data was gathered from a prod mw host with requests like the following cat curl-format.txt time_namelookup: %{time_namelookup}\n time_connect: %{time_connect}\n time_appconnect: %{time_appconnect}\n time_pretransfer: %{time_pretransfer}\n time_redirect: %{time_redirect}\n time_starttransfer: %{time_starttransfer}\n ----------\n time_total: %{time_total}\n curl -w "@curl-format.txt" -o /dev/null -s "https://www.wikidata.org/wiki/Special:EntityData/Q2911127.ttl?flavour=dump&addshore=17" I guess the wdqs internal machines would have comparable response times? It's hard to really figure anything concrete out from this but the wdqs updater / updaters would potentially spend a lot less time waiting for responses (maybe they already do them async?) if they hit varnish more? Doing some terrible maths and looking at the smallest possible time saving for a short page, so 0.04s saved by hitting the cache and assuming 1 million edits in a day (based on the comment above, even though right now the wdqs updater does a small amount of batching so makes less requests) 1000000*0.04 = 40,000s = =~11 hours per host? This doesn't really help if the slowest part of the process is actually writing the data to blazegraph, but 11 hours in a 24 hour period is still pretty significant. I hope the Java updater does some amount of async work (writing to blazegraph while getting the next data ready?) TASK DETAIL https://phabricator.wikimedia.org/T217897 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Addshore, Smalyshev, BBlack, Aklapper, Gehel, alaa_wmde, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs