Smalyshev added a comment.
> After talking with Stas this apparently makes updating within the updater harder etc as it might result in more writes to sparql? ( Yes because it would have to do SPARQL Update for each individual revision. > I guess the wdqs internal machines would have comparable response times? You can see response times for RDF loading in the dashboard: https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&from=now-24h&to=now&panelId=26&fullscreen > but 11 hours in a 24 hour period is still pretty significant I'm not sure I understand how this figure was obtained but there's absolutely no way Updater spends half time in waiting for RDF loading. In reality, it spends most of its time in SPARQL Update. > I hope the Java updater does some amount of async work All RDF is loaded in parallel of course (10 threads if I remember correctly). It should be relatively easy to see timings by yourself - just run the Updater with verbose logging (DEBUG level I think - `-v` option should do that). > writing to blazegraph while getting the next data ready? That could be possible but doesn't happen now. May be a good idea to try. However, since SPARQL Update dominates the timings pretty heavily it's unlikely we'd save too much. And since we need to validate IDs against database (to ensure we don't already have the revision we're about to fetch) we can not fetch RDF before previous update has finished, thus reducing the parallelizeable part to essentially only Kafka data loading, which doesn't seem to be worth it. > Another thing to consider here is in theory even when using the cache buster method the data the wdqs updater currently gets when passing nocache=ts may not be up to date due to maxlag, not sure if that has been considered in the updater process at all? Yes, see discussion at T210901 <https://phabricator.wikimedia.org/T210901> and T212550 <https://phabricator.wikimedia.org/T212550>. TLDR: we know it happens, we have stopgap measure to counter it, but we haven't implemented the real solution yet. TASK DETAIL https://phabricator.wikimedia.org/T217897 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev Cc: Addshore, Smalyshev, BBlack, Aklapper, Gehel, alaa_wmde, Legado_Shulgin, Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, rosalieper, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Jay8g, fgiunchedi
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
