dcausse created this task. dcausse added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION As a maintainer of the wdqs streaming updater I want all the objects serialized in the pipeline to support schema upgrades so that I don't have to use the StateExtractionJob to build a savepoint compatible with the next version of the pipeline. Initially I thought that using a combination of stop&drain + StateExtractionJob + UpdaterBootstrapJob would be sufficient for dealing with incompatible serialization changes but this is not totally true because the AsyncIO operator holds its in-flight input & output events in a state that is not emptied when draining the pipeline (not driven by timers). StateExtractionJob still has values for extraordinary circumstances (debug, unrecoverable serialization bug, complete refactor), but it is not robust/easy enough to be used for regular upgrades. AC: - decide what serialization format to use (custom or avro) - pipeline can be upgraded event with schema changes without relying on StateExtractionJob (note that with unaligned checkpoints enabled all data transferred between operators can be serialized) TASK DETAIL https://phabricator.wikimedia.org/T283599 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
_______________________________________________ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
