dcausse added a comment.
@JAllemandou I think that is an option as well, the thing is that is it is transitional to help to bootstrap a test of the full pipeline. In the end we won't be using jumbo and thus won't be able to rely on a 30days retention on main so hopefully we'll be able to reset the retention back to 7days once we're done with the test. To circumvent this particular problem (time to make the dumps available > retention) we could either: - send back the events that matter back to kafka and have higher retention like you suggest - create a dedicated job running on the analytics network to read the events stored in HDFS and figure out a way to make the resulting data available in kafka main TASK DETAIL https://phabricator.wikimedia.org/T253753 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: JAllemandou, Ottomata, dcausse, Aklapper, CBogen, 4748kitoko, darthmon_wmde, Nandana, Namenlos314, Akovalyov, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, terrrydactyl, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, jeremyb
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
