[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

Nuria added a comment. Given that retention is not on puppet is this a setting that is communicated to a new node when it joins the cluster by the leader of the partition or similar? TASK DETAIL https://phabricator.wikimedia.org/T253753 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

dcausse added a comment. @JAllemandou I think that is an option as well, the thing is that is it is transitional to help to bootstrap a test of the full pipeline. In the end we won't be using jumbo and thus won't be able to rely on a 30days retention on main so hopefully we'll be able to

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

JAllemandou added a comment. An idea: How about sending back to kafka the update stream and make THAT one retention higher? Moving retention to 30 days for revision-create will make a lot of data stay that wouldn't be necessary (about half of the data), while keeping only the updates

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

Stashbot added a comment. Mentioned in SAL (#wikimedia-analytics) [2020-05-27T13:42:02Z] increased Kafka topic retention in jumbo-eqiad to 31 days for (eqiad|codfw).mediawiki.revision-create - T253753 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T253753: Increase retention for mediawiki.revision-create on the kafka jumbo cluster

Ottomata added a comment. Did 31 days: $ kafka topics --alter --topic eqiad.mediawiki.revision-create --config retention.ms=267840 $ kafka topics --alter --topic codfw.mediawiki.revision-create --config retention.ms=267840 $ kafka topics --describe --topic