So where can I find the current schema definition? > On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics > <[email protected]> wrote: > > yes, it appears that changes like this one > <https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2> > will automatically be picked up by the job, because the dumps job > <https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala> > doesn't specify a set of fields. We'll have to make the job more explicit > and communicate changes like this better. It does look like Isaac updated > the schema definition > <https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583> > on Wikitech, thank you Isaac! > > On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics > <[email protected] <mailto:[email protected]>> wrote: > Has the format of the data lake history files changed? Some code I have > which works with these assumes 70 fields, and that jives with what's > documented at > https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Important_fields:_event_entity_and_event_type > > <https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Important_fields:_event_entity_and_event_type>. > But it looks like there's now 76 fields: > > $ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l > 76 > > _______________________________________________ > Analytics mailing list -- [email protected] > <mailto:[email protected]> > To unsubscribe send an email to [email protected] > <mailto:[email protected]> > _______________________________________________ > Analytics mailing list -- [email protected] > To unsubscribe send an email to [email protected]
_______________________________________________ Analytics mailing list -- [email protected] To unsubscribe send an email to [email protected]
