https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Schema_details
On Mon, Mar 16, 2026 at 8:02 AM Roy Smith <[email protected]> wrote: > So where can I find the current schema definition? > > On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics < > [email protected]> wrote: > > yes, it appears that changes like this one > <https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2> > will > automatically be picked up by the job, because the dumps job > <https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala> > doesn't specify a set of fields. We'll have to make the job more explicit > and communicate changes like this better. It does look like Isaac > updated the schema definition > <https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583> > on Wikitech, thank you Isaac! > > On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics < > [email protected]> wrote: > >> Has the format of the data lake history files changed? Some code I have >> which works with these assumes 70 fields, and that jives with what's >> documented at >> https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Important_fields:_event_entity_and_event_type. >> But it looks like there's now 76 fields: >> >> $ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l >> 76 >> >> _______________________________________________ >> Analytics mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> > _______________________________________________ > Analytics mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > >
_______________________________________________ Analytics mailing list -- [email protected] To unsubscribe send an email to [email protected]
