Not an official definition, but you're also welcome to use my schema in this PAWS notebook (which is what Dan linked but in Python code): https://public-paws.wmcloud.org/User:Isaac_(WMF)/mediawiki-history-dumps-example.ipynb
Good luck with Mediawiki History -- one of my favorite datasets out there and one day at a hackathon, I'll see about making a Python library for it. On Mon, Mar 16, 2026 at 8:32 AM Dan Andreescu via Analytics < [email protected]> wrote: > > https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Schema_details > > On Mon, Mar 16, 2026 at 8:02 AM Roy Smith <[email protected]> wrote: > >> So where can I find the current schema definition? >> >> On Mar 16, 2026, at 6:46 AM, Dan Andreescu via Analytics < >> [email protected]> wrote: >> >> yes, it appears that changes like this one >> <https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/67099aa6850d136f139e474c9429a14447a99ccd%5E%21/#F2> >> will >> automatically be picked up by the job, because the dumps job >> <https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala> >> doesn't specify a set of fields. We'll have to make the job more explicit >> and communicate changes like this better. It does look like Isaac >> updated the schema definition >> <https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FData_Lake%2FEdits%2FMediaWiki_history_dumps&diff=2372750&oldid=2281583> >> on Wikitech, thank you Isaac! >> >> On Sun, Mar 15, 2026 at 3:38 PM Roy Smith via Analytics < >> [email protected]> wrote: >> >>> Has the format of the data lake history files changed? Some code I have >>> which works with these assumes 70 fields, and that jives with what's >>> documented at >>> https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps#Important_fields:_event_entity_and_event_type. >>> But it looks like there's now 76 fields: >>> >>> $ bzcat 2026-01.enwiki.2026-01.tsv.bz2 | head -1 | tr '\t' '\n' | wc -l >>> 76 >>> >>> _______________________________________________ >>> Analytics mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> >> _______________________________________________ >> Analytics mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> >> >> _______________________________________________ > Analytics mailing list -- [email protected] > To unsubscribe send an email to [email protected] > -- Isaac Johnson <https://meta.wikimedia.org/wiki/User:Isaac_(WMF)> (he/him) -- Senior Research Scientist -- Wikimedia Foundation
_______________________________________________ Analytics mailing list -- [email protected] To unsubscribe send an email to [email protected]
