Hi all, I’m sending this on behalf of a few groups at WMF: the Product Safety and Integrity (PSI) team, who work on safety and security issues, as well as our teams who promote responsible reuse and manage how our data platform and data dumps work.
To reduce safety issues and to continue modernizing our data platform, we are introducing a freshness delay to our content dumps, and completing the deprecation of our legacy export system: 1. A 7-day “safety buffer” between content creation and dumps publishing will be added to the MediaWiki Content File Exports <https://dumps.wikimedia.org/other/mediawiki_content_history/readme.html>. As a result of this change, the full content will be published up to 10 days after the first of the month, which allows more time for users with extended rights to respond to calls for deletion and remove abusive content before it is published. 2. Legacy XML database backup dumps <https://dumps.wikimedia.org/backup-index.html> for full content history and latest content versions will be sunset in January 2027. MediaWiki Content File Exports <https://dumps.wikimedia.org/other/mediawiki_content_history/readme.html> currently offer comparable content and will have to be used instead, following the legacy sunsetting date. 3. Support for the 7z compression standard will be added to the MediaWiki Content History File Exports, at the request of the community. Previously only bzip2 file formats were provided. Some more information about this is below, and even more detailed information about the safety buffer, how to migrate to MediaWiki Content File Exports, and what the detailed timeline looks like will be available on the project page <https://wikitech.wikimedia.org/wiki/Legacy_content_dumps_sunset>. What is the “safety buffer”? The safety buffer will add a 7-day delay between content creation and when it is published via the MediaWiki Content File Export process. This change gives time for on-wiki content cleanup processes to happen. This is especially important for users with extended rights to suppress content before it is published. Unlike standard reverts for vandalism or other bad-faith edits, suppressions fully removes the revision so it is no longer visible to standard users. This ‘safety buffer’ is implemented as a cooldown window, where a content snapshot is created, then any subsequent changes to visibility made between the snapshot and publishing date will be respected, which prevents content removed in this window from making it into the dump at all. Why is a ‘safety buffer’ necessary? Once published, data dumps become permanent snapshots that propagate across the internet. Even when content is later redacted on the wikis, those relying on data dumps may miss those changes. This means that especially sensitive or abusive content can easily end up included in data dumps, even if that content was later redacted. Introducing a 7-day buffer allows on-wiki processes some space to operate, and significantly reduces the chances that this kind of content will be widely distributed across the internet. How will the buffer affect availability and publishing cadence? MediaWiki Content File Exports will continue to be generated and published on a monthly basis. However, with the buffer in place, dumps are expected to be available between the 7th and 10th of the month instead of on the 1st. For example, this means that content created between October 1st and October 31st would be available between November 7th and 10th to allow time for both the buffer and the XML generation processes to take place. What specific dumps are being sunset? A huge variety of dumps are available today. This change is specifically targeting XML dumps for all pages (current revision only) and all pages with complete edit history. More specialized dumps, such as those for Wikidata, category, or title dumps are currently unaffected by this change. Why are we sunsetting legacy XML dumps? The legacy dumps generation process is expensive to operate, difficult to maintain, and unreliable in availability. MediaWiki Content File Exports <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports> is a process that relies on the MediaWiki Content History <https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Content/Mediawiki_content_history_v1> data set built by the Data Platform Engineering team in 2025. This data set contains the full content of all revisions, making it a more reliable and maintainable foundation for generating XML dumps. What is the timeline for sunsetting? MediaWiki Content File Exports were launched in January 2026 <https://lists.wikimedia.org/hyperkitty/list/[email protected]/thread/E6D5EU4PMSTSOI2J7A46HJ3YW2W554CS/>as a replacement for XML exports of the full content history and the latest revision for all pages. At that time, the legacy dumps were also announced as deprecated, but a sunsetting date was not provided. Users are now encouraged to proactively migrate to use the MediaWiki Content File Exports, prior to the January 2027 sunsetting date. To ensure that all dumps users are aware of the change and incentivized to migrate, we are also implementing progressive “brown out” periods that limit availability of the legacy dumps through the dumps website ( dumps.wikimedia.org) before they are fully sunset. During these brown-out periods, access to the affected legacy dumps will be unavailable. As a temporary workaround, users may access data earlier in the month to prevent disruption. Legacy dumps will also continue to be published to the Wikimedia Cloud Services fileshare through January 2027. Limited brownouts will start towards the end of October, with progressively longer periods of blocking legacy dump downloads each month. By December, legacy dumps will be unavailable for the majority of the month and are expected to be fully disabled by January. More information about this plan is available on the project page <https://wikitech.wikimedia.org/wiki/Legacy_content_dumps_sunset>. What does the migration path look like from the legacy XML dumps to MediaWiki Content File Exports? MediaWiki Content File Exports are largely compatible with legacy XML dumps. However, the files are available on a different path, offer a different chunking threshold, and have a different mechanism for determining if the dump is fully complete for a given project. More information about how to download MediaWiki Content File Exports can be found on the project page <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports#How_to_download> . What does this mean for dumps accessed through WMCS? The WMCS fileshares will remain unaffected until January 2027. Starting in January (ie: December dump), legacy dumps will no longer be available in the WMCS file share. What does this mean for Wikimedia Enterprise Snapshots? Wikimedia Enterprise Snapshots will not be affected by this buffer. Wikimedia Enterprise users are contractually bound to continuously update the content adhering to the policies of the source projects in ways that minimize these issues. What does this change mean for mirrors? Starting in December, the rsync config will exclusively publish MediaWiki File Exports for the in-scope XML dumps. Previous versions will continue to reflect legacy dumps, until the last 5 are fully replaced with v2. Changes to the rsync process will happen automatically. Mirrors that rely on rsync updates may need to notify their users of the changes to adopt the MediaWiki Content File Exports. A dedicated email will also be sent to mirror managers directly with more specific details so they know what to expect. If you have any additional questions, comments, or concerns, please feel free to reach out directly here, on the talk page <https://wikitech.wikimedia.org/w/index.php?title=Talk:Legacy_content_dumps_sunset&action=edit&redlink=1> for the project <https://wikitech.wikimedia.org/wiki/MediaWiki_Content_File_Exports#How_to_download>, or on Discord. Thanks, Eric Mill, Guilherme Gonçalves, Halley Coplin, Andreas Hoelzl, and Xabriel Collazo Mojica _______________________________________________ Wiki-research-l mailing list -- [email protected] To unsubscribe send an email to [email protected]
