Hi Binaris,
We actually have better hardware than 4 years ago [0]. However, we have
more projects with more content than 4 years ago. Wikidata did not exist
in 2011; today it has almost 1/2 the revisions of the English language
Wikipedia. The English language Wikipedia itself has increased 51% in size
since early 2012. And the Hungarian language wiki has grown by over 50% as
well.
We should be running two dumps a month going forwards [1], where the second
run each month does not contain full history revisions. This isn't as good
as 2011 but it's not as bad as once a month either.
The main work however to improve the dumps situation will be in a complete
rearchitecturing of the dumps. One big change will be to move to a format
and structure that is truly incremental.
Folks interested in these issues are welcome to subscribe to or watch the
Phabricator project for the current dumps [3] and/or the future dumps [4].
There is also a dedicated (low-traffic) list for users and contributors to
the xml dumps [4].
Lastly, your email reminds me that I should update the dumps information at
Meta; the documentation there has fallen a bit behind. Thanks!
Ariel
[0] For current hardware, see
https://wikitech.wikimedia.org/wiki/Dumps/Snapshot_hosts
[1] https://phabricator.wikimedia.org/T126339
[2] https://phabricator.wikimedia.org/tag/dumps-generation/
[3] https://phabricator.wikimedia.org/tag/dumps-rewrite/
[4] https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l
On Wed, Aug 3, 2016 at 8:31 AM, Bináris wrote:
> Hi folks,
>
> we in Hungarian Wikipedia have been watching new huwiki dumps by bot since
> 2011, so this page history:
>
> https://hu.wikipedia.org/w/index.php?title=Sablon:A_dump_d%C3%A1tuma&offset=&limit=250&action=history
> clearly shows the freqency. Back in 2012 it took 8-10 days to create the
> new dump. Now it takes one month. Are we less develeoped or do we have less
> hardware than 4 years ago?
>
>
>
> --
> Bináris
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l