Re: [Xmldatadumps-l] [Wikitech-l] Compressing full-history dumps faster

2014-01-21 Thread Randall Farmer
Ack, sorry for the (no subject); again in the right thread: > For external uses like XML dumps integrating the compression > strategy into LZMA would however be very attractive. This would also > benefit other users of LZMA compression like HBase. For dumps or other uses, 7za -mx=3 / xz -3 is you

Re: [Xmldatadumps-l] (no subject)

2014-01-21 Thread Randall Farmer
> For external uses like XML dumps integrating the compression > strategy into LZMA would however be very attractive. This would also > benefit other users of LZMA compression like HBase. For dumps or other uses, 7za -mx=3 / xz -3 is your best bet. That has a 4 MB buffer, compression ratios withi

Re: [Xmldatadumps-l] [Wikitech-l] Compressing full-history dumps faster

2014-01-21 Thread Randall Farmer
> > That does not sound like much economically. Do keep in mind the cost of > porting, deploying, maintaining, obtaining, and so on, new tools. Briefly, yes, CPU-hours don't cost too much, but I don't think the potential win is limited to the direct CPU-hours saved. In more detail: For Wikimedia