On Wed, Nov 21, 2018 at 09:59:51AM -0600, Eric Blake wrote: > On 11/21/18 9:46 AM, Richard W.M. Jones wrote: > >Matt asked if xz should really be a filter rather than a plugin. The > >answer is yes, of course it should be! That's been something in the > >todo file for a while. > > > >The commit converts the xz plugin code into a filter (leaving the > >plugin around, but deprecating it). > > > > plugin: nbdkit xz file.xz > > filter: nbdkit --filter=xz file file.xz > > > > plugin: # can't be done > > filter: nbdkit --filter=xz curl url=https://example.com/disk.xz > > And further: > > nbdkit --filter=cache --filter=xz curl url=... > > to take advantage of local caching rather than repeated curl requests.
The xz plugin includes a block cache, and now so does the filter. (The plugin long predates our addition of filters into nbdkit). I suppose there is a case for removing the block cache code from the xz plugin, and relying on the cache filter instead. I'll test that out to see if it makes a difference. It will certainly simplify the xz filter code if we did that, but at the cost of making it a bit more complex to use. > >This is only lightly tested but it works for local files and for the > >curl example given in the commit message. Unfortunately because of > >the very large block size used in the Fedora cloud image, the curl > >example is barely usable. We should get them to use a more reasonable > >block size such as 16M (currently 192M). > > This may be the first real random-access of a remote xz file that > makes the argument for a smaller block size :) > > Of course, when you switch to a smaller block size, the xz image > can't compress quite as far, but hopefully the size difference is > not that bad. Do you have actual numbers comparing the file size, > vs. the speed changes made possible by the difference in block size? In fact yes I do. I measured < 1% overhead with a 16M block size (see nbdkit-xz-plugin man page for details). This is not particularly surprising since 16M is still pretty large. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v _______________________________________________ Libguestfs mailing list [email protected] https://www.redhat.com/mailman/listinfo/libguestfs
