Re: ZFS subvolume support inside Bhyve vm
> On Mar 10, 2016, at 5:31 PM, Paul Vixie wrote: > > Сергей Мамонов wrote: >> Hello! >> >> Yes - zvols looks awesome. But what driver you use for it? > > virtio-blk. > >> And what >> about disk usage overhead in guest? > > ufs on zvol is faster, either in the parent or a bhyve using virtio-blk, than > zfs. at least for writing, which is my dominant work load. i expect that this > is due to zfs's compression logic rather than anything having to do with > creating/extending files to accommodate writes. > >> virtio-blk doesnt support fstrim (ahci-hd support it, but slower? "/At >> this point virtio-blk is indeed faster then ahci-hd on high IOPS/"). >> In linux && kvm we try used virtio-scsi driver with support fstrim, but >> how I see it not availble now in 10-2 stable for bhyve. >> And I not lonely with this question - >> https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-March/003442.html > > i'm just going to live without fstrim until it's supported in virtio-blk. i > know that this option isn't available to everybody, but at the moment storage > is cheap enough to waste. At the risk of getting farther off-topic.. Virtio-blk can't and won't support fstrim. If/when bhyve were to have virtio-scsi support that could work with trim, but last I was aware there wasn't a whole lot of momentum in that direction. The virtual AHCI controller in bhyve does support trim and performs remarkably well. That's what I use for my vol-backed VMs and I haven't had any complaints. Worth testing, IMO. JN ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
Сергей Мамонов wrote: Hello! Yes - zvols looks awesome. But what driver you use for it? virtio-blk. And what about disk usage overhead in guest? ufs on zvol is faster, either in the parent or a bhyve using virtio-blk, than zfs. at least for writing, which is my dominant work load. i expect that this is due to zfs's compression logic rather than anything having to do with creating/extending files to accommodate writes. virtio-blk doesnt support fstrim (ahci-hd support it, but slower? "/At this point virtio-blk is indeed faster then ahci-hd on high IOPS/"). In linux && kvm we try used virtio-scsi driver with support fstrim, but how I see it not availble now in 10-2 stable for bhyve. And I not lonely with this question - https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-March/003442.html i'm just going to live without fstrim until it's supported in virtio-blk. i know that this option isn't available to everybody, but at the moment storage is cheap enough to waste. -- P Vixie ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
Hello!
Yes - zvols looks awesome. But what driver you use for it? And what about
disk usage overhead in guest?
virtio-blk doesnt support fstrim (ahci-hd support it, but slower? "*At this
point virtio-blk is indeed faster then ahci-hd on high IOPS*").
In linux && kvm we try used virtio-scsi driver with support fstrim, but how
I see it not availble now in 10-2 stable for bhyve.
And I not lonely with this question -
https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-March/003442.html
2016-03-11 2:45 GMT+03:00 Paul Vixie :
>
>
> Pavel Odintsov wrote:
>
>> Hello, Dear Community!
>>
>> I would like to ask about plans for this storage engine approach. I like
>> ZFS so much and we are storing about half petabyte of data here.
>>
>> But when we are speaking about vm's we should use zvols or even raw file
>> based images and they are discarding all ZFS benefits.
>>
>
> i use zvols for my bhyves and they have two of the most important zfs
> advantages:
>
> 1. snapshots.
>
> root@mm1:/home/vixie # zfs list|grep fam
>> zroot1/vms/family55.7G 3.84T 5.34G -
>> root@mm1:/home/vixie # zfs snap zroot1/vms/family@before
>>
>> [family.redbarn:amd64] touch /var/tmp/after
>>
>> root@mm1:/home/vixie # zfs snap zroot1/vms/family@after
>> root@mm1:/home/vixie # mkdir /mnt/before /mnt/after
>> root@mm1:/home/vixie # zfs clone zroot1/vms/family@before zroot1/before
>> root@mm1:/home/vixie # fsck_ffs -p /dev/zvol/zroot1/beforep2
>> ...
>> /dev/zvol/zroot1/beforep2: 264283 files, 1118905 used, 11575625 free
>> (28697 frags, 1443366 blocks, 0.2% fragmentation)
>> root@mm1:/home/vixie # mount -r /dev/zvol/zroot1/beforep2 /mnt/before
>> root@mm1:/home/vixie # mount -r /dev/zvol/zroot1/beforep2 /mnt/before
>>
>> root@mm1:/home/vixie # zfs clone zroot1/vms/family@after zroot1/after
>> root@mm1:/home/vixie # fsck_ffs -p /dev/zvol/zroot1/afterp2
>> ...
>> /dev/zvol/zroot1/afterp2: 264284 files, 1118905 used, 11575625 free
>> (28697 frags, 1443366 blocks, 0.2% fragmentation)
>> root@mm1:/home/vixie # mount -r /dev/zvol/zroot1/afterp2 /mnt/after
>>
>> root@mm1:/home/vixie # ls -l /mnt/{before,after}/var/tmp/after
>> ls: /mnt/before/var/tmp/after: No such file or directory
>> -rw-rw-r-- 1 vixie wheel 0 Mar 10 22:52 /mnt/after/var/tmp/after
>>
>
> 2. storage redundancy, read caching, and write caching:
>
> root@mm1:/home/vixie # zpool status | tr -d '\t'
>> pool: zroot1
>> state: ONLINE
>> scan: scrub repaired 0 in 2h24m with 0 errors on Thu Mar 10 12:24:13
>> 2016
>> config:
>>
>> NAMESTATE READ WRITE CKSUM
>> zroot1 ONLINE 0 0 0
>> mirror-0 ONLINE 0 0 0
>> gptid/2427e651-d9cc-11e3-b8a1-002590ea750a ONLINE 0 0 0
>> gptid/250b0f01-d9cc-11e3-b8a1-002590ea750a ONLINE 0 0 0
>> mirror-1 ONLINE 0 0 0
>> gptid/d35bb315-da08-11e3-b17f-002590ea750a ONLINE 0 0 0
>> gptid/d85ad8be-da08-11e3-b17f-002590ea750a ONLINE 0 0 0
>> logs
>> mirror-2 ONLINE 0 0 0
>> ada0s1 ONLINE 0 0 0
>> ada1s1 ONLINE 0 0 0
>> cache
>> ada0s2ONLINE 0 0 0
>> ada1s2ONLINE 0 0 0
>>
>> errors: No known data errors
>>
>
> so while i'd love to chroot a bhyve driver to some place in the middle of
> the host's file system and then pass VFS right on through, more or less the
> way mount_nullfs does, i am pretty comfortable with zvol UFS, and i think
> it's misleading to say that zvol UFS lacks all ZFS benefits.
>
> --
> P Vixie
>
> ___
> freebsd-virtualization@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscr...@freebsd.org"
>
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
Pavel Odintsov wrote:
Hello, Dear Community!
I would like to ask about plans for this storage engine approach. I like
ZFS so much and we are storing about half petabyte of data here.
But when we are speaking about vm's we should use zvols or even raw file
based images and they are discarding all ZFS benefits.
i use zvols for my bhyves and they have two of the most important zfs
advantages:
1. snapshots.
root@mm1:/home/vixie # zfs list|grep fam
zroot1/vms/family55.7G 3.84T 5.34G -
root@mm1:/home/vixie # zfs snap zroot1/vms/family@before
[family.redbarn:amd64] touch /var/tmp/after
root@mm1:/home/vixie # zfs snap zroot1/vms/family@after
root@mm1:/home/vixie # mkdir /mnt/before /mnt/after
root@mm1:/home/vixie # zfs clone zroot1/vms/family@before zroot1/before
root@mm1:/home/vixie # fsck_ffs -p /dev/zvol/zroot1/beforep2
...
/dev/zvol/zroot1/beforep2: 264283 files, 1118905 used, 11575625 free (28697
frags, 1443366 blocks, 0.2% fragmentation)
root@mm1:/home/vixie # mount -r /dev/zvol/zroot1/beforep2 /mnt/before
root@mm1:/home/vixie # mount -r /dev/zvol/zroot1/beforep2 /mnt/before
root@mm1:/home/vixie # zfs clone zroot1/vms/family@after zroot1/after
root@mm1:/home/vixie # fsck_ffs -p /dev/zvol/zroot1/afterp2
...
/dev/zvol/zroot1/afterp2: 264284 files, 1118905 used, 11575625 free (28697
frags, 1443366 blocks, 0.2% fragmentation)
root@mm1:/home/vixie # mount -r /dev/zvol/zroot1/afterp2 /mnt/after
root@mm1:/home/vixie # ls -l /mnt/{before,after}/var/tmp/after
ls: /mnt/before/var/tmp/after: No such file or directory
-rw-rw-r-- 1 vixie wheel 0 Mar 10 22:52 /mnt/after/var/tmp/after
2. storage redundancy, read caching, and write caching:
root@mm1:/home/vixie # zpool status | tr -d '\t'
pool: zroot1
state: ONLINE
scan: scrub repaired 0 in 2h24m with 0 errors on Thu Mar 10 12:24:13 2016
config:
NAMESTATE READ WRITE CKSUM
zroot1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/2427e651-d9cc-11e3-b8a1-002590ea750a ONLINE 0 0 0
gptid/250b0f01-d9cc-11e3-b8a1-002590ea750a ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/d35bb315-da08-11e3-b17f-002590ea750a ONLINE 0 0 0
gptid/d85ad8be-da08-11e3-b17f-002590ea750a ONLINE 0 0 0
logs
mirror-2 ONLINE 0 0 0
ada0s1 ONLINE 0 0 0
ada1s1 ONLINE 0 0 0
cache
ada0s2ONLINE 0 0 0
ada1s2ONLINE 0 0 0
errors: No known data errors
so while i'd love to chroot a bhyve driver to some place in the middle
of the host's file system and then pass VFS right on through, more or
less the way mount_nullfs does, i am pretty comfortable with zvol UFS,
and i think it's misleading to say that zvol UFS lacks all ZFS benefits.
--
P Vixie
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to
"freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
Jakub Klama wrote on 03/11/2016 00:13: Wiadomość napisana przez Miroslav Lachman <000.f...@quip.cz> w dniu 10.03.2016, o godz. 23:30: Pavel Odintsov wrote on 03/10/2016 23:24: Hello! Songs interesting! But I could not find any information regarding p9fs state in FreeBSD and their protocol overhead. Coukd you aim me to more details? You can find something at https://github.com/wca/p9fs Wow, that's a nice find! It should be pretty easy to teach it speak virtio transport :) I've already started kernel client implementation from scratch, but since that one is in definitely better shape than mine mostly nonexistent code, I'll look if I could make it work with virtio easily. I am looking forward to seeing the fruits of your work. I hope it will be soon. :) Miroslav Lachman ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
> Wiadomość napisana przez Miroslav Lachman <000.f...@quip.cz> w dniu > 10.03.2016, o godz. 23:30: > > Pavel Odintsov wrote on 03/10/2016 23:24: >> Hello! >> >> Songs interesting! But I could not find any information regarding p9fs >> state in FreeBSD and their protocol overhead. >> >> Coukd you aim me to more details? > > You can find something at https://github.com/wca/p9fs Wow, that's a nice find! It should be pretty easy to teach it speak virtio transport :) I've already started kernel client implementation from scratch, but since that one is in definitely better shape than mine mostly nonexistent code, I'll look if I could make it work with virtio easily. Jakub ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
Pavel Odintsov wrote on 03/10/2016 23:24: Hello! Songs interesting! But I could not find any information regarding p9fs state in FreeBSD and their protocol overhead. Coukd you aim me to more details? You can find something at https://github.com/wca/p9fs On Friday, 11 March 2016, Peter Grehan wrote: [...] To be clear on this: do you mean something like ZFS DMU passthru to a guest ? There's already host filesystem access to bhyve with NFS, and the p9fs patch. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
>Coukd you aim me to more details? I believe Jakub was working on this. https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-February/004102.html Here's the GitHub Branch: https://github.com/jceel/freebsd/tree/virtfs Here's the diff: https://github.com/freebsd/freebsd/compare/master...jceel:virtfs -Trent ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
Hello! Songs interesting! But I could not find any information regarding p9fs state in FreeBSD and their protocol overhead. Coukd you aim me to more details? Thanks! On Friday, 11 March 2016, Peter Grehan wrote: > As we have support for ZFS on almost all platforms (I'm speaking about >> Linux and FreeBSD) we could run ZFS subvolumes inside vm's without so much >> pain. >> >> So this approach looks very promising and I would like to talk about it ;) >> > > To be clear on this: do you mean something like ZFS DMU passthru to a > guest ? There's already host filesystem access to bhyve with NFS, and the > p9fs patch. > > later, > > Peter. > > -- Sincerely yours, Pavel Odintsov ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: ZFS subvolume support inside Bhyve vm
As we have support for ZFS on almost all platforms (I'm speaking about Linux and FreeBSD) we could run ZFS subvolumes inside vm's without so much pain. So this approach looks very promising and I would like to talk about it ;) To be clear on this: do you mean something like ZFS DMU passthru to a guest ? There's already host filesystem access to bhyve with NFS, and the p9fs patch. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
ZFS subvolume support inside Bhyve vm
Hello, Dear Community! I would like to ask about plans for this storage engine approach. I like ZFS so much and we are storing about half petabyte of data here. But when we are speaking about vm's we should use zvols or even raw file based images and they are discarding all ZFS benefits. We have tons of vm's per server with another virtualization engines (up to 500 with openvz and up to 150 with kvm). But filesystems and storage system become bottleneck here. With our estimations smart file based COW could reduce required amount of storage up to 30%: http://www.stableit.ru/2015/07/effectiveness-of-zfs-usage-for-openvz.html?m=1 Actually, we have only single stable and open source filesystem with cow in mind and we definitely coukd offer this killer feature to virtualization world which lacks something like this. As we have support for ZFS on almost all platforms (I'm speaking about Linux and FreeBSD) we could run ZFS subvolumes inside vm's without so much pain. So this approach looks very promising and I would like to talk about it ;) -- Sincerely yours, Pavel Odintsov ___ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
