On Sat, Mar 28, 2020 at 9:47 PM Strahil Nikolov <hunter86...@yahoo.com> wrote: > > On March 28, 2020 7:26:33 PM GMT+02:00, Nir Soffer <nsof...@redhat.com> wrote: > >On Sat, Mar 28, 2020 at 1:59 PM Strahil Nikolov <hunter86...@yahoo.com> > >wrote: > >> > >> On March 28, 2020 11:03:54 AM GMT+02:00, Gianluca Cecchi > ><gianluca.cec...@gmail.com> wrote: > >> >On Sat, Mar 28, 2020 at 8:39 AM Strahil Nikolov > ><hunter86...@yahoo.com> > >> >wrote: > >> > > >> >> On March 28, 2020 3:21:45 AM GMT+02:00, Gianluca Cecchi < > >> >> gianluca.cec...@gmail.com> wrote: > >> >> > >> >> > >> >[snip] > >> > > >> >>Actually it only happened with empty disk (thin provisioned) and > >> >sudden > >> >> >high I/O during the initial phase of install of the OS; it didn't > >> >> >happened > >> >> >then during normal operaton (even with 600MB/s of throughput). > >> >> > >> > > >> >[snip] > >> > > >> > > >> >> Hi Gianluca, > >> >> > >> >> Is it happening to machines with preallocated disks or on machines > >> >with > >> >> thin disks ? > >> >> > >> >> Best Regards, > >> >> Strahil Nikolov > >> >> > >> > > >> >thin provisioned. But as I have tro create many VMs with 120Gb of > >disk > >> >size > >> >of which probably only a part during time will be allocated, it > >would > >> >be > >> >unfeasible to make them all preallocated. I learned that thin is not > >> >good > >> >for block based storage domains and heavy I/O, but I would hope that > >it > >> >is > >> >not the same with file based storage domains... > >> >Thanks, > >> >Gianluca > >> > >> This is normal - gluster cannot allocate fast enough the needed > >shards (due to high IO), so the qemu pauses the VM until storage is > >available again . > > > >I don't know glusterfs internals, but I think this is very unlikely. > > > >For block storage thin provisioning in vdsm, vdsm is responsible for > >allocating > >more space, but vdsm is not in the datapath, it is monitoring the > >allocation and > >allocate more data when free space reaches a limit. It has no way to > >block I/O > >before more space is available. Gluster is in the datapath and can > >block I/O until > >it can process it. > > > >Can you explain what is the source for this theory? > > > >> You can think about VDO (with deduplication ) as a PV for the Thin > >LVM and this way you can preallocate your VMs , while saving space > >(deduplication, zero-block elimination and even compression). > >> Of course, VDO will reduce performance (unless you have > >battery-backed write cache and compression is disabled), but tbe > >benefits will be alot more. > >> > >> Another approach is to increase the shard size - so gluster will > >create fewer shards, but allocation on disk will be higher. > >> > >> Best Regards, > >> Strahil Nikolov > >> _______________________________________________ > >> Users mailing list -- users@ovirt.org > >> To unsubscribe send an email to users-le...@ovirt.org > >> Privacy Statement: https://www.ovirt.org/privacy-policy.html > >> oVirt Code of Conduct: > >https://www.ovirt.org/community/about/community-guidelines/ > >> List Archives: > >https://lists.ovirt.org/archives/list/users@ovirt.org/message/77DYUF7A5D6BIAYGVCBDKRBX2YWWJDJ4/ > >_______________________________________________ > >Users mailing list -- users@ovirt.org > >To unsubscribe send an email to users-le...@ovirt.org > >Privacy Statement: https://www.ovirt.org/privacy-policy.html > >oVirt Code of Conduct: > >https://www.ovirt.org/community/about/community-guidelines/ > >List Archives: > >https://lists.ovirt.org/archives/list/users@ovirt.org/message/2LC5HGDMXJPOMVIYABLM77BRWG6LYOZJ/ > > Hey Nir, > You are right ... This is just a theory based on my knowledge and it might > not be valid. > We nees the libvirt logs to confirm or reject the theory, but I'm convinced > that is the reason. > > Yet, it's quite possible. > Qemu tries to write to the qcow disk on gluster. > Gluster is creating shards based of the ofset, as it was not done initially > (preallocated disk take the full size on gluster and all shards are > created immediately). This takes time and requires to be done on all bricks. > As the shard size is too small (default 64MB), gluster has to create the > next shard almost immediately, but if it can't do it as fast as qemu is > filling it's qcow2 disk
Gluster can block the I/O until it can write the data to a new shard. There is no reason to return an error unless a real error happened. Also the VMs mentioned here are using raw disks, not qcow2: <disk device="disk" snapshot="no" type="file"> <target bus="scsi" dev="sda"/> <source file="/rhev/data-center/mnt/glusterSD/ovirtst.mydomain.storage:_vmstore/81b97244-4b69-4d49-84c4-c822387adc6a/images/0a91c346-23a5-4432-8af7-ae0a28f9c208/2741af0b-27fe-4f7b-a8bc-8b34b9e31cb6"> <seclabel model="dac" relabel="no" type="none"/> </source> <driver cache="none" error_policy="stop" io="threads" name="qemu" type="raw"/> <alias name="ua-0a91c346-23a5-4432-8af7-ae0a28f9c208"/> <address bus="0" controller="0" target="0" type="drive" unit="0"/> <boot order="1"/> <serial>0a91c346-23a5-4432-8af7-ae0a28f9c208</serial> </disk> Note type="raw" > - qemu will get an I/O error and we know what happens there. > Later gluster manages to create the shard(s) , and the VM is unpaused. > > That's why the oVirt team made all gluster-based disks to be fully > preallocated. Gluster disks are thin (raw-sparse) by default just like any other file based storage. If this theory was correct, this would fail consistently on gluster: 1. create raw sparse image truncate -s 100g /rhev/data-center/mnt/glusterSD/server:_path/test 2. Fill image quickly with data dd if=/dev/zero bs=1M | tr "\0" "U" | dd of=/rhev/data-center/mnt/glusterSD/server:_path/test bs=1M count=12800 iflag=fullblock oflag=direct conv=notrunc According to your theory gluster will fail to allocate shards fast enough and fail the I/O. Nir _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6H6OHOTPTEFNPRG5QDAZCT2JRG4TAFO3/