Also when testing with dd i get the following: *Testing on the gluster mount: * dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/test2.img oflag=direct bs=512 count=1 dd: error writing β/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/test2.imgβ: *Transport endpoint is not connected* 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00336755 s, 0.0 kB/s
*Testing on the /root directory (XFS): * dd if=/dev/zero of=/test2.img oflag=direct bs=512 count=1 dd: error writing β/test2.imgβ:* Invalid argument* 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.000321239 s, 0.0 kB/s Seems that the gluster is trying to do the same and fails. On Mon, Jun 5, 2017 at 10:10 PM, Abi Askushi <rightkickt...@gmail.com> wrote: > The question that rises is what is needed to make gluster aware of the 4K > physical sectors presented to it (the logical sector is also 4K). The > offset (127488) at the log does not seem aligned at 4K. > > Alex > > On Mon, Jun 5, 2017 at 2:47 PM, Abi Askushi <rightkickt...@gmail.com> > wrote: > >> Hi Krutika, >> >> I am saying that I am facing this issue with 4k drives. I never >> encountered this issue with 512 drives. >> >> Alex >> >> On Jun 5, 2017 14:26, "Krutika Dhananjay" <kdhan...@redhat.com> wrote: >> >>> This seems like a case of O_DIRECT reads and writes gone wrong, judging >>> by the 'Invalid argument' errors. >>> >>> The two operations that have failed on gluster bricks are: >>> >>> [2017-06-05 09:40:39.428979] E [MSGID: 113072] >>> [posix.c:3453:posix_writev] 0-engine-posix: write failed: offset 0, >>> [Invalid argument] >>> [2017-06-05 09:41:00.865760] E [MSGID: 113040] >>> [posix.c:3178:posix_readv] 0-engine-posix: read failed on >>> gfid=8c94f658-ac3c-4e3a-b368-8c038513a914, fd=0x7f408584c06c, >>> offset=127488 size=512, buf=0x7f4083c0b000 [Invalid argument] >>> >>> But then, both the write and the read have 512byte-aligned offset, size >>> and buf address (which is correct). >>> >>> Are you saying you don't see this issue with 4K block-size? >>> >>> -Krutika >>> >>> On Mon, Jun 5, 2017 at 3:21 PM, Abi Askushi <rightkickt...@gmail.com> >>> wrote: >>> >>>> Hi Sahina, >>>> >>>> Attached are the logs. Let me know if sth else is needed. >>>> >>>> I have 5 disks (with 4K physical sector) in RAID5. The RAID has 64K >>>> stripe size at the moment. >>>> I have prepared the storage as below: >>>> >>>> pvcreate --dataalignment 256K /dev/sda4 >>>> vgcreate --physicalextentsize 256K gluster /dev/sda4 >>>> >>>> lvcreate -n engine --size 120G gluster >>>> mkfs.xfs -f -i size=512 /dev/gluster/engine >>>> >>>> Thanx, >>>> Alex >>>> >>>> On Mon, Jun 5, 2017 at 12:14 PM, Sahina Bose <sab...@redhat.com> wrote: >>>> >>>>> Can we have the gluster mount logs and brick logs to check if it's the >>>>> same issue? >>>>> >>>>> On Sun, Jun 4, 2017 at 11:21 PM, Abi Askushi <rightkickt...@gmail.com> >>>>> wrote: >>>>> >>>>>> I clean installed everything and ran into the same. >>>>>> I then ran gdeploy and encountered the same issue when deploying >>>>>> engine. >>>>>> Seems that gluster (?) doesn't like 4K sector drives. I am not sure >>>>>> if it has to do with alignment. The weird thing is that gluster volumes >>>>>> are >>>>>> all ok, replicating normally and no split brain is reported. >>>>>> >>>>>> The solution to the mentioned bug (1386443 >>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1386443>) was to format >>>>>> with 512 sector size, which for my case is not an option: >>>>>> >>>>>> mkfs.xfs -f -i size=512 -s size=512 /dev/gluster/engine >>>>>> illegal sector size 512; hw sector is 4096 >>>>>> >>>>>> Is there any workaround to address this? >>>>>> >>>>>> Thanx, >>>>>> Alex >>>>>> >>>>>> >>>>>> On Sun, Jun 4, 2017 at 5:48 PM, Abi Askushi <rightkickt...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Maor, >>>>>>> >>>>>>> My disk are of 4K block size and from this bug seems that gluster >>>>>>> replica needs 512B block size. >>>>>>> Is there a way to make gluster function with 4K drives? >>>>>>> >>>>>>> Thank you! >>>>>>> >>>>>>> On Sun, Jun 4, 2017 at 2:34 PM, Maor Lipchuk <mlipc...@redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Alex, >>>>>>>> >>>>>>>> I saw a bug that might be related to the issue you encountered at >>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1386443 >>>>>>>> >>>>>>>> Sahina, maybe you have any advise? Do you think that BZ1386443is >>>>>>>> related? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Maor >>>>>>>> >>>>>>>> On Sat, Jun 3, 2017 at 8:45 PM, Abi Askushi < >>>>>>>> rightkickt...@gmail.com> wrote: >>>>>>>> > Hi All, >>>>>>>> > >>>>>>>> > I have installed successfully several times oVirt (version 4.1) >>>>>>>> with 3 nodes >>>>>>>> > on top glusterfs. >>>>>>>> > >>>>>>>> > This time, when trying to configure the same setup, I am facing >>>>>>>> the >>>>>>>> > following issue which doesn't seem to go away. During >>>>>>>> installation i get the >>>>>>>> > error: >>>>>>>> > >>>>>>>> > Failed to execute stage 'Misc configuration': Cannot acquire host >>>>>>>> id: >>>>>>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>>>>>> 'Sanlock >>>>>>>> > lockspace add failure', 'Invalid argument')) >>>>>>>> > >>>>>>>> > The only different in this setup is that instead of standard >>>>>>>> partitioning i >>>>>>>> > have GPT partitioning and the disks have 4K block size instead of >>>>>>>> 512. >>>>>>>> > >>>>>>>> > The /var/log/sanlock.log has the following lines: >>>>>>>> > >>>>>>>> > 2017-06-03 19:21:15+0200 23450 [943]: s9 lockspace >>>>>>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:250:/rhev/data-center/m >>>>>>>> nt/_var_lib_ovirt-hosted-engin-setup_tmptjkIDI/ba6bd862-c2b8 >>>>>>>> -46e7-b2c8-91e4a5bb2047/dom_md/ids:0 >>>>>>>> > 2017-06-03 19:21:36+0200 23471 [944]: s9:r5 resource >>>>>>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047:SDM:/rhev/data-center/m >>>>>>>> nt/_var_lib_ovirt-hosted-engine-setup_tmptjkIDI/ba6bd862-c2b >>>>>>>> 8-46e7-b2c8-91e4a5bb2047/dom_md/leases:1048576 >>>>>>>> > for 2,9,23040 >>>>>>>> > 2017-06-03 19:21:36+0200 23471 [943]: s10 lockspace >>>>>>>> > a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922:250:/rhev/data-center/m >>>>>>>> nt/glusterSD/10.100.100.1:_engine/a5a6b0e7-fc3f-4838-8e26-c8 >>>>>>>> b4d5e5e922/dom_md/ids:0 >>>>>>>> > 2017-06-03 19:21:36+0200 23471 [23522]: a5a6b0e7 aio collect RD >>>>>>>> > 0x7f59b00008c0:0x7f59b00008d0:0x7f59b0101000 result -22:0 match >>>>>>>> res >>>>>>>> > 2017-06-03 19:21:36+0200 23471 [23522]: read_sectors delta_leader >>>>>>>> offset >>>>>>>> > 127488 rv -22 >>>>>>>> > /rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/a5a6b0e >>>>>>>> 7-fc3f-4838-8e26-c8b4d5e5e922/dom_md/ids >>>>>>>> > 2017-06-03 19:21:37+0200 23472 [930]: s9 host 250 1 23450 >>>>>>>> > 88c2244c-a782-40ed-9560-6cfa4d46f853.v0.neptune >>>>>>>> > 2017-06-03 19:21:37+0200 23472 [943]: s10 add_lockspace fail >>>>>>>> result -22 >>>>>>>> > >>>>>>>> > And /var/log/vdsm/vdsm.log says: >>>>>>>> > >>>>>>>> > 2017-06-03 19:19:38,176+0200 WARN (jsonrpc/3) >>>>>>>> > [storage.StorageServer.MountConnection] Using user specified >>>>>>>> > backup-volfile-servers option (storageServer:253) >>>>>>>> > 2017-06-03 19:21:12,379+0200 WARN (periodic/1) [throttled] MOM >>>>>>>> not >>>>>>>> > available. (throttledlog:105) >>>>>>>> > 2017-06-03 19:21:12,380+0200 WARN (periodic/1) [throttled] MOM >>>>>>>> not >>>>>>>> > available, KSM stats will be missing. (throttledlog:105) >>>>>>>> > 2017-06-03 19:21:14,714+0200 WARN (jsonrpc/1) >>>>>>>> > [storage.StorageServer.MountConnection] Using user specified >>>>>>>> > backup-volfile-servers option (storageServer:253) >>>>>>>> > 2017-06-03 19:21:15,515+0200 ERROR (jsonrpc/4) >>>>>>>> [storage.initSANLock] Cannot >>>>>>>> > initialize SANLock for domain a5a6b0e7-fc3f-4838-8e26-c8b4d5 >>>>>>>> e5e922 >>>>>>>> > (clusterlock:238) >>>>>>>> > Traceback (most recent call last): >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/clusterlock.py", line >>>>>>>> > 234, in initSANLock >>>>>>>> > sanlock.init_lockspace(sdUUID, idsPath) >>>>>>>> > SanlockException: (107, 'Sanlock lockspace init failure', >>>>>>>> 'Transport >>>>>>>> > endpoint is not connected') >>>>>>>> > 2017-06-03 19:21:15,515+0200 WARN (jsonrpc/4) >>>>>>>> > [storage.StorageDomainManifest] lease did not initialize >>>>>>>> successfully >>>>>>>> > (sd:557) >>>>>>>> > Traceback (most recent call last): >>>>>>>> > File "/usr/share/vdsm/storage/sd.py", line 552, in >>>>>>>> initDomainLock >>>>>>>> > self._domainLock.initLock(self.getDomainLease()) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/clusterlock.py", line >>>>>>>> > 271, in initLock >>>>>>>> > initSANLock(self._sdUUID, self._idsPath, lease) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/clusterlock.py", line >>>>>>>> > 239, in initSANLock >>>>>>>> > raise se.ClusterLockInitError() >>>>>>>> > ClusterLockInitError: Could not initialize cluster lock: () >>>>>>>> > 2017-06-03 19:21:37,867+0200 ERROR (jsonrpc/2) >>>>>>>> [storage.StoragePool] Create >>>>>>>> > pool hosted_datacenter canceled (sp:655) >>>>>>>> > Traceback (most recent call last): >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 652, in create >>>>>>>> > self.attachSD(sdUUID) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/securable.py", line >>>>>>>> > 79, in wrapper >>>>>>>> > return method(self, *args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 971, in attachSD >>>>>>>> > dom.acquireHostId(self.id) >>>>>>>> > File "/usr/share/vdsm/storage/sd.py", line 790, in >>>>>>>> acquireHostId >>>>>>>> > self._manifest.acquireHostId(hostId, async) >>>>>>>> > File "/usr/share/vdsm/storage/sd.py", line 449, in >>>>>>>> acquireHostId >>>>>>>> > self._domainLock.acquireHostId(hostId, async) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/clusterlock.py", line >>>>>>>> > 297, in acquireHostId >>>>>>>> > raise se.AcquireHostIdFailure(self._sdUUID, e) >>>>>>>> > AcquireHostIdFailure: Cannot acquire host id: >>>>>>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>>>>>> 'Sanlock >>>>>>>> > lockspace add failure', 'Invalid argument')) >>>>>>>> > 2017-06-03 19:21:37,870+0200 ERROR (jsonrpc/2) >>>>>>>> [storage.StoragePool] Domain >>>>>>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047 detach from MSD >>>>>>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047 Ver 1 failed. (sp:528) >>>>>>>> > Traceback (most recent call last): >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 525, in >>>>>>>> __cleanupDomains >>>>>>>> > self.detachSD(sdUUID) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/securable.py", line >>>>>>>> > 79, in wrapper >>>>>>>> > return method(self, *args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 1046, in detachSD >>>>>>>> > raise se.CannotDetachMasterStorageDomain(sdUUID) >>>>>>>> > CannotDetachMasterStorageDomain: Illegal action: >>>>>>>> > (u'ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047',) >>>>>>>> > 2017-06-03 19:21:37,872+0200 ERROR (jsonrpc/2) >>>>>>>> [storage.StoragePool] Domain >>>>>>>> > a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922 detach from MSD >>>>>>>> > ba6bd862-c2b8-46e7-b2c8-91e4a5bb2047 Ver 1 failed. (sp:528) >>>>>>>> > Traceback (most recent call last): >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 525, in >>>>>>>> __cleanupDomains >>>>>>>> > self.detachSD(sdUUID) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/securable.py", line >>>>>>>> > 79, in wrapper >>>>>>>> > return method(self, *args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 1043, in detachSD >>>>>>>> > self.validateAttachedDomain(dom) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/securable.py", line >>>>>>>> > 79, in wrapper >>>>>>>> > return method(self, *args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 542, in >>>>>>>> validateAttachedDomain >>>>>>>> > self.validatePoolSD(dom.sdUUID) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/securable.py", line >>>>>>>> > 79, in wrapper >>>>>>>> > return method(self, *args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 535, in >>>>>>>> validatePoolSD >>>>>>>> > raise se.StorageDomainNotMemberOfPool(self.spUUID, sdUUID) >>>>>>>> > StorageDomainNotMemberOfPool: Domain is not member in pool: >>>>>>>> > u'pool=a1e7e9dd-0cf4-41ae-ba13-36297ed66309, >>>>>>>> > domain=a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922' >>>>>>>> > 2017-06-03 19:21:40,063+0200 ERROR (jsonrpc/2) >>>>>>>> [storage.TaskManager.Task] >>>>>>>> > (Task='a2476a33-26f8-4ebd-876d-02fe5d13ef78') Unexpected error >>>>>>>> (task:870) >>>>>>>> > Traceback (most recent call last): >>>>>>>> > File "/usr/share/vdsm/storage/task.py", line 877, in _run >>>>>>>> > return fn(*args, **kargs) >>>>>>>> > File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line >>>>>>>> 52, in >>>>>>>> > wrapper >>>>>>>> > res = f(*args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/hsm.py", line 959, in >>>>>>>> createStoragePool >>>>>>>> > leaseParams) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 652, in create >>>>>>>> > self.attachSD(sdUUID) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/securable.py", line >>>>>>>> > 79, in wrapper >>>>>>>> > return method(self, *args, **kwargs) >>>>>>>> > File "/usr/share/vdsm/storage/sp.py", line 971, in attachSD >>>>>>>> > dom.acquireHostId(self.id) >>>>>>>> > File "/usr/share/vdsm/storage/sd.py", line 790, in >>>>>>>> acquireHostId >>>>>>>> > self._manifest.acquireHostId(hostId, async) >>>>>>>> > File "/usr/share/vdsm/storage/sd.py", line 449, in >>>>>>>> acquireHostId >>>>>>>> > self._domainLock.acquireHostId(hostId, async) >>>>>>>> > File "/usr/lib/python2.7/site-packa >>>>>>>> ges/vdsm/storage/clusterlock.py", line >>>>>>>> > 297, in acquireHostId >>>>>>>> > raise se.AcquireHostIdFailure(self._sdUUID, e) >>>>>>>> > AcquireHostIdFailure: Cannot acquire host id: >>>>>>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>>>>>> 'Sanlock >>>>>>>> > lockspace add failure', 'Invalid argument')) >>>>>>>> > 2017-06-03 19:21:40,067+0200 ERROR (jsonrpc/2) >>>>>>>> [storage.Dispatcher] >>>>>>>> > {'status': {'message': "Cannot acquire host id: >>>>>>>> > (u'a5a6b0e7-fc3f-4838-8e26-c8b4d5e5e922', SanlockException(22, >>>>>>>> 'Sanlock >>>>>>>> > lockspace add failure', 'Invalid argument'))", 'code': 661}} >>>>>>>> (dispatcher:77) >>>>>>>> > >>>>>>>> > The gluster volume prepared for engine storage is online and no >>>>>>>> split brain >>>>>>>> > is reported. I don't understand what needs to be done to overcome >>>>>>>> this. Any >>>>>>>> > idea will be appreciated. >>>>>>>> > >>>>>>>> > Thank you, >>>>>>>> > Alex >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > Users mailing list >>>>>>>> > Users@ovirt.org >>>>>>>> > http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>> >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users