Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
The CI setup did not change recently. All standard-CI jobs run inside mock (chroot) which is stored on top of a regular FS, so they should not be affected by the slave OS at all as far as FS settings go. But perhaps some slave-OS/mock-OS combination is acting strangely, so could you be more specific and point to particular job runs that fail? On 28 September 2016 at 22:00, Nir Soffer wrote: > Hi all, > > It seems that the CI setup has changed, and /var/tmp is using now tempfs. > > This is not compatible with vdsm tests, assuming that /var/tmp is a real file > system. This is the reason we do not use /tmp. > > We have lot of storage tests using directio, and directio cannot work on > tempfs. > > Please check the slaves and make sure /var/tmp is using file system supporting > directio. > > See example failure bellow. > > Nir > > > > 12:33:20 > == > 12:33:20 ERROR: test_create_fail_creating_lease > (storage_volume_artifacts_test.BlockVolumeArtifactsTests) > 12:33:20 > -- > 12:33:20 Traceback (most recent call last): > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", > line 485, in test_create_fail_creating_lease > 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) > 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in > assertRaises > 12:33:20 callableObj(*args, **kwargs) > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", > line 391, in create > 12:33:20 desc, parent) > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", > line 482, in _create_metadata > 12:33:20 sc.LEGAL_VOL) > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", > line 427, in newMetadata > 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", > line 420, in createMetadata > 12:33:20 cls._putMetadata(metaId, meta) > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", > line 242, in _putMetadata > 12:33:20 f.write(data) > 12:33:20 File > "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", > line 161, in write > 12:33:20 raise OSError(err, msg) > 12:33:20 OSError: [Errno 22] Invalid argument > 12:33:20 >> begin captured logging << > > 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] > (MainThread) Created a persistent dict with VGTagMetadataRW backend > 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] > (MainThread) read lines (VGTagMetadataRW)=[] > 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] > (MainThread) Empty metadata > 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] > (MainThread) Starting transaction > 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] > (MainThread) Flushing changes > 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] > (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', > 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', > 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', > '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] > 12:33:20 2016-09-28 05:32:03,257 DEBUG > [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata > adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, > MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, > MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, > MDT_VERSION=3 removing= > 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] > (MainThread) Finished transaction > 12:33:20 2016-09-28 05:32:03,261 INFO > [storage.BlockVolumeArtifacts] (MainThread) Create placeholder > /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 > for image's volumes > 12:33:20 2016-09-28 05:32:03,264 WARNING > [storage.StorageDomainManifest] (MainThread) Could not find mapping > for lv > c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 > 12:33:20 2016-09-28 05:32:03,264 DEBUG > [storage.StorageDomainManifest] (MainThread) Found freeSlot 4 in VG > c088020e-91b2-45e6-85fd-eea3fce58764 > 12:33:20 - >> end captured logging << > - > 1 > ___ > Infra mailing list > in...@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra -- Barak Korren bkor...@redhat.com RHEV-CI Team ___ Devel mailing list
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren wrote: > The CI setup did not change recently. Great > All standard-CI jobs run inside mock (chroot) which is stored on top > of a regular FS, so they should not be affected by the slave OS at all > as far as FS settings go. > > But perhaps some slave-OS/mock-OS combination is acting strangely, so > could you be more specific and point to particular job runs that fail? This jobs failed, but it was deleted (I get 404 now): http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/ Then I triggered the tests and they passed: http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/ Thanks Nir > > On 28 September 2016 at 22:00, Nir Soffer wrote: >> Hi all, >> >> It seems that the CI setup has changed, and /var/tmp is using now tempfs. >> >> This is not compatible with vdsm tests, assuming that /var/tmp is a real file >> system. This is the reason we do not use /tmp. >> >> We have lot of storage tests using directio, and directio cannot work on >> tempfs. >> >> Please check the slaves and make sure /var/tmp is using file system >> supporting >> directio. >> >> See example failure bellow. >> >> Nir >> >> >> >> 12:33:20 >> == >> 12:33:20 ERROR: test_create_fail_creating_lease >> (storage_volume_artifacts_test.BlockVolumeArtifactsTests) >> 12:33:20 >> -- >> 12:33:20 Traceback (most recent call last): >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", >> line 485, in test_create_fail_creating_lease >> 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) >> 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in >> assertRaises >> 12:33:20 callableObj(*args, **kwargs) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", >> line 391, in create >> 12:33:20 desc, parent) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", >> line 482, in _create_metadata >> 12:33:20 sc.LEGAL_VOL) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", >> line 427, in newMetadata >> 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", >> line 420, in createMetadata >> 12:33:20 cls._putMetadata(metaId, meta) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", >> line 242, in _putMetadata >> 12:33:20 f.write(data) >> 12:33:20 File >> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", >> line 161, in write >> 12:33:20 raise OSError(err, msg) >> 12:33:20 OSError: [Errno 22] Invalid argument >> 12:33:20 >> begin captured logging << >> >> 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] >> (MainThread) Created a persistent dict with VGTagMetadataRW backend >> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >> (MainThread) read lines (VGTagMetadataRW)=[] >> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >> (MainThread) Empty metadata >> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >> (MainThread) Starting transaction >> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] >> (MainThread) Flushing changes >> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] >> (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', >> 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', >> 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', >> '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] >> 12:33:20 2016-09-28 05:32:03,257 DEBUG >> [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata >> adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, >> MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, >> MDT_SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764, MDT_CLASS=Data, >> MDT_VERSION=3 removing= >> 12:33:20 2016-09-28 05:32:03,257 DEBUG [storage.PersistentDict] >> (MainThread) Finished transaction >> 12:33:20 2016-09-28 05:32:03,261 INFO >> [storage.BlockVolumeArtifacts] (MainThread) Create placeholder >> /var/tmp/tmp3UKYqC/52fe3782-ed7a-4d84-be3d-236faebdca2d/c088020e-91b2-45e6-85fd-eea3fce58764/images/a1e3aa74-2304-4e0c-b8e8-f5b86f8d3ac7 >> for image's volumes >> 12:33:20 2016-09-28 05:32:03,264 WARNING >> [storage.StorageDomainManifest] (MainThread) Could not find mapping >> for lv >> c088020e-91b2-45e6-85fd-eea3fce58764/3267d54d-c0fa-4fe1-b82b-b88dd5f90de3 >> 12:33:20 2016-09-28 05:32:03,264 DEBUG >> [storage.StorageDomain
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer wrote: > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren wrote: >> The CI setup did not change recently. > > Great > >> All standard-CI jobs run inside mock (chroot) which is stored on top >> of a regular FS, so they should not be affected by the slave OS at all >> as far as FS settings go. >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so >> could you be more specific and point to particular job runs that fail? > > This jobs failed, but it was deleted (I get 404 now): > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/ Oops, wrong build. This is the failing build: http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console And this is probably the reason - using a ram disk: 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical integ-tests ram_disk fc23) in workspace /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 We cannot run the storage tests using a ramdisk. We are creating (tiny) volumes and storage domains and doing copies, this code cannot work with ramdisk. Attaching full console log before the job is deleted. > > Then I triggered the tests and they passed: > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/ > > Thanks > Nir > >> >> On 28 September 2016 at 22:00, Nir Soffer wrote: >>> Hi all, >>> >>> It seems that the CI setup has changed, and /var/tmp is using now tempfs. >>> >>> This is not compatible with vdsm tests, assuming that /var/tmp is a real >>> file >>> system. This is the reason we do not use /tmp. >>> >>> We have lot of storage tests using directio, and directio cannot work on >>> tempfs. >>> >>> Please check the slaves and make sure /var/tmp is using file system >>> supporting >>> directio. >>> >>> See example failure bellow. >>> >>> Nir >>> >>> >>> >>> 12:33:20 >>> == >>> 12:33:20 ERROR: test_create_fail_creating_lease >>> (storage_volume_artifacts_test.BlockVolumeArtifactsTests) >>> 12:33:20 >>> -- >>> 12:33:20 Traceback (most recent call last): >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", >>> line 485, in test_create_fail_creating_lease >>> 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) >>> 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in >>> assertRaises >>> 12:33:20 callableObj(*args, **kwargs) >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", >>> line 391, in create >>> 12:33:20 desc, parent) >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", >>> line 482, in _create_metadata >>> 12:33:20 sc.LEGAL_VOL) >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", >>> line 427, in newMetadata >>> 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", >>> line 420, in createMetadata >>> 12:33:20 cls._putMetadata(metaId, meta) >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", >>> line 242, in _putMetadata >>> 12:33:20 f.write(data) >>> 12:33:20 File >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", >>> line 161, in write >>> 12:33:20 raise OSError(err, msg) >>> 12:33:20 OSError: [Errno 22] Invalid argument >>> 12:33:20 >> begin captured logging << >>> >>> 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] >>> (MainThread) Created a persistent dict with VGTagMetadataRW backend >>> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >>> (MainThread) read lines (VGTagMetadataRW)=[] >>> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >>> (MainThread) Empty metadata >>> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] >>> (MainThread) Starting transaction >>> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] >>> (MainThread) Flushing changes >>> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] >>> (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', >>> 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', >>> 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VERSION=3', >>> '_SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139'] >>> 12:33:20 2016-09-28 05:32:03,257 DEBUG >>> [storage.Metadata.VGTagMetadataRW] (MainThread) Updating metadata >>> adding=MDT_POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d, >>> MDT__SHA_CKSUM=3f26f105271d3f7af6ea7458fb179418e7f9c139, >>> MDT_SDUU
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: > > On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer wrote: > > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren wrote: > >> The CI setup did not change recently. > > > > Great > > > >> All standard-CI jobs run inside mock (chroot) which is stored on top > >> of a regular FS, so they should not be affected by the slave OS at all > >> as far as FS settings go. > >> > >> But perhaps some slave-OS/mock-OS combination is acting strangely, so > >> could you be more specific and point to particular job runs that fail? > > > > This jobs failed, but it was deleted (I get 404 now): > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/ > > Oops, wrong build. > > This is the failing build: > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console > > And this is probably the reason - using a ram disk: > > 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > integ-tests ram_disk fc23) in workspace > /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > We cannot run the storage tests using a ramdisk. We are creating > (tiny) volumes and storage domains and doing copies, this code cannot > work with ramdisk. Will it work on zram? What if we configure ram based iSCSI targets? Y. > > Attaching full console log before the job is deleted. > > > > > Then I triggered the tests and they passed: > > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2548/ > > > > Thanks > > Nir > > > >> > >> On 28 September 2016 at 22:00, Nir Soffer wrote: > >>> Hi all, > >>> > >>> It seems that the CI setup has changed, and /var/tmp is using now tempfs. > >>> > >>> This is not compatible with vdsm tests, assuming that /var/tmp is a real file > >>> system. This is the reason we do not use /tmp. > >>> > >>> We have lot of storage tests using directio, and directio cannot work on > >>> tempfs. > >>> > >>> Please check the slaves and make sure /var/tmp is using file system supporting > >>> directio. > >>> > >>> See example failure bellow. > >>> > >>> Nir > >>> > >>> > >>> > >>> 12:33:20 == > >>> 12:33:20 ERROR: test_create_fail_creating_lease > >>> (storage_volume_artifacts_test.BlockVolumeArtifactsTests) > >>> 12:33:20 -- > >>> 12:33:20 Traceback (most recent call last): > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/tests/storage_volume_artifacts_test.py", > >>> line 485, in test_create_fail_creating_lease > >>> 12:33:20 *BASE_PARAMS[sc.RAW_FORMAT]) > >>> 12:33:20 File "/usr/lib64/python2.7/unittest/case.py", line 513, in > >>> assertRaises > >>> 12:33:20 callableObj(*args, **kwargs) > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", > >>> line 391, in create > >>> 12:33:20 desc, parent) > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/sdm/volume_artifacts.py", > >>> line 482, in _create_metadata > >>> 12:33:20 sc.LEGAL_VOL) > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", > >>> line 427, in newMetadata > >>> 12:33:20 cls.createMetadata(metaId, meta.legacy_info()) > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/volume.py", > >>> line 420, in createMetadata > >>> 12:33:20 cls._putMetadata(metaId, meta) > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/vdsm/storage/blockVolume.py", > >>> line 242, in _putMetadata > >>> 12:33:20 f.write(data) > >>> 12:33:20 File > >>> "/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/lib/vdsm/storage/directio.py", > >>> line 161, in write > >>> 12:33:20 raise OSError(err, msg) > >>> 12:33:20 OSError: [Errno 22] Invalid argument > >>> 12:33:20 >> begin captured logging << > >>> 12:33:20 2016-09-28 05:32:03,254 DEBUG [storage.PersistentDict] > >>> (MainThread) Created a persistent dict with VGTagMetadataRW backend > >>> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] > >>> (MainThread) read lines (VGTagMetadataRW)=[] > >>> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] > >>> (MainThread) Empty metadata > >>> 12:33:20 2016-09-28 05:32:03,255 DEBUG [storage.PersistentDict] > >>> (MainThread) Starting transaction > >>> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] > >>> (MainThread) Flushing changes > >>> 12:33:20 2016-09-28 05:32:03,256 DEBUG [storage.PersistentDict] > >>> (MainThread) about to write lines (VGTagMetadataRW)=['CLASS=Data', > >>> 'POOL_UUID=52fe3782-ed7a-4d84-be3d-236faebdca2d', > >>> 'SDUUID=c088020e-91b2-45e6-85fd-eea3fce58764', 'VER
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul wrote: > On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: >> >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer wrote: >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren >> > wrote: >> >> The CI setup did not change recently. >> > >> > Great >> > >> >> All standard-CI jobs run inside mock (chroot) which is stored on top >> >> of a regular FS, so they should not be affected by the slave OS at all >> >> as far as FS settings go. >> >> >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so >> >> could you be more specific and point to particular job runs that fail? >> > >> > This jobs failed, but it was deleted (I get 404 now): >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/2530/ >> >> Oops, wrong build. >> >> This is the failing build: >> >> >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/1054/console >> >> And this is probably the reason - using a ram disk: >> >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical >> integ-tests ram_disk fc23) in workspace >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 >> >> We cannot run the storage tests using a ramdisk. We are creating >> (tiny) volumes and storage domains and doing copies, this code cannot >> work with ramdisk. > > Will it work on zram? > What if we configure ram based iSCSI targets? I don't know, but it is easy to test - it this works the tests will work: dd if=/dev/zero of=file bs=512 count=1 oflag=direct ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Evgheni, Can you try switching the current RAM drive with zram? On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer wrote: > On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul wrote: > > On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: > >> > >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer > wrote: > >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren > >> > wrote: > >> >> The CI setup did not change recently. > >> > > >> > Great > >> > > >> >> All standard-CI jobs run inside mock (chroot) which is stored on top > >> >> of a regular FS, so they should not be affected by the slave OS at > all > >> >> as far as FS settings go. > >> >> > >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so > >> >> could you be more specific and point to particular job runs that > fail? > >> > > >> > This jobs failed, but it was deleted (I get 404 now): > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > x86_64/2530/ > >> > >> Oops, wrong build. > >> > >> This is the failing build: > >> > >> > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- > x86_64/1054/console > >> > >> And this is probably the reason - using a ram disk: > >> > >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > >> integ-tests ram_disk fc23) in workspace > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > >> > >> We cannot run the storage tests using a ramdisk. We are creating > >> (tiny) volumes and storage domains and doing copies, this code cannot > >> work with ramdisk. > > > > Will it work on zram? > > What if we configure ram based iSCSI targets? > > I don't know, but it is easy to test - it this works the tests will work: > > dd if=/dev/zero of=file bs=512 count=1 oflag=direct > ___ > Infra mailing list > in...@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Hi, Indeed the proposed dd test does not work on zRAM slaves. Can we modify the job not to run on nodes with ram_disk label? The node will be offline for now until we agree on what to do. An option is to abandon RAM disks completely as we didn't find any performance benefits from using them so far. Regards, Evgheni Dereveanchin - Original Message - From: "Eyal Edri" To: "Nir Soffer" , "Evgheni Dereveanchin" Cc: "Yaniv Kaul" , "devel" , "infra" Sent: Thursday, 29 September, 2016 8:08:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI Evgheni, Can you try switching the current RAM drive with zram? On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer wrote: > On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul wrote: > > On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: > >> > >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer > wrote: > >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren > >> > wrote: > >> >> The CI setup did not change recently. > >> > > >> > Great > >> > > >> >> All standard-CI jobs run inside mock (chroot) which is stored on top > >> >> of a regular FS, so they should not be affected by the slave OS at > all > >> >> as far as FS settings go. > >> >> > >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so > >> >> could you be more specific and point to particular job runs that > fail? > >> > > >> > This jobs failed, but it was deleted (I get 404 now): > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > x86_64/2530/ > >> > >> Oops, wrong build. > >> > >> This is the failing build: > >> > >> > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- > x86_64/1054/console > >> > >> And this is probably the reason - using a ram disk: > >> > >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > >> integ-tests ram_disk fc23) in workspace > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > >> > >> We cannot run the storage tests using a ramdisk. We are creating > >> (tiny) volumes and storage domains and doing copies, this code cannot > >> work with ramdisk. > > > > Will it work on zram? > > What if we configure ram based iSCSI targets? > > I don't know, but it is easy to test - it this works the tests will work: > > dd if=/dev/zero of=file bs=512 count=1 oflag=direct > ___ > Infra mailing list > in...@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ) ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Sep 29, 2016 10:28 AM, "Evgheni Dereveanchin" wrote: > > Hi, > > Indeed the proposed dd test does not work on zRAM slaves. > Can we modify the job not to run on nodes with ram_disk label? Are those zram based or ram based *virtio-blk* disks, or zram/ram disks within the VMs? The former should work. The latter - no idea. > > The node will be offline for now until we agree on what to do. > An option is to abandon RAM disks completely as we didn't find > any performance benefits from using them so far. That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours. Y. > > Regards, > Evgheni Dereveanchin > > - Original Message - > From: "Eyal Edri" > To: "Nir Soffer" , "Evgheni Dereveanchin" < edere...@redhat.com> > Cc: "Yaniv Kaul" , "devel" , "infra" < in...@ovirt.org> > Sent: Thursday, 29 September, 2016 8:08:45 AM > Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI > > Evgheni, > Can you try switching the current RAM drive with zram? > > On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer wrote: > > > On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul wrote: > > > On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: > > >> > > >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer > > wrote: > > >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren > > >> > wrote: > > >> >> The CI setup did not change recently. > > >> > > > >> > Great > > >> > > > >> >> All standard-CI jobs run inside mock (chroot) which is stored on top > > >> >> of a regular FS, so they should not be affected by the slave OS at > > all > > >> >> as far as FS settings go. > > >> >> > > >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so > > >> >> could you be more specific and point to particular job runs that > > fail? > > >> > > > >> > This jobs failed, but it was deleted (I get 404 now): > > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > x86_64/2530/ > > >> > > >> Oops, wrong build. > > >> > > >> This is the failing build: > > >> > > >> > > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- > > x86_64/1054/console > > >> > > >> And this is probably the reason - using a ram disk: > > >> > > >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > > >> integ-tests ram_disk fc23) in workspace > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > >> > > >> We cannot run the storage tests using a ramdisk. We are creating > > >> (tiny) volumes and storage domains and doing copies, this code cannot > > >> work with ramdisk. > > > > > > Will it work on zram? > > > What if we configure ram based iSCSI targets? > > > > I don't know, but it is easy to test - it this works the tests will work: > > > > dd if=/dev/zero of=file bs=512 count=1 oflag=direct > > ___ > > Infra mailing list > > in...@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Hi Yaniv, this is a physical server with work directories created on a zRAM device, here's the patch: https://gerrit.ovirt.org/#/c/62249/2/site/ovirt_jenkins_slave/templates/prepare-ram-disk.service.erb I'll still need to read up on this but the only slave having this class (ovirt-srv08) is now offline and should not cause issues. I tested on VM slaves and did not see errors from the dd test command you provided. Please tell me if you see errors on other nodes and I'll check what's going on but it must be something else than RAM disks. Regards, Evgheni Dereveanchin - Original Message - From: "Yaniv Kaul" To: "Evgheni Dereveanchin" Cc: "infra" , "devel" , "Eyal Edri" , "Nir Soffer" Sent: Thursday, 29 September, 2016 9:32:45 AM Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI On Sep 29, 2016 10:28 AM, "Evgheni Dereveanchin" wrote: > > Hi, > > Indeed the proposed dd test does not work on zRAM slaves. > Can we modify the job not to run on nodes with ram_disk label? Are those zram based or ram based *virtio-blk* disks, or zram/ram disks within the VMs? The former should work. The latter - no idea. > > The node will be offline for now until we agree on what to do. > An option is to abandon RAM disks completely as we didn't find > any performance benefits from using them so far. That's very surprising. On my case it doubles the performance, at least. But I assume my storage (single disk) is far slower than yours. Y. > > Regards, > Evgheni Dereveanchin > > - Original Message - > From: "Eyal Edri" > To: "Nir Soffer" , "Evgheni Dereveanchin" < edere...@redhat.com> > Cc: "Yaniv Kaul" , "devel" , "infra" < in...@ovirt.org> > Sent: Thursday, 29 September, 2016 8:08:45 AM > Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI > > Evgheni, > Can you try switching the current RAM drive with zram? > > On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer wrote: > > > On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul wrote: > > > On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: > > >> > > >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer > > wrote: > > >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren > > >> > wrote: > > >> >> The CI setup did not change recently. > > >> > > > >> > Great > > >> > > > >> >> All standard-CI jobs run inside mock (chroot) which is stored on top > > >> >> of a regular FS, so they should not be affected by the slave OS at > > all > > >> >> as far as FS settings go. > > >> >> > > >> >> But perhaps some slave-OS/mock-OS combination is acting strangely, so > > >> >> could you be more specific and point to particular job runs that > > fail? > > >> > > > >> > This jobs failed, but it was deleted (I get 404 now): > > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > x86_64/2530/ > > >> > > >> Oops, wrong build. > > >> > > >> This is the failing build: > > >> > > >> > > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- > > x86_64/1054/console > > >> > > >> And this is probably the reason - using a ram disk: > > >> > > >> 12:24:53 Building remotely on ovirt-srv08.phx.ovirt.org (phx physical > > >> integ-tests ram_disk fc23) in workspace > > >> /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 > > >> > > >> We cannot run the storage tests using a ramdisk. We are creating > > >> (tiny) volumes and storage domains and doing copies, this code cannot > > >> work with ramdisk. > > > > > > Will it work on zram? > > > What if we configure ram based iSCSI targets? > > > > I don't know, but it is easy to test - it this works the tests will work: > > > > dd if=/dev/zero of=file bs=512 count=1 oflag=direct > > ___ > > Infra mailing list > > in...@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
zram does not support direct IO (tested, indeed fails). What I do is host the VMs there, though - this is working - but I'm using Lago (and not oVirt). does oVirt need direct IO for the temp disks? I thought we are doing them on the libvirt level? This is the command I use: sudo modprobe zram num_devices=1 && sudo zramctl --find --size 12G && sudo mkfs.xfs -K /dev/zram0 && sudo mount -o nobarrier /dev/zram0 /home/zram && sudo chmod 777 /home/zram And then I run lago with: ./run_suite.sh -o /home/zram basic_suite_master Y. On Thu, Sep 29, 2016 at 10:47 AM, Evgheni Dereveanchin wrote: > Hi Yaniv, > > this is a physical server with work directories > created on a zRAM device, here's the patch: > https://gerrit.ovirt.org/#/c/62249/2/site/ovirt_jenkins_ > slave/templates/prepare-ram-disk.service.erb > > I'll still need to read up on this but the only > slave having this class (ovirt-srv08) is now offline > and should not cause issues. I tested on VM slaves > and did not see errors from the dd test command you provided. > > Please tell me if you see errors on other nodes and I'll > check what's going on but it must be something else than RAM disks. > > Regards, > Evgheni Dereveanchin > > - Original Message - > From: "Yaniv Kaul" > To: "Evgheni Dereveanchin" > Cc: "infra" , "devel" , "Eyal Edri" < > ee...@redhat.com>, "Nir Soffer" > Sent: Thursday, 29 September, 2016 9:32:45 AM > Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI > > On Sep 29, 2016 10:28 AM, "Evgheni Dereveanchin" > wrote: > > > > Hi, > > > > Indeed the proposed dd test does not work on zRAM slaves. > > Can we modify the job not to run on nodes with ram_disk label? > > Are those zram based or ram based *virtio-blk* disks, or zram/ram disks > within the VMs? > The former should work. The latter - no idea. > > > > > The node will be offline for now until we agree on what to do. > > An option is to abandon RAM disks completely as we didn't find > > any performance benefits from using them so far. > > That's very surprising. On my case it doubles the performance, at least. > But I assume my storage (single disk) is far slower than yours. > Y. > > > > > Regards, > > Evgheni Dereveanchin > > > > - Original Message - > > From: "Eyal Edri" > > To: "Nir Soffer" , "Evgheni Dereveanchin" < > edere...@redhat.com> > > Cc: "Yaniv Kaul" , "devel" , "infra" > < > in...@ovirt.org> > > Sent: Thursday, 29 September, 2016 8:08:45 AM > > Subject: Re: [ovirt-devel] [VDSM] All tests using directio fail on CI > > > > Evgheni, > > Can you try switching the current RAM drive with zram? > > > > On Wed, Sep 28, 2016 at 11:43 PM, Nir Soffer wrote: > > > > > On Wed, Sep 28, 2016 at 11:39 PM, Yaniv Kaul wrote: > > > > On Sep 28, 2016 11:37 PM, "Nir Soffer" wrote: > > > >> > > > >> On Wed, Sep 28, 2016 at 11:20 PM, Nir Soffer > > > wrote: > > > >> > On Wed, Sep 28, 2016 at 10:31 PM, Barak Korren < > bkor...@redhat.com> > > > >> > wrote: > > > >> >> The CI setup did not change recently. > > > >> > > > > >> > Great > > > >> > > > > >> >> All standard-CI jobs run inside mock (chroot) which is stored on > top > > > >> >> of a regular FS, so they should not be affected by the slave OS > at > > > all > > > >> >> as far as FS settings go. > > > >> >> > > > >> >> But perhaps some slave-OS/mock-OS combination is acting > strangely, so > > > >> >> could you be more specific and point to particular job runs that > > > fail? > > > >> > > > > >> > This jobs failed, but it was deleted (I get 404 now): > > > >> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24- > > > x86_64/2530/ > > > >> > > > >> Oops, wrong build. > > > >> > > > >> This is the failing build: > > > >> > > > >> > > > >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7- > > > x86_64/1054/console > > > >> > > > >> And this is probably the reason - using a ram disk: > > > >> > > > >> 12:24:53 Bu
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On 29 September 2016 at 11:08, Yaniv Kaul wrote: > zram does not support direct IO (tested, indeed fails). > What I do is host the VMs there, though - this is working - but I'm using > Lago (and not oVirt). does oVirt need direct IO for the temp disks? I > thought we are doing them on the libvirt level? Its not oVirt that needs those, its some specific VDSM unit tests. -- Barak Korren bkor...@redhat.com RHEV-CI Team ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Thu, Sep 29, 2016 at 11:21 AM, Barak Korren wrote: > On 29 September 2016 at 11:08, Yaniv Kaul wrote: >> zram does not support direct IO (tested, indeed fails). >> What I do is host the VMs there, though - this is working - but I'm using >> Lago (and not oVirt). does oVirt need direct IO for the temp disks? I >> thought we are doing them on the libvirt level? > > Its not oVirt that needs those, its some specific VDSM unit tests. oVirt needs use directio when accessing shared storage. In the tests we have small integration tests that create storage domains and volume metadata using directio, or test code that used to work with directio (e.g. vdsm.storage.directio.py). Same for ovirt-imageio tests, most tests uses directio. And there are also some ioprocess tests using directio. We can use a loop device for these tests: $ truncate -s 10G /tmp/backing-file $ sudo losetup -f /tmp/backing-file --show /dev/loop0 $ sudo time dd if=/dev/zero of=/dev/loop0 bs=1M count=1024 oflag=direct 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.408083 s, 2.6 GB/s 0.00user 0.10system 0:00.40elapsed 25%CPU (0avgtext+0avgdata 3128maxresident)k 0inputs+2097152outputs (0major+333minor)pagefaults 0swaps Maybe we can mount a loop device at /var/tmp in the test environment? Nir ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
> > > > The node will be offline for now until we agree on what to do. > > An option is to abandon RAM disks completely as we didn't find > > any performance benefits from using them so far. > > That's very surprising. On my case it doubles the performance, at least. > But I assume my storage (single disk) is far slower than yours. > What amount of RAM you had available to Linux file system cache and were there any previous runs so Linux were able to put any mock caches into the RAM cache? Besides the possible difference in disk speeds I think the second factor is this Linux fs cache that basically create an analog of RAM disk on the fly. Those two things might explain why we do not see any performance improvement from RAM drives in our case. Anton. ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
On Thu, Sep 29, 2016 at 12:55 PM, Anton Marchukov wrote: > >> > The node will be offline for now until we agree on what to do. >> > An option is to abandon RAM disks completely as we didn't find >> > any performance benefits from using them so far. >> >> That's very surprising. On my case it doubles the performance, at least. >> But I assume my storage (single disk) is far slower than yours. >> > > What amount of RAM you had available to Linux file system cache and were > there any previous runs so Linux were able to put any mock caches into the > RAM cache? > I don't do mock. And if I run everything in RAM (whether directly under /dev/shm/ or in a zram disk), I honestly don't need the Linux system cache. > > Besides the possible difference in disk speeds I think the second factor > is this Linux fs cache that basically create an analog of RAM disk on the > fly. > Well, theoretically, if you have enough RAM and you keep re-running, many of the data is indeed going to be cached. I'd argue that it's a better use of RAM to just run it there. > > Those two things might explain why we do not see any performance > improvement from RAM drives in our case. > Indeed. Y. > > Anton. > > ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Re: [ovirt-devel] [VDSM] All tests using directio fail on CI
Hello Yaniv. I don't do mock. And if I run everything in RAM (whether directly under > /dev/shm/ or in a zram disk), I honestly don't need the Linux > system cache. > >> >> Besides the possible difference in disk speeds I think the second factor >> is this Linux fs cache that basically create an analog of RAM disk on the >> fly. >> > > Well, theoretically, if you have enough RAM and you keep re-running, many > of the data is indeed going to be cached. I'd argue that it's a better use > of RAM to just run it there. > > Ok then I guess we traced down the reason. I did not do any tests to confirm this yet but I believe the "mock" is causing this effect. On our side we use mock that is at the moment essential part of the standard CI offering. Mock utilizes file system cache of the chroots and those cashes are shared between runs that along with Linux fs cache means that they are probably already were identified to be good candidates for RAM fs cache and put there. So since mock has own cache it should be trivial for Linux to effectively identify them and put into RAM and this benefits all runs that use mock. If we remove the mock than it is completely different. Anton. -- Anton Marchukov Senior Software Engineer - RHEV CI - Red Hat ___ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
