On 28.2.2014, 20:05, Nir Soffer wrote: > ----- Original Message ----- >> From: "Boyan Tabakov" <bl...@alslayer.net> >> To: "Nir Soffer" <nsof...@redhat.com> >> Cc: users@ovirt.org >> Sent: Tuesday, February 25, 2014 11:53:45 AM >> Subject: Re: [Users] SD Disk's Logical Volume not visible/activated on some >> nodes >> >> Hello, >> >> On 22.2.2014, 22:19, Nir Soffer wrote: >>> ----- Original Message ----- >>>> From: "Boyan Tabakov" <bl...@alslayer.net> >>>> To: "Nir Soffer" <nsof...@redhat.com> >>>> Cc: users@ovirt.org >>>> Sent: Wednesday, February 19, 2014 7:18:36 PM >>>> Subject: Re: [Users] SD Disk's Logical Volume not visible/activated on >>>> some nodes >>>> >>>> Hello, >>>> >>>> On 19.2.2014, 17:09, Nir Soffer wrote: >>>>> ----- Original Message ----- >>>>>> From: "Boyan Tabakov" <bl...@alslayer.net> >>>>>> To: users@ovirt.org >>>>>> Sent: Tuesday, February 18, 2014 3:34:49 PM >>>>>> Subject: [Users] SD Disk's Logical Volume not visible/activated on some >>>>>> nodes >>>>> >>>>>> Consequently, when creating/booting >>>>>> a VM with the said disk attached, the VM fails to start on host2, >>>>>> because host2 can't see the LV. Similarly, if the VM is started on >>>>>> host1, it fails to migrate to host2. Extract from host2 log is in the >>>>>> end. The LV in question is 6b35673e-7062-4716-a6c8-d5bf72fe3280. >>>>>> >>>>>> As far as I could track quickly the vdsm code, there is only call to lvs >>>>>> and not to lvscan or lvchange so the host2 LVM doesn't fully refresh. > > lvs should see any change on the shared storage. > >>>>>> The only workaround so far has been to restart VDSM on host2, which >>>>>> makes it refresh all LVM data properly. > > When vdsm starts, it calls multipath -r, which ensure that we see all > physical volumes. > >>>>>> >>>>>> When is host2 supposed to pick up any newly created LVs in the SD VG? >>>>>> Any suggestions where the problem might be? >>>>> >>>>> When you create a new lv on the shared storage, the new lv should be >>>>> visible on the other host. Lets start by verifying that you do see >>>>> the new lv after a disk was created. >>>>> >>>>> Try this: >>>>> >>>>> 1. Create a new disk, and check the disk uuid in the engine ui >>>>> 2. On another machine, run this command: >>>>> >>>>> lvs -o vg_name,lv_name,tags >>>>> >>>>> You can identify the new lv using tags, which should contain the new disk >>>>> uuid. >>>>> >>>>> If you don't see the new lv from the other host, please provide >>>>> /var/log/messages >>>>> and /var/log/sanlock.log. >>>> >>>> Just tried that. The disk is not visible on the non-SPM node. >>> >>> This means that storage is not accessible from this host. >> >> Generally, the storage seems accessible ok. For example, if I restart >> the vdsmd, all volumes get picked up correctly (become visible in lvs >> output and VMs can be started with them). > > Lests repeat this test, but now, if you do not see the new lv, please > run: > > multipath -r > > And report the results. >
Running multipath -r helped and the disk was properly picked up by the second host. Is running multipath -r safe while host is not in maintenance mode? If yes, as a temporary workaround I can patch vdsmd to run multipath -r when e.g. monitoring the storage domain. >>>> Here's the full >>>> sanlock.log for that host: >> ... >>>> 0x7fc37c0008c0:0x7fc37c0008d0:0x7fc391f5f000 ioto 10 to_count 1 >>>> 2014-02-06 05:24:10+0200 563065 [31453]: s1 delta_renew read rv -202 >>>> offset 0 /dev/3307f6fa-dd58-43db-ab23-b1fb299006c7/ids >>> >>> Sanlock cannot write to the ids lockspace >> >> Which line shows that sanlock can't write? The messages are not very >> "human readable". > > The one above my comment at 2014-02-06 05:24:10+0200 > > I suggest to set sanlock debug level on the sanlock log to get more detailed > output. > > Edit /etc/sysconfig/sanlock and add: > > # -L 7: use debug level logging to sanlock log file > SANLOCKOPTS="$SANLOCKOPTS -L 7" > >>>> >>>> Last entry is from yesterday, while I just created a new disk. >>> >>> What was the status of this host in the engine from 2014-02-06 >>> 05:24:10+0200 to 2014-02-18 14:22:16? >>> >>> vdsm.log and engine.log for this time frame will make it more clear. >> >> Host was up and running. The vdsm and engine logs are quite large, as we >> were running some VM migrations between the hosts. Any pointers at what >> to look for? For example, I noticed many entries in engine.log like this: > > It will be hard to make any progress without the logs. > >> >> One warning that I keep seeing in vdsm logs on both nodes is this: >> >> Thread-1617881::WARNING::2014-02-24 >> 16:57:50,627::sp::1553::Storage.StoragePool::(getInfo) VG >> 3307f6fa-dd58-43db-ab23-b1fb299006c7's metadata size exceeded >> critical size: mdasize=134217728 mdafree=0 > > Can you share the output of the command bellow? > > lvs -o > uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name Here's the output for both hosts. host1: [root@host1 ~]# lvs -o uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count LV UUID LV Attr LSize VFree Ext #Ext Free LV Tags VMdaSize VMdaFree #LV #PV jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL 3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-ao--- 2.00g 114.62g 128.00m 1596 917 IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-ac73492ac465 128.00m 0 13 2 ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC 3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi------ 7.00g 114.62g 128.00m 1596 917 IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-000000000000 128.00m 0 13 2 YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n 57668a89-c5ca-4231-b18a-dd0607e459a5 -wi------ 10.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-f02d683f07f8 128.00m 0 13 2 FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf 637bdaf7-61cb-4feb-838d-24355367dcec -wi------ 7.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2feb223,MD_9 128.00m 0 13 2 9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE 65a711a2-d08a-496a-93de-e47aec19cfb1 -wi------ 12.00g 114.62g 128.00m 1596 917 IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000000000,MD_8 128.00m 0 13 2 Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd f4231952-76c5-4764-9c8b-ac73492ac465 -wi-ao--- 3.00g 114.62g 128.00m 1596 917 MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d42e7b760036 128.00m 0 13 2 u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0 f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-ao--- 40.00g 114.62g 128.00m 1596 917 MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-eae7f136d7e1 128.00m 0 13 2 h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids -wi-ao--- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases -wi-a---- 2.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master -wi-ao--- 1.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata -wi-a---- 512.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 JTDXAM-vMQI-UY0o-Axy9-UQKv-7Hl7-M9XSwQ mysql -wi-a---- 40.00g 0 4.00m 10241 0 188.00k 0 1 1 nQHcrF-KWhE-s8pY-90bc-jPSN-3qZR-pxHe0O iso-domain -wi-ao--- 10.00g 97.80g 4.00m 34596 25036 1020.00k 0 3 2 qElr6v-lPNE-Kvc5-0g9W-I41J-dsBD-7QfZLv root -wi-ao--- 19.53g 97.80g 4.00m 34596 25036 1020.00k 0 3 2 FSaRPB-WiZo-Ddqo-hPsB-jUe9-bf4v-eJBTJW swap -wi-ao--- 7.81g 97.80g 4.00m 34596 25036 1020.00k 0 3 2 host2: [root@host2 log]# lvs -o uuid,name,attr,size,vg_free,vg_extent_size,vg_extent_count,vg_free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count LV UUID LV Attr LSize VFree Ext #Ext Free LV Tags VMdaSize VMdaFree #LV #PV jGEpVm-oPW8-XyxI-l2yi-YF4X-qteQ-dm8SqL 3d362bf2-20f4-438d-9ba9-486bd2e8cedf -wi-a---- 2.00g 114.62g 128.00m 1596 917 IU_0227da98-34b2-4b0c-b083-d42e7b760036,MD_5,PU_f4231952-76c5-4764-9c8b-ac73492ac465 128.00m 0 13 2 ltJOs7-T7yE-faQd-MkD8-PS5F-wGXU-nmDQrC 3e147e83-0b0c-4cb8-b46f-67c8dc62756e -wi-ao--- 7.00g 114.62g 128.00m 1596 917 IU_bdf2c305-194c-45b5-bd62-1bf01b05e286,MD_7,PU_00000000-0000-0000-0000-000000000000 128.00m 0 13 2 YxR9Df-YfvQ-0W5s-5CnC-JGN6-gQug-vi5R4n 57668a89-c5ca-4231-b18a-dd0607e459a5 -wi-a---- 10.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,MD_10,IU_90f2b1e5-59f7-4bed-97a9-f02d683f07f8 128.00m 0 13 2 FeqoWr-aYs1-N4M6-Miss-lMec-afKk-7XJKpf 637bdaf7-61cb-4feb-838d-24355367dcec -wi-ao--- 7.00g 114.62g 128.00m 1596 917 PU_00000000-0000-0000-0000-000000000000,IU_e2dafaea-71f5-4f60-abc1-0c7bf2feb223,MD_9 128.00m 0 13 2 9xWhSv-g5ps-utzj-eXfz-G8Zz-BdkN-hN8wGE 65a711a2-d08a-496a-93de-e47aec19cfb1 -wi-ao--- 12.00g 114.62g 128.00m 1596 917 IU_91d741e3-19b7-40bd-8b96-de48caa161f1,PU_00000000-0000-0000-0000-000000000000,MD_8 128.00m 0 13 2 Isn11q-cokl-op59-vQcl-l4Rc-Uaeg-1lcGqd f4231952-76c5-4764-9c8b-ac73492ac465 -wi-a---- 3.00g 114.62g 128.00m 1596 917 MD_4,PU_00000000-0000-0000-0000-000000000000,IU_0227da98-34b2-4b0c-b083-d42e7b760036 128.00m 0 13 2 u80jcl-PPVX-nRp6-KDqH-ZcRI-IAuU-JMrqK0 f8f7d099-3e54-45c0-8f4b-3ee25901522e -wi-a---- 40.00g 114.62g 128.00m 1596 917 MD_6,PU_00000000-0000-0000-0000-000000000000,IU_d5a37063-66bb-4bf1-9269-eae7f136d7e1 128.00m 0 13 2 h9SB9I-J9Dm-mTIL-lT6T-ZxvF-8mHj-eAs59A ids -wi-ao--- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 rrycOM-WkBu-1VLg-RYrd-z7bO-8nS9-CkLpqc inbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 IDjyMT-Quxu-NxG8-dKfz-CSlc-i1EQ-o8rlSb leases -wi-a---- 2.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 sTxbVK-5YDT-H7nw-53b9-Av7J-YIzP-res2b1 master -wi-a---- 1.00g 114.62g 128.00m 1596 917 128.00m 0 13 2 2lwYGv-cZop-7Jj9-vYFI-RDEZ-1hKy-oXNbei metadata -wi-a---- 512.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 NznWRR-kLhp-HEMw-0c7P-9XIj-YiXb-oYFovC outbox -wi-a---- 128.00m 114.62g 128.00m 1596 917 128.00m 0 13 2 G2zjlF-QO3X-tznc-BZE8-mY63-8I9n-gZQ7k5 root -wi-ao--- 19.53g 4.00m 4.00m 7001 1 1020.00k 0 2 1 vtuPP3-itfy-DtFX-j2gJ-wZBI-3RpF-fsWBJc swap -wi-ao--- 7.81g 4.00m 4.00m 7001 1 1020.00k 0 2 1 > > I suggest that you open a bug and attach there engine.log, > /var/log/messages, vdsm.log and sanlock.log. > > Please also give detailed info on the host os, vdsm version etc. > > Nir > I'll gather logs and will create bug report. Thanks! Boyan
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users