Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
02.07.2016 03:02, Lindsay Mathieson пишет: On 1/07/2016 3:46 PM, Dmitry Melekhov wrote: It's strange, but we have operating-version=30710 in /var/lib/glusterd/glusterd.info Chiming in late, but the op-version has to be manually upgraded, it won't happen automatically. A safety precaution basically. Very strange, from my point of view... Thank you! ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
On 1/07/2016 3:46 PM, Dmitry Melekhov wrote: It's strange, but we have operating-version=30710 in /var/lib/glusterd/glusterd.info Chiming in late, but the op-version has to be manually upgraded, it won't happen automatically. A safety precaution basically. -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
>01.07.2016 11:35, Dmitry Melekhov пишет: >> 01.07.2016 11:21, Dmitry Melekhov пишет: >>> OK, in our case we solved this problem by installing 3.7.12 again and >>> restarting libvirtd after that. >>> Now we have all this working. >>> As I said this is centos7. >>> >>> I guess we did not restart libvirtd during upgrade to 3.7.11 too, so >>> I don't know why libvirt doesn't work >>> with 3.7.11 libs... :-( >>> >> >OK, finally downgrade to 3.7.11 is successful. >Had to kill all gluster processess before downgrade, for some reason >systemd systemctl does not stop them, >and upgrade scripts from rpms too... > >Thank you! I think you wanted to first stop all of the volumes: gluster volume stop YourVolume1 gluster volume stop yourVolume2 gluster volume stop … Then on each host, systemctl stop glusterd yum downgrade glusterfs-3.7.11 ... ... As noted in one of these threads, need to revise the op_version: vi /var/lib/glusterd/glusterd.info I had: operating-version=30712 Revised to: operating-version=30710 start it up: systemctl daemon-reload ; systemctl start glusterd (my CentOS 7 systemctl complained about wanting the “systemctl daemon-reload” first.) Cheers! > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
On 1/07/2016 7:22 PM, Dmitry Melekhov wrote: OK, finally downgrade to 3.7.11 is successful. Had to kill all gluster processess before downgrade, for some reason systemd systemctl does not stop them, and upgrade scripts from rpms too... Phew! -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
01.07.2016 11:35, Dmitry Melekhov пишет: 01.07.2016 11:21, Dmitry Melekhov пишет: OK, in our case we solved this problem by installing 3.7.12 again and restarting libvirtd after that. Now we have all this working. As I said this is centos7. I guess we did not restart libvirtd during upgrade to 3.7.11 too, so I don't know why libvirt doesn't work with 3.7.11 libs... :-( OK, finally downgrade to 3.7.11 is successful. Had to kill all gluster processess before downgrade, for some reason systemd systemctl does not stop them, and upgrade scripts from rpms too... Thank you! ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
01.07.2016 11:21, Dmitry Melekhov пишет: OK, in our case we solved this problem by installing 3.7.12 again and restarting libvirtd after that. Now we have all this working. As I said this is centos7. I guess we did not restart libvirtd during upgrade to 3.7.11 too, so I don't know why libvirt doesn't work with 3.7.11 libs... :-( Oops, by "solved" I mean that now we can start VMs :-D But they don't work, they dies: [2016-07-01 06:57:51.836023] I [MSGID: 108019] [afr-transaction.c:1339:afr_post_blocking_inodelk_cbk] 0-pool-replicate-0: Blocking inodelks failed. block I/O error in device 'drive-ide0-0-0': Operation not permitted (1) [2016-07-01 06:57:51.837022] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-7: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837086] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-6: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837306] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-8: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837658] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-6: remote operation failed [Invalid argument] [2016-07-01 06:57:51.837940] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-7: remote operation failed [Invalid argument] [2016-07-01 06:57:51.838533] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-pool-client-8: remote operation failed [Invalid argument] [2016-07-01 06:57:51.838583] W [MSGID: 108019] [afr-lk-common.c:1082:is_blocking_locks_count_sufficient] 0-pool-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:c45544e7-f449-48cf-bbc1 -cfd9f3ebc401. End on gluster side in brick log: [2016-07-01 07:31:44.013224] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325937: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013306] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325938: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013410] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325939: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013485] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325940: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013535] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325941: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013628] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325943: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013703] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325944: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013732] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325945: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013734] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325946: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013782] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325947: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013793] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325948: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013895] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325949: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] [2016-07-01 07:31:44.013918] E [MSGID: 115053] [server-rpc-fops.c:314:server_finodelk_cbk] 0-pool-server: 3325950: FINODELK 0 (52dfd80c-ef02-4749-a43a-db9573113420) ==> (Invalid argument) [Invalid argument] So, gfapi is broken... ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
01.07.2016 10:51, Dmitry Melekhov пишет: 01.07.2016 09:46, Dmitry Melekhov пишет: 01.07.2016 09:23, Lindsay Mathieson пишет: How did you do your downgrade? downloaded packages from https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.11/CentOS/epel-7.2/x86_64/ and https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.11/CentOS/epel-7.2/noarch/ and yum downgrade * from this directory OK, forgot to restart libvirtd after this downgrade. Rebooted one of servers to be completely sure. Now I get virsh create w8test.xml error: Failed to create domain from w8test.xml error: End of file while reading data: Input/output error :-( looks like some problems with libgfs packages for centos... OK, in our case we solved this problem by installing 3.7.12 again and restarting libvirtd after that. Now we have all this working. As I said this is centos7. I guess we did not restart libvirtd during upgrade to 3.7.11 too, so I don't know why libvirt doesn't work with 3.7.11 libs... :-( ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
01.07.2016 09:46, Dmitry Melekhov пишет: 01.07.2016 09:23, Lindsay Mathieson пишет: How did you do your downgrade? downloaded packages from https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.11/CentOS/epel-7.2/x86_64/ and https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.11/CentOS/epel-7.2/noarch/ and yum downgrade * from this directory OK, forgot to restart libvirtd after this downgrade. Rebooted one of servers to be completely sure. Now I get virsh create w8test.xml error: Failed to create domain from w8test.xml error: End of file while reading data: Input/output error :-( looks like some problems with libgfs packages for centos... ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
01.07.2016 09:23, Lindsay Mathieson пишет: How did you do your downgrade? downloaded packages from https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.11/CentOS/epel-7.2/x86_64/ and https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.11/CentOS/epel-7.2/noarch/ and yum downgrade * from this directory Had you changed the op-version to 3071? It's strange, but we have operating-version=30710 in /var/lib/glusterd/glusterd.info so I decided to not change it. I can mount volumes using fuse, but can't use libvirt, strange but true. If I write wrong pool name in VM config , then I get failed to initialize gluster connection to server: 'localhost': No such file or directory But if I write right pool name, but any file name then I get failed to initialize gluster connection to server: 'localhost': Invalid argument Looks like upgrade to 3.7.12 changed something in volumes config and this change is not compatible with gfapi, but I don't see any changes. Here is volume info for volume with VM images: Volume Name: pool Type: Replicate Volume ID: 6748e47c-4a2e-4a80-a42d-61b6917f4a41 Status: Started Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: father:/wall/pool/brick Brick2: son:/wall/pool/brick Brick3: spirit:/wall/pool/brick Options Reconfigured: network.ping-timeout: 10 performance.readdir-ahead: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off network.remote-dio: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.eager-lock: enable nfs.disable: on storage.owner-uid: 107 storage.owner-gid: 107 auth.allow: 127.0.0.1,192.168.22.26,192.168.22.27,192.168.22.28 features.barrier: disable May be there are any ideas what we can try to change? On 1 July 2016 at 15:03, Dmitry Melekhov wrote: 30.06.2016 15:42, Dmitry Melekhov пишет: 30.06.2016 15:40, Lindsay Mathieson пишет: On 30/06/2016 6:54 PM, Dmitry Melekhov wrote: After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument This looks similar to the errors we are seeing in the "3.7.12 disaster" thread, with the debian 3.7.12 packages. For myself things were fine with a rolling ugrade and running VM's, until we shut them down. Thank you! I guess we have the same issue here, although I see no errors in logs. As I see you reverted back to 3.7.11, we are planning to go this way tomorrow :-( Downgrade to 3.7.11 doesn't help in our case, don't know why yet. fuse mount works OK. But virsh still says error: failed to initialize gluster connection to server: 'localhost': Invalid argument. Looks like upgrade to 3.7.12 changed some lib, which was not upgraded before. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
On 1 July 2016 at 15:23, Lindsay Mathieson wrote: > Had you changed the op-version to 3071? 30712 I mean -- Lindsay ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
How did you do your downgrade? Had you changed the op-version to 3071? On 1 July 2016 at 15:03, Dmitry Melekhov wrote: > 30.06.2016 15:42, Dmitry Melekhov пишет: > >> 30.06.2016 15:40, Lindsay Mathieson пишет: >>> >>> On 30/06/2016 6:54 PM, Dmitry Melekhov wrote: After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument >>> >>> >>> This looks similar to the errors we are seeing in the "3.7.12 disaster" >>> thread, with the debian 3.7.12 packages. For myself things were fine with a >>> rolling ugrade and running VM's, until we shut them down. >>> >> Thank you! >> >> I guess we have the same issue here, although I see no errors in logs. >> >> As I see you reverted back to 3.7.11, we are planning to go this way >> tomorrow :-( >> > > Downgrade to 3.7.11 doesn't help in our case, don't know why yet. > fuse mount works OK. But virsh still says error: failed to initialize > gluster connection to server: 'localhost': Invalid argument. > Looks like upgrade to 3.7.12 changed some lib, which was not upgraded > before. > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users -- Lindsay ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
30.06.2016 15:42, Dmitry Melekhov пишет: 30.06.2016 15:40, Lindsay Mathieson пишет: On 30/06/2016 6:54 PM, Dmitry Melekhov wrote: After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument This looks similar to the errors we are seeing in the "3.7.12 disaster" thread, with the debian 3.7.12 packages. For myself things were fine with a rolling ugrade and running VM's, until we shut them down. Thank you! I guess we have the same issue here, although I see no errors in logs. As I see you reverted back to 3.7.11, we are planning to go this way tomorrow :-( Downgrade to 3.7.11 doesn't help in our case, don't know why yet. fuse mount works OK. But virsh still says error: failed to initialize gluster connection to server: 'localhost': Invalid argument. Looks like upgrade to 3.7.12 changed some lib, which was not upgraded before. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
30.06.2016 15:40, Lindsay Mathieson пишет: On 30/06/2016 6:54 PM, Dmitry Melekhov wrote: After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument This looks similar to the errors we are seeing in the "3.7.12 disaster" thread, with the debian 3.7.12 packages. For myself things were fine with a rolling ugrade and running VM's, until we shut them down. Thank you! I guess we have the same issue here, although I see no errors in logs. As I see you reverted back to 3.7.11, we are planning to go this way tomorrow :-( ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
On 30/06/2016 6:54 PM, Dmitry Melekhov wrote: After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument This looks similar to the errors we are seeing in the "3.7.12 disaster" thread, with the debian 3.7.12 packages. For myself things were fine with a rolling ugrade and running VM's, until we shut them down. -- Lindsay Mathieson ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
30.06.2016 13:22, Dmitry Melekhov пишет: 30.06.2016 13:09, Dmitry Melekhov пишет: 30.06.2016 12:54, Dmitry Melekhov пишет: Hello! After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument Disk config: It worked with 3.7.11 and still works for VMs still running. Could you tell me is there any solution for this problem? btw, I see in brick log: [2016-06-30 09:06:05.002914] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-pool-posix: null gfid for path (null) [2016-06-30 09:06:05.002971] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-pool-posix: lstat on null failed [Invalid argument] No, this is not related, because there are no new messages in log if I retry. And, yes, it is, I see this on other bricks. Looks like bug in 3.7.12. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
30.06.2016 13:09, Dmitry Melekhov пишет: 30.06.2016 12:54, Dmitry Melekhov пишет: Hello! After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument Disk config: It worked with 3.7.11 and still works for VMs still running. Could you tell me is there any solution for this problem? btw, I see in brick log: [2016-06-30 09:06:05.002914] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-pool-posix: null gfid for path (null) [2016-06-30 09:06:05.002971] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-pool-posix: lstat on null failed [Invalid argument] No, this is not related, because there are no new messages in log if I retry. ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] 3.7.12, centos 7 , libvirt problem
30.06.2016 12:54, Dmitry Melekhov пишет: Hello! After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument Disk config: It worked with 3.7.11 and still works for VMs still running. Could you tell me is there any solution for this problem? btw, I see in brick log: [2016-06-30 09:06:05.002914] E [MSGID: 113091] [posix.c:178:posix_lookup] 0-pool-posix: null gfid for path (null) [2016-06-30 09:06:05.002971] E [MSGID: 113018] [posix.c:196:posix_lookup] 0-pool-posix: lstat on null failed [Invalid argument] ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] 3.7.12, centos 7 , libvirt problem
Hello! After upgrade from 3.7.11 to 3.7.12 we can't start VMs after VM shutdown: virsh create w8test.xml error: Failed to create domain from w8test.xml error: failed to initialize gluster connection to server: 'localhost': Invalid argument Disk config: It worked with 3.7.11 and still works for VMs still running. Could you tell me is there any solution for this problem? ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
