Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
If anyone has encountered this issue - I was graciously given a solution by some of the gluster folks. The short of it is: gluster volume set yournovashared remote-dio on https://ask.openstack.org/en/question/25371/gluster-volume-for-nova-instance/ On Mar 18, 2014, at 5:04 PM, Ryan Aydelott ry...@anl.gov wrote: I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
Resending it as I suppose it was not delievered :( Hi Ryan/Narayan, Apologies for top posting. After today's call, we had a chat with Vijay and he added a hint. You need to enable/*remote-dio*/ for that volume that serves for virt-store. gluster volume set vol-name remote-dio enable Hope this could solve your issue most probably. Could you test it with this option 'on' and let us know the results ? -- sas On 03/20/2014 12:40 AM, Aydelott, Ryan M. wrote: So a little further down the road I have destroyed/rebuilt the gluster volume and restarted glusterd on each of the nodes. Shortly after the instances directory is created and the console.log file is placed, I receive the following fuse errors on the client: [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: remote operation failed: Invalid argument [2014-03-19 19:06:35.326487] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: remote operation failed: Invalid argument [2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106504: WRITE = -1 (Invalid argument) [2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106505: FTRUNCATE() ERR = -1 (Bad file descriptor) [2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106507: WRITE = -1 (Bad file descriptor) [2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106508: FTRUNCATE() ERR = -1 (Bad file descriptor) [2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106510: WRITE = -1 (Bad file descriptor) [2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106511: FTRUNCATE() ERR = -1 (Bad file descriptor) [2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106513: WRITE = -1 (Bad file descriptor) [2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106514: FTRUNCATE() ERR = -1 (Bad file descriptor) One one of the servers the normal messages apply: [2014-03-13 21:47:33.735042] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:17:42.763076] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:17:42.763101] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:50:02.206635] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:50:23.037886] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:50:23.051446] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:50:23.051466] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:56:22.215898] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:56:22.215923] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:57:04.821397] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 3.4.2) [2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:32.135629] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:32.135652] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server:
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
This rings a bell... I mapped a remote nfs share to one of my nodes and mapped the local nfs share, to the local gluster volume as well and started to copy (with cp) files from one mount to the other. Somewhere down the line all stalled, the server was irresponsive, even restarting glusterd solved nothing. Had to reboot the nodes. CentOS 6.5 and Gluster 3.4.2. Since then, I haven't tried those copies, but, I also, I am now using the native glusterfs client just in case. On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell rpowe...@uchicago.edu wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 I have not seen that with the fuse client. I had it happen repeatedly when we were using gluster/NFS for /var/lib/nova/instances and/or /var/lib/nova/instances/_base. The access to instances-XX/disk (or its qcow2 backing image) would hang completely or be very slow under heavy load. The VM interprets the delay as a problem on /dev/vda and remounts RO. On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote: I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users - -- Raymond Powell Systems Administrator Laboratory for Advanced Computing University of Chicago, KCBD 10160 900 East 57th Street Chicago, IL 60637 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9 lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/ HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe +UhAq6+74ie1MokNssB8 =un7j -END PGP SIGNATURE- ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
So a little further down the road I have destroyed/rebuilt the gluster volume and restarted glusterd on each of the nodes. Shortly after the instances directory is created and the console.log file is placed, I receive the following fuse errors on the client: [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: remote operation failed: Invalid argument [2014-03-19 19:06:35.326487] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: remote operation failed: Invalid argument [2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106504: WRITE = -1 (Invalid argument) [2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106505: FTRUNCATE() ERR = -1 (Bad file descriptor) [2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106507: WRITE = -1 (Bad file descriptor) [2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106508: FTRUNCATE() ERR = -1 (Bad file descriptor) [2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106510: WRITE = -1 (Bad file descriptor) [2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106511: FTRUNCATE() ERR = -1 (Bad file descriptor) [2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106513: WRITE = -1 (Bad file descriptor) [2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106514: FTRUNCATE() ERR = -1 (Bad file descriptor) One one of the servers the normal messages apply: [2014-03-13 21:47:33.735042] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:17:42.763076] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:17:42.763101] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 [2014-03-18 21:50:02.206635] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:50:23.037886] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 3.4.2) [2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:50:23.051446] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:50:23.051466] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 [2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:56:22.215898] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:56:22.215923] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 [2014-03-18 21:57:04.821397] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 3.4.2) [2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:32.135629] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:32.135652] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 [2014-03-19 16:48:47.712973] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 (version: 3.4.2) [2014-03-19 18:07:44.332240] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 [2014-03-19 18:07:44.332272] I
[Gluster-users] Openstack Havana Gluster /var/lib/instances issues
I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan signature.asc Description: Message signed with OpenPGP using GPGMail ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
It might be more useful to look at your hypervisor log and/or client log. On March 18, 2014 3:04:48 PM PDT, Aydelott, Ryan M. ry...@mcs.anl.gov wrote: I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Openstack Havana Gluster /var/lib/instances issues
From the pastebin it seems like its getting a ton of I/O errors. It would be good to look at gluster logs (/var/log/glusterfs) and whether that carries any errors, since I/O error is a generic error that glusterfs can return for many reasons. On Wed, Mar 19, 2014 at 3:34 AM, Aydelott, Ryan M. ry...@mcs.anl.govwrote: I was wondering if anyone had seen similar issues before, when booting a vm with a root volume on local disk things work fine. When I try to do the same with /var/lib/nova/instances mounted via gluster/fuse I get the following: http://paste.ubuntu.com/7116594/ I have of course insured perms were correct with chown -R nova:nova /var/lib/nova/instances, additionally app armor is completely unloaded/disabled. Has anyone encountered this before? Gluster version is 3.4.2 -ryan ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
