Re: [Gluster-users] Upgrade from 3.5 to 3.6: issue with socket
I've seen this when the subprocesses running glustershd and nfs were not restarted after the upgrade. I've also seen it if nfs is disabled on all volumes. On 03/08/2015 09:10 PM, Krishnan Parthasarathi wrote: I just upgraded from 3.5.3 to 3.6.2 and have issues mounting my volume on a client. On the server side I found this error message which might be the cause of my issues: Could you describe the issues you are facing? [2015-03-08 13:22:36.383715] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/6b8f1f2526c6af8a87f1bb611ae5a86f.socket failed (Invalid argument) Anyony has any idea what that could be ? This log message indicates that there was no (server) process listening on the socket identified by the path, when glusterd tried to connect(3). This happens if the connect(3) happened sooner than the server process starts listening. glusterd will attempt a (re)connect within the next 3 seconds and should successfully establish connection with the server. This wouldn't result in any sort of 'service disruption' in the I/O path. Could you confirm that this log snippet is from glusterd's log file? ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Upgrade from 3.5 to 3.6: issue with socket
I just upgraded from 3.5.3 to 3.6.2 and have issues mounting my volume on a client. On the server side I found this error message which might be the cause of my issues: Could you describe the issues you are facing? [2015-03-08 13:22:36.383715] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/6b8f1f2526c6af8a87f1bb611ae5a86f.socket failed (Invalid argument) Anyony has any idea what that could be ? This log message indicates that there was no (server) process listening on the socket identified by the path, when glusterd tried to connect(3). This happens if the connect(3) happened sooner than the server process starts listening. glusterd will attempt a (re)connect within the next 3 seconds and should successfully establish connection with the server. This wouldn't result in any sort of 'service disruption' in the I/O path. Could you confirm that this log snippet is from glusterd's log file? ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster errors create zombie processes [LOGS ATTACHED]
I don't have volfiles, they are not on our machines as I said previously we don't have impact on gluster servers. I saw some graph that looks similiar to volume file on logs. I will paste it here but we don't really have any impact on that. We are just using client to connect to gluster servers, we are not in control of. *1: volume drslk-prod-client-0* * 2: type protocol/client* * 3: option ping-timeout 20* * 4: option remote-host brick13.gluster.iadm* * 5: option remote-subvolume /GLUSTERFS/drslk-prod* * 6: option transport-type socket* * 7: option frame-timeout 60* * 8: option send-gids true* * 9: end-volume* * 10: * * 11: volume drslk-prod-client-1* * 12: type protocol/client* * 13: option ping-timeout 20* * 14: option remote-host brick14.gluster.iadm* * 15: option remote-subvolume /GLUSTERFS/drslk-prod* * 16: option transport-type socket* * 17: option frame-timeout 60* * 18: option send-gids true* * 19: end-volume* * 20: * * 21: volume drslk-prod-client-2* * 22: type protocol/client* * 23: option ping-timeout 20* * 24: option remote-host brick15.gluster.iadm* * 25: option remote-subvolume /GLUSTERFS/drslk-prod* * 26: option transport-type socket* * 27: option frame-timeout 60* * 28: option send-gids true* * 29: end-volume* * 30: * * 31: volume drslk-prod-replicate-0* * 32: type cluster/replicate* * 33: option read-hash-mode 2* * 34: option data-self-heal-window-size 128* * 35: option quorum-type auto* * 36: subvolumes drslk-prod-client-0 drslk-prod-client-1 drslk-prod-client-2* * 37: end-volume* * 38: * * 39: volume drslk-prod-client-3* * 40: type protocol/client* * 41: option ping-timeout 20* * 42: option remote-host brick16.gluster.iadm* * 43: option remote-subvolume /GLUSTERFS/drslk-prod* * 44: option transport-type socket* * 45: option frame-timeout 60* * 46: option send-gids true* * 47: end-volume* * 48: * * 49: volume drslk-prod-client-4* * 50: type protocol/client* * 51: option ping-timeout 20* * 52: option remote-host brick17.gluster.iadm* * 53: option remote-subvolume /GLUSTERFS/drslk-prod* * 54: option transport-type socket* * 55: option frame-timeout 60* * 56: option send-gids true* * 57: end-volume* * 58: * * 59: volume drslk-prod-client-5* * 60: type protocol/client* * 61: option ping-timeout 20* * 62: option remote-host brick18.gluster.iadm* * 63: option remote-subvolume /GLUSTERFS/drslk-prod* * 64: option transport-type socket* * 65: option frame-timeout 60* * 66: option send-gids true* * 67: end-volume* * 68: * * 69: volume drslk-prod-replicate-1* * 70: type cluster/replicate* * 71: option read-hash-mode 2* * 72: option data-self-heal-window-size 128* * 73: option quorum-type auto* * 74: subvolumes drslk-prod-client-3 drslk-prod-client-4 drslk-prod-client-5* * 75: end-volume* * 76: * * 77: volume drslk-prod-client-6* * 78: type protocol/client* * 79: option ping-timeout 20* * 80: option remote-host brick19.gluster.iadm* * 81: option remote-subvolume /GLUSTERFS/drslk-prod* * 82: option transport-type socket* * 83: option frame-timeout 60* * 84: option send-gids true* * 85: end-volume* * 86: * * 87: volume drslk-prod-client-7* * 88: type protocol/client* * 89: option ping-timeout 20* * 90: option remote-host brick20.gluster.iadm* * 91: option remote-subvolume /GLUSTERFS/drslk-prod* * 92: option transport-type socket* * 93: option frame-timeout 60* * 94: option send-gids true* * 95: end-volume* * 96: * * 97: volume drslk-prod-client-8* * 98: type protocol/client* * 99: option ping-timeout 20* *100: option remote-host brick21.gluster.iadm* *101: option remote-subvolume /GLUSTERFS/drslk-prod* *102: option transport-type socket* *103: option frame-timeout 60* *104: option send-gids true* *105: end-volume* *106: * *107: volume drslk-prod-replicate-2* *108: type cluster/replicate* *109: option read-hash-mode 2* *110: option data-self-heal-window-size 128* *111: option quorum-type auto* *112: subvolumes drslk-prod-client-6 drslk-prod-client-7 drslk-prod-client-8* *113: end-volume* *114: * *115: volume drslk-prod-client-9* *116: type protocol/client* *117: option ping-timeout 20* *118: option remote-host brick22.gluster.iadm* *119: option remote-subvolume /GLUSTERFS/drslk-prod* *120: option transport-type socket* *121: option frame-timeout 60* *122: option send-gids true* *123: end-volume* *124: * *125: volume drslk-prod-client-10* *126: type protocol/client* *127: option ping-timeout 20* *128: option remote-host brick23.gluster.iadm* *129: option remote-subvolume /GLUSTERFS/drslk-prod* *130: option transport-type socket* *131: option frame-timeout 60* *132: option send-gids true* *133: end-volume* *134: *
[Gluster-users] Upgrade from 3.5 to 3.6: issue with socket
Hello, I just upgraded from 3.5.3 to 3.6.2 and have issues mounting my volume on a client. On the server side I found this error message which might be the cause of my issues: [2015-03-08 13:22:36.383715] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/6b8f1f2526c6af8a87f1bb611ae5a86f.socket failed (Invalid argument) Anyony has any idea what that could be ? Regards Uli ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] no data available
On 03/06/2015 02:34 AM, Niels de Vos wrote: This should be addressed with the next 3.6 update. See bug 1188064 for more details: https://bugzilla.redhat.com/1188064 Niels Thanks, tamas ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster errors create zombie processes [LOGS ATTACHED]
On 03/08/2015 09:36 AM, Przemysław Mroczek wrote: I don't have volfiles, they are not on our machines as I said previously we don't have impact on gluster servers. I saw some graph that looks similiar to volume file on logs. I will paste it here but we don't really have any impact on that. We are just using client to connect to gluster servers, we are not in control of. I would recommend to not alter the default for frame timeout. Btw, do you think that different versions of gluster client and gluster server could be an issue here? It can potentially be. What versions are you using on the servers and the client? -Vijay 2015-03-08 1:29 GMT+01:00 Vijay Bellur vbel...@redhat.com mailto:vbel...@redhat.com: On 03/07/2015 06:20 PM, Przemysław Mroczek wrote: Hi guys, We have rails app, which is using gluster for our distributed file system. The glusters servers are hosted independently as part of deal with other, we don't have any impact on them, we are connected o them by using gluster native client. We tried to resolve this issue using help from the admins of the company that is hosting our gluster servers, but they say that's the client issue and we ran out of ideas how that's possible if we are not doing anything special here. Information about independent gluster servers: -version: 3.6.0.42.1 - They are using red hat -They are enterprise so the are always using older versions Our servers: System version: Ubuntu 14.04 Our gluster client version: 3.6.2 The exact problem is that it often happens(couple times a week) that errors in gluster causes proceses to become zombies. It happens with our application server(unicorn), nginx and our crawling script that is run as daemon. Our fstab file: 10.10.11.17:/drslk-prod /mnt/storage glusterfs defaults,_netdev,nobootwait,__fetch-attempts=10 0 0 10.10.11.17:/drslk-backup /mnt/backup glusterfs defaults,_netdev,nobootwait,__fetch-attempts=10 0 0 Logs from gluster: 2015-02-18 12:36:12.375695] E [rpc-clnt.c:362:saved_frames___unwind] (-- /usr/lib/x86_64-linux-gnu/__libglusterfs.so.0(_gf_log___callingfn+0x186)[__0x7fb41ddeada6] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(saved_frames___unwind+0x1de)[0x7fb41d bc1c7e] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(saved_frames___destroy+0xe)[0x7fb41dbc1d8e] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(rpc_clnt___connection_cleanup+0x82)[__0x7fb41dbc3602] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(rpc _clnt_notify+0x48)[__0x7fb41dbc3d98] ) 0-drslk-prod-client-10: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-02-18 12:36:12.361489 (xid=0x5d475da) [2015-02-18 12:36:12.375765] W [client-rpc-fops.c:2766:__client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/71/77/59.__jpg (2ad81c2b-a141-478d-9dd4-__253345edbce b) [2015-02-18 12:36:12.376288] E [rpc-clnt.c:362:saved_frames___unwind] (-- /usr/lib/x86_64-linux-gnu/__libglusterfs.so.0(_gf_log___callingfn+0x186)[__0x7fb41ddeada6] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(saved_frames___unwind+0x1de)[0x7fb41d bc1c7e] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(saved_frames___destroy+0xe)[0x7fb41dbc1d8e] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(rpc_clnt___connection_cleanup+0x82)[__0x7fb41dbc3602] (-- /usr/lib/x86_64-linux-gnu/__libgfrpc.so.0(rpc _clnt_notify+0x48)[__0x7fb41dbc3d98] ) 0-drslk-prod-client-10: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-02-18 12:36:12.361858 (xid=0x5d475db) [2015-02-18 12:36:12.376355] W [client-rpc-fops.c:2766:__client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/08 (f5c33a99-719e-4ea2-ad1f-__33b893af103d) [2015-02-18 12:36:12.376711] I [socket.c:3292:socket_submit___request] 0-drslk-prod-client-10: not connected (priv-connected = 0) [2015-02-18 12:36:12.376749] W [rpc-clnt.c:1562:rpc_clnt___submit] 0-drslk-prod-client-10: failed to submit rpc-request (XID: 0x5d475dc Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (drslk-prod-client-10) [2015-02-18 12:36:12.376814] W [client-rpc-fops.c:2766:__client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed:
