Re: [Gluster-users] glusterfsd not starting at boot with Debian 7
As promised here the log file of the brick itself: [2015-02-05 20:38:41.614491] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.5.3 (/usr/sbin/glusterfsd -s gfs1a --volfile-id myvol.gfs1a.data-myvol-brick1 -p /var/lib/glusterd/vols/myvol/run/gfs1a-data-myvol-brick1.pid -S /var/run/23669fa92fc09b0dd09f4f997f6a3adf.socket --brick-name /data/myvol/brick1 -l /var/log/glusterfs/bricks/data-myvol-brick1.log --xlator-option *-posix.glusterd-uuid=459ca0c5-8400-4392-9846-fcef98772f44 --brick-port 49153 --xlator-option myvol-server.listen-port=49153) [2015-02-05 20:38:41.616083] I [socket.c:3645:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled [2015-02-05 20:38:41.616111] I [socket.c:3660:socket_init] 0-socket.glusterfsd: using system polling thread [2015-02-05 20:38:41.616184] I [socket.c:3645:socket_init] 0-glusterfs: SSL support is NOT enabled [2015-02-05 20:38:41.616193] I [socket.c:3660:socket_init] 0-glusterfs: using system polling thread [2015-02-05 20:38:41.616234] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-02-05 20:38:41.616242] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1a [2015-02-05 20:38:41.616273] E [glusterfsd-mgmt.c:1601:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: gfs1a (Success) [2015-02-05 20:38:41.616283] I [glusterfsd-mgmt.c:1607:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers [2015-02-05 20:38:41.616382] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/sbin/glusterfsd(glusterfs_mgmt_init+0x16d) [0x7f0d393ac31d] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f0d3815c492] (-->/usr/sbin/glusterfsd(+0xd588) [0x7f0d393ac588]))) 0-: received signum (1), shutting down [2015-02-05 20:38:41.616435] E [common-utils.c:223:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known) [2015-02-05 20:38:41.616449] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host gfs1a [2015-02-05 20:38:41.616455] I [socket.c:3143:socket_submit_request] 0-glusterfs: not connected (priv->connected = -1) [2015-02-05 20:38:41.616460] W [rpc-clnt.c:1542:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x1 Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs) Looks like a DNS issue to me. I have to mention here that I use gfs1a as hostname in my brick but do not have it into my DNS server, I just added it to my /etc/hosts file which in theory should be enough. Also note here that I am using ZFS and looking at the startup scripts of rc2.d ZFS gets started before glusterfs so the device should be available when gluster starts. On Thursday, February 5, 2015 5:03 PM, Pranith Kumar Karampuri wrote: On 02/05/2015 06:46 PM, ML mail wrote: > Dear Pranith > > As asked you will find below the glusterd logfile including the log entries > at the boot of the server. On Debian the logfile is located in the > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file. > > > [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: > Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p > /var/run/glusterd.pid) > [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] > 0-socket.management: using system polling thread > [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed > to initialize IB Device > [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] > 0-socket.management: using system polling thread > [2015-02-05 13:13:06.111967] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 30501 > [2015-02-05 13:13:06.114871] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] > 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 > Final graph: > +--+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-gluste
Re: [Gluster-users] glusterfsd not starting at boot with Debian 7
On 02/05/2015 06:46 PM, ML mail wrote: Dear Pranith As asked you will find below the glusterd logfile including the log entries at the boot of the server. On Debian the logfile is located in the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file. [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p /var/run/glusterd.pid) [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-02-05 13:13:06.111967] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501 [2015-02-05 13:13:06.114871] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 Final graph: +--+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option rpc-auth-allow-insecure on 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +--+ [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-02-05 13:13:06.285982] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1a:/data/myvol/brick [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now Log above suggests that glusterd launched the brick but it failed to come up. Let us see the logfile of the brick as well. It should be present in /var/log/glusterfs/bricks Pranith On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri wrote: On 02/05/2015 04:08 AM, ML mail wrote: Hi, I have installed Gluster 3.5.3 on Debian 7 and have one single test volume right now. Unfortunately after a reboot this volume does not get get started automatically: the glusterfsd process for that volume is inexistant although the glusterd process is running. After a boot running "gluster volume status" shows: Status of volume: myvol Gluster processPortOnlinePid -- Brick gfs1a:/data/myvol/brickN/ANN/A Task Status of Volume myvol -- There are no active volume tasks as you can see all N/A there. In order to make it work I have to run "gluster volume stop myvol" and "gluster volume start myvol". Then it is ready and working. What could the problem be? Is the init.d startup srcript on Debian 7 buggy? If glusterd is starting then there shouldn't be any bug in the init scripts. Could you attach the log file of glusterd? I am not sure where it is stored in debian may be /var/log/glusterfs or /usr/local/var/log/glusterfs it will have glu
Re: [Gluster-users] glusterfsd not starting at boot with Debian 7
On Thu, Feb 05, 2015 at 01:16:54PM +, ML mail wrote: > Dear Pranith > > As asked you will find below the glusterd logfile including the log > entries at the boot of the server. On Debian the logfile is located in > the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file. This is the log file of the glusterd process, whereas the issue you reported is with the glusterfsd (brick) process. There should be a log for each brick under /var/log/glusterfs/bricks/. The most common problem, is that the mount point is not available yet when glusterd starts the brick processes. You mention that you can get the processes started with a 'gluster volume stop/start' command. An easier way would be 'gluster volume start $VOLUME force' which start only the missing processes (and does not stop any brick processes on other storage servers). Depending on how you mount the filesystems for your bricks, you may be able to add the x-systemd.automount option to /etc/fstab to trigger a mount of the filesystem upon access (when the brick process starts). See 'man systemd.mount' for some more details. HTH, Niels > > > [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: > Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p > /var/run/glusterd.pid) > [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using > /var/lib/glusterd as working directory > [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] > 0-socket.management: using system polling thread > [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] > 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) > [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed > to initialize IB Device > [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] > 0-rpc-transport: 'rdma' initialization failed > [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] > 0-socket.management: SSL support is NOT enabled > [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] > 0-socket.management: using system polling thread > [2015-02-05 13:13:06.111967] I > [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved > op-version: 30501 > [2015-02-05 13:13:06.114871] E > [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] > 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 > Final graph: > +--+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option rpc-auth-allow-insecure on > 8: option transport.socket.read-fail-log off > 9: option transport.socket.keepalive-interval 2 > 10: option transport.socket.keepalive-time 10 > 11: option transport-type rdma > 12: option working-directory /var/lib/glusterd > 13: end-volume > 14: > +--+ > [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL > support is NOT enabled > [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: > using system polling thread > [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] > 0-transport: disconnecting now > [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] > 0-management: setting frame-timeout to 600 > [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL > support is NOT enabled > [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: > using system polling thread > [2015-02-05 13:13:06.285982] I > [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: > Disconnected from gfs1a:/data/myvol/brick > [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] > 0-transport: disconnecting now > > > > > > On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri > wrote: > > On 02/05/2015 04:08 AM, ML mail wrote: > > Hi, > > > > I have installed Gluster 3.5.3 on Debian 7 and have one single test volume > > right now. Unfortunately after a reboot this volume does not get get > > started automatically: the glusterfsd process for that volume is inexistant > > although the glusterd process is running. > > > > > > After a boot running "gluster volume status"
Re: [Gluster-users] glusterfsd not starting at boot with Debian 7
Dear Pranith As asked you will find below the glusterd logfile including the log entries at the boot of the server. On Debian the logfile is located in the /var/log/glusterfs/etc-glusterfs-glusterd.vol.log file. [2015-02-05 13:13:03.566059] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.3 (/usr/sbin/glusterd -p /var/run/glusterd.pid) [2015-02-05 13:13:03.580131] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory [2015-02-05 13:13:03.583040] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-05 13:13:03.583060] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-02-05 13:13:03.585326] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2015-02-05 13:13:03.585343] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2015-02-05 13:13:03.585350] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2015-02-05 13:13:03.585392] W [rpcsvc.c:1535:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-02-05 13:13:03.585446] I [socket.c:3645:socket_init] 0-socket.management: SSL support is NOT enabled [2015-02-05 13:13:03.585454] I [socket.c:3660:socket_init] 0-socket.management: using system polling thread [2015-02-05 13:13:06.111967] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501 [2015-02-05 13:13:06.114871] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2015-02-05 13:13:06.277826] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 459ca0c5-8400-4392-9846-fcef98772f44 Final graph: +--+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option rpc-auth-allow-insecure on 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +--+ [2015-02-05 13:13:06.281802] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-05 13:13:06.281889] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-02-05 13:13:06.281902] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-02-05 13:13:06.282359] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now [2015-02-05 13:13:06.285676] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-02-05 13:13:06.285770] I [socket.c:3645:socket_init] 0-management: SSL support is NOT enabled [2015-02-05 13:13:06.285786] I [socket.c:3660:socket_init] 0-management: using system polling thread [2015-02-05 13:13:06.285982] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1a:/data/myvol/brick [2015-02-05 13:13:06.286042] I [socket.c:2321:socket_event_handler] 0-transport: disconnecting now On Thursday, February 5, 2015 12:02 PM, Pranith Kumar Karampuri wrote: On 02/05/2015 04:08 AM, ML mail wrote: > Hi, > > I have installed Gluster 3.5.3 on Debian 7 and have one single test volume > right now. Unfortunately after a reboot this volume does not get get started > automatically: the glusterfsd process for that volume is inexistant although > the glusterd process is running. > > > After a boot running "gluster volume status" shows: > Status of volume: myvol > Gluster processPortOnlinePid > -- > Brick gfs1a:/data/myvol/brickN/ANN/A > > Task Status of Volume myvol > -- > > There are no active volume tasks > > as you can see all N/A there. In order to make it work I have to run "gluster > volume stop myvol" and "gluster volume start myvol". Then it is ready and > working. What could the problem be? Is the init.d startup srcript on Debian 7 > buggy? If glusterd is starting then there shouldn't be any bug in the init scripts. Could you attach the log file of glusterd? I am not sure where it is stored in debian may be /var/log/glusterfs or /usr/local/var/log/glusterfs it will have glusterd in the name of the file. Pranith > > Best regards > ML > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.glust
Re: [Gluster-users] glusterfsd not starting at boot with Debian 7
On 02/05/2015 04:08 AM, ML mail wrote: Hi, I have installed Gluster 3.5.3 on Debian 7 and have one single test volume right now. Unfortunately after a reboot this volume does not get get started automatically: the glusterfsd process for that volume is inexistant although the glusterd process is running. After a boot running "gluster volume status" shows: Status of volume: myvol Gluster processPortOnlinePid -- Brick gfs1a:/data/myvol/brickN/ANN/A Task Status of Volume myvol -- There are no active volume tasks as you can see all N/A there. In order to make it work I have to run "gluster volume stop myvol" and "gluster volume start myvol". Then it is ready and working. What could the problem be? Is the init.d startup srcript on Debian 7 buggy? If glusterd is starting then there shouldn't be any bug in the init scripts. Could you attach the log file of glusterd? I am not sure where it is stored in debian may be /var/log/glusterfs or /usr/local/var/log/glusterfs it will have glusterd in the name of the file. Pranith Best regards ML ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
