Re: [Gluster-users] segfault in glustershd
Hi, Thanks for notifying the issue. It is fixed now. The patch needs to be reviewed and will be merged soon. https://bugzilla.redhat.com/show_bug.cgi?id=947824 Regards, S.Venkatesh - Original Message - From: "Vijay Bellur" To: "Venkatesh Somyajulu" Sent: Wednesday, April 3, 2013 5:20:14 PM Subject: Fwd: [Gluster-users] segfault in glustershd Original Message ---- Subject: [Gluster-users] segfault in glustershd Date: Wed, 03 Apr 2013 21:13:14 +1100 From: Toby Corkindale To: gluster-users@gluster.org We experienced the gluster self-heal daemon crashing on us after a reboot of the server.. Logs are below. Is this a known issue: [2013-04-03 20:54:06.500173] E [dict.c:2424:dict_unserialize] (-->/lib/libc.so.6(+0x41600) [0x7f6fff533600] (-->/usr/lib/libglusterfs.so.0(synctask_wrap+0x12) [0x7f700028d9d2] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x7f70006e520f]))) 0-dict: undersized buffer passed. available (140118480393855) < required (140118936926502) pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2013-04-03 20:54:06 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.3.1 /lib/libc.so.6(+0x32230)[0x7f6fff524230] /lib/libc.so.6(memcpy+0x3f3)[0x7f6fff571f23] /usr/lib/libglusterfs.so.0(dict_unserialize+0x27f)[0x7f7000258ccf] /usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f)[0x7f70006e520f] /usr/lib/libglusterfs.so.0(synctask_wrap+0x12)[0x7f700028d9d2] /lib/libc.so.6(+0x41600)[0x7f6fff533600] - ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] segfault in glustershd
This crash is because of some other reason. Fix for which is already merged in upstream. http://review.gluster.com/4767 Thanks for reporting the issue. Pranith. - Original Message - From: "符永涛" To: "Toby Corkindale" Cc: gluster-users@gluster.org Sent: Monday, April 8, 2013 6:46:45 PM Subject: Re: [Gluster-users] segfault in glustershd Hi Toby, According to the crash log, the cause may be the following: glusterfsd/src/glusterfsd-mgmt.c line 1394 static char oldvolfile[131072]; so if the volume file(/var/lib/glusterd/glustershd/glustershd-server.vol) is larger than 128K then it simply crashes. This happens if there're a lot of volumes on the server and the server volume file is larger than 128k. on the line 1629 memcpy (oldvolfile, rsp.spec, size); It should be a bug. You can download glusterfs*.tar.gz and fix the problem and rebuild rpm. Then upgrade glusterfs server one by one. rpm -ivhU *.rpm --force Make sure execute above command on all the server one by one and check the /var/log/glusterfs/glusterd.log util the upgrade finished on one server then do it on another server. FYI 2013/4/3, Toby Corkindale : > We experienced the gluster self-heal daemon crashing on us after a > reboot of the server.. Logs are below. > Is this a known issue: > > > [2013-04-03 20:54:06.500173] E [dict.c:2424:dict_unserialize] > (-->/lib/libc.so.6(+0x41600) [0x7f6fff533600] > (-->/usr/lib/libglusterfs.so.0(synctask_wrap+0x12) [0x7f700028d9d2] > (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) > [0x7f70006e520f]))) 0-dict: undersized buffer passed. available > (140118480393855) < required (140118936926502) > pending frames: > > patchset: git://git.gluster.com/glusterfs.git > signal received: 11 > time of crash: 2013-04-03 20:54:06 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > fdatasync 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 3.3.1 > /lib/libc.so.6(+0x32230)[0x7f6fff524230] > /lib/libc.so.6(memcpy+0x3f3)[0x7f6fff571f23] > /usr/lib/libglusterfs.so.0(dict_unserialize+0x27f)[0x7f7000258ccf] > /usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f)[0x7f70006e520f] > /usr/lib/libglusterfs.so.0(synctask_wrap+0x12)[0x7f700028d9d2] > /lib/libc.so.6(+0x41600)[0x7f6fff533600] > - > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > -- 符永涛 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] segfault in glustershd
Hi Toby, According to the crash log, the cause may be the following: glusterfsd/src/glusterfsd-mgmt.c line 1394 static char oldvolfile[131072]; so if the volume file(/var/lib/glusterd/glustershd/glustershd-server.vol) is larger than 128K then it simply crashes. This happens if there're a lot of volumes on the server and the server volume file is larger than 128k. on the line 1629 memcpy (oldvolfile, rsp.spec, size); It should be a bug. You can download glusterfs*.tar.gz and fix the problem and rebuild rpm. Then upgrade glusterfs server one by one. rpm -ivhU *.rpm --force Make sure execute above command on all the server one by one and check the /var/log/glusterfs/glusterd.log util the upgrade finished on one server then do it on another server. FYI 2013/4/3, Toby Corkindale : > We experienced the gluster self-heal daemon crashing on us after a > reboot of the server.. Logs are below. > Is this a known issue: > > > [2013-04-03 20:54:06.500173] E [dict.c:2424:dict_unserialize] > (-->/lib/libc.so.6(+0x41600) [0x7f6fff533600] > (-->/usr/lib/libglusterfs.so.0(synctask_wrap+0x12) [0x7f700028d9d2] > (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) > [0x7f70006e520f]))) 0-dict: undersized buffer passed. available > (140118480393855) < required (140118936926502) > pending frames: > > patchset: git://git.gluster.com/glusterfs.git > signal received: 11 > time of crash: 2013-04-03 20:54:06 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > fdatasync 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 3.3.1 > /lib/libc.so.6(+0x32230)[0x7f6fff524230] > /lib/libc.so.6(memcpy+0x3f3)[0x7f6fff571f23] > /usr/lib/libglusterfs.so.0(dict_unserialize+0x27f)[0x7f7000258ccf] > /usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f)[0x7f70006e520f] > /usr/lib/libglusterfs.so.0(synctask_wrap+0x12)[0x7f700028d9d2] > /lib/libc.so.6(+0x41600)[0x7f6fff533600] > - > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users > -- 符永涛 ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] segfault in glustershd
We experienced the gluster self-heal daemon crashing on us after a reboot of the server.. Logs are below. Is this a known issue: [2013-04-03 20:54:06.500173] E [dict.c:2424:dict_unserialize] (-->/lib/libc.so.6(+0x41600) [0x7f6fff533600] (-->/usr/lib/libglusterfs.so.0(synctask_wrap+0x12) [0x7f700028d9d2] (-->/usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f) [0x7f70006e520f]))) 0-dict: undersized buffer passed. available (140118480393855) < required (140118936926502) pending frames: patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2013-04-03 20:54:06 configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.3.1 /lib/libc.so.6(+0x32230)[0x7f6fff524230] /lib/libc.so.6(memcpy+0x3f3)[0x7f6fff571f23] /usr/lib/libglusterfs.so.0(dict_unserialize+0x27f)[0x7f7000258ccf] /usr/sbin/glusterfs(glusterfs_handle_translator_op+0x16f)[0x7f70006e520f] /usr/lib/libglusterfs.so.0(synctask_wrap+0x12)[0x7f700028d9d2] /lib/libc.so.6(+0x41600)[0x7f6fff533600] - ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
