Re: [Gluster-devel] Upstream regression crash : https://build.gluster.org/job/rackspace-regression-2GB-triggered/16191/consoleFull

2015-11-27 Thread Manikandan Selvaganesh 
Hi,
Even I am facing the same issue. The test 
./tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t
 has failed with a core with no symbols attached. Stack seems to be corrupted. 
Could you please look into the issue and the NFS log says

[2015-11-27 06:02:49.947501] W [socket.c:583:__socket_rwv] 0-patchy-client-0: 
readv on 166.78.247.4:49154 failed (No data available)
[2015-11-27 06:02:49.947535] I [MSGID: 114018] 
[client.c:2018:client_rpc_notify] 0-patchy-client-0: disconnected from 
patchy-client-0. Client process will keep >  \trying to connect to 
glusterd until brick's port is available
[2015-11-27 06:02:49.951462] W [glusterfsd.c:1231:cleanup_and_exit] 
(-->/lib64/libpthread.so.0(+0x7a51) [0x7f1611210a51] -->/build/install/sbin/
>  \glusterfs(glusterfs_sigwaiter+0xe4) [0x40986c] 
-->/build/install/sbin/glusterfs(cleanup_and_exit+0x87) [0x407cdf] ) 0-: 
received signum (15), shutting down  


Regression crash : 
https://build.gluster.org/job/rackspace-regression-2GB-triggered/16262/consoleFull

Thank you :-)

--
Regards,
Manikandan Selvaganesh.

- Original Message -
From: "Nithya Balachandran" 
To: "Rajesh Joseph" , "Soumya Koduri" 
Cc: "Gluster Devel" 
Sent: Thursday, November 26, 2015 10:43:54 AM
Subject: [Gluster-devel] Upstream regression crash : 
https://build.gluster.org/job/rackspace-regression-2GB-triggered/16191/consoleFull

Hi,


The test 
tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t
 has failed with a core.Can you please take a look?


The NFS log says:
gluster/02c803ff8630bd12cc8dc9dc043a6103.socket)
[2015-11-25 19:06:41.561937] I [MSGID: 101190] 
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with 
index 1
[2015-11-25 19:06:41.566469] I [rpcsvc.c:2211:rpcsvc_set_outstanding_rpc_limit] 
0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
[2015-11-25 19:06:41.568896] I [MSGID: 112153] [mount3.c:3924:mnt3svc_init] 
0-nfs-mount: Exports auth has been disabled!
[2015-11-25 19:06:41.592393] I [rpc-drc.c:689:rpcsvc_drc_init] 0-rpc-service: 
DRC is turned OFF
[2015-11-25 19:06:41.592441] I [MSGID: 112110] [nfs.c:1513:init] 0-nfs: NFS 
service started
[2015-11-25 19:06:41.592905] E [crypt.c:4298:master_set_master_vol_key] 
0-patchy-crypt: FATAL: missing master key
[2015-11-25 19:06:41.592928] E [MSGID: 101019] [xlator.c:429:xlator_init] 
0-patchy-crypt: Initialization of volume 'patchy-crypt' failed, review your 
volfile again
[2015-11-25 19:06:41.592953] E [MSGID: 101066] 
[graph.c:324:glusterfs_graph_init] 0-patchy-crypt: initializing translator 
failed
[2015-11-25 19:06:41.592976] E [MSGID: 101176] 
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2015-11-25 19:06:41
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2015-11-25 19:06:41
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8dev
[2015-11-25 19:06:41.594542] W [glusterfsd.c:1231:cleanup_and_exit] 
(-->/build/install/sbin/glusterfs(mgmt_getspec_cbk+0x34d) [0x40e71d] -->/b
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Upstream regression crash : https://build.gluster.org/job/rackspace-regression-2GB-triggered/16191/consoleFull

2015-11-26 Thread Soumya Koduri 

Below are the findings from the core and the logs

1) [2015-11-25 19:06:41.592905] E 
[crypt.c:4298:master_set_master_vol_key] 0-patchy-crypt: FATAL: missing 
master key


xlator_init() of crypt xlator fails, which I assume gets loaded when 
features.encryption is on (which the below mentioned .t test does)


Request anyone familiar with crypt xlator to take a look at it.

This results in the shutdown of NFS (glusterfsd) process calling 
cleanup_an_exit()



2) There is a crash in libgcc

Thread 1 (LWP 11584):
#0  0x7f922ad34867 in ?? () from ./lib64/libgcc_s.so.1
#1  0x7f922ad35119 in _Unwind_Backtrace () from ./lib64/libgcc_s.so.1
#2  0x7f923660b936 in backtrace () from ./lib64/libc.so.6
---Type  to continue, or q  to quit---
#3  0x7f92379abf73 in _gf_msg_backtrace_nomem (level=GF_LOG_ALERT, 
stacksize=200)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/logging.c:1090

#4  0x7f92379b1d38 in gf_print_trace (signum=11, ctx=0x2331010)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/common-utils.c:740

#5  0x004098d6 in glusterfsd_print_trace (signum=11)
at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:2033

#6  
#7  0x7f9229eb8cec in ?? ()
#8  0x7f9236c8ba51 in start_thread () from ./lib64/libpthread.so.0
#9  0x7f92365f593d in clone () from ./lib64/libc.so.6
(gdb)

This looks similar to the issue pointed earlier by Kotresh [1]

As mentioned in the last mail of that thread, [2] doesn't seem to have 
fixed it completely.


[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10866
[2] http://review.gluster.org/#/c/10417/


Thanks,
Soumya

On 11/26/2015 10:43 AM, Nithya Balachandran wrote:

Hi,


The test 
tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t
 has failed with a core.Can you please take a look?


The NFS log says:
gluster/02c803ff8630bd12cc8dc9dc043a6103.socket)
[2015-11-25 19:06:41.561937] I [MSGID: 101190] 
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with 
index 1
[2015-11-25 19:06:41.566469] I [rpcsvc.c:2211:rpcsvc_set_outstanding_rpc_limit] 
0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
[2015-11-25 19:06:41.568896] I [MSGID: 112153] [mount3.c:3924:mnt3svc_init] 
0-nfs-mount: Exports auth has been disabled!
[2015-11-25 19:06:41.592393] I [rpc-drc.c:689:rpcsvc_drc_init] 0-rpc-service: 
DRC is turned OFF
[2015-11-25 19:06:41.592441] I [MSGID: 112110] [nfs.c:1513:init] 0-nfs: NFS 
service started
[2015-11-25 19:06:41.592905] E [crypt.c:4298:master_set_master_vol_key] 
0-patchy-crypt: FATAL: missing master key
[2015-11-25 19:06:41.592928] E [MSGID: 101019] [xlator.c:429:xlator_init] 
0-patchy-crypt: Initialization of volume 'patchy-crypt' failed, review your 
volfile again
[2015-11-25 19:06:41.592953] E [MSGID: 101066] 
[graph.c:324:glusterfs_graph_init] 0-patchy-crypt: initializing translator 
failed
[2015-11-25 19:06:41.592976] E [MSGID: 101176] 
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2015-11-25 19:06:41
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2015-11-25 19:06:41
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8dev
[2015-11-25 19:06:41.594542] W [glusterfsd.c:1231:cleanup_and_exit] 
(-->/build/install/sbin/glusterfs(mgmt_getspec_cbk+0x34d) [0x40e71d] -->/b


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Upstream regression crash : https://build.gluster.org/job/rackspace-regression-2GB-triggered/16191/consoleFull

2015-11-25 Thread Nithya Balachandran 
Hi,


The test 
tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t
 has failed with a core.Can you please take a look?


The NFS log says:
gluster/02c803ff8630bd12cc8dc9dc043a6103.socket)
[2015-11-25 19:06:41.561937] I [MSGID: 101190] 
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with 
index 1
[2015-11-25 19:06:41.566469] I [rpcsvc.c:2211:rpcsvc_set_outstanding_rpc_limit] 
0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
[2015-11-25 19:06:41.568896] I [MSGID: 112153] [mount3.c:3924:mnt3svc_init] 
0-nfs-mount: Exports auth has been disabled!
[2015-11-25 19:06:41.592393] I [rpc-drc.c:689:rpcsvc_drc_init] 0-rpc-service: 
DRC is turned OFF
[2015-11-25 19:06:41.592441] I [MSGID: 112110] [nfs.c:1513:init] 0-nfs: NFS 
service started
[2015-11-25 19:06:41.592905] E [crypt.c:4298:master_set_master_vol_key] 
0-patchy-crypt: FATAL: missing master key
[2015-11-25 19:06:41.592928] E [MSGID: 101019] [xlator.c:429:xlator_init] 
0-patchy-crypt: Initialization of volume 'patchy-crypt' failed, review your 
volfile again
[2015-11-25 19:06:41.592953] E [MSGID: 101066] 
[graph.c:324:glusterfs_graph_init] 0-patchy-crypt: initializing translator 
failed
[2015-11-25 19:06:41.592976] E [MSGID: 101176] 
[graph.c:670:glusterfs_graph_activate] 0-graph: init failed
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2015-11-25 19:06:41
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 
2015-11-25 19:06:41
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8dev
[2015-11-25 19:06:41.594542] W [glusterfsd.c:1231:cleanup_and_exit] 
(-->/build/install/sbin/glusterfs(mgmt_getspec_cbk+0x34d) [0x40e71d] -->/b
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel