[Gluster-users] gluster v3.0.0 log showing errors
Hello, Recently we've put a gluster v3.0.0 file system into use for a group of apache2/tomcat6 servers. Today one of those servers started hanging and we noticed a lot of lines like the following two in the log file: 1) W [fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 285653: GETATTR 140556365266128 (fuse_loc_fill() failed) 2) E [afr.c:179:afr_read_child] mirror-0: invalid argument: inode I don't think we're doing anything very strange with this setup. I'm at a loss as to why we are seeing this "hanging" behavior and these errors in the logs. All machines are running v3.0.0 from the same source build. We have 3 bricks using jfs for their backend storage setup to replicate to all three. During testing we discovered that we needed to turn off write-behind to make things like CVS and xsl transformations more reliable in our setup. Write speed of course suffers but that is a compromise we've been willing to live with. There are 7 clients using the file system for web files, java classes, and log files. While this was going on there was an rsync of more images and PDF files copying from an old web server onto the new gluster file system using the client that was "hanging" as it's receiving side. If you could shed any light on what might be causing our problems, it would be very helpful. Thanks! Kind regards, -Chris -- Chris Ely Hosting Department IQnection Internet Services, Inc. 23 Taylor Avenue Doylestown, PA 18901 Ph. 1-866-400-4678/Fax. 215-489-1872 http://www.iqnection.com/ ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] open() issues on 3.0.0
Aaron,
Nice piece of information !! ...
Regards,
Tejas.
- Original Message -
From: "Aaron Knister"
To: b...@usf.edu
Cc: gluster-users@gluster.org
Sent: Sunday, January 17, 2010 11:42:14 PM GMT +05:30 Chennai, Kolkata, Mumbai,
New Delhi
Subject: Re: [Gluster-users] open() issues on 3.0.0
I ran into a similar problem when compiling code on a gluster mount with the
intel compiler. The problem was intermittent and after digging into it further
it appeared to only happy when compiling from source files that had very high
inode numbers. There's a command run by icc that's a 32 bit executable and my
guess is that somewhere internally the extremely large inode number wasn't able
to be handled and caused the compilation to fail. The "fix" was to upgrade the
compiler. I can verify that version 11.0.064 works without a hitch (at least in
this regard).
More Detail:
I can reproduce this error with version 10.0.026 of the intel compiler.
I have file test.c on a glusterfs with inode num 21479794854 (greater than 32
bits)
21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
Running icc fails:
...
$ icc test.c
Catastrophic error: could not open source file "test.c"
compilation aborted for test.c (code 4)
...
However if I copy the same file to a directory on the same fs that has a 32 bit
inode number (le 4294967295) (hoping the new file will also have a 32 bit inode
number):
11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
Compilation is a success:
...
$ !icc
icc test.c
$ echo $?
0
...
If you run icc with the -v option you'll see that a program called mcpcom gets
run and produces the error. At least with version 10.0.026 it was a 32 bit
program:
$ file /opt/intel/cce/10.0.026/bin/mcpcom
/opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel 80386,
version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs),
for GNU/Linux 2.2.5, not stripped
Anyway I hope that helps.
-Aaron
On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
> Hi,
>
> I'm new to Gluster, so please forgive any ignorance on my part. I've
> tried reading over everything I could find to resolve this myself.
>
> I'm having an issue building executables on our gluster volume (there
> are some other similar issues as well, but we'll deal with this one
> first). I have the presta infiniband utilities extracted into a
> directory in my glusterfs volume and I'm attempting to build them. One
> of the source files, util.c (coincidentally, the first one referenced in
> the make file) fails to build. I get an error message:
>
> Catastrophic error: could not open source file "util.c"
>
> Some further investigating reveals the command that causes the issue:
>
> icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib -lmpi
> -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath -lnuma
> -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
> Catastrophic error: could not open source file "util.c"
>
> compilation aborted for util.c (code 4)
>
> And further, we see an strace of the command:
> ...
> 2717 6601 brk(0x957b000)= 0x957b000
> 2718 6601 open("util.c", O_RDONLY) = 3
> 2719 6601 stat64("util.c", {st_mode=S_IFREG|0600, st_size=14370, ...})
> = 0
> 2720 6601 close(3) = 0
> 2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> 1), ...}) = 0
> 2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> MAP_ANONYMOUS, -1, 0x1000) = 0xf0bce000
> 2723 6601 write(2, "Catastrophic error: could not open source file
> \"util.c\"\n", 56) = 56
> 2724 6601 write(2, "\n", 1) = 1
> 2725 6601 exit_group(4)
> ...
>
> We can also see that the file is indeed present:
>
> $ ls -l util.c
> -rw--- 1 brs users 14370 Apr 8 2002 util.c
> $ stat util.c
> File: `util.c'
> Size: 14370 Blocks: 32 IO Block: 4096 regular file
> Device: 18h/24d Inode: 4295774314 Links: 1
> Access: (0600/-rw---) Uid: (1229806/ brs) Gid: (10001/
> users)
> Access: 2010-01-17 00:27:43.0 -0500
> Modify: 2002-04-08 21:13:57.0 -0400
> Change: 2010-01-17 00:27:43.0 -0500
>
> If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> above works correctly.
>
> diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
>
> also appears clean. Any ideas what is happening here? Is there an
> issue with mmap2() and glusterfs?
>
> Many thanks in advance,
>
> -Brian
>
>
>
> --
> Brian Smith
> Senior Systems Administrator
> IT Research Computing, University of South Florida
> 4202 E. Fowler Ave. ENB308
> Office Phone: +1 813 974-1467
> Organization URL: http://rc.usf.edu
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Re: [Gluster-users] open() issues on 3.0.0
Good find! That seems to be exactly the issue. My inode numbers for
the files in that directory are literally just 100k away from the 32-bit
cut-off. I've rm'ed and re-extraced to get lesser inode values and
voila! I guess intel must have thought that 2^32 inodes outta be enough
for everyone.
Many thanks,
-Brian
--
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu
On Sun, 2010-01-17 at 13:12 -0500, Aaron Knister wrote:
> I ran into a similar problem when compiling code on a gluster mount
> with the intel compiler. The problem was intermittent and after
> digging into it further it appeared to only happy when compiling from
> source files that had very high inode numbers. There's a command run
> by icc that's a 32 bit executable and my guess is that somewhere
> internally the extremely large inode number wasn't able to be handled
> and caused the compilation to fail. The "fix" was to upgrade the
> compiler. I can verify that version 11.0.064 works without a hitch (at
> least in this regard).
>
>
> More Detail:
>
>
> I can reproduce this error with version 10.0.026 of the intel
> compiler.
>
>
> I have file test.c on a glusterfs with inode num 21479794854 (greater
> than 32 bits)
> 21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
>
>
> Running icc fails:
> ...
> $ icc test.c
> Catastrophic error: could not open source file "test.c"
>
>
> compilation aborted for test.c (code 4)
> ...
>
>
> However if I copy the same file to a directory on the same fs that has
> a 32 bit inode number (le 4294967295) (hoping the new file will also
> have a 32 bit inode number):
> 11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
>
>
> Compilation is a success:
> ...
> $ !icc
> icc test.c
> $ echo $?
> 0
> ...
>
>
>
>
> If you run icc with the -v option you'll see that a program called
> mcpcom gets run and produces the error. At least with version 10.0.026
> it was a 32 bit program:
> $ file /opt/intel/cce/10.0.026/bin/mcpcom
> /opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel
> 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses
> shared libs), for GNU/Linux 2.2.5, not stripped
>
>
> Anyway I hope that helps.
>
>
> -Aaron
>
>
>
> On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
>
> > Hi,
> >
> > I'm new to Gluster, so please forgive any ignorance on my part.
> > I've
> > tried reading over everything I could find to resolve this myself.
> >
> > I'm having an issue building executables on our gluster volume
> > (there
> > are some other similar issues as well, but we'll deal with this one
> > first). I have the presta infiniband utilities extracted into a
> > directory in my glusterfs volume and I'm attempting to build them.
> > One
> > of the source files, util.c (coincidentally, the first one
> > referenced in
> > the make file) fails to build. I get an error message:
> >
> > Catastrophic error: could not open source file "util.c"
> >
> > Some further investigating reveals the command that causes the
> > issue:
> >
> > icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> > -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib
> > -lmpi
> > -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath
> > -lnuma
> > -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
> > Catastrophic error: could not open source file "util.c"
> >
> > compilation aborted for util.c (code 4)
> >
> > And further, we see an strace of the command:
> > ...
> > 2717 6601 brk(0x957b000)= 0x957b000
> > 2718 6601 open("util.c", O_RDONLY) = 3
> > 2719 6601 stat64("util.c", {st_mode=S_IFREG|0600,
> > st_size=14370, ...})
> > = 0
> > 2720 6601 close(3) = 0
> > 2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> > 1), ...}) = 0
> > 2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> > MAP_ANONYMOUS, -1, 0x1000) = 0xf0bce000
> > 2723 6601 write(2, "Catastrophic error: could not open source file
> > \"util.c\"\n", 56) = 56
> > 2724 6601 write(2, "\n", 1) = 1
> > 2725 6601 exit_group(4)
> > ...
> >
> > We can also see that the file is indeed present:
> >
> > $ ls -l util.c
> > -rw--- 1 brs users 14370 Apr 8 2002 util.c
> > $ stat util.c
> > File: `util.c'
> > Size: 14370 Blocks: 32 IO Block: 4096 regular file
> > Device: 18h/24d Inode: 4295774314 Links: 1
> > Access: (0600/-rw---) Uid: (1229806/ brs) Gid: (10001/
> > users)
> > Access: 2010-01-17 00:27:43.0 -0500
> > Modify: 2002-04-08 21:13:57.0 -0400
> > Change: 2010-01-17 00:27:43.0 -0500
> >
> > If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> > above works correctly.
> >
> > diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
> >
> > also appears cle
Re: [Gluster-users] open() issues on 3.0.0
I ran into a similar problem when compiling code on a gluster mount with the
intel compiler. The problem was intermittent and after digging into it further
it appeared to only happy when compiling from source files that had very high
inode numbers. There's a command run by icc that's a 32 bit executable and my
guess is that somewhere internally the extremely large inode number wasn't able
to be handled and caused the compilation to fail. The "fix" was to upgrade the
compiler. I can verify that version 11.0.064 works without a hitch (at least in
this regard).
More Detail:
I can reproduce this error with version 10.0.026 of the intel compiler.
I have file test.c on a glusterfs with inode num 21479794854 (greater than 32
bits)
21479794854 -rw-r--r-- 1 aaron users 26 Jan 17 12:58 test.c
Running icc fails:
...
$ icc test.c
Catastrophic error: could not open source file "test.c"
compilation aborted for test.c (code 4)
...
However if I copy the same file to a directory on the same fs that has a 32 bit
inode number (le 4294967295) (hoping the new file will also have a 32 bit inode
number):
11889296 -rw-r--r-- 1 aaron users 26 Jan 17 13:02 test.c
Compilation is a success:
...
$ !icc
icc test.c
$ echo $?
0
...
If you run icc with the -v option you'll see that a program called mcpcom gets
run and produces the error. At least with version 10.0.026 it was a 32 bit
program:
$ file /opt/intel/cce/10.0.026/bin/mcpcom
/opt/intel/cce/10.0.026/bin/mcpcom: ELF 32-bit LSB executable, Intel 80386,
version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs),
for GNU/Linux 2.2.5, not stripped
Anyway I hope that helps.
-Aaron
On Jan 17, 2010, at 11:13 AM, Brian Smith wrote:
> Hi,
>
> I'm new to Gluster, so please forgive any ignorance on my part. I've
> tried reading over everything I could find to resolve this myself.
>
> I'm having an issue building executables on our gluster volume (there
> are some other similar issues as well, but we'll deal with this one
> first). I have the presta infiniband utilities extracted into a
> directory in my glusterfs volume and I'm attempting to build them. One
> of the source files, util.c (coincidentally, the first one referenced in
> the make file) fails to build. I get an error message:
>
> Catastrophic error: could not open source file "util.c"
>
> Some further investigating reveals the command that causes the issue:
>
> icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
> -L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib -lmpi
> -lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath -lnuma
> -ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
> Catastrophic error: could not open source file "util.c"
>
> compilation aborted for util.c (code 4)
>
> And further, we see an strace of the command:
> ...
> 2717 6601 brk(0x957b000)= 0x957b000
> 2718 6601 open("util.c", O_RDONLY) = 3
> 2719 6601 stat64("util.c", {st_mode=S_IFREG|0600, st_size=14370, ...})
> = 0
> 2720 6601 close(3) = 0
> 2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
> 1), ...}) = 0
> 2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
> MAP_ANONYMOUS, -1, 0x1000) = 0xf0bce000
> 2723 6601 write(2, "Catastrophic error: could not open source file
> \"util.c\"\n", 56) = 56
> 2724 6601 write(2, "\n", 1) = 1
> 2725 6601 exit_group(4)
> ...
>
> We can also see that the file is indeed present:
>
> $ ls -l util.c
> -rw--- 1 brs users 14370 Apr 8 2002 util.c
> $ stat util.c
> File: `util.c'
> Size: 14370 Blocks: 32 IO Block: 4096 regular file
> Device: 18h/24d Inode: 4295774314 Links: 1
> Access: (0600/-rw---) Uid: (1229806/ brs) Gid: (10001/
> users)
> Access: 2010-01-17 00:27:43.0 -0500
> Modify: 2002-04-08 21:13:57.0 -0400
> Change: 2010-01-17 00:27:43.0 -0500
>
> If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
> above works correctly.
>
> diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
>
> also appears clean. Any ideas what is happening here? Is there an
> issue with mmap2() and glusterfs?
>
> Many thanks in advance,
>
> -Brian
>
>
>
> --
> Brian Smith
> Senior Systems Administrator
> IT Research Computing, University of South Florida
> 4202 E. Fowler Ave. ENB308
> Office Phone: +1 813 974-1467
> Organization URL: http://rc.usf.edu
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
[Gluster-users] open() issues on 3.0.0
Hi,
I'm new to Gluster, so please forgive any ignorance on my part. I've
tried reading over everything I could find to resolve this myself.
I'm having an issue building executables on our gluster volume (there
are some other similar issues as well, but we'll deal with this one
first). I have the presta infiniband utilities extracted into a
directory in my glusterfs volume and I'm attempting to build them. One
of the source files, util.c (coincidentally, the first one referenced in
the make file) fails to build. I get an error message:
Catastrophic error: could not open source file "util.c"
Some further investigating reveals the command that causes the issue:
icc -I/opt/priv/openmpi-1.4.1/intel-10.1.018/include -pthread
-L/opt/priv/mx/lib -L/opt/priv/openmpi-1.4.1/intel-10.1.018/lib -lmpi
-lopen-rte -lopen-pal -lmyriexpress -libverbs -lpsm_infinipath -lnuma
-ldl -Wl,--export-dynamic -lnsl -lutil -c util.c
Catastrophic error: could not open source file "util.c"
compilation aborted for util.c (code 4)
And further, we see an strace of the command:
...
2717 6601 brk(0x957b000)= 0x957b000
2718 6601 open("util.c", O_RDONLY) = 3
2719 6601 stat64("util.c", {st_mode=S_IFREG|0600, st_size=14370, ...})
= 0
2720 6601 close(3) = 0
2721 6601 fstat64(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136,
1), ...}) = 0
2722 6601 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|
MAP_ANONYMOUS, -1, 0x1000) = 0xf0bce000
2723 6601 write(2, "Catastrophic error: could not open source file
\"util.c\"\n", 56) = 56
2724 6601 write(2, "\n", 1) = 1
2725 6601 exit_group(4)
...
We can also see that the file is indeed present:
$ ls -l util.c
-rw--- 1 brs users 14370 Apr 8 2002 util.c
$ stat util.c
File: `util.c'
Size: 14370 Blocks: 32 IO Block: 4096 regular file
Device: 18h/24d Inode: 4295774314 Links: 1
Access: (0600/-rw---) Uid: (1229806/ brs) Gid: (10001/
users)
Access: 2010-01-17 00:27:43.0 -0500
Modify: 2002-04-08 21:13:57.0 -0400
Change: 2010-01-17 00:27:43.0 -0500
If I extract the tar-ball in /tmp, a local ext3 fs, the compile line
above works correctly.
diff /work/b/brs/presta1.2/util.c /tmp/presta1.2/util.c
also appears clean. Any ideas what is happening here? Is there an
issue with mmap2() and glusterfs?
Many thanks in advance,
-Brian
--
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
