[Gluster-users] Kernel panic when using FUSE
Hi, I've run into a kernel bug, triggered by using gluster through FUSE. Wondering if any of you have encountered a similar problem and have worked around it by using an updated version of the kernel, gluster, or FUSE. Here's the bug: KERNEL: /usr/lib/debug/lib/modules/2.6.32-220.el6.x86_64/vmlinux DUMPFILE: /var/crash/127.0.0.1-2013-12-21-13:55:32/vmcore [PARTIAL DUMP] CPUS: 64 DATE: Sat Dec 21 13:52:33 2013 UPTIME: 16 days, 23:17:25 LOAD AVERAGE: 3.22, 3.36, 3.57 TASKS: 5598 NODENAME: uca-amm3.cnda.avaya.com RELEASE: 2.6.32-220.el6.x86_64 VERSION: #1 SMP Wed Nov 9 08:03:13 EST 2011 MACHINE: x86_64 (2892 Mhz) MEMORY: 32 GB PANIC: kernel BUG at fs/inode.c:322! PID: 259 COMMAND: kswapd1 TASK: 880433108b00 [THREAD_INFO: 88043310e000] CPU: 8 STATE: TASK_RUNNING (PANIC) crash bt PID: 259TASK: 880433108b00 CPU: 8 COMMAND: kswapd1 #0 [88043310f7b0] machine_kexec at 81031fcb #1 [88043310f810] crash_kexec at 810b8f72 #2 [88043310f8e0] oops_end at 814f04b0 #3 [88043310f910] die at 8100f26b #4 [88043310f940] do_trap at 814efda4 #5 [88043310f9a0] do_invalid_op at 8100ce35 #6 [88043310fa40] invalid_op at 8100bedb [exception RIP: clear_inode+248] RIP: 81190fb8 RSP: 88043310faf0 RFLAGS: 00010202 RAX: RBX: 8800245ce980 RCX: 0001 RDX: RSI: 0001 RDI: 8800245ce980 RBP: 88043310fb00 R8: R9: 000d R10: 0002 R11: 0001 R12: 81fbf340 R13: R14: 8800888014f8 R15: 0001 ORIG_RAX: CS: 0010 SS: 0018 #7 [88043310fb08] generic_delete_inode at 811916f6 #8 [88043310fb38] iput at 81190612 #9 [88043310fb58] dentry_iput at 8118d170 #10 [88043310fb78] d_kill at 8118d2d1 #11 [88043310fb98] __shrink_dcache_sb at 8118d666 #12 [88043310fc38] shrink_dcache_memory at 8118d7e9 #13 [88043310fc98] shrink_slab at 811299aa #14 [88043310fcf8] balance_pgdat at 8112c75d #15 [88043310fe28] kswapd at 8112caf6 #16 [88043310fee8] kthread at 81090886 #17 [88043310ff48] kernel_thread at 8100c14a This bug is reported here: http://fuse.996288.n3.nabble.com/Kernel-panic-when-using-fuse-with-distributed-filesystem-while-doing-heavy-i-o-td10373.html, which shows that this bug is reported on two different bugzilla bug reports https://bugzilla.redhat.com/show_bug.cgi?id=644085 and https://bugzilla.kernel.org/show_bug.cgi?id=15927. All of these reports mention using FUSE with high volume on the 2.6 kernel (which we are using). There's another entry on the red hat network, but I can't see it (don't have the subscription) https://access.redhat.com/site/solutions/72383 Thanks, Anne ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] adding bricks to replicated volume
Hi, I've seen this question asked a couple of times here http://www.gluster.org/pipermail/gluster-users/2013-July/036414.html and here http://supercolony.gluster.org/pipermail/gluster-users/2013-June/036272.html but never fully answered, unless I'm reading the answers wrongly. I understand that bricks are replicated in pairs (with rep 2) in the order in which they are added, so for a 3-node cluster we're planning on having 6 bricks, 2 per node, and replicating in a pair-wise fashion, so that one node never holds both bricks of a replica. My question is what to do when you add another node to the cluster. It would have two bricks that are replicas of each other. a) Is there a way of changing these pairings later on, even if just for the new data? b) Could I take down the volume and add the bricks in one at a time again? Thanks, Anne ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Can't access volume during self-healing
I ended up changing the directory structure so instead of having everything in one long directory, it was hierarchical, one dir per day, then one dir per 10 minutes. Most accesses are in the most recent directory. This gives just a 4-second interval where everything is locked up. Works for me. Anne From: gluster-users-boun...@gluster.org [mailto:gluster-users-boun...@gluster.org] On Behalf Of Harshavardhana Sent: Friday, October 11, 2013 3:36 AM To: Toby Corkindale Cc: gluster-users Subject: Re: [Gluster-users] Can't access volume during self-healing I've posted to the list about this issue before actually. We had/have a similar requirement for storing a very large number of fairly small files, and originally had them all in just a few directories in glusterfs. Directory layout also matters here number of files v/s number of directory hierarchy, also necessary to know is how does the application reach to these individual files (access patterns) It turns out that Glusterfs is really badly suited to directories with large numbers of files in them. If you can split them up, do so, and performance will become tolerable again. But even then it wasn't great.. Self-heal can swamp the network, making access for clients so slow as to cause problems. This analysis is wrong - self-heal daemon runs in lower priority threads and shouldn't be swamping the network at all. It never competes by default against User i/o traffic. Which was the version this was tested against? For your use case (wanting distributed, replicated storage for large numbers of 1mb files) I suggest you check out Riak and the Riak CS add-on. It's proven to be great for that particular use-case for us. Including all of that there is a fair amount of tuning which should be done at kernel, network and filesystem level as well. NoSQL's such as Riak could be beneficial but again are based on use-case basis. -- Religious confuse piety with mere ritual, the virtuous confuse regulation with outcomes ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
