Re: [Gluster-users] GPG key missing
thanks, working now Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-04-09 18:51 GMT+02:00 Chen Chen : > Hi Mathieu, > > It changed to " > http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/rsa.pub"; > now. > > Best wishes, > Chen > > > On 4/10/2016 12:45 AM, Mathieu Chateau wrote: > >> Hello, >> >> when trying to update a centos with Gluster: >> >> warning: >> >> /var/cache/yum/x86_64/7/glusterfs-epel/packages/glusterfs-3.7.10-1.el7.x86_64.rpm: >> Header V4 RSA/SHA256 Signature, key ID d5dc52dc: NOKEY >> Retrieving key from >> http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key >> >> >> GPG key retrieval failed: [Errno 14] HTTP Error 404 - Not Found >> >> >> Can someone fix this missing gpg key ? >> >> thanks >> >> Mathieu CHATEAU >> http://www.lotp.fr >> >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> >> > -- > Chen Chen > Shanghai SmartQuerier Biotechnology Co., Ltd. > Add: Room 410, 781 Cai Lun Road, China (Shanghai) Pilot Free Trade Zone > Shanghai 201203, P. R. China > Mob: +86 15221885893 > Email: chenc...@smartquerier.com > Web: www.smartquerier.com > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] GPG key missing
Hello, when trying to update a centos with Gluster: warning: /var/cache/yum/x86_64/7/glusterfs-epel/packages/glusterfs-3.7.10-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID d5dc52dc: NOKEY Retrieving key from http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key GPG key retrieval failed: [Errno 14] HTTP Error 404 - Not Found Can someone fix this missing gpg key ? thanks Mathieu CHATEAU http://www.lotp.fr ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] GlusterFS v3.7.8 client leaks summary — part II
Hello, when will these patch be included in gluster official version ? I upgraded my client to 3.7.8, but still have big leak following rsync jobs PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 1570 root 20 0 7404740 *6.210g* 4108 S 0.0 *40.0* 106:24.02 glusterfs 1573 root 20 0 3796044 2.924g 3580 S 0.0 18.8 7:07.05 glusterfs 1571 root 20 0 2469924 1.695g 3588 S 0.0 10.9 1:19.75 glusterfs thanks Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-02-16 18:54 GMT+01:00 Soumya Koduri : > > > On 02/16/2016 08:06 PM, Oleksandr Natalenko wrote: > >> Hmm, OK. I've rechecked 3.7.8 with the following patches (latest >> revisions): >> >> === >> Soumya Koduri (3): >>gfapi: Use inode_forget in case of handle objects >>inode: Retire the inodes from the lru list in inode_table_destroy >>rpc: Fix for rpc_transport_t leak >> === >> >> Here is Valgrind output: [1] >> >> It seems that all leaks are gone, and that is very nice. >> > > At least major chunk of leaks seem to have gone. Many thanks to you too > for very detailed tests and analysis :) > > -Soumya > > > >> Many thanks to all devs. >> >> [1] https://gist.github.com/anonymous/eddfdaf3eb7bff458326 >> >> 16.02.2016 15:30, Soumya Koduri wrote: >> >>> I have tested using your API app (I/Os done - create,write and stat). >>> I still do not see any inode related leaks. However I posted another >>> fix for rpc_transport object related leak [1]. >>> >>> I request you to re-check if you have the latest patch of [2] applied >>> in your build. >>> >>> [1] http://review.gluster.org/#/c/13456/ >>> [2] http://review.gluster.org/#/c/13125/ >>> >> ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Backup of 48126852 files / 9.1 TB data
Hello, On gluster client and server, did you disable atime & co ? Did you check for network bottleneck ? You are now using network twice: -One way to read data through glusterfs, -One way to push data remotely somewhere I am using rsnapshot on my side, so it's doing hardlink to same files, maybe it goes faster than true full copy. Also problem raise with folder containing a lot of small files in replicated setup. Which gluster version are you using ? I also experience memory leakage from server doing rsync (glusterfs client leak), but we are aware and patched has been pushed for version 3.7.8 Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-02-14 10:56 GMT+01:00 Nico Schottelius < nico-gluster-us...@schottelius.org>: > Hello everyone, > > we have a 2 brick setup running on a raid6 with 19T storage. > > We are currently facing the problem that the backup (9.1 TB data in > 48126852 files) is taking more than a week when being backed up by > means of rsync (actually, ccollect[0]). > > During backup the rsync process is continously in D state (expected), > but cpu load is far from 100% and disk is also only about 15-30% busy. > > (this is snapshot from right now) > > I have two questions, the second one more important: > > a) Is there a good way to identify the bottleneck? > b) Is it "safe" to backup data directly from the underlying > filesystem instead of going via the glusterfs mount? > > The reason why I ask about (b) is that we used to backup from those > servers *before* we switched to glusterfs within about a day and thus > I suspect backing up from the xfs filesystem again should do the job. > > Thanks for any hints, > > Nico > > > [0] http://www.nico.schottelius.org/software/ccollect/ > > -- > Werde Teil des modernen Arbeitens im Glarnerland auf www.digitalglarus.ch! > Lese Neuigkeiten auf Twitter: www.twitter.com/DigitalGlarus > Diskutiere mit auf Facebook: www.facebook.com/digitalglarus > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Data reconstruction from an EC volume
Hello, not sure to understand. Gluster store files as regular ones, which is one big difference against ceph and others that store container/blocks. Just mount disk as normal one, your data should be there. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-24 11:37 GMT+01:00 Serkan Çoban : > Hi, > > I would like to know if it is possible to reconstruct data from an EC > volume without gluster online? If it is possible do you know any tool > that does this? > > Thanks, > Serkan > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] [Gluster-devel] Memory leak in GlusterFS FUSE client
Thanks for all your tests and times, it looks promising :) Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko : > OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the > following > patches: > > === > Kaleb S KEITHLEY (1): > fuse: use-after-free fix in fuse-bridge, revisited > > Pranith Kumar K (1): > mount/fuse: Fix use-after-free crash > > Soumya Koduri (3): > gfapi: Fix inode nlookup counts > inode: Retire the inodes from the lru list in inode_table_destroy > upcall: free the xdr* allocations > === > > I run rsync from one GlusterFS volume to another. While memory started from > under 100 MiBs, it stalled at around 600 MiBs for source volume and does > not > grow further. As for target volume it is ~730 MiBs, and that is why I'm > going > to do several rsync rounds to see if it grows more (with no patches bare > 3.7.6 > could consume more than 20 GiBs). > > No "kernel notifier loop terminated" message so far for both volumes. > > Will report more in several days. I hope current patches will be > incorporated > into 3.7.7. > > On пʼятниця, 22 січня 2016 р. 12:53:36 EET Kaleb S. KEITHLEY wrote: > > On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote: > > > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote: > > >> I presume by this you mean you're not seeing the "kernel notifier loop > > >> terminated" error in your logs. > > > > > > Correct, but only with simple traversing. Have to test under rsync. > > > > Without the patch I'd get "kernel notifier loop terminated" within a few > > minutes of starting I/O. With the patch I haven't seen it in 24 hours > > of beating on it. > > > > >> Hmmm. My system is not leaking. Last 24 hours the RSZ and VSZ are > > >> stable: > > >> > http://download.gluster.org/pub/gluster/glusterfs/dynamic-analysis/longev > > >> ity /client.out > > > > > > What ops do you perform on mounted volume? Read, write, stat? Is that > > > 3.7.6 + patches? > > > > I'm running an internally developed I/O load generator written by a guy > > on our perf team. > > > > it does, create, write, read, rename, stat, delete, and more. > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] RAID on GLUSTER node
Hello, You should also check how are performance when 96TB volume is degraded (remove one disk) and rebuilding (after putting back disk). You can adjust performance impact of rebuild rate, but it will take longer to complete (during which another disk may fail). Also reboot one node while copying file, to ensure that files healing can handle your files count. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-18 4:21 GMT+01:00 Pawan Devaiah : > Hi Guys, > > Sorry I was busy with setting up those 2 machines for GlusterFS > So my machines now have > 32 GB memory > 7 TB RAID 10 Storage > 94 TB Raid 6 Storage > > I have setup the initials bricks and cluster is formed and working well. > Although I haven't checked from client side. > > I just wanted to ask what should be the idle size of the brick, for now I > have made the entire 7TB volume as one brick, is that ok? > Is there a best practice for brick size? > > @ Pranith: Yes for now I want to test this system. > > @Mathieu : Even I think it will be perform better with RAID 10 for read > write intensive workloads. > > Cheers, > Dev > > On Wed, Jan 13, 2016 at 9:05 PM, Mathieu Chateau > wrote: > >> RAID 10 provide best performance (much better than raid 6) >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2016-01-13 5:35 GMT+01:00 Pranith Kumar Karampuri : >> >>> +gluster-users >>> >>> On 01/13/2016 09:44 AM, Pawan Devaiah wrote: >>> >>> We would be looking for redundancy so replicated volumes I guess >>> >>> If replication is going to be there, why additional RAID10? You can do >>> just RAID6, it saves on space and replication in glusterfs will give >>> redundancy anyways. >>> >>> Pranith >>> >>> >>> Thanks, >>> Dev >>> >>> On Wed, Jan 13, 2016 at 5:07 PM, Pranith Kumar Karampuri < >>> pkara...@redhat.com> wrote: >>> >>>> >>>> >>>> On 01/13/2016 02:21 AM, Pawan Devaiah wrote: >>>> >>>> Thanks for the response Pranith >>>> >>>> If we take EC out of the equation and say I go with RAID on the >>>> physical disk, do you think GlusterFS is good for the 2 workloads that I >>>> mentioned before. >>>> >>>> Basically it is going to be a NFS storage for VM and data but with >>>> different RAIDs, 10 for VM and 6 for data. >>>> >>>> What will be the kind of volume you will be using with these disks? >>>> >>>> Pranith >>>> >>>> Thanks >>>> Dev >>>> >>>> On Tue, Jan 12, 2016 at 9:46 PM, Pranith Kumar Karampuri < >>>> pkara...@redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On 01/12/2016 01:26 PM, Pawan Devaiah wrote: >>>>> >>>>> Thanks for your response Pranith and Mathieu, >>>>> >>>>> Pranith: To answer your question, I am planning to use this storage >>>>> for two main workloads. >>>>> >>>>> 1. As a shared storage for VMs. >>>>> >>>>> EC as it is today is not good for this. >>>>> >>>>> 2. As a NFS Storage for files. >>>>> >>>>> If the above is for storing archive data. EC is nice here. >>>>> >>>>> Pranith >>>>> >>>>> >>>>> We are a online backup company so we store few hundred Terra bytes of >>>>> data. >>>>> >>>>> >>>>> Mathieu: I appreciate your concern, however as a system admins >>>>> sometimes we get paranoid and try to control everything under the Sun. >>>>> I know I can only control what I can. >>>>> >>>>> Having said that, No, I have pair of servers to start with so at the >>>>> moment I am just evaluating and preparing for proof of concept, after >>>>> which >>>>> I am going to propose to my management, if they are happy then we will >>>>> proceed further. >>>>> >>>>> Regards, >>>>> Dev >>>>> >>>>> On Tue, Jan 12, 2016 at 7:30 PM, Mathieu Chateau < >>>>> mathieu.chat...@lotp.fr> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> For any system, 36 disks raise disk failure probability. Do you plan >>>>&
Re: [Gluster-users] RAID on GLUSTER node
RAID 10 provide best performance (much better than raid 6) Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-13 5:35 GMT+01:00 Pranith Kumar Karampuri : > +gluster-users > > On 01/13/2016 09:44 AM, Pawan Devaiah wrote: > > We would be looking for redundancy so replicated volumes I guess > > If replication is going to be there, why additional RAID10? You can do > just RAID6, it saves on space and replication in glusterfs will give > redundancy anyways. > > Pranith > > > Thanks, > Dev > > On Wed, Jan 13, 2016 at 5:07 PM, Pranith Kumar Karampuri < > pkara...@redhat.com> wrote: > >> >> >> On 01/13/2016 02:21 AM, Pawan Devaiah wrote: >> >> Thanks for the response Pranith >> >> If we take EC out of the equation and say I go with RAID on the physical >> disk, do you think GlusterFS is good for the 2 workloads that I mentioned >> before. >> >> Basically it is going to be a NFS storage for VM and data but with >> different RAIDs, 10 for VM and 6 for data. >> >> What will be the kind of volume you will be using with these disks? >> >> Pranith >> >> Thanks >> Dev >> >> On Tue, Jan 12, 2016 at 9:46 PM, Pranith Kumar Karampuri < >> pkara...@redhat.com> wrote: >> >>> >>> >>> On 01/12/2016 01:26 PM, Pawan Devaiah wrote: >>> >>> Thanks for your response Pranith and Mathieu, >>> >>> Pranith: To answer your question, I am planning to use this storage for >>> two main workloads. >>> >>> 1. As a shared storage for VMs. >>> >>> EC as it is today is not good for this. >>> >>> 2. As a NFS Storage for files. >>> >>> If the above is for storing archive data. EC is nice here. >>> >>> Pranith >>> >>> >>> We are a online backup company so we store few hundred Terra bytes of >>> data. >>> >>> >>> Mathieu: I appreciate your concern, however as a system admins sometimes >>> we get paranoid and try to control everything under the Sun. >>> I know I can only control what I can. >>> >>> Having said that, No, I have pair of servers to start with so at the >>> moment I am just evaluating and preparing for proof of concept, after which >>> I am going to propose to my management, if they are happy then we will >>> proceed further. >>> >>> Regards, >>> Dev >>> >>> On Tue, Jan 12, 2016 at 7:30 PM, Mathieu Chateau < >>> mathieu.chat...@lotp.fr> wrote: >>> >>>> Hello, >>>> >>>> For any system, 36 disks raise disk failure probability. Do you plan >>>> GlusterFS with only one server? >>>> >>>> You should think about failure at each level and be prepared for it: >>>> >>>>- Motherboard failure (full server down) >>>>- Disks failure >>>>- Network cable failure >>>>- File system corruption (time needed for fsck) >>>>- File/folder removed by mistake (backup) >>>> >>>> Using or not raid depend on your answer on these questions and >>>> performance needed. >>>> It also depend how "good" is raid controller in your server, like if it >>>> has battery and 1GB of cache. >>>> >>>> When many disks are bought at same time (1 order, serial number close >>>> to each other), they may fail in near time to each other (if something bad >>>> happened in manufactory). >>>> I already saw like 3 disks failing in few days. >>>> >>>> just my 2 cents, >>>> >>>> >>>> >>>> Cordialement, >>>> Mathieu CHATEAU >>>> http://www.lotp.fr >>>> >>>> 2016-01-12 4:36 GMT+01:00 Pranith Kumar Karampuri >>>> : >>>> >>>>> >>>>> >>>>> On 01/12/2016 04:34 AM, Pawan Devaiah wrote: >>>>> >>>>> Hi All, >>>>> >>>>> We have a fairly powerful server sitting at office with 128 Gig RAM >>>>> and 36 X 4 TB drives. I am planning to utilize this server as a backend >>>>> storage with GlusterFS on it. >>>>> I have been doing lot of reading on Glusterfs, but I do not see any >>>>> definite recommendation on having RAID on GLUSTER nodes. >>>>> Is it recommended to have RAID on GL
Re: [Gluster-users] Poor and very slow performance on glusterfs mounted volume
Hello, we are aware of this, as you can find in mail list archives. You can get things better, but not completely remove this issue for now. It mostly depend on number of files in folder when doing ls. Small files provide much lower performance than big one during copy & co. Replicate option slow down read, as your client contact both servers to get metada. Things I did to get things better in this setup: fstab option (on client) : defaults,direct-io-mode=disable,_netdev,backupvolfile-server=mysecondservername fstab option (on gluster server volumes): defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier *sysctl (clients and servers):* vm.swappiness=0 net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem=4096 87380 33554432 net.ipv4.tcp_wmem=4096 65536 33554432 # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp *Gluster volume options:* performance.cache-size: 1GB performance.io-thread-count: 16 performance.readdir-ahead: enable performance.read-ahead: disable performance.client-io-threads: on performance.write-behind-window-size: 1MB cluster.lookup-optimize: on client.event-threads: 4 server.event-threads: 4 cluster.readdir-optimize: on Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-13 7:39 GMT+01:00 Arash Shams : > Hello > > Im using 2 standalone servers as share storage servers with glusterfs 3.7 > when i change directory and run command inside a mounted volume for each > operation i have to wait at least 5 seconds for result and prompt > Im using replicate option and delay maybe for write but not for reading > > im using gluster as my /var/www/ share along all of my instance and > servers but for a simple ls or cd command to this directory i have to wait > 5 seconds any suggestion for me ? > > here is some information about the servers and glusterfs : > > OS : Ubuntu 14.04.3 LTS > Version info : glusterfs 3.7.6 built on Nov 9 2015 15:17:05 > > Ping between 2 nodes result : > 4 packets transmitted, 4 received, 0% packet loss, time 2998ms > rtt min/avg/max/mdev = 0.042/0.056/0.071/0.014 ms > > iperf : > Interval TransferBandwidth > 0.0-10.0 sec 10.5 GBytes 9.03 Gbits/sec > > dd test : > 1573468160 bytes (1.6 GB) copied, 70.8977 s, 22.2 MB/s > > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Memory leak in GlusterFS FUSE client
I tried like suggested: echo 3 > /proc/sys/vm/drop_caches sync It lower a bit usage: before: [image: Images intégrées 2] after: [image: Images intégrées 1] Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-12 7:34 GMT+01:00 Mathieu Chateau : > Hello, > > I also experience high memory usage on my gluster clients. Sample : > [image: Images intégrées 1] > > Can I help in testing/debugging ? > > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2016-01-12 7:24 GMT+01:00 Soumya Koduri : > >> >> >> On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote: >> >>> Brief test shows that Ganesha stopped leaking and crashing, so it seems >>> to be good for me. >>> >>> Thanks for checking. >> >> Nevertheless, back to my original question: what about FUSE client? It >>> is still leaking despite all the fixes applied. Should it be considered >>> another issue? >>> >> >> For fuse client, I tried vfs drop_caches as suggested by Vijay in an >> earlier mail. Though all the inodes get purged, I still doesn't see much >> difference in the memory footprint drop. Need to investigate what else is >> consuming so much memory here. >> >> Thanks, >> Soumya >> >> >> >>> 11.01.2016 12:26, Soumya Koduri написав: >>> >>>> I have made changes to fix the lookup leak in a different way (as >>>> discussed with Pranith) and uploaded them in the latest patch set #4 >>>> - http://review.gluster.org/#/c/13096/ >>>> >>>> Please check if it resolves the mem leak and hopefully doesn't result >>>> in any assertion :) >>>> >>>> Thanks, >>>> Soumya >>>> >>>> On 01/08/2016 05:04 PM, Soumya Koduri wrote: >>>> >>>>> I could reproduce while testing deep directories with in the mount >>>>> point. I root caus'ed the issue & had discussion with Pranith to >>>>> understand the purpose and recommended way of taking nlookup on inodes. >>>>> >>>>> I shall make changes to my existing fix and post the patch soon. >>>>> Thanks for your patience! >>>>> >>>>> -Soumya >>>>> >>>>> On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote: >>>>> >>>>>> OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most >>>>>> recent >>>>>> revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision >>>>>> too). >>>>>> >>>>>> On traversing GlusterFS volume with many files in one folder via NFS >>>>>> mount I >>>>>> get an assertion: >>>>>> >>>>>> === >>>>>> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >>>>>> >= >>>>>> nlookup' failed. >>>>>> === >>>>>> >>>>>> I used GDB on NFS-Ganesha process to get appropriate stacktraces: >>>>>> >>>>>> 1. short stacktrace of failed thread: >>>>>> >>>>>> https://gist.github.com/7f63bb99c530d26ded18 >>>>>> >>>>>> 2. full stacktrace of failed thread: >>>>>> >>>>>> https://gist.github.com/d9bc7bc8f6a0bbff9e86 >>>>>> >>>>>> 3. short stacktrace of all threads: >>>>>> >>>>>> https://gist.github.com/f31da7725306854c719f >>>>>> >>>>>> 4. full stacktrace of all threads: >>>>>> >>>>>> https://gist.github.com/65cbc562b01211ea5612 >>>>>> >>>>>> GlusterFS volume configuration: >>>>>> >>>>>> https://gist.github.com/30f0129d16e25d4a5a52 >>>>>> >>>>>> ganesha.conf: >>>>>> >>>>>> https://gist.github.com/9b5e59b8d6d8cb84c85d >>>>>> >>>>>> How I mount NFS share: >>>>>> >>>>>> === >>>>>> mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o >>>>>> defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100 >>>>>> === >>>>>> >>>>>> On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote: >>>>>> >>>>>>> Entries_HWMark = 500; >>>>>>> >>>>>> >>>>>> >>>>>> ___ >>>>> Gluster-users mailing list >>>>> Gluster-users@gluster.org >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Memory leak in GlusterFS FUSE client
Hello, I also experience high memory usage on my gluster clients. Sample : [image: Images intégrées 1] Can I help in testing/debugging ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-12 7:24 GMT+01:00 Soumya Koduri : > > > On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote: > >> Brief test shows that Ganesha stopped leaking and crashing, so it seems >> to be good for me. >> >> Thanks for checking. > > Nevertheless, back to my original question: what about FUSE client? It >> is still leaking despite all the fixes applied. Should it be considered >> another issue? >> > > For fuse client, I tried vfs drop_caches as suggested by Vijay in an > earlier mail. Though all the inodes get purged, I still doesn't see much > difference in the memory footprint drop. Need to investigate what else is > consuming so much memory here. > > Thanks, > Soumya > > > >> 11.01.2016 12:26, Soumya Koduri написав: >> >>> I have made changes to fix the lookup leak in a different way (as >>> discussed with Pranith) and uploaded them in the latest patch set #4 >>> - http://review.gluster.org/#/c/13096/ >>> >>> Please check if it resolves the mem leak and hopefully doesn't result >>> in any assertion :) >>> >>> Thanks, >>> Soumya >>> >>> On 01/08/2016 05:04 PM, Soumya Koduri wrote: >>> >>>> I could reproduce while testing deep directories with in the mount >>>> point. I root caus'ed the issue & had discussion with Pranith to >>>> understand the purpose and recommended way of taking nlookup on inodes. >>>> >>>> I shall make changes to my existing fix and post the patch soon. >>>> Thanks for your patience! >>>> >>>> -Soumya >>>> >>>> On 01/07/2016 07:34 PM, Oleksandr Natalenko wrote: >>>> >>>>> OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most >>>>> recent >>>>> revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision >>>>> too). >>>>> >>>>> On traversing GlusterFS volume with many files in one folder via NFS >>>>> mount I >>>>> get an assertion: >>>>> >>>>> === >>>>> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >= >>>>> nlookup' failed. >>>>> === >>>>> >>>>> I used GDB on NFS-Ganesha process to get appropriate stacktraces: >>>>> >>>>> 1. short stacktrace of failed thread: >>>>> >>>>> https://gist.github.com/7f63bb99c530d26ded18 >>>>> >>>>> 2. full stacktrace of failed thread: >>>>> >>>>> https://gist.github.com/d9bc7bc8f6a0bbff9e86 >>>>> >>>>> 3. short stacktrace of all threads: >>>>> >>>>> https://gist.github.com/f31da7725306854c719f >>>>> >>>>> 4. full stacktrace of all threads: >>>>> >>>>> https://gist.github.com/65cbc562b01211ea5612 >>>>> >>>>> GlusterFS volume configuration: >>>>> >>>>> https://gist.github.com/30f0129d16e25d4a5a52 >>>>> >>>>> ganesha.conf: >>>>> >>>>> https://gist.github.com/9b5e59b8d6d8cb84c85d >>>>> >>>>> How I mount NFS share: >>>>> >>>>> === >>>>> mount -t nfs4 127.0.0.1:/mail_boxes /mnt/tmp -o >>>>> defaults,_netdev,minorversion=2,noac,noacl,lookupcache=none,timeo=100 >>>>> === >>>>> >>>>> On четвер, 7 січня 2016 р. 12:06:42 EET Soumya Koduri wrote: >>>>> >>>>>> Entries_HWMark = 500; >>>>>> >>>>> >>>>> >>>>> ___ >>>> Gluster-users mailing list >>>> Gluster-users@gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>> ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] RAID on GLUSTER node
Hello, For any system, 36 disks raise disk failure probability. Do you plan GlusterFS with only one server? You should think about failure at each level and be prepared for it: - Motherboard failure (full server down) - Disks failure - Network cable failure - File system corruption (time needed for fsck) - File/folder removed by mistake (backup) Using or not raid depend on your answer on these questions and performance needed. It also depend how "good" is raid controller in your server, like if it has battery and 1GB of cache. When many disks are bought at same time (1 order, serial number close to each other), they may fail in near time to each other (if something bad happened in manufactory). I already saw like 3 disks failing in few days. just my 2 cents, Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-12 4:36 GMT+01:00 Pranith Kumar Karampuri : > > > On 01/12/2016 04:34 AM, Pawan Devaiah wrote: > > Hi All, > > We have a fairly powerful server sitting at office with 128 Gig RAM and 36 > X 4 TB drives. I am planning to utilize this server as a backend storage > with GlusterFS on it. > I have been doing lot of reading on Glusterfs, but I do not see any > definite recommendation on having RAID on GLUSTER nodes. > Is it recommended to have RAID on GLUSTER nodes specially for the bricks? > If Yes, is it not contrary to the latest Erasure code implemented in > Gluster or is it still not ready for production environment? > I am happy to implement RAID but my two main concern are > 1. I want to make most of the disk space available. > 2. I am also concerned about the rebuild time after disk failure on the > RAID. > > What is the workload you have? > > We found in our testing that random read/write workload with Erasure coded > volumes is not as good as we get with replication. There are enhancements > in progress at the moment to address these things which we are yet to merge > and re-test. > > Pranith > > > Thanks > Dev > > > > > ___ > Gluster-users mailing > listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster - Performance issue while copying bulk files/folders
Hello, did you test performance of storage brick itself before setting up ? How are they connecting to NetApp for their storage? nfs?iscsi? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-12-11 7:02 GMT+01:00 Vijay Bellur : > On 12/09/2015 07:03 AM, Srikanth Mampilakal wrote: > >> However, if I do dd to check the copy speed, I get the below result. >> >> [root@ClientServer ~]# time sh -c "dd if=/dev/zero >> of=/mnt/testmount/test.tmp bs=4k count=2 && sync" >> 2+0 records in >> 2+0 records out >> 8192 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s >> >> real0m17.337s >> user0m0.031s >> sys 0m0.317s >> > > > The block size used is not ideal for the gluster native client. Can you > please attempt with a higher block size - 64k or 128k & check the > throughput? > > Regards, > Vijay > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS Performance tuning
Hello, is it for small files with lot of them, or mainly big ones? When mounting from client, i add direct-io-mode=disable in fstab sysctl setting I use (I don't have 10G network, may not be enough numbers) (on clients and servers): vm.swappiness=0 net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem=4096 87380 33554432 net.ipv4.tcp_wmem=4096 65536 33554432 # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp Options I have reconfigured for small files tuning: performance.cache-size: 1GB nfs.disable: on performance.client-io-threads: on performance.io-cache: on performance.io-thread-count: 16 performance.readdir-ahead: enable performance.read-ahead: disable server.allow-insecure: on cluster.lookup-optimize: on client.event-threads: 4 server.event-threads: 4 cluster.readdir-optimize: on performance.write-behind-window-size: 1MB Again, you should try but even with bigger number based on your hardware specs. Did you set MTU to 9000 ? (jumbo frame) Did you set in bios power settings to high static mode (name depend on vendor) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-11-25 15:44 GMT+01:00 Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] < uthra.r@nasa.gov>: > Thank you all for taking the time to reply to my email: > > > > Here is some more information on our setup: > > - Number of Nodes à 2 Gluster servers and 1 client for testing. After > testing we will mount the GlusterFS volume on 3 clients. > > - CPU & RAM on Each Node à 2 CPUs 3.4MHz, 384GB RAM on each Gluster Server > > - What else is running on the nodes à Nothing it is only our data server > > - Number of bricks à Two > > - output of "gluster volume info" & "gluster volume status" > > > > *Storage server1:* > > # *gluster volume info gtower* > > Volume Name: gtower > > Type: Replicate > > Volume ID: 838ab806-06d9-45c5-8d88-2a905c167dba > > Status: Started > > Number of Bricks: 1 x 2 = 2 > > Transport-type: tcp > > Bricks: > > Brick1: storage1.sci.gsfc.nasa.gov:/tower7/gluster1/brick > > Brick2: storage2.sci.gsfc.nasa.gov:/tower8/gluster2/brick > > Options Reconfigured: > > nfs.export-volumes: off > > nfs.addr-namelookup: off > > performance.readdir-ahead: on > > performance.cache-size: 2GB > > > > --- > > > > *Storage server 2:* > > # *gluster volume info gtower* > > Volume Name: gtower > > Type: Replicate > > Volume ID: 838ab806-06d9-45c5-8d88-2a905c167dba > > Status: Started > > Number of Bricks: 1 x 2 = 2 > > Transport-type: tcp > > Bricks: > > Brick1: storage1.sci.gsfc.nasa.gov:/tower7/gluster1/brick > > Brick2: storage2.sci.gsfc.nasa.gov:/tower8/gluster2/brick > > Options Reconfigured: > > nfs.export-volumes: off > > nfs.addr-namelookup: off > > performance.readdir-ahead: on > > performance.cache-size: 2GB > > > > - > > > > We have made a raidz3 consisting of 6 vdevs each consisting of 12 (6TB) > drives and assigned one 200GB SSD drive for ZFS caching. > > > > Our attached storage has 60 (6TB) drives for which I have done > multipathing. We are also using 12 drives in the server for which I have > set-up vdevs. So we are using 60+12 = 72 drives for ZFS (raidz3) > > > > > > If you have any other suggestions based on our configuration please let me > know. > > > > Thank you. > > Uthra > > > > > > > > > > > > *From:* Gmail [mailto:b.s.mikh...@gmail.com] > *Sent:* Tuesday, November 24, 2015 4:50 PM > *To:* Pierre MGP Pro > *Cc:* Lindsay Mathieson; Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC]; > gluster-users@gluster.org > *Subject:* Re: [Gluster-users] GlusterFS Performance tuning > > > > you can do the following: > > > > # gluster volume set $vol performance.o-thread-count 64 > > Today’s CPU are powerful enough to handle 64 threads per volume. > > > > # gluster volume set $vol client.event-threads XX > > XX depend on the number of connections from the FUSE client to the server, > you can get this number by running netstat and grep on the server IP and > count the number of connections. > > > > # gluster volume set $vol server.event-threads XX > > > > XX depend on the number of connections from the server to the client(s), > you can get this number by running netstat and grep on “gluster" and c
Re: [Gluster-users] Gluster considerations - replicated volumes in different sites
Hello, I suggest Tom to clarify if he intends geo-rep or replica. Tom, do you expect automatic failover to remote site if local is down ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-11-23 14:37 GMT+01:00 Kaleb S. KEITHLEY : > On 11/23/2015 03:26 AM, Tom Farrar wrote: > > Good Morning All, > > > > I'm looking at deploying 3 Gluster nodes, two in one location and the > > third in another. The link between these two locations is fast and > > fairly low latency, around ~4ms. The volumes are all low write/high read > > with the largest being a few TBs (with lots of small files). While the > > primary location will have two nodes in, the secondary location (with > > one node) will see local reads and writes. > > > > I'm a little concerned about running the replication in separate > > locations given that from what I've read Gluster doesn't like latency. > > Is this a valid concern? I've seen a few options for what I believe is > > keeping reads local so they don't go to the distant node, but I'm > > struggling to find a definitive answer (cluster.read-subvolumeperhaps). > > For reads, the clients will read from the server that responds the > fastest to the Lookup call. Usually that will be from the closest, > lowest latency, server. That's automatic, there is no configuration > required for that. > > Aside from that, using geo-rep to replicate to the remote system is > probably a good fit for your use case; clients only know about the > closest systems and never incur the hit for the synchronous write to the > remote system. > > -- > > Kaleb > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster considerations - replicated volumes in different sites
Hello, except for NFS, client will synchronously writes to each replica all time. They are also getting meta data from both. As it's synchronous, it's going as slower as slowest replica. replica with 2 nodes won't be useful as it's the slowest one that is important. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-11-23 9:26 GMT+01:00 Tom Farrar : > Good Morning All, > > I'm looking at deploying 3 Gluster nodes, two in one location and the > third in another. The link between these two locations is fast and fairly > low latency, around ~4ms. The volumes are all low write/high read with the > largest being a few TBs (with lots of small files). While the primary > location will have two nodes in, the secondary location (with one node) > will see local reads and writes. > > I'm a little concerned about running the replication in separate locations > given that from what I've read Gluster doesn't like latency. Is this a > valid concern? I've seen a few options for what I believe is keeping reads > local so they don't go to the distant node, but I'm struggling to find a > definitive answer (cluster.read-subvolume perhaps). > > Also, in terms of configuration for low write/high read and a large volume > of small files, the following seems to be the recommended from what I can > cobble together - does this seem ok? > > Options Reconfigured: > performance.readdir-ahead: on > cluster.ensure-durability: off > server.event-threads: 4 > cluster.lookup-optimize: on > performance.quick-read: on > cluster.readdir-optimize: on > > Many thanks, > > Tom > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster & Nagios False Alerts
Hello, I am monitoring with nagios (Centreon to be exact), but not yet with script provided by gluster dev. I currently monitor through nrpe: - Gluster brick: - Disk space - glusterd daemon (present and #) - glusterfs daemon (present and #) - glusterfsd daemon (present and #) - Gluster client: - checking if all mount points are present using this script <https://github.com/echocat/nagios-plugin-check_mountpoints> Monitoring do same things all the time. If it's reporting issue, but resolving quick, you may have a repeating short issue. Best shot in this case is to manually execute nagios script inside a loop, like every 10s, to cach it by yourself. If you wait for nagios alert to log on and try, short issue may already be gone. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-10-20 13:23 GMT+02:00 Gary Armstrong : > Hi Gluster experts, > > I have a three brick setup that I'm monitoring with Gluster, which throws > a warning when 2 bricks are detected and critical when 1 or less are > detected. > > Every now and again, seemingly randomly, I will get a critical warning on > one of the three servers, saying that no bricks are found. This is > incorrect as when you log onto the server and check gluster vol status all > three bricks are online and healthy. After a few minutes Nagios returns to > reporting a healthy volume. > > Does anyone else monitor their gluster volume with Nagios, and see random > critical alerts? > > Cheers, > Gary > > -- > GARY ARMSTRONG > Systems Engineer > Tibus > > T: +44 (0)28 9033 1122 > F: +44 (0)28 9042 4709 > E: garmstr...@tibus.com > W: www.tibus.com > > Follow us on Twitter @tibus > > Tibus is a trading name of The Internet Business Ltd, a company limited by > share capital and registered in Northern Ireland, NI31235. Tibus is a part > of UTV Media Plc. > > > > This e-mail and any attachment may contain confidential and privileged > information for the sole use of the intended recipient. Any review, use, > distribution or disclosure by others is strictly prohibited. If you are not > the intended recipient (or authorised to receive for the recipient), please > contact the sender by reply e-mail and delete all copies of this message. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] File limit in glusterfs
Hello, For now, main issue is number of files per folder, and the "ls -l" becoming super slow. All of this in replication mode, distributed setup are not impacted as far as I know. In replication mode, client check against all replicated node for metadata info, to ensure it's all good. My 2 cents Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-10-05 8:01 GMT+02:00 Kaamesh Kamalaaharan : > Hi everyone, > I was wondering if anyone has done any performance testing on gluster 3.62 > and above where they test the maximum number of files and folders you can > have before performance starts to drop. > > I read that gluster can support as many files as there are inodes and it > should support 2^64 files. Is there a cut off point where after a certain > number of files are put in a folder , the read/write performance drops off? > > Any relevant information would be helpful. > > Thanks, > Kaamesh > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster-Nagios
Hello, from internet I get "Not Found" for gluster-nagios* url Is it published ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-28 14:21 GMT+02:00 Ramesh Nachimuthu : > > > On 09/24/2015 10:21 PM, André Bauer wrote: > > I would also love to see packages for Ubuntu. > > Are the sources of the Nagios plugins available somewhere? > > > Yes. You can find them at http://review.gluster.org/ > > gluster-nagios-common : http://review.gluster.org/gluster-nagios-common > gluster-nagios-addons: http://review.gluster.org/gluster-nagios-addons > nagios-server-addons: http://review.gluster.org/nagios-server-addons > > Regards > André > > Am 20.09.2015 um 11:02 schrieb Prof. Dr. Michael Schefczyk: > > Dear All, > > In June 2014, the gluster-nagios team (thanks!) published the availability of > gluster-nagios-common and gluster-nagios-addons on this list. As far as I can > tell, this quite extensive gluster nagios monitoring tool is available for > el6 only. Are there known plans to make this available for el7 outside the > RHEL-repos > (http://ftp.redhat.de/pub/redhat/linux/enterprise/7Server/en/RHS/SRPMS/), > e.g. for use with oVirt / Centos 7 also? It would be good to be able to > monitor gluster without playing around with scripts from sources other than a > rpm repo. > > Regards, > > Michael > ___ > Gluster-users mailing > listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Losing disks on Glusterfs
Hello, As You will already have 5 copies (one per server), adding rendundancy on storage is wasty. I would do a 2 node distributed + replication: 2 servers * 4 disk = 8TB replicated to 2 others servers with same setup. The last server can be abitrer for split brain maybe, or you have to get another server to get 3 and 3. Yes in this setup it's a raid 0, so if you loose a disk, the whole 2 replicated server are down. But you will get much better performance and much more usable space, and still survive a single server crash. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-21 15:51 GMT+02:00 Bruno Andrade : > Hi there, > > We are setting up 5 new servers with 4 disks of 1 TB each. > > We are planning of using Glusterfs Distributed Replicated so we have > protection over our data. > > My question is, if I lose a disk, since we are using LVM mirror, are we > still able to access the data normally until we replace the broken disk. > And, in this scenario, out of the 20 disks, how many usable ones will I > have? > > Thanks > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Advice for auto-scaling
Hello, I am doing that in production for web farm. My experience: - Gluster is synchronous (client writes to all replicated nodes), so no issue with old content - Gluster is slwww with small files in replicated mode due to metadata - for configuration, I ended replicating locally instead for availability So it work as you can imagine (good), just slow Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-16 14:23 GMT+02:00 Paul Thomas : > Hi, > > I’m new to shared file systems and horizontal cloud scaling. > > I have already played with auto-scaling on aws/ec2. In term of spawning a > destroying and I can achieve that. > > I just want to some advice of how best implement syncing for web files, > infrastructure, data, etc. > > I have pretty much decided to put the database side of things on a private > instance. > I'll worry about db clustering later I’m not to bothered about this not, > because the software supports it. > > It seems logical to put the web folder / application layer on a shared > file system, maybe some configuration too. > > What I'm really unsure about is how to ensure that the current system is > up to date and the configuration tweaked for the physical specs. > > How do people typically approach this? I'm guessing it not always viable > to have a shared file system for everything. > > Is the approach a disciplined one? Where say I have development instance > for infrastructure changes. > Then there is a deployment flow where production instances are somehow > refreshed without downtime. > > Or is there some other approach? > > I notice on sites like yahoo, things are often noticeably unsynced, mostly > on the data front, but also other things. > This would be unacceptable in my case. > > I appreciate any help I can get regarding this. > > My typical load is from php-fpm/nginx processes, mysql bellow this. > > Should the memory cache also be separated, or as I think it is quite good > for this to be divided up with the infrastructure to support each public > instance individually? > > Paul > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Expanding replicated volume
Hello, in replicated setup, all bricks have all data. So you need to grow all bricks, or do a distributed + replicated volume I would stay with 2 bricks and extend current volume or add another volume to the current brick No need to change fstab. It will try first server1 and server2 as a backup if first is offline. Then gluster client get all nodes from the one it's connected too. So even if you have 50 bricks, no change in fstab. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-07 2:58 GMT+02:00 Antun Krasic : > Hello, > > I have two servers with replica 2 setup, with total of 380GB of disk space > ( gluster volume create datastore replica 2 transport tcp > server1:/datastore server2:/datastore). > > I'd need to add more capacity to the servers, I know I need to add 2 more > servers to the configuration, however do they need to be with the same disk > size ? Meaning, if I add a 2 VMs with 20GB bricks, that would expand the > volume by 20GB ? > > Second part is mounting, I'm curious if anything would change in the > current fstab entry: > > server1:/datastore /mnt/datastore glusterfs > default,_netdev,backupvolfile-server=server2 0 0 > > Gluster would properly utilize all 4 servers in this case, or fstab entry > needs to be modified ? > > Thank you! > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] What is the recommended backup strategy for GlusterFS?
Hello, for my needs, it's about having a simple "photo" of files present 5 days ago for example. But i do not want to store file data twice, as most file didn't change. Using snapshot is convenient of course, but it's risky as you loose both data and snapshot in case of failure (snapshot only contains delta blocks). Rsync with hardlink is more resistant (inode stay until last reference is removed) But interested to hear about production setup relying on it Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-05 21:03 GMT+02:00 M S Vishwanath Bhat : > MS > On 5 Sep 2015 12:57 am, "Mathieu Chateau" wrote: > > > > Hello, > > > > so far I use rsnapshot. This script do rsync with rotation, and most > important same files are stored only once through hard link (inode). I save > space, but still rsync need to parse all folders to know for new files. > > > > I am also interested in solution 1), but need to be stored on distinct > drives/servers. We can't afford to loose data and snapshot in case of human > error or disaster. > > > > > > > > Cordialement, > > Mathieu CHATEAU > > http://www.lotp.fr > > > > 2015-09-03 13:05 GMT+02:00 Merlin Morgenstern < > merlin.morgenst...@gmail.com>: > >> > >> I have about 1M files in a GlusterFS with rep 2 on 3 nodes runnnig > gluster 3.7.3. > >> > >> What would be a recommended automated backup strategy for this setup? > >> > >> I already considered the following: > > Have you considered glusterfs geo-rep? It's actually for disaster > recovery. But might suit your backup use case as well. > > My two cents > > //MS > > >> > >> 1) glusterfs snapshots in combination with dd. This unfortunatelly was > not possible so far as I could not find any info on how to make a image > file out of the snapshots and how to automate the snapshot procedure. > >> > >> 2) rsync the mounted file share to a second directory and do a tar on > the entire directory after rsync completed > >> > >> 3) combination of 1 and 2. Doing a snapshot that gets mounted > automaticaly and then rsync from there. Problem: How to automate snapshots > and how to know the mount path > >> > >> Currently I am only able to do the second option, but the fist option > seems to be the most atractive. > >> > >> Thank you for any help on this. > >> > >> ___ > >> Gluster-users mailing list > >> Gluster-users@gluster.org > >> http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] What is the recommended backup strategy for GlusterFS?
Hello, so far I use rsnapshot <https://github.com/rsnapshot/rsnapshot>. This script do rsync with rotation, and most important same files are stored only once through hard link (inode). I save space, but still rsync need to parse all folders to know for new files. I am also interested in solution 1), but need to be stored on distinct drives/servers. We can't afford to loose data and snapshot in case of human error or disaster. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-03 13:05 GMT+02:00 Merlin Morgenstern : > I have about 1M files in a GlusterFS with rep 2 on 3 nodes runnnig gluster > 3.7.3. > > What would be a recommended automated backup strategy for this setup? > > I already considered the following: > > 1) glusterfs snapshots in combination with dd. This unfortunatelly was not > possible so far as I could not find any info on how to make a image file > out of the snapshots and how to automate the snapshot procedure. > > 2) rsync the mounted file share to a second directory and do a tar on the > entire directory after rsync completed > > 3) combination of 1 and 2. Doing a snapshot that gets mounted automaticaly > and then rsync from there. Problem: How to automate snapshots and how to > know the mount path > > Currently I am only able to do the second option, but the fist option > seems to be the most atractive. > > Thank you for any help on this. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Gluster 3.6.3 performance.cache-size not working as expected in some cases
Hello, What I could say from my few knowledge on gluster: - If each server have itself as a mount point (name of server in fstab), then it should only ask others for metadata but grab file locally from itself. Use backupvolfile-server to provide an alternate one in case of issue at boot - Did you disable atime flag ? Maybe the file get this attribute updated on each read that may invalidate cache (just a clue) - if a lot small files and good server perf, you can increase number of thread through performance.io-thread-count You also have these settings that impact what get or not in the cache: performance.*cache*-max-file-size 0 performance.*cache*-min-file-size 0 just my 2cents Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-09-01 21:15 GMT+02:00 Christian Rice : > This is still an issue for me, I don’t need anyone to tear the code apart, > but I’d be grateful if someone would even chime in and say “yeah, we’ve > seen that too." > > From: Christian Rice > Date: Sunday, August 30, 2015 at 11:18 PM > To: "gluster-users@gluster.org" > Subject: [Gluster-users] Gluster 3.6.3 performance.cache-size not working > as expected in some cases > > I am confused about my caching problem. I’ll try to keep this as > straightforward as possible and include the basic details... > > I have a sixteen node distributed volume, one brick per node, XFS > isize=512, Debian 7/Wheezy, 32GB RAM minimally. Every brick node is also a > gluster client, and also importantly an HTTP server. We use a back-end > 1GbE network for gluster traffic (eth1). There are a couple dozen gluster > client-only systems accessing this volume, as well. > > We had a really hot spot on one brick due to an oft-requested file, and > every time any httpd process on any gluster client was asked to deliver the > file, it was physically fetching it (we could see this traffic using, say, > ‘iftop -i eth1’,) so we thought to increase the volume cache timeout and > cache size. We set the following values for testing: > > performance.cache-size 16GB > performance.cache-refresh-timeout: 30 > > This test was run from a node that didn’t have the requested file on the > local brick: > > while(true); do cat /path/to/file > /dev/null; done > > and what had been very high traffic on the gluster backend network, > delivering the data repeatedly to my requesting node, dropped to nothing > visible. > > I thought good, problem fixed. Caching works. My colleague had run a > test early on to show this perf issue, so he ran it again to sign off. > > His testing used curl, because all the real front end traffic is HTTP, and > all the gluster nodes are web servers, which are of course using the fuse > mount to access the document root. Even with our performance tuning, the > traffic on the gluster backend subnet was continuous and undiminished. I > saw no evidence of cache (again using ‘iftop -i eth1’, which showed a > steady 75+% of line rate on a 1GbE link. > > Does that make sense at all? We had theorized that we wouldn’t get to use > VFS/kernel page cache on any node except maybe the one which held the data > in the local brick. That’s what drove us to setting the gluster > performance cache. But it doesn’t seem to come into play with http access. > > > Volume info: > Volume Name: DOCROOT > Type: Distribute > Volume ID: 3aecd277-4d26-44cd-879d-cffbb1fec6ba > Status: Started > Number of Bricks: 16 > Transport-type: tcp > Bricks: > > Options Reconfigured: > performance.cache-refresh-timeout: 30 > performance.cache-size: 16GB > > The net result of being overwhelmed by a hot spot is all the gluster > client nodes lose access to the gluster volume—it becomes so busy it > hangs. When the traffic goes away (failing health checks by load balancers > causes requests to be redirected elsewhere), the volume eventually > unfreezes and life goes on. > > I wish I could type ALL that into a google query and get a lucid answer :) > > Regards, > Christian > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to partition directory structur for 300K files?
Hello, this is to do on brick, not on client side Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 17:57 GMT+02:00 Merlin Morgenstern : > thank you for the recommendation on parameters. > > I tried: > > gs2:/volume1/data/nfs glusterfs > defaults,_netdev,backupvolfile-server=gs1,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier > 0 0 > > The system reports: > > #sudo mount -a > > Invalid option noatime > > Same with nodiratime and logbufs as soon as I remove noatime > > > > 2015-08-24 16:48 GMT+02:00 Mathieu Chateau : > >> Re, >> >> putting back mailing list so they keep up. >> >> With newer version, fuse perform much better, and provide transparent >> failover. As both brick are 2 VM in same host, latency will not be an issue. >> Most important is to ensure you have drivers /tools inside VM to get best >> perf. >> >> on client and servers, I use this in /etc/sysctl.conf >> >> vm.swappiness=0 >> net.core.rmem_max=67108864 >> net.core.wmem_max=67108864 >> # increase Linux autotuning TCP buffer limit to 32MB >> net.ipv4.tcp_rmem="4096 87380 33554432" >> net.ipv4.tcp_wmem="4096 65536 33554432" >> # increase the length of the processor input queue >> net.core.netdev_max_backlog=3 >> # recommended default congestion control is htcp >> net.ipv4.tcp_congestion_control=htcp >> >> >> options I set on gluster volumes: >> >> server.allow-insecure: on >> performance.client-io-threads: on >> performance.read-ahead: on >> performance.readdir-ahead: enable >> performance.cache-size: 1GB >> performance.io-thread-count: 16 >> >> Options I set on brick in fstab for XFS mounted volumes used by gluster: >> >> >> defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-08-24 16:11 GMT+02:00 Merlin Morgenstern < >> merlin.morgenst...@gmail.com>: >> >>> re your questions: >>> >>> > did you do some basic tuning to help anyway ? >>> >>> no, this is a basica setup. Can you please direct me to the most >>> important tuning parameters to look at? >>> >>> > using latest version ? >>> glusterfs 3.7.3 built on Jul 28 2015 15:14:43 >>> >>> > in replication or only distributed ? >>> re 2 >>> >>> > Why using NFS and not native fuse client to mount volume? >>> I was reading that the NFS-Client is better with small files. (typical >>> 2-20KB in my case) >>> >>> >>> > did you install VM tools (if using VMware fusion) ? >>> I am using virtualbox 5.0.3 on Mac OS X 10.10 >>> >>> >>> >>> 2015-08-24 16:01 GMT+02:00 Mathieu Chateau : >>> >>>> Hello, >>>> >>>> did you do some basic tuning to help anyway ? >>>> using latest version ? >>>> in replication or only distributed ? >>>> Why using NFS and not native fuse client to mount volume? >>>> did you install VM tools (if using VMware fusion) ? >>>> >>>> Cordialement, >>>> Mathieu CHATEAU >>>> http://www.lotp.fr >>>> >>>> 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern < >>>> merlin.morgenst...@gmail.com>: >>>> >>>>> I am running into trouble while syncing (rsync, cp ... ) my files to >>>>> glusterfs. After about 50K files, one machine dies and has to be rebooted. >>>>> >>>>> As there are about 300K files in one directory, I am thinking about to >>>>> cluster that in a directory structure in order to overcome that problem. >>>>> >>>>> e.g. /0001/filename /0002/filename >>>>> >>>>> That would cut down the amount of files in one directory. However this >>>>> is something I would like to avoid if possible due to SEO - changing the >>>>> url of the file brings a lot of trouble. >>>>> >>>>> The system underneith are 2 seperate VM instances, each running ubuntu >>>>> 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro >>>>> 13 >>>>> retina with capable SSD and 1G internal network between the VMs. >>>>> >>>>> Thank you for any help on this. >>>>> >>>>> ___ >>>>> Gluster-users mailing list >>>>> Gluster-users@gluster.org >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>> >> > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to partition directory structur for 300K files?
Re, putting back mailing list so they keep up. With newer version, fuse perform much better, and provide transparent failover. As both brick are 2 VM in same host, latency will not be an issue. Most important is to ensure you have drivers /tools inside VM to get best perf. on client and servers, I use this in /etc/sysctl.conf vm.swappiness=0 net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem="4096 87380 33554432" net.ipv4.tcp_wmem="4096 65536 33554432" # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp options I set on gluster volumes: server.allow-insecure: on performance.client-io-threads: on performance.read-ahead: on performance.readdir-ahead: enable performance.cache-size: 1GB performance.io-thread-count: 16 Options I set on brick in fstab for XFS mounted volumes used by gluster: defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio,inode64,swalloc,allocsize=131072k,nobarrier Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 16:11 GMT+02:00 Merlin Morgenstern : > re your questions: > > > did you do some basic tuning to help anyway ? > > no, this is a basica setup. Can you please direct me to the most important > tuning parameters to look at? > > > using latest version ? > glusterfs 3.7.3 built on Jul 28 2015 15:14:43 > > > in replication or only distributed ? > re 2 > > > Why using NFS and not native fuse client to mount volume? > I was reading that the NFS-Client is better with small files. (typical > 2-20KB in my case) > > > > did you install VM tools (if using VMware fusion) ? > I am using virtualbox 5.0.3 on Mac OS X 10.10 > > > > 2015-08-24 16:01 GMT+02:00 Mathieu Chateau : > >> Hello, >> >> did you do some basic tuning to help anyway ? >> using latest version ? >> in replication or only distributed ? >> Why using NFS and not native fuse client to mount volume? >> did you install VM tools (if using VMware fusion) ? >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern < >> merlin.morgenst...@gmail.com>: >> >>> I am running into trouble while syncing (rsync, cp ... ) my files to >>> glusterfs. After about 50K files, one machine dies and has to be rebooted. >>> >>> As there are about 300K files in one directory, I am thinking about to >>> cluster that in a directory structure in order to overcome that problem. >>> >>> e.g. /0001/filename /0002/filename >>> >>> That would cut down the amount of files in one directory. However this >>> is something I would like to avoid if possible due to SEO - changing the >>> url of the file brings a lot of trouble. >>> >>> The system underneith are 2 seperate VM instances, each running ubuntu >>> 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 >>> retina with capable SSD and 1G internal network between the VMs. >>> >>> Thank you for any help on this. >>> >>> ___ >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to partition directory structur for 300K files?
Hello, did you do some basic tuning to help anyway ? using latest version ? in replication or only distributed ? Why using NFS and not native fuse client to mount volume? did you install VM tools (if using VMware fusion) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-24 15:20 GMT+02:00 Merlin Morgenstern : > I am running into trouble while syncing (rsync, cp ... ) my files to > glusterfs. After about 50K files, one machine dies and has to be rebooted. > > As there are about 300K files in one directory, I am thinking about to > cluster that in a directory structure in order to overcome that problem. > > e.g. /0001/filename /0002/filename > > That would cut down the amount of files in one directory. However this is > something I would like to avoid if possible due to SEO - changing the url > of the file brings a lot of trouble. > > The system underneith are 2 seperate VM instances, each running ubuntu > 14.04. Cluster NFS client on same machine as Gluster server. Macbook Pro 13 > retina with capable SSD and 1G internal network between the VMs. > > Thank you for any help on this. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] cluster.min-free-disk is not working in distributed disperse volume
720 brick! Respect ! Le 24 août 2015 09:48, "Mohamed Pakkeer" a écrit : > Hi, > >I have a cluster of 720 bricks, all bricks are 4TB in size. I have > change the cluster.min-free-disk default value 10% to 3%. So all the disks > should have 3% minimum disk space free. But some cluster disks are getting > full now. Is there any additional configuration for keeping some > percentage of disk space kept free? > > > Volume Name: glustertest > Type: Distributed-Disperse > Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609 > Status: Started > Number of Bricks: 72 x (8 + 2) = 720 > Transport-type: tcp > > > Options Reconfigured: > features.default-soft-limit: 95% > *cluster.min-free-disk: 3%* > performance.readdir-ahead: on > > df -h of one node > > /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1 > /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2 > */dev/sdd1 3.7T 3.6T 109G 98% /media/disk3* > > Any help will be greatly appreciated. > > > Regards > Backer > > > > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS and DFS
Hello, If you have an Active Directory domain, I would either create a domain DFS root. Then create in DFS a share that will point to both node. So Windows clients, using DFS client, will figure out to use available node from all enabled. You can also use cost & co to direct client. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-17 5:54 GMT+02:00 Colin Coe : > Hi all > > I've setup a two node replicated system that is providing NFS and > SMB/CIFS services to clients (RHEL5,6,7 with NFS and > WinXP,7,2008R2,2012R2). > > I'm try to create a DFS mount point for Windows nodes to use. I'm > doing this so that if I have to take down one of the Gluster nodes for > maintenance, Windows nodes won't be affected. > > I've googled but not found how to make Gluster and Samba DFS play nicely. > > Version info: > OS - RHEL7.1 > Samba - 4.1.12 > Gluster - 3.7.3 > > smb.conf (sanitized) > --- > [global] > workgroup = DOMAIN > realm = DOMAIN.COMPANY.COM > server string = Samba Server Version %v > security = ADS > dedicated keytab file = /etc/krb5.keytab > kerberos method = secrets and keytab > log file = /var/log/samba/log.%m > max log size = 50 > client signing = if_required > load printers = No > winbind enum users = Yes > winbind enum groups = Yes > winbind use default domain = Yes > winbind nss info = rfc2307 > winbind refresh tickets = Yes > winbind offline logon = Yes > idmap config DOMAIN:range = 1-99 > idmap config DOMAIN:schema_mode = rfc2307 > idmap config DOMAIN:backend = ad > idmap config *:range = 2000- > idmap config * : backend = tdb > map acl inherit = Yes > printing = bsd > cups options = raw > print command = lpr -r -P'%p' %s > lpq command = lpq -P'%p' > lprm command = lprm -P'%p' %j > store dos attributes = Yes > vfs objects = acl_xattr > > [fileshare] > comment = For samba share of volume gv_fileshare > path = /export/dfsroot > admin users = @'DOMAIN\Domain, Admins' > read only = No > guest ok = Yes > vfs objects = glusterfs, acl_xattr > msdfs root = Yes > glusterfs:loglevel = 7 > glusterfs:logfile = /var/log/samba/gv_fileshare.log > glusterfs:volume = gv_fileshare > --- > > gluster volume status > --- > Status of volume: gv_fileshare > Gluster process TCP Port RDMA Port Online > Pid > > -- > Brick benfil01:/export/vdb1/brick1 49153 0 Y > 2286 > Brick benfil02:/export/vdb1/brick1 49154 0 Y > 3629 > NFS Server on localhost 2049 0 Y > 19731 > Self-heal Daemon on localhost N/A N/AY > 19739 > NFS Server on fs02 N/A N/A >N N/A > Self-heal Daemon onfs02 N/A N/A > Y 3595 > > Task Status of Volume gv_fileshare > > -- > There are no active volume tasks > --- > > Lastly, I can't work out why the NFS service generally doesn't start > on boot. It seems to be somewhat automagical... I've been through > > http://www.gluster.org/community/documentation/index.php/Gluster_3.1:_NFS_Frequently_Asked_Questions > but can't see whats wrong. How do I go about trouble shooting this? > > > Thanks > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] GluserFS error with ls (File list)
Hello, Just in case, Did you create and test from the client (and not locally on any brick)? Envoyé de mon iPad > Le 21 août 2015 à 13:36, "tecni...@gmx.de" a écrit : > > Hi all, > > I have the following problem with GlusterFS: > > Setup: > > 1x Client (Ubuntu 14.04 / VirtualBox) > 3x GlusterFS Replications (Ubuntu 14.04 / VirtualBox) > > In GlusterFS I've created a volume (volumeTest). > This also works as expected. > Data is stored. > Data is replicated. > > But if I create mote than 20 files in the volume (volumeTest) the volume > is empty. > Not realy empty, the files are readable, but the files cannot be listed > anymore. > "sudo ls /mnt/test volume/" then leads to an I/O error. > > This behavior exists also, when I add a new volumes and even if I put > subdirectories in /testVolume/. > > What could be wrong? > > Best Regards Tobi > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Need help making a decision choosing MS DFS or Gluster+SAMBA+CTDB
Hello, what do you mean by "true" clustering ? We can do a Windows Failover cluster (1 virtual ip, 1 virtual name), but this mean using a shared storage like SAN. Then it depends on your network topology. If you have multiple geographical sites / datacenter, then DFS-R behave a lot better than Gluster in replicated mode. Users won't notice any latency, At the price that replication is async. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-10 7:26 GMT+02:00 Ira Cooper : > Mathieu Chateau writes: > > > I do have DFS-R in production, that replaced sometimes netapp ones. > > But no similar workload as my current GFS. > > > > In active/active, the most common issue is file changed on both side (no > > global lock) > > Will users access same content from linux & windows ? > > If you want to go active/active. I'd recommend Samba + CTDB + Gluster. > > You want true clustering, and a system that can handle the locking etc. > > I'd layer normal DFS to do "namespace" control, and to help with > handling failover, or just use round robin DNS. > > Thanks, > > -Ira > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Need help making a decision choosing MS DFS or Gluster+SAMBA+CTDB
Hello, Yes it's much like a standard Windows File servers. You will only monitor DFS-R replication through nagios or so, to check backlog/latency in replication. For performance, it's all about what you do with a standard Windows file server anyway: - Check raid controller settings - NTFS formatted in 64K, not the 4K default, - Defrag since the beginning through the Windows one (scheduled). O&O can be a great invest for that if 1 volume go much beyond than 2M files - Setup antivirus to only do realtime check on file change, not on access. Exclude DFS database from antivirus scan - Tune network card (send and receive buffer) - ... Start with 2012 R2, to get SMB v3 and all latest stuff Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-10 7:11 GMT+02:00 David : > No, but files can be accessed from different clients from different nodes. > > OK, so from what you are saying, there is no stability or other issues > with DFS keeping an eye on workload, and as long as files accessed from one > node, right? > > On Sun, Aug 9, 2015 at 11:24 PM, Mathieu Chateau > wrote: > >> I do have DFS-R in production, that replaced sometimes netapp ones. >> But no similar workload as my current GFS. >> >> In active/active, the most common issue is file changed on both side (no >> global lock) >> Will users access same content from linux & windows ? >> >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-08-09 21:18 GMT+02:00 David : >> >>> Hi, >>> >>> Thank you very much for detailed answer. >>> >>> Most of the clients are Windows based OS's, but Linux will come in the >>> future. >>> Now I know that Windows does a bad job with NFS, so this is one concern >>> that I have, but I also worried about performance and stability. >>> I used to work with NFS clustered environments, also with GPFS and CTDB >>> exporting both NFS and CIFS servicing 100s of users and render farms with >>> no issues. >>> >>> Are you aware of any performance challenges/limitations of DFS-R, I mean >>> real world ones compared to Linux? I aware of MS official DFS docs. >>> Wonder if there is a comparison of same HW, one runs Gluster replica >>> (ontop of XFS/Ext4) compared to two nodes DFS-R. (checking concurrent CIFS >>> session, IOps, network utilization while sync etc..) >>> >>> Thanks again, >>> David >>> >>> >>> On Sun, Aug 9, 2015 at 9:05 PM, Mathieu Chateau >> > wrote: >>> >>>> Hello, >>>> By DFS, you mean DFS-R. >>>> Because DFS can also be used only as domain space (DFS-N). This allow >>>> to publish share that hide real server name and so allow to move target >>>> somewhere else as needed. >>>> >>>> As I do quite a lot of DFS-R, here are the differences using DFS-R >>>> instead of Gluster: >>>> >>>>- Replication occurs between servers. Client only connect to one of >>>>ther server (can be based on AD topology), and is not aware that it's >>>> DFS-R. >>>>- Replication only transmit block changed in files, not the whole >>>>file >>>>- Replication is tracked using an internal Jet database >>>>- You have reporting tool to see differences & co between servers >>>>- This is active/active. If a client can/connect to a server, it >>>>will work there. >>>>- Lock on files are not replicated. If same file is changed on 2 >>>>servers at same time, replication will log that in event log and put >>>> file >>>>that lost in lost&found folder (the more recent win) >>>>- Not any issue browsing files/folder tree. Everything act like if >>>>it was just a file server. >>>>- You can use NTFS permission to fine grain access (Go further than >>>>unix style from my point of view) >>>>- Quota are working >>>>- You can prevent file based on extension >>>>- New version can deduplicate content (file server standard) >>>>- Writes are not synchronous. Once file is written, it's replicated >>>>in the background. >>>> >>>> >>>> Main difference is that you can't strip inside share content over >>>> multiple servers like if they were just one (the distributed feature of >>>> Gluster). Things are evolving with Windows Server 2016, but not yet RTM.
Re: [Gluster-users] Need help making a decision choosing MS DFS or Gluster+SAMBA+CTDB
I do have DFS-R in production, that replaced sometimes netapp ones. But no similar workload as my current GFS. In active/active, the most common issue is file changed on both side (no global lock) Will users access same content from linux & windows ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-09 21:18 GMT+02:00 David : > Hi, > > Thank you very much for detailed answer. > > Most of the clients are Windows based OS's, but Linux will come in the > future. > Now I know that Windows does a bad job with NFS, so this is one concern > that I have, but I also worried about performance and stability. > I used to work with NFS clustered environments, also with GPFS and CTDB > exporting both NFS and CIFS servicing 100s of users and render farms with > no issues. > > Are you aware of any performance challenges/limitations of DFS-R, I mean > real world ones compared to Linux? I aware of MS official DFS docs. > Wonder if there is a comparison of same HW, one runs Gluster replica > (ontop of XFS/Ext4) compared to two nodes DFS-R. (checking concurrent CIFS > session, IOps, network utilization while sync etc..) > > Thanks again, > David > > > On Sun, Aug 9, 2015 at 9:05 PM, Mathieu Chateau > wrote: > >> Hello, >> By DFS, you mean DFS-R. >> Because DFS can also be used only as domain space (DFS-N). This allow to >> publish share that hide real server name and so allow to move target >> somewhere else as needed. >> >> As I do quite a lot of DFS-R, here are the differences using DFS-R >> instead of Gluster: >> >>- Replication occurs between servers. Client only connect to one of >>ther server (can be based on AD topology), and is not aware that it's >> DFS-R. >>- Replication only transmit block changed in files, not the whole file >>- Replication is tracked using an internal Jet database >>- You have reporting tool to see differences & co between servers >>- This is active/active. If a client can/connect to a server, it will >>work there. >>- Lock on files are not replicated. If same file is changed on 2 >>servers at same time, replication will log that in event log and put file >>that lost in lost&found folder (the more recent win) >>- Not any issue browsing files/folder tree. Everything act like if it >>was just a file server. >>- You can use NTFS permission to fine grain access (Go further than >>unix style from my point of view) >>- Quota are working >>- You can prevent file based on extension >>- New version can deduplicate content (file server standard) >>- Writes are not synchronous. Once file is written, it's replicated >>in the background. >> >> >> Main difference is that you can't strip inside share content over >> multiple servers like if they were just one (the distributed feature of >> Gluster). Things are evolving with Windows Server 2016, but not yet RTM. >> You can also use shared storage in a more cluster way, with or without >> DFS replication (to survive a server down). >> >> We nearly always use both DFS-R and DFS-N, so we can migrate share to a >> different server without changes on client side. >> >> NAS & SAN vendors don't have choice. NetApp can't use Windows, else they >> can't customize it deeper enough, and they would have to pay license to MS. >> I always found that using a linux for CIFS is far away from feature you >> have on Windows side, and issue it generate (like robocopy diff. not >> working without /FFT flag). >> >> Backup of NetApp or EMC CIFS is just not working if doing that through >> share, you have to use NDMP, which is proprietary and generate others issue >> to backup (NDMP license, need to write directly itself on tape...). >> >> What will be your clients ? Windows box ? Linux ? Both ? If both, going >> to same shares? >> >> >> >> >> >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-08-09 17:56 GMT+02:00 David : >> >>> Hi, >>> >>> I need some help in making this call choosing between the two. >>> I have no experience with MS DFS or with Windows server OS as a file >>> server. >>> >>> There are some developers that pushing the DFS direction, mostly because >>> the applications that will use it and access will be from Microsoft using >>> CIFS. >>> >>> Now I know that most serious storage, NAS and SAN vendors work with >>> Linux or Unix based because of performance and flexibility, and I'm afraid >>> that DFS will just won't carry the expected load. >>> >>> Does anyone has experience with it? >>> Can some tell what are the PROS and CONS of each that can help us to >>> make a call? >>> >>> Many thanks, >>> David >>> >>> >>> ___ >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Need help making a decision choosing MS DFS or Gluster+SAMBA+CTDB
Hello, By DFS, you mean DFS-R. Because DFS can also be used only as domain space (DFS-N). This allow to publish share that hide real server name and so allow to move target somewhere else as needed. As I do quite a lot of DFS-R, here are the differences using DFS-R instead of Gluster: - Replication occurs between servers. Client only connect to one of ther server (can be based on AD topology), and is not aware that it's DFS-R. - Replication only transmit block changed in files, not the whole file - Replication is tracked using an internal Jet database - You have reporting tool to see differences & co between servers - This is active/active. If a client can/connect to a server, it will work there. - Lock on files are not replicated. If same file is changed on 2 servers at same time, replication will log that in event log and put file that lost in lost&found folder (the more recent win) - Not any issue browsing files/folder tree. Everything act like if it was just a file server. - You can use NTFS permission to fine grain access (Go further than unix style from my point of view) - Quota are working - You can prevent file based on extension - New version can deduplicate content (file server standard) - Writes are not synchronous. Once file is written, it's replicated in the background. Main difference is that you can't strip inside share content over multiple servers like if they were just one (the distributed feature of Gluster). Things are evolving with Windows Server 2016, but not yet RTM. You can also use shared storage in a more cluster way, with or without DFS replication (to survive a server down). We nearly always use both DFS-R and DFS-N, so we can migrate share to a different server without changes on client side. NAS & SAN vendors don't have choice. NetApp can't use Windows, else they can't customize it deeper enough, and they would have to pay license to MS. I always found that using a linux for CIFS is far away from feature you have on Windows side, and issue it generate (like robocopy diff. not working without /FFT flag). Backup of NetApp or EMC CIFS is just not working if doing that through share, you have to use NDMP, which is proprietary and generate others issue to backup (NDMP license, need to write directly itself on tape...). What will be your clients ? Windows box ? Linux ? Both ? If both, going to same shares? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-09 17:56 GMT+02:00 David : > Hi, > > I need some help in making this call choosing between the two. > I have no experience with MS DFS or with Windows server OS as a file > server. > > There are some developers that pushing the DFS direction, mostly because > the applications that will use it and access will be from Microsoft using > CIFS. > > Now I know that most serious storage, NAS and SAN vendors work with Linux > or Unix based because of performance and flexibility, and I'm afraid that > DFS will just won't carry the expected load. > > Does anyone has experience with it? > Can some tell what are the PROS and CONS of each that can help us to make > a call? > > Many thanks, > David > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Cascading errors and very bad write performance
Maybe related to the insecure port issue reported ? try with : gluster volume set xxx server.allow-insecure on Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-07 23:47 GMT+02:00 Geoffrey Letessier : > I’m not really sure to well understand your answer. > > I try to set inode-lru-limit to 1, I can not notice any good effect. > > When i re-run ddt application, I can note 2 kinds of messages: > [2015-08-07 21:29:21.792156] W > [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is > NULL for , aborting updation txn > [2015-08-07 21:29:21.792176] W > [marker-quota.c:3379:_mq_initiate_quota_txn] 0-vol_home-marker: parent is > NULL for , aborting updation txn > > and/or: > [2015-08-07 21:44:19.279971] E [marker-quota.c:2990:mq_start_quota_txn_v2] > 0-vol_home-marker: contribution node list is empty > (31d7bf88-b63a-4731-a737-a3dce73b8cd1) > [2015-08-07 21:41:26.177095] E [dict.c:1418:dict_copy_with_ref] > (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) > [0x7f85e9a6a410] > -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) > [0x7f85e9a6a188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) > [0x3e99c20674] ) 0-dict: invalid argument: dict [Argument invalide] > > And concerning the bad IO performance? > > [letessier@node031 ~]$ ddt -t 35g /home/admin_team/letessier/ > Writing to /home/admin_team/letessier/ddt.25259 ... syncing ... done. > sleeping 10 seconds ... done. > Reading from /home/admin_team/letessier/ddt.25259 ... done. > 35840MiBKiB/s CPU% > Write 277451 3 > Read 188682 1 > [letessier@node031 ~]$ logout > [root@node031 ~]# ddt -t 35g /home/ > Writing to /home/ddt.25559 ... syncing ... done. > sleeping 10 seconds ... done. > Reading from /home/ddt.25559 ... done. > 35840MiBKiB/s CPU% > Write 196539 2 > Read 438944 3 > Notice the read/write throughput differences when i’m root and when i’m a > simple user. > > Thanks. > Geoffrey > -- > Geoffrey Letessier > Responsable informatique & ingénieur système > UPR 9080 - CNRS - Laboratoire de Biochimie Théorique > Institut de Biologie Physico-Chimique > 13, rue Pierre et Marie Curie - 75005 Paris > Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@ibpc.fr > > Le 7 août 2015 à 14:57, Vijaikumar M a écrit : > > > > On Friday 07 August 2015 05:34 PM, Geoffrey Letessier wrote: > > Hi Vijay, > > My brick logs issue and big performance problem have begun when I upgraded > Gluster into 3.7.3 version; before write throughput was good enough > (~500MBs) -but not as good as with GlusterFS 3.5.3 (especially with > distributed volumes)- and didn’t notice these problème with brick-logs. > > OK… in live: > > i just disable to quota for my home volume and now my performance appears > to be relatively better (around 300MBs) but i still see the logs (from > storage1 and its replicate storage2) growing up with only this kind of > lines: > [2015-08-07 11:16:51.746142] E [dict.c:1418:dict_copy_with_ref] > (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) > [0x7f85e9a6a410] > -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) > [0x7f85e9a6a188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) > [0x3e99c20674] ) 0-dict: invalid argument: dict [Argument invalide] > > We have root caused log issue, bug# 1244613 tracks this issue > > > After a few minutes: my write throughput seems to be now correct (~550MBs) > but the log are still growing up (to not say exploding). So one part of the > problem looks like taking its origin in the quota system management. > … after a few minutes (and still only 1 client connected), now it is the > read operation which is very very slow… -I’m gonna become crazy! :/- > # ddt -t 50g /home/ > Writing to /home/ddt.11293 ... syncing ... done. > sleeping 10 seconds ... done. > Reading from /home/ddt.11293 ... done. > 35840MiBKiB/s CPU% > Write 568201 5 > Read 567008 4 > # ddt -t 50g /home/ > Writing to /home/ddt.11397 ... syncing ... done. > sleeping 10 seconds ... done. > Reading from /home/ddt.11397 ... done. > 51200MiBKiB/s CPU% > Write 573631 5 > Read 164716 1 > > and my log are still exploding… > > After having re-enabled the quota on my volume: > # ddt -t 50g /home/ > Writing to /home/ddt.11817 ... syncing ... done. > sleeping 10 seconds ... done. > Reading from /home/ddt.11817 ... done. > 51200MiBKiB/s CPU% > Write 269608 3 > Read 160219 1 > > Thanks > Geoffr
Re: [Gluster-users] Very slow ls
Sorry only read replicated in your first mail, I squashed the distributed one :'( Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-04 9:47 GMT+02:00 Florian Oppermann : > In my current configuration I have a distributed and replicated volume > which is (to my understanding) similar to a raid 10. > > On 04.08.2015 08:51, Mathieu Chateau wrote: > > In a replicated scheme, it's like a raid 1 (mirror). > > You write as slow as the slowest disk. Client will wait for all brick > > writes confirmation. > > > > In this scheme, you wouldn't much more than 3 bricks. > > > > I think you mox up with distributed scheme, which is like a raid 0 > stripped. > > This one get more perf when adding bricks. But a single file is > > present only in one brick. > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Very slow ls
In a replicated scheme, it's like a raid 1 (mirror). You write as slow as the slowest disk. Client will wait for all brick writes confirmation. In this scheme, you wouldn't much more than 3 bricks. I think you mox up with distributed scheme, which is like a raid 0 stripped. This one get more perf when adding bricks. But a single file is present only in one brick. Envoyé de mon iPad Le 4 août 2015 à 08:32, Florian Oppermann a écrit : >> As you are in replicate mode, all write will be send synchronously to all >> bricks, and in your case to a single hdd. > > I thought that every file will be sent to 2 bricks synchronously but if > I write several files they are distributed between the three pairs of > bricks. Therefore the performance should become better with more bricks > (note that the 3×2 bricks are not final but only a test setup, more > bricks will be added when going to production). > >> For sure I wouldn't go for 60+ users with this setup, maybe except if these >> hdd are ssd > > What would be a suitable setup? Or: Which use cases are typical for > Gluster setups? Maybe I misunderstood the target of Gluster. > > Best regards > Florian > >> On 04.08.2015 07:25, Mathieu Chateau wrote: >> Hello, >> >> As you are in replicate mode, all write will be send synchronously to >> all bricks, and in your case to a single hdd. >> >> Writes: you are going to have same perf as 1 single hdd (best case >> possible, you will have less) >> read: all brick will be queried for metadata, one will send the file (if >> I am correct) >> >> For sure I wouldn't go for 60+ users with this setup, maybe except if >> these hdd are ssd >> >> just my 2 cents >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-08-03 23:29 GMT+02:00 Florian Oppermann > <mailto:gluster-us...@flopwelt.de>>: >> >>> If starting setup right now, you should start with current version (3.7.X) >> >>Is 3.7 stable? I have 60+ potential users and dont want to risk too >>much. ;-) >> >>> Filesystem >> >>XFS partitions on all bricks >> >>> network type (lan, VM...) >> >>Gigabit LAN >> >>> where is client (same lan?) >> >>Yep >> >>> MTU >> >>1500 >> >>> storage (raid, # of disks...) >> >>The bricks are all on separate servers. On each is a XFS partition on a >>single HDD (together with other partitions for system etc.). All in all >>there are currently seven machines involved. >> >>I just noticed that on all servers the >>/var/log/glusterfs/etc-glusterfs-glusterd.vol.log is full of >>messages like >> >>> [2015-08-03 21:24:59.879820] W [socket.c:620:__socket_rwv] >>0-management: readv on >>/var/run/a91fc43b47272ffaace2a6989e7b5e85.socket failed (Invalid >>argument) >> >>I assume this to be part of the problem… >> >>Regards :-) >>Florian >> >>>On 03.08.2015 22 :41, Mathieu Chateau wrote: >>> Hello, >>> >>> If starting setup right now, you should start with current version (3.7.X) >>> >>> We need more data/context as you were able to feed 150GB before having >>> issue. >>> >>> Info: >>> Filesystem >>> network type (lan, VM...) >>> where is client (same lan?) >>> MTU >>> storage (raid, # of disks...) >>> >>> Cordialement, >>> Mathieu CHATEAU >>> http://www.lotp.fr >>> >>> 2015-08-03 21:44 GMT+02:00 Florian Oppermann >> <mailto:gluster-us...@flopwelt.de> >>> <mailto:gluster-us...@flopwelt.de >><mailto:gluster-us...@flopwelt.de>>>: >>> >>>Dear Gluster users, >>> >>>after setting up a distributed replicated volume (3x2 bricks) on gluster >>>3.6.4 on Ubuntu systems and populating it with some data (about 150 GB >>>in 20k files) I experience extreme delay when navigating through >>>directories or trying to ls the contents (actually the process seems to >>>hang completely now until I kill the /usr/sbin/glusterfs process on the >>>mounting machine). >>> >>>Is there some common misconfiguration or any performance tuning option >>>that I could try? >>> >>>I mount via automount with fstype=glusterfs option (using the native >>>fuse mount). >>> >>>Any tips? >>> >>>Best regards, >>>Florian Oppermann >>>___ >>>Gluster-users mailing list >>>Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> >><mailto:Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>> >>>http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >> >> ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Very slow ls
Hello, As you are in replicate mode, all write will be send synchronously to all bricks, and in your case to a single hdd. Writes: you are going to have same perf as 1 single hdd (best case possible, you will have less) read: all brick will be queried for metadata, one will send the file (if I am correct) For sure I wouldn't go for 60+ users with this setup, maybe except if these hdd are ssd just my 2 cents Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-03 23:29 GMT+02:00 Florian Oppermann : > > If starting setup right now, you should start with current version > (3.7.X) > > Is 3.7 stable? I have 60+ potential users and dont want to risk too > much. ;-) > > > Filesystem > > XFS partitions on all bricks > > > network type (lan, VM...) > > Gigabit LAN > > > where is client (same lan?) > > Yep > > > MTU > > 1500 > > > storage (raid, # of disks...) > > The bricks are all on separate servers. On each is a XFS partition on a > single HDD (together with other partitions for system etc.). All in all > there are currently seven machines involved. > > I just noticed that on all servers the > /var/log/glusterfs/etc-glusterfs-glusterd.vol.log is full of messages like > > > [2015-08-03 21:24:59.879820] W [socket.c:620:__socket_rwv] 0-management: > readv on /var/run/a91fc43b47272ffaace2a6989e7b5e85.socket failed (Invalid > argument) > > I assume this to be part of the problem… > > Regards :-) > Florian > > On 03.08.2015 22:41, Mathieu Chateau wrote: > > Hello, > > > > If starting setup right now, you should start with current version > (3.7.X) > > > > We need more data/context as you were able to feed 150GB before having > > issue. > > > > Info: > > Filesystem > > network type (lan, VM...) > > where is client (same lan?) > > MTU > > storage (raid, # of disks...) > > > > Cordialement, > > Mathieu CHATEAU > > http://www.lotp.fr > > > > 2015-08-03 21:44 GMT+02:00 Florian Oppermann > <mailto:gluster-us...@flopwelt.de>>: > > > > Dear Gluster users, > > > > after setting up a distributed replicated volume (3x2 bricks) on > gluster > > 3.6.4 on Ubuntu systems and populating it with some data (about 150 > GB > > in 20k files) I experience extreme delay when navigating through > > directories or trying to ls the contents (actually the process seems > to > > hang completely now until I kill the /usr/sbin/glusterfs process on > the > > mounting machine). > > > > Is there some common misconfiguration or any performance tuning > option > > that I could try? > > > > I mount via automount with fstype=glusterfs option (using the native > > fuse mount). > > > > Any tips? > > > > Best regards, > > Florian Oppermann > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Very slow ls
Hello, If starting setup right now, you should start with current version (3.7.X) We need more data/context as you were able to feed 150GB before having issue. Info: Filesystem network type (lan, VM...) where is client (same lan?) MTU storage (raid, # of disks...) Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-03 21:44 GMT+02:00 Florian Oppermann : > Dear Gluster users, > > after setting up a distributed replicated volume (3x2 bricks) on gluster > 3.6.4 on Ubuntu systems and populating it with some data (about 150 GB > in 20k files) I experience extreme delay when navigating through > directories or trying to ls the contents (actually the process seems to > hang completely now until I kill the /usr/sbin/glusterfs process on the > mounting machine). > > Is there some common misconfiguration or any performance tuning option > that I could try? > > I mount via automount with fstype=glusterfs option (using the native > fuse mount). > > Any tips? > > Best regards, > Florian Oppermann > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] File out of sync
date is not the same but is content different ? You may have disable the mtime attribue to get better perf ? What are these 2 GFID ? You can use this script to find who they are: https://gist.github.com/semiosis/4392640 Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-03 12:30 GMT+02:00 shacky : > 2015-08-03 12:16 GMT+02:00 Mathieu Chateau : > > > You should start a heal > > gluster volume xxx heal > > or even a full one if not enough > > gluster volume xxx heal full > > Thanks, I tried both but this not solved the problem: > > root@web1:~# ls -l /mnt/data/web/system/web/config.inc.php > -rw-r--r-- 1 1010 1010 4041 Jul 31 17:42 > /mnt/data/web/system/web/config.inc.php > > root@web2:~# ls -l /mnt/data/web/system/web/config.inc.php > -rw-r--r-- 1 1010 1010 4041 Jul 24 09:20 > /mnt/data/web/system/web/config.inc.php > > root@web3:~# ls -l /mnt/data/web/system/web/config.inc.php > -rw-r--r-- 1 1010 1010 4041 Jul 24 09:20 > /mnt/data/web/system/web/config.inc.php > > This is the heal info: > > root@web1:~# gluster volume heal data info > Brick web1:/data/ > > > Number of entries: 2 > > Brick web2:/data/ > Number of entries: 0 > > Brick web3:/data/ > Number of entries: 0 > > > I guess clients wrote some files while node was down or rebooting? > > I don't think this happened, because when I updated that file all > nodes was running. > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] File out of sync
Hello, You should start a heal gluster volume xxx heal or even a full one if not enough gluster volume xxx heal full I guess clients wrote some files while node was down or rebooting? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-08-03 12:14 GMT+02:00 shacky : > Hi. > I have a Gluster cluster with 3 nodes and one volume. > > All nodes are connected (gluster peer status) and the volume is > started on all nodes (gluster volume info), but I realized that one > file is different between nodes (it is updated only on node1 but it's > old on node2 and node3). > > Could you help me, please? > > Thank you very much! > Bye > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Any issue using version 3.6.1 along with 3.7.2?
Hello, sorry operating-version is a variable like others, just need to find the good name: op-version: gluster volume get all cluster.op-version then to set version (global to all volumes): gluster volume set all cluster.op-version 30702 Value for version are here but not updated: http://www.gluster.org/community/documentation/index.php/OperatingVersions Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-28 20:34 GMT+02:00 Mathieu Chateau : > Hello, > > any official doc to do that ? I can't find any so far. Just indications on > what it is used for > > thanks > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-07-28 20:27 GMT+02:00 Atin Mukherjee : > >> That's shouldn't be a problem. Ensure that you bump up the op-version to >> 30702 post up gradation of all the nodes. >> >> -Atin >> Sent from one plus one >> On Jul 28, 2015 11:30 PM, "John Kennedy" wrote: >> >>> I have to reinstall glusterfs-server on a node in a 4 node environment. >>> The existing servers use glusterfs-server 3.6.1 and the latest version is >>> 3.7.2. Are there any issues if I upgrade the servers one at a time? >>> >>> Also, I will be using the EPEL rpm's. Will a yum update do the trick or >>> do I need to prep anything? >>> >>> Thanks, >>> John >>> >>> John Kennedy (_8(|) >>> I have a yellow dog: >>> http://www.theyellowdogproject.com/The_Yellow_Dog_Project/About.html >>> >>> Sometimes it happens, sometimes it doesn't - Pedro Catacora >>> >>> Just because nobody disagrees with you doesn't mean you are correct. >>> >>> Anatidaephobia is the fear that somehow, somewhere a duck is watching >>> you - urbandictionary.com >>> >>> The Dunning-Kruger effect occurs when incompetent people not only fail >>> to realize their incompetence, but consider themselves much more competent >>> than everyone else. Basically - they're too stupid to know that they're >>> stupid. >>> >>> VGSR Disclaimer: The opinions expressed in this email are mine and do >>> not reflect the opinions of VGSR or their board of directors. >>> >>> ___ >>> Gluster-users mailing list >>> Gluster-users@gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Performance issues with one node
Hello, sorry operating-version is a variable like others, just need to find the good name: op-version: gluster volume get all cluster.op-version then to set version (global to all volumes): gluster volume set all cluster.op-version 30702 Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-28 8:03 GMT+02:00 Mathieu Chateau : > Hello, > > thanks for this guidance, I wasn't aware of! > > Any doc that describe all settings values ? > For example, I can't find documentation for cluster.lookup-optimize > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-07-27 14:58 GMT+02:00 André Bauer : > >> Some more infos: >> >> http://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/index.html?highlight=small%20file%20performance >> >> Am 24.07.2015 um 20:15 schrieb Mathieu Chateau: >> > Hello, >> > >> > gluster performance are not good with large number of small files. >> > Recent version do a better job with them, but not yet what I would >> enjoy. >> > >> > As you are starting at gluster having an existing architecture, you >> > should first setup a lab to learn about it Else you will learn the hard >> way. >> > Don't play with turning off nodes, as you may create more issues than >> solve. >> > >> > just my 2cents >> > >> > Cordialement, >> > Mathieu CHATEAU >> > http://www.lotp.fr >> > >> > 2015-07-24 19:34 GMT+02:00 John Kennedy > > <mailto:skeb...@gmail.com>>: >> > >> > I am new to Gluster and have not found anything useful from my >> > friend Google. I have not dealt with physical hardware in a few >> > years (my last few jobs have been VM's and AWS based) >> > >> > I inherited a 4 node gluster configuration. There are 2 bricks, one >> > is 9TB the other 11TB. >> > >> > The 11TB brick has a HUGE number of small files taking up only 1.3TB >> > of the brick. For some reason, even a simple ls command can take >> > hours to even start listing files. I removed a node by shutting down >> > gluster on that node. The change in performance is dramatic. If I >> > try and do ls on the 11TB brick on the downed node, I am still >> > getting the slow response. I have narrowed the issue down to this >> > one node as a result. >> > >> > When I start gluster on the bad node, glusterfsd hits over 1000%CPU >> > use (The server has dual 8 core CPU's) and the load will jump to >> > 25-30 within 5 minutes. As such, I think this is a gluster issue and >> > not a hardware issue. I am trying to not reinstall gluster yet. >> > >> > Is there something I am missing in my checks or will I need to >> > reinstall gluster on that node? >> > >> > Thanks, >> > John >> > >> > John Kennedy (_8(|) >> > Sometimes it happens, sometimes it doesn't - Pedro Catacora >> > >> > Just because nobody disagrees with you doesn't mean you are correct. >> > >> > Anatidaephobia is the fear that somehow, somewhere a duck is >> > watching you - urbandictionary.com <http://urbandictionary.com> >> > >> > The Dunning-Kruger effect occurs when incompetent people not only >> > fail to realize their incompetence, but consider themselves much >> > more competent than everyone else. Basically - they're too stupid to >> > know that they're stupid. >> > >> > ___ >> > Gluster-users mailing list >> > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> > >> > >> > >> > ___ >> > Gluster-users mailing list >> > Gluster-users@gluster.org >> > http://www.gluster.org/mailman/listinfo/gluster-users >> > >> >> >> -- >> Mit freundlichen Grüßen >> André Bauer >> >> MAGIX Software GmbH >> André Bauer >> Administrator >> August-Bebel-Straße 48 >> 01219 Dresden >> GERMANY >> >> tel.: 0351 41884875 >> e-mail: aba...@magix.net >> aba...@magix.net <mailto:Email> >> www.magix.com <http://www.magix.com/> >> >> >> Geschäftsführer | M
Re: [Gluster-users] Any issue using version 3.6.1 along with 3.7.2?
Hello, any official doc to do that ? I can't find any so far. Just indications on what it is used for thanks Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-28 20:27 GMT+02:00 Atin Mukherjee : > That's shouldn't be a problem. Ensure that you bump up the op-version to > 30702 post up gradation of all the nodes. > > -Atin > Sent from one plus one > On Jul 28, 2015 11:30 PM, "John Kennedy" wrote: > >> I have to reinstall glusterfs-server on a node in a 4 node environment. >> The existing servers use glusterfs-server 3.6.1 and the latest version is >> 3.7.2. Are there any issues if I upgrade the servers one at a time? >> >> Also, I will be using the EPEL rpm's. Will a yum update do the trick or >> do I need to prep anything? >> >> Thanks, >> John >> >> John Kennedy (_8(|) >> I have a yellow dog: >> http://www.theyellowdogproject.com/The_Yellow_Dog_Project/About.html >> >> Sometimes it happens, sometimes it doesn't - Pedro Catacora >> >> Just because nobody disagrees with you doesn't mean you are correct. >> >> Anatidaephobia is the fear that somehow, somewhere a duck is watching you >> - urbandictionary.com >> >> The Dunning-Kruger effect occurs when incompetent people not only fail to >> realize their incompetence, but consider themselves much more competent >> than everyone else. Basically - they're too stupid to know that they're >> stupid. >> >> VGSR Disclaimer: The opinions expressed in this email are mine and do not >> reflect the opinions of VGSR or their board of directors. >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Performance issues with one node
Hello, thanks for this guidance, I wasn't aware of! Any doc that describe all settings values ? For example, I can't find documentation for cluster.lookup-optimize Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-27 14:58 GMT+02:00 André Bauer : > Some more infos: > > http://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/index.html?highlight=small%20file%20performance > > Am 24.07.2015 um 20:15 schrieb Mathieu Chateau: > > Hello, > > > > gluster performance are not good with large number of small files. > > Recent version do a better job with them, but not yet what I would enjoy. > > > > As you are starting at gluster having an existing architecture, you > > should first setup a lab to learn about it Else you will learn the hard > way. > > Don't play with turning off nodes, as you may create more issues than > solve. > > > > just my 2cents > > > > Cordialement, > > Mathieu CHATEAU > > http://www.lotp.fr > > > > 2015-07-24 19:34 GMT+02:00 John Kennedy > <mailto:skeb...@gmail.com>>: > > > > I am new to Gluster and have not found anything useful from my > > friend Google. I have not dealt with physical hardware in a few > > years (my last few jobs have been VM's and AWS based) > > > > I inherited a 4 node gluster configuration. There are 2 bricks, one > > is 9TB the other 11TB. > > > > The 11TB brick has a HUGE number of small files taking up only 1.3TB > > of the brick. For some reason, even a simple ls command can take > > hours to even start listing files. I removed a node by shutting down > > gluster on that node. The change in performance is dramatic. If I > > try and do ls on the 11TB brick on the downed node, I am still > > getting the slow response. I have narrowed the issue down to this > > one node as a result. > > > > When I start gluster on the bad node, glusterfsd hits over 1000%CPU > > use (The server has dual 8 core CPU's) and the load will jump to > > 25-30 within 5 minutes. As such, I think this is a gluster issue and > > not a hardware issue. I am trying to not reinstall gluster yet. > > > > Is there something I am missing in my checks or will I need to > > reinstall gluster on that node? > > > > Thanks, > > John > > > > John Kennedy (_8(|) > > Sometimes it happens, sometimes it doesn't - Pedro Catacora > > > > Just because nobody disagrees with you doesn't mean you are correct. > > > > Anatidaephobia is the fear that somehow, somewhere a duck is > > watching you - urbandictionary.com <http://urbandictionary.com> > > > > The Dunning-Kruger effect occurs when incompetent people not only > > fail to realize their incompetence, but consider themselves much > > more competent than everyone else. Basically - they're too stupid to > > know that they're stupid. > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > ___ > > Gluster-users mailing list > > Gluster-users@gluster.org > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > > -- > Mit freundlichen Grüßen > André Bauer > > MAGIX Software GmbH > André Bauer > Administrator > August-Bebel-Straße 48 > 01219 Dresden > GERMANY > > tel.: 0351 41884875 > e-mail: aba...@magix.net > aba...@magix.net <mailto:Email> > www.magix.com <http://www.magix.com/> > > > Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael Keith > Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205 > > Find us on: > > <http://www.facebook.com/MAGIX> <http://www.twitter.com/magix_de> > <http://www.youtube.com/wwwmagixcom> <http://www.magixmagazin.de> > -- > The information in this email is intended only for the addressee named > above. Access to this email by anyone else is unauthorized. If you are > not the intended recipient of this message any disclosure, copying, > distribution or any action taken in reliance on it is prohibited and > may be unlawful. MAGIX does not warrant that any attachments are free > from viruses or other defects and accepts no liability for any losses > resulting from infected email transmissions. Please note that any > views expressed in this email may be those of the originator and do > not necessarily represent the agenda of the company. > -- > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] ram as cache for brick
Hello, I know cache occur on client side. My bricks have 32GB of ram allocated. I can see that Linux use it as cache on brick. Do you think I can expect benefits from it and may improve performance by adding more ? If buff/cache memory help gluster bricks what about tuning these: /proc/sys/vm/vfs_cache_pressure (I have mainly many small files and things are slow). I am using version 3.7.1, upgrading to 3.7.2-3 Thanks for your help Regards, Mathieu CHATEAU http://www.lotp.fr ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Performance issues with one node
Hello, gluster performance are not good with large number of small files. Recent version do a better job with them, but not yet what I would enjoy. As you are starting at gluster having an existing architecture, you should first setup a lab to learn about it Else you will learn the hard way. Don't play with turning off nodes, as you may create more issues than solve. just my 2cents Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-24 19:34 GMT+02:00 John Kennedy : > I am new to Gluster and have not found anything useful from my friend > Google. I have not dealt with physical hardware in a few years (my last few > jobs have been VM's and AWS based) > > I inherited a 4 node gluster configuration. There are 2 bricks, one is 9TB > the other 11TB. > > The 11TB brick has a HUGE number of small files taking up only 1.3TB of > the brick. For some reason, even a simple ls command can take hours to even > start listing files. I removed a node by shutting down gluster on that > node. The change in performance is dramatic. If I try and do ls on the 11TB > brick on the downed node, I am still getting the slow response. I have > narrowed the issue down to this one node as a result. > > When I start gluster on the bad node, glusterfsd hits over 1000%CPU use > (The server has dual 8 core CPU's) and the load will jump to 25-30 within 5 > minutes. As such, I think this is a gluster issue and not a hardware issue. > I am trying to not reinstall gluster yet. > > Is there something I am missing in my checks or will I need to reinstall > gluster on that node? > > Thanks, > John > > John Kennedy (_8(|) > Sometimes it happens, sometimes it doesn't - Pedro Catacora > > Just because nobody disagrees with you doesn't mean you are correct. > > Anatidaephobia is the fear that somehow, somewhere a duck is watching you > - urbandictionary.com > > The Dunning-Kruger effect occurs when incompetent people not only fail to > realize their incompetence, but consider themselves much more competent > than everyone else. Basically - they're too stupid to know that they're > stupid. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster write performance
These numbers are not helpfull without context : - Raid level - Number of disks - nature & size of files What do you get for this sample? dd if=/dev/zero of=sample.txt bs=50M count=1 Test it accross differents cases to get a baseline: - Local disk on client - Locak disk on each brick - through gluster from client Once you have your baseline, you can start to tune system and see changes based on it. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-18 22:11 GMT+02:00 Roman : > But when I'm inside of VM, I'm getting 50-53 MB/s for replica and 120 MB > for distributed volumes... > whats the logic? that slow write speed directly to mounted gfs volume > slows down backup and restore process of VMs by VE. > > 2015-07-18 22:47 GMT+03:00 Roman : > >> Hi, >> >> Looked a lot from forums, but didn't find much.. >> So I've got glusterfs 3.6.4 >> 1 Gbps network >> replicate and distributed volumes >> >> so when i read from any of them, I get maxed-out network performance >> (around 80-120 MB/sec) >> If i write to distributed volume, I get 16 MB/sec >> If i write to replicated volume, I get around half of it (which is >> logical) 8-10MB/sec >> >> i use XFS (with all set recommended options) and EXT4 for gluster. no >> difference. >> Disks are in HW RAID5 for Distributed volume and RAID0 for Replicated >> volume >> >> volume options: >> >> performance.write-behind: off >> server.allow-insecure: on >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> >> Am I missing something? >> >> -- >> Best regards, >> Roman. >> > > > > -- > Best regards, > Roman. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] gluster write performance
Hello, you can try tuning your TCP settings to be more aggressives (both on 1 client and all gluster bricks) in /etc/systctl.conf: net.core.rmem_max=67108864 net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB net.ipv4.tcp_rmem="4096 87380 33554432" net.ipv4.tcp_wmem="4096 65536 33554432" # increase the length of the processor input queue net.core.netdev_max_backlog=3 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-18 22:11 GMT+02:00 Roman : > But when I'm inside of VM, I'm getting 50-53 MB/s for replica and 120 MB > for distributed volumes... > whats the logic? that slow write speed directly to mounted gfs volume > slows down backup and restore process of VMs by VE. > > 2015-07-18 22:47 GMT+03:00 Roman : > >> Hi, >> >> Looked a lot from forums, but didn't find much.. >> So I've got glusterfs 3.6.4 >> 1 Gbps network >> replicate and distributed volumes >> >> so when i read from any of them, I get maxed-out network performance >> (around 80-120 MB/sec) >> If i write to distributed volume, I get 16 MB/sec >> If i write to replicated volume, I get around half of it (which is >> logical) 8-10MB/sec >> >> i use XFS (with all set recommended options) and EXT4 for gluster. no >> difference. >> Disks are in HW RAID5 for Distributed volume and RAID0 for Replicated >> volume >> >> volume options: >> >> performance.write-behind: off >> server.allow-insecure: on >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> network.remote-dio: enable >> cluster.eager-lock: enable >> performance.stat-prefetch: off >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> >> Am I missing something? >> >> -- >> Best regards, >> Roman. >> > > > > -- > Best regards, > Roman. > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Newbie: Exploring Gluster for large-scale deployment in AWS, large media files, high performance I/O
Hello, Ok you can stick with NFS, will just have to manage failover if needed. So they use 4TB hard drive (80TB/20 disks). each disk can provide let's say 150 io/s max. that means 3000 io/s max, without raid cost & co. >From your explaination, I guess you have many workloads running in parallel, and so 20 disks may not be enough anyway. You first must be sure that storage can physically provide your needs in terms or capacity and performance. Then you can choose solution that fit best your needs. just my 2cts Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-14 22:21 GMT+02:00 Forrest Aldrich : > The instances we use via Direct Connect (a third party company) have > upwards 20 disks and a total of 80T. That part is covered. > > If we were to experiment with EBS, that would be a different case as we'd > need to stripe them. > > Our present model requires one single namespace via NFS. The Instances > are running CentOS 6.x.The Instances mount the Direct Connect disk > space via NFS, the only other alternative we'd have is iSCSI which wouldn't > work for the level of sharing we need. > > > > > On 7/14/15 4:18 PM, Mathieu Chateau wrote: > > by NFS i think you just mean "all servers seeing and changing sames files" > ? That can be done with fuse, without nfs. > NFS is harder for failover while automatic with fuse (no need for dynamic > dns or virtual IP). > > for redundancy I mean : What failure do you want to survive ? > >- Loosing a disk >- Filesystem corrupt >- Server lost or in maintenance >- Whole region down > > Depending on your needs, then you may have to replicate data accross > gluster brick or even use a geo dispersed brick. > > Will network between your servers and node be able to handle that > traffic (380MB/s = 3040Mb/s) ? > > I guess gluster can handle that load, you are using big files and this > is where gluster deliver highest output. Nevertheless, you will need many > disk to provide these i/o, even more if using replicated bricks. > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-07-14 21:15 GMT+02:00 Forrest Aldrich : > >> Sorry, I should have noted that. 380MB is both read and write (I >> confirmed this with a developer). >> >> We do need the NFS stack, as that's how all the code and various many >> Instances work -- we have several "workers" that chop up video on the same >> namespace. It's not efficient, but that's how it has to be for now. >> >> Redundancy, in terms of the server?  We have RAIDED volumes if that's >> what you're referring to. >> >> Here's a basic outline of the flow (as I understand it): >> >> >> Video Capture Agent sends in large file of video (30gb +/-) >> >> Administrative host receives and writes to NFS >> >> A process copies this over to another point in the namespace >> >> Another Instance picks up the file, reads and starts processing and >> writes (FFMPEG is involved) >> >> >> Something like that -- I may not have all the steps, but essentially >> there's a ton of I/O going on.  I know our code model is not efficient, >> but it's complicated and can't just be changed (it's based on an open >> source product and there's some code baggage). >> >> We looked into another product that allegedly scaled out using multiple >> NFS heads with massive local cache (AWS instances) and sharing the same >> space, but it was horrible and just didn't work for us. >> >> >> >> Thank you. >> >> >> >> >> On 7/14/15 3:06 PM, Mathieu Chateau wrote: >> >> Hello, >> >> is it 380MB in read or write ? What level of redundancy do you need? >> do you really need nfs stack or just a mount point (and so be able to use >> native gluster protocol) ? >> >> Gluster load is mostly put on clients, not server (clients do the sync >> writes to all replica, and do the memory cache) >> >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-07-14 20:49 GMT+02:00 Forrest Aldrich < >> for...@gmail.com>: >> >>> I'm exploring solutions to help us achieve high throughput and >>> scalability within the AWS environment.  Specifically, I work in a >>> department where we handle and produce video content that results in very >>> large files (30GB etc) that must be written to NFS, chopped up and copied >>> over on the same mount (there are some odd limits to the code we u
Re: [Gluster-users] Newbie: Exploring Gluster for large-scale deployment in AWS, large media files, high performance I/O
by NFS i think you just mean "all servers seeing and changing sames files" ? That can be done with fuse, without nfs. NFS is harder for failover while automatic with fuse (no need for dynamic dns or virtual IP). for redundancy I mean : What failure do you want to survive ? - Loosing a disk - Filesystem corrupt - Server lost or in maintenance - Whole region down Depending on your needs, then you may have to replicate data accross gluster brick or even use a geo dispersed brick. Will network between your servers and node be able to handle that traffic (380MB/s = 3040Mb/s) ? I guess gluster can handle that load, you are using big files and this is where gluster deliver highest output. Nevertheless, you will need many disk to provide these i/o, even more if using replicated bricks. Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-14 21:15 GMT+02:00 Forrest Aldrich : > Sorry, I should have noted that. 380MB is both read and write (I > confirmed this with a developer). > > We do need the NFS stack, as that's how all the code and various many > Instances work -- we have several "workers" that chop up video on the same > namespace. It's not efficient, but that's how it has to be for now. > > Redundancy, in terms of the server? We have RAIDED volumes if that's > what you're referring to. > > Here's a basic outline of the flow (as I understand it): > > > Video Capture Agent sends in large file of video (30gb +/-) > > Administrative host receives and writes to NFS > > A process copies this over to another point in the namespace > > Another Instance picks up the file, reads and starts processing and writes > (FFMPEG is involved) > > > Something like that -- I may not have all the steps, but essentially > there's a ton of I/O going on. I know our code model is not efficient, > but it's complicated and can't just be changed (it's based on an open > source product and there's some code baggage). > > We looked into another product that allegedly scaled out using multiple > NFS heads with massive local cache (AWS instances) and sharing the same > space, but it was horrible and just didn't work for us. > > > > Thank you. > > > > > On 7/14/15 3:06 PM, Mathieu Chateau wrote: > > Hello, > > is it 380MB in read or write ? What level of redundancy do you need? > do you really need nfs stack or just a mount point (and so be able to use > native gluster protocol) ? > > Gluster load is mostly put on clients, not server (clients do the sync > writes to all replica, and do the memory cache) > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-07-14 20:49 GMT+02:00 Forrest Aldrich : > >> I'm exploring solutions to help us achieve high throughput and >> scalability within the AWS environment. Specifically, I work in a >> department where we handle and produce video content that results in very >> large files (30GB etc) that must be written to NFS, chopped up and copied >> over on the same mount (there are some odd limits to the code we use, but >> that's outside the scope of this question). >> >> Currently, we're using a commercial vendor with AWS, with dedicated >> Direct Connect instances as the back end to our production. We're maxing >> out at 350 to 380 MB/s which is not enough. We expect our capacity will >> double or even triple when we bring on more classes or even other entities >> and we need to find a way to squeeze out as much I/O as we can. >> >> Our software model depends on NFS, there's no way around that presently. >> >> Since GlusterFS uses FUSE, I'm concerned about performance, which is a >> key issue. Sounds like a STRIPE would be appropriate. >> >> My basic understanding of Gluster is the ability to include several >> "bricks" which could be multiples of either dedicated EBS volumes or even >> multiple instances of the above commercial vendor, served up via NFS >> namespace, which would be transparently a single namespace to client >> connections. The I/O could be distributed in this manner. >> >> I wonder if someone here with more experience with the above might >> elaborate on whether GlusterFS could be used in the above scenario. >> Specifically, performance I/O. We'd really like to gain upwards as much as >> possible, like 700 Mb/s and 1 GB/s and up if possible. >> >> >> >> Thanks in advance. >> >> >> >> >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Newbie: Exploring Gluster for large-scale deployment in AWS, large media files, high performance I/O
Hello, is it 380MB in read or write ? What level of redundancy do you need? do you really need nfs stack or just a mount point (and so be able to use native gluster protocol) ? Gluster load is mostly put on clients, not server (clients do the sync writes to all replica, and do the memory cache) Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-14 20:49 GMT+02:00 Forrest Aldrich : > I'm exploring solutions to help us achieve high throughput and scalability > within the AWS environment. Specifically, I work in a department where we > handle and produce video content that results in very large files (30GB > etc) that must be written to NFS, chopped up and copied over on the same > mount (there are some odd limits to the code we use, but that's outside the > scope of this question). > > Currently, we're using a commercial vendor with AWS, with dedicated Direct > Connect instances as the back end to our production. We're maxing out at > 350 to 380 MB/s which is not enough. We expect our capacity will double or > even triple when we bring on more classes or even other entities and we > need to find a way to squeeze out as much I/O as we can. > > Our software model depends on NFS, there's no way around that presently. > > Since GlusterFS uses FUSE, I'm concerned about performance, which is a key > issue. Sounds like a STRIPE would be appropriate. > > My basic understanding of Gluster is the ability to include several > "bricks" which could be multiples of either dedicated EBS volumes or even > multiple instances of the above commercial vendor, served up via NFS > namespace, which would be transparently a single namespace to client > connections. The I/O could be distributed in this manner. > > I wonder if someone here with more experience with the above might > elaborate on whether GlusterFS could be used in the above scenario. > Specifically, performance I/O. We'd really like to gain upwards as much as > possible, like 700 Mb/s and 1 GB/s and up if possible. > > > > Thanks in advance. > > > > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Request for information
Yes, after it depend on your average file size Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-12 12:43 GMT+02:00 Geoffrey Letessier : > Hello Mathieu, > > Thank you for replying. > What do you think about reconstructing every RAID VD with 128KB stripe > size? Indeed, with this value, if one day i decide to use stripped volume, > 128KB is the common value of logical stripe size. > > Thanks again, > Have a nice end-of-weekend > Amicalement, > Geoffrey > -- > Geoffrey Letessier > Responsable informatique & ingénieur système > UPR 9080 - CNRS - Laboratoire de Biochimie Théorique > Institut de Biologie Physico-Chimique > 13, rue Pierre et Marie Curie - 75005 Paris > Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@ibpc.fr > > Le 12 juil. 2015 à 11:49, Mathieu Chateau a > écrit : > > Hello, > > If these 4 virtual drives are then put together to get one big volume, it > will goes as the slower one. > I would put the LSI to 64K to get same size everywhere anyway, even if > it's transparent/hidden from an OS perspective > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-07-12 11:16 GMT+02:00 Geoffrey Letessier > : > >> Dear all, >> >> Because I’m currently reconstructing all my GlusterFS volumes, I just >> noticed that all of my RAID virtual drives did not necessarily have the >> same stripe size and I wonder whether it would be better to >> reconstruct/rebuild these virtual drives with the same stripe size ... What >> do you recommend ? >> >> Actually, 2 virtual drives of 4 have a stripe size set to 64KB (Dell Perc >> H8xx RAID controller default settings) and the 2 left are set to 256KB (LSI >> MegaRAID 9286-8e default settings). With iozone tests, i notice Dell Perc >> VD have better results than LSI ones. (~1.3-4GBs vs ~1.2GBs) and I wonder >> if these parameters are not in cause... >> >> It’s very urgent for me to know what i have to do because I need to >> reconstruct and set all my volumes and back transfert roughly 40TBs in >> distributed and replicated volumes as quickly as possible… >> >> Thanks by advance, >> Geoffrey >> -- >> Geoffrey Letessier >> Responsable informatique & ingénieur système >> UPR 9080 - CNRS - Laboratoire de Biochimie Théorique >> Institut de Biologie Physico-Chimique >> 13, rue Pierre et Marie Curie - 75005 Paris >> Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@ibpc.fr >> >> ___ >> Gluster-users mailing list >> Gluster-users@gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Request for information
Hello, If these 4 virtual drives are then put together to get one big volume, it will goes as the slower one. I would put the LSI to 64K to get same size everywhere anyway, even if it's transparent/hidden from an OS perspective Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-12 11:16 GMT+02:00 Geoffrey Letessier : > Dear all, > > Because I’m currently reconstructing all my GlusterFS volumes, I just > noticed that all of my RAID virtual drives did not necessarily have the > same stripe size and I wonder whether it would be better to > reconstruct/rebuild these virtual drives with the same stripe size ... What > do you recommend ? > > Actually, 2 virtual drives of 4 have a stripe size set to 64KB (Dell Perc > H8xx RAID controller default settings) and the 2 left are set to 256KB (LSI > MegaRAID 9286-8e default settings). With iozone tests, i notice Dell Perc > VD have better results than LSI ones. (~1.3-4GBs vs ~1.2GBs) and I wonder > if these parameters are not in cause... > > It’s very urgent for me to know what i have to do because I need to > reconstruct and set all my volumes and back transfert roughly 40TBs in > distributed and replicated volumes as quickly as possible… > > Thanks by advance, > Geoffrey > -- > Geoffrey Letessier > Responsable informatique & ingénieur système > UPR 9080 - CNRS - Laboratoire de Biochimie Théorique > Institut de Biologie Physico-Chimique > 13, rue Pierre et Marie Curie - 75005 Paris > Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@ibpc.fr > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] How to regenerate all metadata with brick content
Hello Geoffrey, If gluster dev couldn't fix it, I would go the dumb way: 1. Remove the 2 replicated brick 2. Reset these 2 bricks using new uuid/volumes (like if they were just installed) 3. rsync data directly from current nodes to this new 2 nodes cluster 4. reset the 2 remaining bricks 5. setup them to replicate the others one. If you are installing 3.7.2, all clients should be updated before Using fuse (native gluster protocol/client), load is pushed on clients (they write to both replica). So they have to be tuned too. just my 2cents on how I would try to resolve this issue Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-07-07 6:43 GMT+02:00 Geoffrey Letessier : > Dear all, > > We are currently in a very critical situation because all our scientific > production in our computing center is stopped since more than a week for > maintenance (because of metadata "corruption"), I tried some scripts > provided my GlusterFS support team with no effect and I can not continue in > this way. > > So,I wonder if it exists a way to regenerate all bricks metadata > (.glusterfs directories) after having remove ".glusterfs" directory on each > brick (probably after having upgraded GlusterFS from v3.5.3 to v3.7.2) > > Your volume is distributed and replicated on 4 servers with 2 bricks each > (i.e. distributed on 2 servers and replicated on the 2 others). My volume > is not stripped > > In addition, because my the past I noticed a lot of performance issue, > probably because some users play with a big files and some others with a > lot (thousands, hundreds of thousands nay more) small files (hundreds of > kB), could you advice me about how to best tune the volume with performance > parameters? > > Thanks in advance, > Geoffrey > -- > Geoffrey Letessier > Responsable informatique & ingénieur système > UPR 9080 - CNRS - Laboratoire de Biochimie Théorique > Institut de Biologie Physico-Chimique > 13, rue Pierre et Marie Curie - 75005 Paris > Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@ibpc.fr > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS 3.7.2 released (RPMs yet to be available)
Great :) Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-06-22 14:52 GMT+02:00 Kaleb S. KEITHLEY : > On 06/21/2015 01:07 AM, Atin Mukherjee wrote: > > All, > > > > GlusterFS 3.7.2 has been released. The packages for Centos, Debian, > > Fedora and RHEL will be available at > > http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.2/ in their > > respective directories in a day or so. I will be notifying all of you > > once all the RPMs are in place. > > RPMs and .debs are there now for Fedora 20, 21, and 22; RHEL and CentOS > (and clones) 6 and 7; Debian stretch and jessie; SLES 12; and OpenSuSE > 13.2. > > The Ubuntu Launchpad ~gluster PPA has .debs for trusty, utopic, and vivid. > > -- > > Kaleb > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance
I am afraid I am not experienced enough to be much more useful. My guess is that, since client is writing synchronously to all node (to keep data coherent), it's going as fast as the slowest brick. Then small files are often slow because TCP windows doesn't have time to grow up. That's why I gave you some kernel tuning to help TCP Windows to get bigger faster. Do you use latest version (3.7.1) ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-06-20 11:01 GMT+02:00 Geoffrey Letessier : > Hello Mathieu, > > Thanks for replying. > > Previously, i’ve never notice such throughput performances (around 1GBs > for 1 big file) but The situation with a « big » set of small files > wasn’t amazing but not such bad than today. > > The problem seems to concern exclusively the size of each file. > "proof": > [root@node056 tmp]# dd if=/dev/zero of=masterfile bs=1M count=1000 > 1000+0 enregistrements lus > 1000+0 enregistrements écrits > 1048576000 octets (1,0 GB) copiés, 2,09139 s, 501 MB/s > [root@node056 tmp]# time split -b 100 -a 12 masterfile # 1MB per file > > real 0m42.841s > user 0m0.004s > sys 0m1.416s > [root@node056 tmp]# rm -f xa* && sync > [root@node056 tmp]# time split -b 500 -a 12 masterfile # 5 MB per > file > > real 0m17.801s > user 0m0.008s > sys 0m1.396s > [root@node056 tmp]# rm -f xa* && sync > [root@node056 tmp]# time split -b 1000 -a 12 masterfile # 10MB per > file > > real 0m9.686s > user 0m0.008s > sys 0m1.451s > [root@node056 tmp]# rm -f xa* && sync > [root@node056 tmp]# time split -b 2000 -a 12 masterfile # 20MB per > file > > real 0m9.717s > user 0m0.003s > sys 0m1.399s > [root@node056 tmp]# rm -f xa* && sync > [root@node056 tmp]# time split -b 100 -a 12 masterfile # 10MB per > file > > real 0m40.283s > user 0m0.007s > sys 0m1.390s > [root@node056 tmp]# rm -f xa* && sync > > Higher is the generated file size, best is the performance (IO throughput > and running time)… ifstat output is coherent from both client/node and > server side.. > > a new test: > [root@node056 tmp]# dd if=/dev/zero of=masterfile bs=1M count=1 > 1+0 enregistrements lus > 1+0 enregistrements écrits > 1048576 octets (10 GB) copiés, 23,0044 s, 456 MB/s > [root@node056 tmp]# rm -f xa* && sync > [root@node056 tmp]# time split -b 1000 -a 12 masterfile # 10MB per > file > > real 1m43.216s > user 0m0.038s > sys 0m13.407s > > > So the performance per file is the same (despite of 10x more files) > > So, i dont understand why, to get the best performance, i need to create > file with a size of 10MB or more. > > Here are my volume reconfigured options: > performance.cache-max-file-size: 64MB > performance.read-ahead: on > performance.write-behind: on > features.quota-deem-statfs: on > performance.stat-prefetch: on > performance.flush-behind: on > features.default-soft-limit: 90% > features.quota: on > diagnostics.brick-log-level: CRITICAL > auth.allow: localhost,127.0.0.1,10.* > nfs.disable: on > performance.cache-size: 1GB > performance.write-behind-window-size: 4MB > performance.quick-read: on > performance.io-cache: on > performance.io-thread-count: 64 > nfs.enable-ino32: off > > It’s not a local cache trouble because: > 1- it’s disabled in my mount command mount -t glusterfs -o transport=rdma, > direct-io-mode=disable,enable-ino32 ib-storage1:vol_home /home > 2- i made my test also playing with /proc/sys/vm/drop_caches > 3- I note the same ifstat output from both client and server side which is > coherent with the computing of bandwidth (file sizes / time (considering > the replication). > > I think it’s not an infiniband network trouble but here are my [not > default] settings: > connected mode with MTU set to 65520 > > Do you confirm my feelings? If yes, do you have any other idea? > > Thanks again and thanks by advance, > Geoffrey > --- > Geoffrey Letessier > > Responsable informatique & ingénieur système > CNRS - UPR 9080 - Laboratoire de Biochimie Théorique > Institut de Biologie Physico-Chimique > 13, rue Pierre et Marie Curie - 75005 Paris > Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@cnrs.fr > > Le 20 juin 2015 à 09:12, Mathieu Chateau a > écrit : > > Hello, > > for the replicated one, is it a new issue or you just didn't notice before > ? Same baseline as before? > > I also have slowness with small files/many files. > > For now I could only tune up things with: > > On gluster level: > gluster volume set
Re: [Gluster-users] GlusterFS 3.5.3 - untar: very poor performance
Hello, for the replicated one, is it a new issue or you just didn't notice before ? Same baseline as before? I also have slowness with small files/many files. For now I could only tune up things with: On gluster level: gluster volume set myvolume performance.io-thread-count 16 gluster volume set myvolume performance.cache-size 1GB gluster volume set myvolume nfs.disable on gluster volume set myvolume readdir-ahead enable gluster volume set myvolume read-ahead disable On network level (client and server) (I don't have infiniband): sysctl -w vm.swappiness=0 sysctl -w net.core.rmem_max=67108864 sysctl -w net.core.wmem_max=67108864 # increase Linux autotuning TCP buffer limit to 32MB sysctl -w net.ipv4.tcp_rmem="4096 87380 33554432" sysctl -w net.ipv4.tcp_wmem="4096 65536 33554432" # increase the length of the processor input queue sysctl -w net.core.netdev_max_backlog=3 # recommended default congestion control is htcp sysctl -w net.ipv4.tcp_congestion_control=htcp But it's still really slow, even if better Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-06-20 2:34 GMT+02:00 Geoffrey Letessier : > Re, > > For comparison, here is the output of the same script run on a distributed > only volume (2 servers of the 4 previously described, 2 bricks each): > ### > UNTAR time consumed > ### > > > real 1m44.698s > user 0m8.891s > sys 0m8.353s > > ### > # DU time consumed ## > ### > > 554M linux-4.1-rc6 > > real 0m21.062s > user 0m0.100s > sys 0m1.040s > > ### > # FIND time consumed > ### > > 52663 > > real 0m21.325s > user 0m0.104s > sys 0m1.054s > > ### > # GREP time consumed > ### > > 7952 > > real 0m43.618s > user 0m0.922s > sys 0m3.626s > > ### > # TAR time consumed # > ### > > > real 0m50.577s > user 0m29.745s > sys 0m4.086s > > ### > # RM time consumed ## > ### > > > real 0m41.133s > user 0m0.171s > sys 0m2.522s > > The performances are amazing different! > > Geoffrey > --- > Geoffrey Letessier > > Responsable informatique & ingénieur système > CNRS - UPR 9080 - Laboratoire de Biochimie Théorique > Institut de Biologie Physico-Chimique > 13, rue Pierre et Marie Curie - 75005 Paris > Tel: 01 58 41 50 93 - eMail: geoffrey.letess...@cnrs.fr > > Le 20 juin 2015 à 02:12, Geoffrey Letessier > a écrit : > > Dear all, > > I just noticed on my main volume of my HPC cluster my IO operations become > impressively poor.. > > Doing some file operations above a linux kernel sources compressed file, > the untar operation can take more than 1/2 hours for this file (roughly > 80MB and 52 000 files inside) as you read below: > ### > UNTAR time consumed > ### > > > real 32m42.967s > user 0m11.783s > sys 0m15.050s > > ### > # DU time consumed ## > ### > > 557M linux-4.1-rc6 > > real 0m25.060s > user 0m0.068s > sys 0m0.344s > > ### > # FIND time consumed > ### > > 52663 > > real 0m25.687s > user 0m0.084s > sys 0m0.387s > > ### > # GREP time consumed > ### > > 7952 > > real 2m15.890s > user 0m0.887s > sys 0m2.777s > > ### > # TAR time consumed # > ### > > &
Re: [Gluster-users] gluster client crash
Where should be the dump file ? in the log file (nothing this same day before): [2015-06-17 05:57:32.034533] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-06-17 05:57:32.085366] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-06-17 05:57:32.085966] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-06-17 05:57:32.086848] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed pending frames: frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2015-06-17 05:57:32 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.6.3 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f66b17a4362] /lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f66b17bb85d] /lib64/libc.so.6(+0x35650)[0x7f66b07bd650] /lib64/libc.so.6(_IO_vfprintf+0x1564)[0x7f66b07d0a94] /lib64/libc.so.6(__vasprintf_chk+0xb5)[0x7f66b0895425] /lib64/libglusterfs.so.0(_gf_log+0x48c)[0x7f66b17a4e4c] /lib64/libglusterfs.so.0(graphyyerror+0xbf)[0x7f66b17fa41f] /lib64/libglusterfs.so.0(graphyyparse+0x337)[0x7f66b17fa867] /lib64/libglusterfs.so.0(glusterfs_graph_construct+0x404)[0x7f66b17fb604] /lib64/libglusterfs.so.0(glusterfs_volfile_reconfigure+0x4a)[0x7f66b17dac6a] /usr/sbin/glusterfs(mgmt_getspec_cbk+0x317)[0x7f66b1c574b7] /lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f66b1578100] /lib64/libgfrpc.so.0(rpc_clnt_notify+0x174)[0x7f66b1578374] /lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f66b15742c3] /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so(+0x8500)[0x7f66a687b500] /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so(+0xacf4)[0x7f66a687dcf4] /lib64/libglusterfs.so.0(+0x766c2)[0x7f66b17f96c2] /usr/sbin/glusterfs(main+0x502)[0x7f66b1c4efb2] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f66b07a9af5] /usr/sbin/glusterfs(+0x6351)[0x7f66b1c4f351] - Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-06-17 16:53 GMT+02:00 Krutika Dhananjay : > Hi, > > Looks like the process crashed. > Could you provide the logs associated with this process along with the > volume configuration? > The process must have dumped a core file. Could you attach the core to gdb > and provide its backtrace as well? > > -Krutika > > ---------- > > *From: *"Mathieu Chateau" > *To: *"gluster-users" > *Sent: *Wednesday, June 17, 2015 7:16:03 PM > *Subject: *[Gluster-users] gluster client crash > > > Hello, > > On on my gluster mount crashed this morning on one of my gluster client > for this share. > I am use fuse. > > Gluster server wad updated to 3.7.1 and this client too but not rebooted. > Trying to mount again this share failed. After reboot, everything is ok > again. > > Any clue ? > > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: pending frames: > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: patchset: git:// > git.gluster.com/glusterfs.git > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: signal received: 11 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: time of crash: > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: 2015-06-17 05:57:32 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: configuration details: > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: argp 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: backtrace 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: dlfcn 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: libpthread 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: llistxattr 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: setfsid 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: spinlock 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: epoll.h 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: xattr.h 1 > Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: st_atim.tv_nsec 1 > Jun 17 07:57:32 mycl
[Gluster-users] gluster client crash
Hello, On on my gluster mount crashed this morning on one of my gluster client for this share. I am use fuse. Gluster server wad updated to 3.7.1 and this client too but not rebooted. Trying to mount again this share failed. After reboot, everything is ok again. Any clue ? Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: pending frames: Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: frame : type(0) op(0) Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: patchset: git:// git.gluster.com/glusterfs.git Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: signal received: 11 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: time of crash: Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: 2015-06-17 05:57:32 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: configuration details: Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: argp 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: backtrace 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: dlfcn 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: libpthread 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: llistxattr 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: setfsid 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: spinlock 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: epoll.h 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: xattr.h 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: st_atim.tv_nsec 1 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: package-string: glusterfs 3.6.3 Jun 17 07:57:32 myclientLinux mnt-gluster-xxx[1615]: - Cordialement, Mathieu CHATEAU http://www.lotp.fr ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] performance.cache-size
Hello, I can't find explicit information, so I am trying here... Is cache only done on client side ? I saw it for quota feature (that I am not using). My bricks have 32GB of ram, but only kernel cache is using it. Gluster daemon are taking very few (0.4% max for one of them). I have 5 volumes, all set up with performance.cache-size to 1GB But on clients, some gluster daemon are taking actually 1GB of ram and even more on some. Can someone confirm this setting only make cache on clients ? Is kernel cache on bricks much useful ? I have memory, so I can add more on clients, I just want to know impacts before setting performance.cache-size to 4GB for example. I have small files and I am trying to cache whenever possible to speed up things Thanks Regards, Mathieu CHATEAU http://www.lotp.fr ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] using a preferred node ?
>From this slide (maybe outdated) it says that reads are also balanced (in replication scenario slide 22): http://www.gluster.org/community/documentation/images/8/80/GlusterFS_Architecture_%26_Roadmap-Vijay_Bellur-LinuxCon_EU_2013.pdf Except for write, having an option to do only "failover" for reads & lookup would be possible I guess ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-06-08 8:11 GMT+02:00 Ravishankar N : > > > On 06/08/2015 11:34 AM, Mathieu Chateau wrote: > > Hello Ravi, > > thanks for clearing things up. > > Anything on the roadmap that would help my case? > > > > I don't think it would be possible for clients to do I/O only on its local > brick and yet expect the bricks' contents to be in sync in real-time.. > > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-06-08 6:37 GMT+02:00 Ravishankar N : > >> >> >> On 06/06/2015 12:49 AM, Mathieu Chateau wrote: >> >> Hello, >> >> sorry to bother again but I am still facing this issue. >> >> client still looks on the "other side" and not using the node declared >> in fstab: >> prd-sta-sto01:/gluster-preprod /mnt/gluster-preprod glusterfs >> defaults,_netdev,backupvolfile-server=prd-sta-sto02 0 0 >> >> I expect client to use sto01 and not sto02 as it's available. >> >> >> Hi Mathieu, >> When you do lookups (`ls` etc), they are sent to both bricks of the >> replica. If you write to a file, the write is also sent to both bricks. >> This is how it works. Only reads are served from the local brick. >> -Ravi >> >> >> >> If I add a static route to break connectivity to sto02 and do a "df", I >> have around 30s before it works. >> Then it works ok. >> >> Questions: >> >>- How to force node to stick as possible with one specific (local) >>node ? >>- How to know where a client is currently connected? >> >> Thanks for your help :) >> >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-05-11 7:26 GMT+02:00 Mathieu Chateau : >> >>> Hello, >>> >>> thanks for helping :) >>> >>> If gluster server is rebooted, any way to make client failback on node >>> after reboot ? >>> >>> How to know which node is using a client ? I see TCP connection to >>> both node >>> >>> Regards, >>> >>> Cordialement, >>> Mathieu CHATEAU >>> http://www.lotp.fr >>> >>> 2015-05-11 7:13 GMT+02:00 Ravishankar N : >>> >>>> >>>> >>>> On 05/10/2015 08:29 PM, Mathieu Chateau wrote: >>>> >>>> Hello, >>>> >>>> Short way: Is there any way to define a preferred Gluster server ? >>>> >>>> Long way: >>>> I have the following setup (version 3.6.3) : >>>> >>>> Gluster A <==> VPN <==> Gluster B >>>> >>>> Volume is replicated between A and B. >>>> >>>> They are in same datacenter, using a 1Gb/s connection, low latency >>>> (0.5ms) >>>> >>>> I have gluster clients in lan A & B. >>>> >>>> When doing a "ls" on big folder (~60k files), both gluster node are >>>> used, and so it need 9mn instead on 1mn if only the local gluster is >>>> reachable. >>>> >>>> >>>> Lookups (and writes of course) from clients are sent to both bricks >>>> because AFR uses the result of the lookup to select which brick to read >>>> from if there is a pending heal etc. >>>> If the file is clean on both A and B, then reads are always served from >>>> the local brick. i.e. reads on clients mounted on A will be served from the >>>> brick in A (and likewise for B). >>>> >>>> Hope that helps, >>>> Ravi >>>> >>>> >>>> It's HA setup, application is present on both side. I would like a >>>> master/master setup, but using only local node as possible. >>>> >>>> >>>> Regards, >>>> Mathieu CHATEAU >>>> http://www.lotp.fr >>>> >>>> >>>> ___ >>>> Gluster-users mailing >>>> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>> >> >> > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] using a preferred node ?
Hello Ravi, thanks for clearing things up. Anything on the roadmap that would help my case? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-06-08 6:37 GMT+02:00 Ravishankar N : > > > On 06/06/2015 12:49 AM, Mathieu Chateau wrote: > > Hello, > > sorry to bother again but I am still facing this issue. > > client still looks on the "other side" and not using the node declared > in fstab: > prd-sta-sto01:/gluster-preprod /mnt/gluster-preprod glusterfs > defaults,_netdev,backupvolfile-server=prd-sta-sto02 0 0 > > I expect client to use sto01 and not sto02 as it's available. > > > Hi Mathieu, > When you do lookups (`ls` etc), they are sent to both bricks of the > replica. If you write to a file, the write is also sent to both bricks. > This is how it works. Only reads are served from the local brick. > -Ravi > > > > If I add a static route to break connectivity to sto02 and do a "df", I > have around 30s before it works. > Then it works ok. > > Questions: > >- How to force node to stick as possible with one specific (local) >node ? >- How to know where a client is currently connected? > > Thanks for your help :) > > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-05-11 7:26 GMT+02:00 Mathieu Chateau : > >> Hello, >> >> thanks for helping :) >> >> If gluster server is rebooted, any way to make client failback on node >> after reboot ? >> >> How to know which node is using a client ? I see TCP connection to both >> node >> >> Regards, >> >> Cordialement, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> 2015-05-11 7:13 GMT+02:00 Ravishankar N : >> >>> >>> >>> On 05/10/2015 08:29 PM, Mathieu Chateau wrote: >>> >>> Hello, >>> >>> Short way: Is there any way to define a preferred Gluster server ? >>> >>> Long way: >>> I have the following setup (version 3.6.3) : >>> >>> Gluster A <==> VPN <==> Gluster B >>> >>> Volume is replicated between A and B. >>> >>> They are in same datacenter, using a 1Gb/s connection, low latency >>> (0.5ms) >>> >>> I have gluster clients in lan A & B. >>> >>> When doing a "ls" on big folder (~60k files), both gluster node are >>> used, and so it need 9mn instead on 1mn if only the local gluster is >>> reachable. >>> >>> >>> Lookups (and writes of course) from clients are sent to both bricks >>> because AFR uses the result of the lookup to select which brick to read >>> from if there is a pending heal etc. >>> If the file is clean on both A and B, then reads are always served from >>> the local brick. i.e. reads on clients mounted on A will be served from the >>> brick in A (and likewise for B). >>> >>> Hope that helps, >>> Ravi >>> >>> >>> It's HA setup, application is present on both side. I would like a >>> master/master setup, but using only local node as possible. >>> >>> >>> Regards, >>> Mathieu CHATEAU >>> http://www.lotp.fr >>> >>> >>> ___ >>> Gluster-users mailing >>> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >> > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] using a preferred node ?
Hello, sorry to bother again but I am still facing this issue. client still looks on the "other side" and not using the node declared in fstab: prd-sta-sto01:/gluster-preprod /mnt/gluster-preprod glusterfs defaults,_netdev,backupvolfile-server=prd-sta-sto02 0 0 I expect client to use sto01 and not sto02 as it's available. If I add a static route to break connectivity to sto02 and do a "df", I have around 30s before it works. Then it works ok. Questions: - How to force node to stick as possible with one specific (local) node ? - How to know where a client is currently connected? Thanks for your help :) Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-05-11 7:26 GMT+02:00 Mathieu Chateau : > Hello, > > thanks for helping :) > > If gluster server is rebooted, any way to make client failback on node > after reboot ? > > How to know which node is using a client ? I see TCP connection to both > node > > Regards, > > Cordialement, > Mathieu CHATEAU > http://www.lotp.fr > > 2015-05-11 7:13 GMT+02:00 Ravishankar N : > >> >> >> On 05/10/2015 08:29 PM, Mathieu Chateau wrote: >> >> Hello, >> >> Short way: Is there any way to define a preferred Gluster server ? >> >> Long way: >> I have the following setup (version 3.6.3) : >> >> Gluster A <==> VPN <==> Gluster B >> >> Volume is replicated between A and B. >> >> They are in same datacenter, using a 1Gb/s connection, low latency >> (0.5ms) >> >> I have gluster clients in lan A & B. >> >> When doing a "ls" on big folder (~60k files), both gluster node are >> used, and so it need 9mn instead on 1mn if only the local gluster is >> reachable. >> >> >> Lookups (and writes of course) from clients are sent to both bricks >> because AFR uses the result of the lookup to select which brick to read >> from if there is a pending heal etc. >> If the file is clean on both A and B, then reads are always served from >> the local brick. i.e. reads on clients mounted on A will be served from the >> brick in A (and likewise for B). >> >> Hope that helps, >> Ravi >> >> >> It's HA setup, application is present on both side. I would like a >> master/master setup, but using only local node as possible. >> >> >> Regards, >> Mathieu CHATEAU >> http://www.lotp.fr >> >> >> ___ >> Gluster-users mailing >> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users >> >> >> > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] using a preferred node ?
Hello, thanks for helping :) If gluster server is rebooted, any way to make client failback on node after reboot ? How to know which node is using a client ? I see TCP connection to both node Regards, Cordialement, Mathieu CHATEAU http://www.lotp.fr 2015-05-11 7:13 GMT+02:00 Ravishankar N : > > > On 05/10/2015 08:29 PM, Mathieu Chateau wrote: > > Hello, > > Short way: Is there any way to define a preferred Gluster server ? > > Long way: > I have the following setup (version 3.6.3) : > > Gluster A <==> VPN <==> Gluster B > > Volume is replicated between A and B. > > They are in same datacenter, using a 1Gb/s connection, low latency > (0.5ms) > > I have gluster clients in lan A & B. > > When doing a "ls" on big folder (~60k files), both gluster node are > used, and so it need 9mn instead on 1mn if only the local gluster is > reachable. > > > Lookups (and writes of course) from clients are sent to both bricks > because AFR uses the result of the lookup to select which brick to read > from if there is a pending heal etc. > If the file is clean on both A and B, then reads are always served from > the local brick. i.e. reads on clients mounted on A will be served from the > brick in A (and likewise for B). > > Hope that helps, > Ravi > > > It's HA setup, application is present on both side. I would like a > master/master setup, but using only local node as possible. > > > Regards, > Mathieu CHATEAU > http://www.lotp.fr > > > ___ > Gluster-users mailing > listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users > > > ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] using a preferred node ?
Hello, Short way: Is there any way to define a preferred Gluster server ? Long way: I have the following setup (version 3.6.3) : Gluster A <==> VPN <==> Gluster B Volume is replicated between A and B. They are in same datacenter, using a 1Gb/s connection, low latency (0.5ms) I have gluster clients in lan A & B. When doing a "ls" on big folder (~60k files), both gluster node are used, and so it need 9mn instead on 1mn if only the local gluster is reachable. It's HA setup, application is present on both side. I would like a master/master setup, but using only local node as possible. Regards, Mathieu CHATEAU http://www.lotp.fr ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
