Re: [Gluster-users] Advice for setup: SW RAID 6 vs JBOD
Good morning, my comment won't help you directly, but i thought i'd send it anyway... Our first glusterfs setup had 3 servers withs 4 disks=bricks (10TB, JBOD) each. Was running fine in the beginning, but then 1 disk failed. The following heal took ~1 month, with a bad performance (quite high IO). Shortly after the heal hat finished another disk failed -> same problems again. Not funny. For our new system we decided to use 3 servers with 10 disks (10 TB) each, but now the 10 disks in a SW RAID 10 (well, we split the 10 disks into 2 SW RAID 10, each of them is a brick, we have 2 gluster volumes). A lot of disk space "wasted", with this type of SW RAID and a replicate 3 setup, but we wanted to avoid the "healing takes a long time with bad performance" problems. Now mdadm takes care of replicating data, glusterfs should always see "good" bricks. And the decision may depend on what kind of data you have. Many small files, like tens of millions? Or not that much, but bigger files? I once watched a video (i think it was this one: https://www.youtube.com/watch?v=61HDVwttNYI). Recommendation there: RAID 6 or 10 for small files, for big files... well, already 2 years "old" ;-) As i said, this won't help you directly. You have to identify what's most important for your scenario; as you said, high performance is not an issue - if this is true even when you have slight performance issues after a disk fail then ok. My experience so far: the bigger and slower the disks are and the more data you have -> healing will hurt -> try to avoid this. If the disks are small and fast (SSDs), healing will be faster -> JBOD is an option. hth, Hubert Am Mi., 5. Juni 2019 um 11:33 Uhr schrieb Eduardo Mayoral : > > Hi, > > I am looking into a new gluster deployment to replace an ancient one. > > For this deployment I will be using some repurposed servers I > already have in stock. The disk specs are 12 * 3 TB SATA disks. No HW > RAID controller. They also have some SSD which would be nice to leverage > as cache or similar to improve performance, since it is already there. > Advice on how to leverage the SSDs would be greatly appreciated. > > One of the design choices I have to make is using 3 nodes for a > replica-3 with JBOD, or using 2 nodes with a replica-2 and using SW RAID > 6 for the disks, maybe adding a 3rd node with a smaller amount of disk > as metadata node for the replica set. I would love to hear advice on the > pros and cons of each setup from the gluster experts. > > The data will be accessed from 4 to 6 systems with native gluster, > not sure if that makes any difference. > > The amount of data I have to store there is currently 20 TB, with > moderate growth. iops are quite low so high performance is not an issue. > The data will fit in any of the two setups. > > Thanks in advance for your advice! > > -- > Eduardo Mayoral Jimeno > Systems engineer, platform department. Arsys Internet. > emayo...@arsys.es - +34 941 620 105 - ext 2153 > > > ___ > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] Geo Replication stops replicating
Hi, Updated link for documentation : -- https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/ You can use this tool as well: http://aravindavk.in/blog/gluster-georep-tools/ -Sunny On Thu, Jun 6, 2019 at 10:29 AM Kotresh Hiremath Ravishankar wrote: > > Hi, > > I think the steps to setup non-root geo-rep is not followed properly. The > following entry is missing in glusterd vol file which is required. > > The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] > 0-management: 'option mountbroker-root' missing in glusterd vol file" > repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 > 08:52:34.019757] > > Could you please the steps from below? > > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave > > And let us know if you still face the issue. > > > > > On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan wrote: >> >> Hi Kotresh, Sunny >> I Have mailed the logs I found in one of the slave machines. Is there >> anything to do with permission? Please help. >> >> On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan wrote: >>> >>> Hi Kotresh, Sunny >>> Found this log in the slave machine. [2019-06-05 08:49:10.632583] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-06-05 08:49:10.632583] and [2019-06-05 08:49:10.670863] The message "I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and [2019-06-05 08:50:37.254063] The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 34 times between [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079] The message "W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]" repeated 34 times between [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080] [2019-06-05 08:50:46.361347] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req [2019-06-05 08:50:46.361384] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file [2019-06-05 08:50:46.361419] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory] The message "I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and [2019-06-05 08:52:34.019741] The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757] The message "W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory]" repeated 33 times between [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758] [2019-06-05 08:52:44.426839] I [MSGID: 106496] [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received mount req [2019-06-05 08:52:44.426886] E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file [2019-06-05 08:52:44.426896] W [MSGID: 106176] [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful mount request [No such file or directory] >>> >>> >>> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan wrote: Thankyou Kotresh On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar wrote: > > Ccing Sunny, who was investing similar issue. > > On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan > wrote: >> >> Have already added the path in bashrc . Still in faulty state >> >> On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar >> wrote: >>> >>> could you please try adding /usr/sbin to $PATH for user 'sas'? If it's >>> bash, add 'export PATH=/usr/sbin:$PATH' in >>> /home/sas/.bashrc >>> >>> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan >>> wrote: Hi Kort
Re: [Gluster-users] Geo Replication stops replicating
Hi, I think the steps to setup non-root geo-rep is not followed properly. The following entry is missing in glusterd vol file which is required. The message "E [MSGID: 106061] [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option mountbroker-root' missing in glusterd vol file" repeated 33 times between [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757] Could you please the steps from below? https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html-single/administration_guide/index#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave And let us know if you still face the issue. On Thu, Jun 6, 2019 at 10:24 AM deepu srinivasan wrote: > Hi Kotresh, Sunny > I Have mailed the logs I found in one of the slave machines. Is there > anything to do with permission? Please help. > > On Wed, Jun 5, 2019 at 2:28 PM deepu srinivasan > wrote: > >> Hi Kotresh, Sunny >> Found this log in the slave machine. >> >>> [2019-06-05 08:49:10.632583] I [MSGID: 106488] >>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: >>> Received get vol req >>> >>> The message "I [MSGID: 106488] >>> [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: >>> Received get vol req" repeated 2 times between [2019-06-05 08:49:10.632583] >>> and [2019-06-05 08:49:10.670863] >>> >>> The message "I [MSGID: 106496] >>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received >>> mount req" repeated 34 times between [2019-06-05 08:48:41.005398] and >>> [2019-06-05 08:50:37.254063] >>> >>> The message "E [MSGID: 106061] >>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >>> mountbroker-root' missing in glusterd vol file" repeated 34 times between >>> [2019-06-05 08:48:41.005434] and [2019-06-05 08:50:37.254079] >>> >>> The message "W [MSGID: 106176] >>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >>> mount request [No such file or directory]" repeated 34 times between >>> [2019-06-05 08:48:41.005444] and [2019-06-05 08:50:37.254080] >>> >>> [2019-06-05 08:50:46.361347] I [MSGID: 106496] >>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received >>> mount req >>> >>> [2019-06-05 08:50:46.361384] E [MSGID: 106061] >>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >>> mountbroker-root' missing in glusterd vol file >>> >>> [2019-06-05 08:50:46.361419] W [MSGID: 106176] >>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >>> mount request [No such file or directory] >>> >>> The message "I [MSGID: 106496] >>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received >>> mount req" repeated 33 times between [2019-06-05 08:50:46.361347] and >>> [2019-06-05 08:52:34.019741] >>> >>> The message "E [MSGID: 106061] >>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >>> mountbroker-root' missing in glusterd vol file" repeated 33 times between >>> [2019-06-05 08:50:46.361384] and [2019-06-05 08:52:34.019757] >>> >>> The message "W [MSGID: 106176] >>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >>> mount request [No such file or directory]" repeated 33 times between >>> [2019-06-05 08:50:46.361419] and [2019-06-05 08:52:34.019758] >>> >>> [2019-06-05 08:52:44.426839] I [MSGID: 106496] >>> [glusterd-handler.c:3187:__glusterd_handle_mount] 0-glusterd: Received >>> mount req >>> >>> [2019-06-05 08:52:44.426886] E [MSGID: 106061] >>> [glusterd-mountbroker.c:555:glusterd_do_mount] 0-management: 'option >>> mountbroker-root' missing in glusterd vol file >>> >>> [2019-06-05 08:52:44.426896] W [MSGID: 106176] >>> [glusterd-mountbroker.c:719:glusterd_do_mount] 0-management: unsuccessful >>> mount request [No such file or directory] >>> >> >> On Wed, Jun 5, 2019 at 1:06 AM deepu srinivasan >> wrote: >> >>> Thankyou Kotresh >>> >>> On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar < >>> khire...@redhat.com> wrote: >>> Ccing Sunny, who was investing similar issue. On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan wrote: > Have already added the path in bashrc . Still in faulty state > > On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar < > khire...@redhat.com> wrote: > >> could you please try adding /usr/sbin to $PATH for user 'sas'? If >> it's bash, add 'export PATH=/usr/sbin:$PATH' in >> /home/sas/.bashrc >> >> On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan >> wrote: >> >>> Hi Kortesh >>> Please find the logs of the above error >>> *Master log snippet* >>> [2019-06-04 11:52:09.254731] I [resource(worker /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing SSH connection between master and slave... [2019-06-04 11:52:09.308923] D [repce(worker /home/sas/gluster/data/code-misc):196:push] RepceClient: c
Re: [Gluster-users] Memory leak in glusterfs
Hi, Writing to a volume should not affect glusterd. The stack you have shown in the valgrind looks like the memory used to initialise the structures glusterd uses and will free only when it is stopped. Can you provide more details to what it is you are trying to test? Regards, Nithya On Tue, 4 Jun 2019 at 15:41, ABHISHEK PALIWAL wrote: > Hi Team, > > Please respond on the issue which I raised. > > Regards, > Abhishek > > On Fri, May 17, 2019 at 2:46 PM ABHISHEK PALIWAL > wrote: > >> Anyone please reply >> >> On Thu, May 16, 2019, 10:49 ABHISHEK PALIWAL >> wrote: >> >>> Hi Team, >>> >>> I upload some valgrind logs from my gluster 5.4 setup. This is writing >>> to the volume every 15 minutes. I stopped glusterd and then copy away the >>> logs. The test was running for some simulated days. They are zipped in >>> valgrind-54.zip. >>> >>> Lots of info in valgrind-2730.log. Lots of possibly lost bytes in >>> glusterfs and even some definitely lost bytes. >>> >>> ==2737== 1,572,880 bytes in 1 blocks are possibly lost in loss record >>> 391 of 391 >>> ==2737== at 0x4C29C25: calloc (in >>> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) >>> ==2737== by 0xA22485E: ??? (in >>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so) >>> ==2737== by 0xA217C94: ??? (in >>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so) >>> ==2737== by 0xA21D9F8: ??? (in >>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so) >>> ==2737== by 0xA21DED9: ??? (in >>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so) >>> ==2737== by 0xA21E685: ??? (in >>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so) >>> ==2737== by 0xA1B9D8C: init (in >>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so) >>> ==2737== by 0x4E511CE: xlator_init (in /usr/lib64/libglusterfs.so.0.0.1) >>> ==2737== by 0x4E8A2B8: ??? (in /usr/lib64/libglusterfs.so.0.0.1) >>> ==2737== by 0x4E8AAB3: glusterfs_graph_activate (in >>> /usr/lib64/libglusterfs.so.0.0.1) >>> ==2737== by 0x409C35: glusterfs_process_volfp (in /usr/sbin/glusterfsd) >>> ==2737== by 0x409D99: glusterfs_volumes_init (in /usr/sbin/glusterfsd) >>> ==2737== >>> ==2737== LEAK SUMMARY: >>> ==2737== definitely lost: 1,053 bytes in 10 blocks >>> ==2737== indirectly lost: 317 bytes in 3 blocks >>> ==2737== possibly lost: 2,374,971 bytes in 524 blocks >>> ==2737== still reachable: 53,277 bytes in 201 blocks >>> ==2737== suppressed: 0 bytes in 0 blocks >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >> > > -- > > > > > Regards > Abhishek Paliwal > ___ > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Advice for setup: SW RAID 6 vs JBOD
Hi, I am looking into a new gluster deployment to replace an ancient one. For this deployment I will be using some repurposed servers I already have in stock. The disk specs are 12 * 3 TB SATA disks. No HW RAID controller. They also have some SSD which would be nice to leverage as cache or similar to improve performance, since it is already there. Advice on how to leverage the SSDs would be greatly appreciated. One of the design choices I have to make is using 3 nodes for a replica-3 with JBOD, or using 2 nodes with a replica-2 and using SW RAID 6 for the disks, maybe adding a 3rd node with a smaller amount of disk as metadata node for the replica set. I would love to hear advice on the pros and cons of each setup from the gluster experts. The data will be accessed from 4 to 6 systems with native gluster, not sure if that makes any difference. The amount of data I have to store there is currently 20 TB, with moderate growth. iops are quite low so high performance is not an issue. The data will fit in any of the two setups. Thanks in advance for your advice! -- Eduardo Mayoral Jimeno Systems engineer, platform department. Arsys Internet. emayo...@arsys.es - +34 941 620 105 - ext 2153 ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Update: GlusterFS code coverage
All, I just wanted to update everyone about one of the initiatives we have undertaken, ie, increasing the overall code coverage of GlusterFS above 70%. You can have a look at current code coverage here: https://build.gluster.org/job/line-coverage/lastCompletedBuild/Line_20Coverage_20Report/ (This shows the latest all the time) The daily job, and its details are captured @ https://build.gluster.org/job/line-coverage/ When we started focus on code coverage 3 months back, our code coverage was around 60% overall. We kept the ambitious goal of increasing the code coverage by 10% before glusterfs-7.0 release, and I am happy to announce that we met this goal, before the branching. Before talking about next goals, I want to thank and call out few developers who made this happen. * Xavier Hernandez - Made EC cross 90% from < 70%. * Glusterd Team (Sanju, Rishub, Mohit, Atin) - Increased CLI/glusterd coverage * Geo-Rep Team (Kotresh, Sunny, Shwetha, Aravinda). * Sheetal (help to increase glfs-api test cases, which indirectly helped cover more code across). Also note that, Some components like AFR/replicate was already at 80%+ before we started the efforts. Now, our next goal is to make sure we have above 80% functions coverage in all of the top level components shown. Once that is done, we will focus on 75% code coverage across all components. (ie, no 'Red' in top level page). While it was possible to meet our goal of increasing the overall code coverage from 60% - 70%, increasing it above 70% is not going to be easy, mainly because it involves adding more tests for negative test cases, and adding tests with different options (currently >300 of them across). We also need to look at details from code coverage tests, and reverse engineer to see how to write a test to hit the particular line in the code. I personally invite everyone who is interested to contribute to gluster project to get involved in this effort. Help us write test cases, suggest how to improve it. Help by assigning interns write them for us (if your team has some of them). This is a good way to understand glusterfs code too. We are happy to organize sessions on how to walk through the code etc if required. Happy to hear feedback and see more contribution in this area. Regards, Amar ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
