[ceph-users] Re: [CEPH] Ceph multi nodes failed
Hello. I get it. I will do a test and let you know. Thank you much. Nguyen Huu Khoi On Fri, Nov 24, 2023 at 5:01 PM Janne Johansson wrote: > Den fre 24 nov. 2023 kl 08:53 skrev Nguyễn Hữu Khôi < > nguyenhuukho...@gmail.com>: > > > > Hello. > > I have 10 nodes. My goal is to ensure that I won't lose data if 2 nodes > > fail. > > Now you are mixing terms here. > > There is a difference between "cluster stops" and "losing data". > > If you have EC 8+2 and min_size 9, then when you stop two hosts, ceph > stops allowing writes exactly so that you do not lose data, making > sure the data is protected until you can get one or two hosts back up > again into the cluster. If you need to keep being able to write to the > cluster with two hosts down, you need EC to be X+3 with min_size = > X+1, this way it will still allow writes when two hosts are down. > > -- > May the most significant bit of your life be positive. > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [CEPH] Ceph multi nodes failed
Hello. I have 10 nodes. My goal is to ensure that I won't lose data if 2 nodes fail. Nguyen Huu Khoi On Fri, Nov 24, 2023 at 2:47 PM Etienne Menguy wrote: > Hello, > > How many nodes do you have? > > > -Original Message- > > From: Nguyễn Hữu Khôi > > Sent: vendredi 24 novembre 2023 07:42 > > To: ceph-users@ceph.io > > Subject: [ceph-users] [CEPH] Ceph multi nodes failed > > > > Hello guys. > > > > I see many docs and threads talking about osd failed. I have a question: > > how many nodes in a cluster can be failed. > > > > I am using ec 8 + 2(10 osd nodes) and when I shutdown 2 nodes then my > > cluster crashes, It cannot write anymore. > > > > Thank you. Regards > > > > Nguyen Huu Khoi > > ___ > > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an > email to > > ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [CEPH] Ceph multi nodes failed
Hello. I am reading. Thank you for information. Nguyen Huu Khoi On Fri, Nov 24, 2023 at 1:56 PM Eugen Block wrote: > Hi, > > basically, with EC pools you usually have a min_size of k + 1 to > prevent data loss. There was a thread about that just a few days ago > on this list. So in your case your min_size is probably 9, which makes > IO pause in case two chunks become unavailable. If your crush failure > domain is host (seems like it is) and you have "only" 10 hosts I'd > recommend to add a host if possible to be able to fully recover while > one host is down. Otherwise the PGs stay degraded until the host comes > back. > So in your case your cluster can handle only one down host, e. g. for > maintenance. If another host goes down (disk, network, whatever) you > hit the min_size limit. Temporarily, you can set min_size = k but you > should not risk anything and increase back to k + 1 after successful > recovery. It's not possible to change the EC profile of a pool, you'd > have to create a new pool and copy the data. > > Check out the EC docs [1] to have some more details. > > Regards, > Eugen > > [1] > > https://docs.ceph.com/en/quincy/rados/operations/erasure-code/?highlight=k%2B1#erasure-coded-pool-recovery > > Zitat von Nguyễn Hữu Khôi : > > > Hello guys. > > > > I see many docs and threads talking about osd failed. I have a question: > > how many nodes in a cluster can be failed. > > > > I am using ec 8 + 2(10 osd nodes) and when I shutdown 2 nodes then my > > cluster crashes, It cannot write anymore. > > > > Thank you. Regards > > > > Nguyen Huu Khoi > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] [CEPH] Ceph multi nodes failed
Hello guys. I see many docs and threads talking about osd failed. I have a question: how many nodes in a cluster can be failed. I am using ec 8 + 2(10 osd nodes) and when I shutdown 2 nodes then my cluster crashes, It cannot write anymore. Thank you. Regards Nguyen Huu Khoi ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [CEPH] OSD Memory Usage
Hello. Great Information, I will keep in mind. Thank you :) Nguyen Huu Khoi On Thu, Nov 16, 2023 at 5:51 PM Janne Johansson wrote: > Den tors 16 nov. 2023 kl 08:43 skrev Nguyễn Hữu Khôi > : > > > > Hello, > > Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is > over > > target. Does Mem Use include cache, I am right? > > > > NAMEHOST PORTSSTATUS REFRESHED > > AGE MEM USE MEM LIM VERSIONIMAGE ID CONTAINER ID > > > > osd.7 sg-osd01 running (3d) 8m ago > > 4w4231M4096M 17.2.6 90a2664234e1 922185643cb8 > > > btw as you said, I feel this value does not have much impact because if > we > > set 1 or 4GB. It still can consume much memory when they need more > memory, > > Don't forget that the tuneable part is only one type of memory > consumption and not ALL parts of the places where and OSD needs ram, > so the absolutely mandatory ram consumption is not covered by the > settings, only the "optional" part, which goes to caches and other > performance improving usages. > During recovery and startup and other events, the required usage can > and will spike, regardless of your memory usage settings. > > If recovery suddenly needs 20G, it would not stop recovery because you > set the limit to say, max 19G ram, it would try to use 20G to recover, > then go back to normal low usage and have as much as possible of the > given 19G as caches again. > > -- > May the most significant bit of your life be positive. > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [CEPH] OSD Memory Usage
Hello. I will read more about it. Thank you :) Nguyen Huu Khoi On Thu, Nov 16, 2023 at 3:21 PM Zakhar Kirpichenko wrote: > Orch ps seems to show virtual size set instead of resident size set. > > /Z > > On Thu, 16 Nov 2023 at 09:43, Nguyễn Hữu Khôi > wrote: > >> Hello, >> Yes, I see it does not exceed RSS but I see in "ceph orch ps". it is >> over target. Does Mem Use include cache, I am right? >> >> NAMEHOST PORTSSTATUS REFRESHED >> AGE MEM USE MEM LIM VERSIONIMAGE ID CONTAINER ID >> >> osd.7 sg-osd01 running (3d) 8m ago >> 4w4231M4096M 17.2.6 90a2664234e1 922185643cb8 >> osd.8 sg-osd03 running (3d) 7m ago >> 4w3407M4096M 17.2.6 90a2664234e1 0ec74fe54bbe >> osd.9 sg-osd01 running (3d) 8m ago >> 4w4575M4096M 17.2.6 90a2664234e1 c2f1c1ee2087 >> osd.10 sg-osd03 running (3d) 7m ago >> 4w3821M4096M 17.2.6 90a2664234e1 fecbd5e910de >> osd.11 sg-osd01 running (3d) 8m ago >> 4w3578M4096M 17.2.6 90a2664234e1 f201704e9026 >> osd.12 sg-osd03 running (3d) 7m ago >> 4w3076M4096M 17.2.6 90a2664234e1 e741b67b6582 >> osd.13 sg-osd01 running (3d) 8m ago >> 4w3688M4096M 17.2.6 90a2664234e1 bffa59278fc2 >> osd.14 sg-osd03 running (3d) 7m ago >> 4w3652M4096M 17.2.6 90a2664234e1 7d9eb3fb9c1e >> osd.15 sg-osd01 running (3d) 8m ago >> 4w3343M4096M 17.2.6 90a2664234e1 d96a425ae5c9 >> osd.16 sg-osd03 running (3d) 7m ago >> 4w2492M4096M 17.2.6 90a2664234e1 637c43176fdc >> osd.17 sg-osd01 running (3d) 8m ago >> 4w3011M4096M 17.2.6 90a2664234e1 a39456dd2c0c >> osd.18 sg-osd03 running (3d) 7m ago >> 4w2341M4096M 17.2.6 90a2664234e1 7b750672391b >> osd.19 sg-osd01 running (3d) 8m ago >> 4w2672M4096M 17.2.6 90a2664234e1 6358234e95f5 >> osd.20 sg-osd03 running (3d) 7m ago >> 4w3297M4096M 17.2.6 90a2664234e1 2ecba6b066fd >> osd.21 sg-osd01 running (3d) 8m ago >> 4w5147M4096M 17.2.6 90a2664234e1 1d0e4efe48bd >> osd.22 sg-osd03 running (3d) 7m ago >> 4w3432M4096M 17.2.6 90a2664234e1 5bb6d4f71f9d >> osd.23 sg-osd03 running (3d) 7m ago >> 4w2893M4096M 17.2.6 90a2664234e1 f7e1948e57d5 >> osd.24 sg-osd02 running (3d) 7m ago >> 12d3007M4096M 17.2.6 90a2664234e1 85d896abe467 >> osd.25 sg-osd02 running (3d) 7m ago >> 12d2666M4096M 17.2.6 90a2664234e1 9800cd8ff1a1 >> osd.26 sg-osd02 running (3d) 7m ago >> 12d2918M4096M 17.2.6 90a2664234e1 f2e0b2d50625 >> osd.27 sg-osd02 running (3d) 7m ago >> 12d3586M4096M 17.2.6 90a2664234e1 ee2fa3a9b40a >> osd.28 sg-osd02 running (3d) 7m ago >> 12d2391M4096M 17.2.6 90a2664234e1 4cf7adf9f60a >> osd.29 sg-osd02 running (3d) 7m ago >> 12d5642M4096M 17.2.6 90a2664234e1 8c1ba98a1738 >> osd.30 sg-osd02 running (3d) 7m ago >> 12d4728M4096M 17.2.6 90a2664234e1 e308497de2e5 >> osd.31 sg-osd02 running (3d) 7m ago >> 12d3615M4096M 17.2.6 90a2664234e1 89b80d464627 >> osd.32 sg-osd02 running (3d) 7m ago >> 12d1703M4096M 17.2.6 90a2664234e1 1e4608786078 >> osd.33 sg-osd02 running (3d) 7m ago >> 12d3039M4096M 17.2.6 90a2664234e1 16e04a1da987 >> osd.34 sg-osd02 running (3d) 7m ago >> 12d2434M4096M 17.2.6 90a2664234e1 014076e28182 >> >> >> >> btw as you said, I feel this value does not have much impact because if >> we set 1 or 4GB. It still can
[ceph-users] Re: [CEPH] OSD Memory Usage
eph-osd > 90297 167 20 0 19.5g 15.6g 12432 S 3.0 4.1 2261:00 > ceph-osd >9799 167 20 0 22.9g 15.4g 12432 S 2.0 4.1 2494:00 > ceph-osd >9778 167 20 0 23.1g 15.3g 12556 S 2.6 4.1 2591:25 > ceph-osd >9815 167 20 0 23.4g 15.1g 12584 S 2.0 4.0 2722:28 > ceph-osd >9809 167 20 0 22.3g 15.1g 12068 S 3.6 4.0 5234:52 > ceph-osd >9811 167 20 0 23.4g 14.9g 12952 S 2.6 4.0 2593:19 > ceph-osd >9819 167 20 0 23.9g 14.9g 12636 S 2.6 4.0 3043:19 > ceph-osd >9820 167 20 0 23.3g 14.8g 12884 S 2.0 3.9 3073:43 > ceph-osd >9769 167 20 0 22.4g 14.7g 12612 S 2.6 3.9 2840:22 > ceph-osd >9836 167 20 0 24.0g 14.7g 12648 S 2.6 3.9 3300:34 > ceph-osd >9818 167 20 0 22.0g 14.7g 12152 S 2.3 3.9 5729:06 > ceph-osd > > Long story short, if you set reasonable targets, OSDs are unlikely to > exceed them during normal operations. If you set memory targets too low, it > is likely that they will be exceeded as OSDs need reasonable amounts of > memory to operate. > > /Z > > On Thu, 16 Nov 2023 at 08:37, Nguyễn Hữu Khôi > wrote: > >> Hello. Thank you very much for your explanation. >> >> Because I thought that osd_memory_target will help me limit OSD memory >> usage which will help prevent memory leak - I tried google and many people >> talked about memory leak. A nice man, @Anthony D'Atri >> , on this forum helped me to understand that it >> wont help to limit OSD usage. >> >> I set it to 1GB because I want to see how this option works. >> >> I will read and test with caches options. >> >> Nguyen Huu Khoi >> >> >> On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko >> wrote: >> >>> Hi, >>> >>> osd_memory_target is a "target", i.e. an OSD make an effort to consume >>> up to the specified amount of RAM, but won't consume less than required for >>> its operation and caches, which have some minimum values such as for >>> example osd_memory_cache_min, bluestore_cache_size, >>> bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended >>> and default OSD memory target is 4 GB. >>> >>> Your nodes have a sufficient amount of RAM, thus I don't see why you >>> would want to reduce OSD memory consumption below the recommended defaults, >>> especially considering that in-memory caches are important for Ceph >>> operations as they're many times faster than the fastest storage devices. I >>> run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, >>> especially with slower HDD-backed OSDs. >>> >>> /Z >>> >>> On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi >>> wrote: >>> >>>> Hello, >>>> I am using a CEPH cluster. After monitoring it, I set: >>>> >>>> ceph config set osd osd_memory_target_autotune false >>>> >>>> ceph config set osd osd_memory_target 1G >>>> >>>> Then restart all OSD services then do test again, I just use fio >>>> commands >>>> from multi clients and I see that OSD memory consume is over 1GB. Would >>>> you >>>> like to help me understand this case? >>>> >>>> Ceph version: Quincy >>>> >>>> OSD: 3 nodes with 11 nvme each and 512GB ram per node. >>>> >>>> CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket. >>>> >>>> Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage >>>> network. >>>> MTU is 9000 >>>> >>>> Thank you very much. >>>> >>>> >>>> Nguyen Huu Khoi >>>> ___ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-le...@ceph.io >>>> >>> ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [CEPH] OSD Memory Usage
Hello. Thank you very much for your explanation. Because I thought that osd_memory_target will help me limit OSD memory usage which will help prevent memory leak - I tried google and many people talked about memory leak. A nice man, @Anthony D'Atri , on this forum helped me to understand that it wont help to limit OSD usage. I set it to 1GB because I want to see how this option works. I will read and test with caches options. Nguyen Huu Khoi On Thu, Nov 16, 2023 at 12:23 PM Zakhar Kirpichenko wrote: > Hi, > > osd_memory_target is a "target", i.e. an OSD make an effort to consume up > to the specified amount of RAM, but won't consume less than required for > its operation and caches, which have some minimum values such as for > example osd_memory_cache_min, bluestore_cache_size, > bluestore_cache_size_hdd, bluestore_cache_size_ssd, etc. The recommended > and default OSD memory target is 4 GB. > > Your nodes have a sufficient amount of RAM, thus I don't see why you would > want to reduce OSD memory consumption below the recommended defaults, > especially considering that in-memory caches are important for Ceph > operations as they're many times faster than the fastest storage devices. I > run my OSDs with osd_memory_target=17179869184 (16 GB) and it helps, > especially with slower HDD-backed OSDs. > > /Z > > On Thu, 16 Nov 2023 at 01:02, Nguyễn Hữu Khôi > wrote: > >> Hello, >> I am using a CEPH cluster. After monitoring it, I set: >> >> ceph config set osd osd_memory_target_autotune false >> >> ceph config set osd osd_memory_target 1G >> >> Then restart all OSD services then do test again, I just use fio commands >> from multi clients and I see that OSD memory consume is over 1GB. Would >> you >> like to help me understand this case? >> >> Ceph version: Quincy >> >> OSD: 3 nodes with 11 nvme each and 512GB ram per node. >> >> CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket. >> >> Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. >> MTU is 9000 >> >> Thank you very much. >> >> >> Nguyen Huu Khoi >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] [CEPH] OSD Memory Usage
Hello, I am using a CEPH cluster. After monitoring it, I set: ceph config set osd osd_memory_target_autotune false ceph config set osd osd_memory_target 1G Then restart all OSD services then do test again, I just use fio commands from multi clients and I see that OSD memory consume is over 1GB. Would you like to help me understand this case? Ceph version: Quincy OSD: 3 nodes with 11 nvme each and 512GB ram per node. CPU: 2 socket xeon gold 6138 cpu with 56 cores per socket. Network: 25Gbps x 2 for public network and 25Gbps x 2 for storage network. MTU is 9000 Thank you very much. Nguyen Huu Khoi ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
