Re: [ceph-users] Ceph and Compute on same hardware?
I think you may also consider risk like kernel crashes etc,since storage and compute node are sharing the same box. Date: Wed, 12 Nov 2014 14:51:47 + From: pieter.koo...@me.com To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] Ceph and Compute on same hardware? Hi, Thanks for the replies. Likely will not choose this method but wanted to make sure that it was a good technical reason rather than just a "best practice". I did not quite think of "conntracker" at the time so this is a good one to consider. Thanks Pieter On 12 November 2014 14:30, Haomai Wang wrote: Actually, our production cluster(up to ten) all are that ceph-osd ran on compute-node(KVM). The primary action is that you need to constrain the cpu and memory. For example, you can alloc a ceph cpu-set and memory group, let ceph-osd run with it within limited cores and memory. The another risk is the network. Because compute-node and ceph-osd shared the same kernel network stack, it exists some risks that VM may ran out of network resources such as conntracker in netfilter framework. On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson wrote: > Technically there's no reason it shouldn't work, but it does complicate > things. Probably the biggest worry would be that if something bad happens > on the compute side (say it goes nuts with network or memory transfers) it > could slow things down enough that OSDs start failing heartbeat checks > causing ceph to go into recovery and maybe cause a vicious cycle of > nastiness. > > You can mitigate some of this with cgroups and try to dedicate specific > sockets and memory banks to Ceph/Compute, but we haven't done a lot of > testing yet afaik. > > Mark > > > On 11/12/2014 07:45 AM, Pieter Koorts wrote: >> >> Hi, >> >> A while back on a blog I saw mentioned that Ceph should not be run on >> compute nodes and in the general sense should be on dedicated hardware. >> Does this really still apply? >> >> An example, if you have nodes comprised of >> >> 16+ cores >> 256GB+ RAM >> Dual 10GBE Network >> 2+8 OSD (SSD log + HDD store) >> >> I understand that Ceph can use a lot of IO and CPU in some cases but if >> the nodes are powerful enough does it not make it an option to run >> compute and storage on the same hardware to either increase density of >> compute or save money on additional hardware? >> >> What are the reasons for not running Ceph on the Compute nodes. >> >> Thanks >> >> Pieter >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Best Regards, Wheat ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph and Compute on same hardware?
> A while back on a blog I saw mentioned that Ceph should not be run on compute > nodes and in the general > sense should be on dedicated hardware. Does this really still apply? In my opinion storage needs to be rock-solid. Running other (complex) software on a Ceph node increases the chances of stuff falling over. Worst-case a cascading effect takes down you whole storage platform. If your storage platform bites the dust your whole compute cloud also falls over (assuming you boot instances from Ceph). Troubleshooting issues (especially those that have no obvious cause) becomes more complex having to rule out more potential causes. Not saying it can not work perfectly fine. I'd rather just not take any chances with the storage system... Cheers, Robert van Leeuwen ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph and Compute on same hardware?
Hi, Thanks for the replies. Likely will not choose this method but wanted to make sure that it was a good technical reason rather than just a "best practice". I did not quite think of "conntracker" at the time so this is a good one to consider. Thanks Pieter On 12 November 2014 14:30, Haomai Wang wrote: > Actually, our production cluster(up to ten) all are that ceph-osd ran > on compute-node(KVM). > > The primary action is that you need to constrain the cpu and memory. > For example, you can alloc a ceph cpu-set and memory group, let > ceph-osd run with it within limited cores and memory. > > The another risk is the network. Because compute-node and ceph-osd > shared the same kernel network stack, it exists some risks that VM may > ran out of network resources such as conntracker in netfilter > framework. > > On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson > wrote: > > Technically there's no reason it shouldn't work, but it does complicate > > things. Probably the biggest worry would be that if something bad > happens > > on the compute side (say it goes nuts with network or memory transfers) > it > > could slow things down enough that OSDs start failing heartbeat checks > > causing ceph to go into recovery and maybe cause a vicious cycle of > > nastiness. > > > > You can mitigate some of this with cgroups and try to dedicate specific > > sockets and memory banks to Ceph/Compute, but we haven't done a lot of > > testing yet afaik. > > > > Mark > > > > > > On 11/12/2014 07:45 AM, Pieter Koorts wrote: > >> > >> Hi, > >> > >> A while back on a blog I saw mentioned that Ceph should not be run on > >> compute nodes and in the general sense should be on dedicated hardware. > >> Does this really still apply? > >> > >> An example, if you have nodes comprised of > >> > >> 16+ cores > >> 256GB+ RAM > >> Dual 10GBE Network > >> 2+8 OSD (SSD log + HDD store) > >> > >> I understand that Ceph can use a lot of IO and CPU in some cases but if > >> the nodes are powerful enough does it not make it an option to run > >> compute and storage on the same hardware to either increase density of > >> compute or save money on additional hardware? > >> > >> What are the reasons for not running Ceph on the Compute nodes. > >> > >> Thanks > >> > >> Pieter > >> > >> > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Best Regards, > > Wheat > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph and Compute on same hardware?
On Wed, Nov 12, 2014 at 5:30 PM, Haomai Wang wrote: > Actually, our production cluster(up to ten) all are that ceph-osd ran > on compute-node(KVM). > > The primary action is that you need to constrain the cpu and memory. > For example, you can alloc a ceph cpu-set and memory group, let > ceph-osd run with it within limited cores and memory. > > The another risk is the network. Because compute-node and ceph-osd > shared the same kernel network stack, it exists some risks that VM may > ran out of network resources such as conntracker in netfilter > framework. > > On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson wrote: >> Technically there's no reason it shouldn't work, but it does complicate >> things. Probably the biggest worry would be that if something bad happens >> on the compute side (say it goes nuts with network or memory transfers) it >> could slow things down enough that OSDs start failing heartbeat checks >> causing ceph to go into recovery and maybe cause a vicious cycle of >> nastiness. >> >> You can mitigate some of this with cgroups and try to dedicate specific >> sockets and memory banks to Ceph/Compute, but we haven't done a lot of >> testing yet afaik. >> >> Mark >> >> >> On 11/12/2014 07:45 AM, Pieter Koorts wrote: >>> >>> Hi, >>> >>> A while back on a blog I saw mentioned that Ceph should not be run on >>> compute nodes and in the general sense should be on dedicated hardware. >>> Does this really still apply? >>> >>> An example, if you have nodes comprised of >>> >>> 16+ cores >>> 256GB+ RAM >>> Dual 10GBE Network >>> 2+8 OSD (SSD log + HDD store) >>> >>> I understand that Ceph can use a lot of IO and CPU in some cases but if >>> the nodes are powerful enough does it not make it an option to run >>> compute and storage on the same hardware to either increase density of >>> compute or save money on additional hardware? >>> >>> What are the reasons for not running Ceph on the Compute nodes. >>> >>> Thanks >>> >>> Pieter >>> >>> >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Best Regards, > > Wheat > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Yes, the essential part is a resource management, which can neither be dynamic or static. In Flops we had implemented dynamic resource control which allows to pack VMs and OSDs more densely than static cg-based jails can allow (and it requires deep orchestration modifications for every open source cloud orchestrator, unfortunately). As long as you are able to manage strong traffic isolation for storage and vm segment, there are absolutely no problem (it can be static limits from linux-qos or tricky flow management for OpenFlow, depends on what your orchestration allows). The possibility of putting together compute and storage roles without significant impact to performance characteristics was one of key features which led our selection to Ceph as a storage backend three years ago. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph and Compute on same hardware?
Actually, our production cluster(up to ten) all are that ceph-osd ran on compute-node(KVM). The primary action is that you need to constrain the cpu and memory. For example, you can alloc a ceph cpu-set and memory group, let ceph-osd run with it within limited cores and memory. The another risk is the network. Because compute-node and ceph-osd shared the same kernel network stack, it exists some risks that VM may ran out of network resources such as conntracker in netfilter framework. On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson wrote: > Technically there's no reason it shouldn't work, but it does complicate > things. Probably the biggest worry would be that if something bad happens > on the compute side (say it goes nuts with network or memory transfers) it > could slow things down enough that OSDs start failing heartbeat checks > causing ceph to go into recovery and maybe cause a vicious cycle of > nastiness. > > You can mitigate some of this with cgroups and try to dedicate specific > sockets and memory banks to Ceph/Compute, but we haven't done a lot of > testing yet afaik. > > Mark > > > On 11/12/2014 07:45 AM, Pieter Koorts wrote: >> >> Hi, >> >> A while back on a blog I saw mentioned that Ceph should not be run on >> compute nodes and in the general sense should be on dedicated hardware. >> Does this really still apply? >> >> An example, if you have nodes comprised of >> >> 16+ cores >> 256GB+ RAM >> Dual 10GBE Network >> 2+8 OSD (SSD log + HDD store) >> >> I understand that Ceph can use a lot of IO and CPU in some cases but if >> the nodes are powerful enough does it not make it an option to run >> compute and storage on the same hardware to either increase density of >> compute or save money on additional hardware? >> >> What are the reasons for not running Ceph on the Compute nodes. >> >> Thanks >> >> Pieter >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Best Regards, Wheat ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph and Compute on same hardware?
Technically there's no reason it shouldn't work, but it does complicate things. Probably the biggest worry would be that if something bad happens on the compute side (say it goes nuts with network or memory transfers) it could slow things down enough that OSDs start failing heartbeat checks causing ceph to go into recovery and maybe cause a vicious cycle of nastiness. You can mitigate some of this with cgroups and try to dedicate specific sockets and memory banks to Ceph/Compute, but we haven't done a lot of testing yet afaik. Mark On 11/12/2014 07:45 AM, Pieter Koorts wrote: Hi, A while back on a blog I saw mentioned that Ceph should not be run on compute nodes and in the general sense should be on dedicated hardware. Does this really still apply? An example, if you have nodes comprised of 16+ cores 256GB+ RAM Dual 10GBE Network 2+8 OSD (SSD log + HDD store) I understand that Ceph can use a lot of IO and CPU in some cases but if the nodes are powerful enough does it not make it an option to run compute and storage on the same hardware to either increase density of compute or save money on additional hardware? What are the reasons for not running Ceph on the Compute nodes. Thanks Pieter ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
