[ceph-users] Re: How you loadbalance your rgw endpoints?

On 9/24/21 07:59, Szabo, Istvan (Agoda) wrote: Hi, Wonder how you guys do it due to we will always have limitation on the network bandwidth of the loadbalancer. You might get rid of the load-balancer entirely, and use DNS based load-balancing. PowerDNS has a powerful feature for this [1]. Th

[ceph-users] Re: How you loadbalance your rgw endpoints?

Hi, How many RGW are you using for this huge cluster? Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From: Svante Kar

[ceph-users] Re: How you loadbalance your rgw endpoints?

I forgot, we actually have two per node, we run a separate cold storage pool as well with erasure encoded data and each pool needs it's own rgw (I think). But they are not load balanced or anything. Den mån 27 sep. 2021 kl 13:18 skrev Svante Karlsson : > > One per kubernetes node (currently 10x 64

[ceph-users] Re: How you loadbalance your rgw endpoints?

One per kubernetes node (currently 10x 64 core nodes). plus one extra outside for non kubernetes stuff. This works for the jobs that we currently run on each node. If they would overload the on-node-rgw we could add more rgw's on the node and put a local haproxy before. same principle. Den mån 27

[ceph-users] Re: How you loadbalance your rgw endpoints?

Hi Robert, I think it was because kubernetes jobs, in docker themselves have an own definitions of 127.0.0.1 (ie inside the container). Den mån 27 sep. 2021 kl 12:50 skrev Robert Sander : > > Am 27.09.21 um 12:44 schrieb Svante Karlsson: > > We added > > a common extra ip address in iptables with

[ceph-users] Re: How you loadbalance your rgw endpoints?

Am 27.09.21 um 12:44 schrieb Svante Karlsson: We added a common extra ip address in iptables with a rule to map that to localhost. Finally each kubernetes job uses this common ip to the "local" rgw server. This way we skip two hops of network traffic to the real gateway and this scales with the

[ceph-users] Re: How you loadbalance your rgw endpoints?

Hi Szabo, we have a 7PB cluster that only servers s3 content for read heavy jobs running on a dedicated kubernetes cluster, all connections are 100G . We overloaded first rgw gateways, and then the loadbalancers. The hackish solution we came up with is to add each kubernetetes node as ceph members

[ceph-users] Re: How you loadbalance your rgw endpoints?

Hi Szabo, I think you can have a look at https://docs.ceph.com/en/latest/cephadm/rgw/#high-availability-service-for-rgw even if you don't deploy ceph using cephadm. Am 24.09.21 um 07:59 schrieb Szabo, Istvan (Ag