[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
Le mar. 23 févr. 2021, à 03 h 07, Janne Johansson a écrit : > >>> Hello, > >>> We have functional ceph swarm with a pair of S3 rgw in front that uses > >>> A.B.C.D domain to be accessed. > >>> > >>> Now a new client asks to have access using the domain : E.C.D, but to > >>> already existing buckets. This is not a scenario discussed in the > docs. > >>> Apparently, looking at the code and by trying it, rgw does not support > >>> multiple domains for the variable rgw_dns_name. > >>> > >>> But reading through parts of the code, I am no dev, and my c++ is 25 > years > >>> rusty, I get the impression that maybe we could just add a second pair > of > >>> rgw S3 servers that would give service to the same buckets, but using a > >>> different domain. > >>> > >>> Am I wrong ? Let's say this works, is this an unconscious behaviour > that > >>> the ceph team would remove down the road ? > >> > >> We run this, a LB sends to one pool for one DNS name and to another pool > >> for a different DNS name, and both rgws serve the "same" buckets. > > > > > > How can they serve the "same" buckets if they are in different ceph > pools ? Am I understanding you correctly ? To me, same bucket means same > objects. > > I mean that a user can go via either one, and it works. > And no, it is not different ceph pools, it is the same ceph pools > underneath, only the rgw name in the conf differs. > > > So if I were to deploy a new pair of RGWS with the new domain, would it > create a bunch of new pools in ceph to store its objects or reuse the > preexisting ones ? > > It reuses the old pools. The pool names are not tied to the DNS name > the rgw is using, so it starts looking for .rgw.root and from there > divines which zones and zonegroups exist and (in our case) that the > pools are default.rgw.buckets.index and so on, which is true for both > sets of rgws. > > Thank you, it confirms what I thought. In my modest opinion, this multi domain is quite important and should be better documented on the various ways to do it using ceph. This seems to be a distinct advantage of ceph compared to amazon's S3. >> Since S3 auth v4 the dns name is very much a part of the hash to make > your > >> access work, so whatever the client thinks is the DNS name is what it > will > >> use to make the hash-of-hash-of-hash* combination to auth itself. > >> > >> We haven't made a huge attempt to break it by doing wacky parallel > accesses > >> from both directions, but it seems to work to move off clients from old > name > >> to new name and the stragglers that will never change will get the old > small > >> LB pool and the clients with a decent config get better service. > > > > I have a need for parallel access, have you tried it ? > > We have not tried since we see it as either you have moved to the new > name or you haven't. > > I don't expect this to be a showstopper, since having N+1 rgws in all > other cases is equally susceptible to races regardless of the DNS name > the client used to reach an rgw. > After auth is done, I expect it to be quite similar if your client and > my client ends up on different rgw daemons. > Since using N+1 rgw daemons is used in many many installations, I > consider that use-case tested well enough. > > -- > May the most significant bit of your life be positive. > -- Simon Pierre Desrosiers Architecte de Solutions Direction technologies, architecture, innovation et sécurité Service des technologies de l'information ___ 801 rue Brennan, 10e étage Montréal (Québec) H3C 0G4 Cellullaire : 514-513-8551 sp.desrosi...@ville.montreal.qc.ca -- **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et peuvent contenir de l’information privilégiée ou confidentielle. Si vous avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que les pièces jointes, le cas échéant. La copie ou la redistribution non autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne peut être interprété qu’en conformité avec les lois et règlements qui régissent les pouvoirs des diverses instances décisionnelles compétentes de la Ville de Montréal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
Le lun. 22 févr. 2021, à 14 h 50, Chris Palmer a écrit : > I'm not sure that the tenant solution is what the OP wants - my reading > is that running under a different tenant allows you have different > tenants use the same bucket and user names but still be distinct, which > wasn't what I thought was meant. > Our need is for the same user/tenant to use the Bucket using two different domains. We deployed Ceph's S3 internally with internal domains not accessible to the outside of our organization with plan to present S3 externally using a homemade api (we have our reasons). Of course, all good plans have a need for exceptions, hence my question. > > You can however get RGW to accept a list of host names regardless of > domain. Assuming you are using the default zone (adjust if otherwise): > > Extract the default zonegroup settings: > > radosgw-admin zonegroup get > zg-default > > Edit zg-default looking for "hostnames" and adjust to be a list of the > required ones (short and/or FQDN according to what the clients use). > Then install the updated settings: > > radosgw-admin zonegroup set --infile zg-default > > and restart all the RGW instances. > > We do this when fronting RGW with HAProxy, to include the individual > hostnames used by the HAProxy backend. > > Regards, Chris > Thank you very much, that worked out beautifully, we shall go with that solution. > > > On 22/02/2021 14:29, Freddy Andersen wrote: > > You need to enable users with tenants … > https://docs.ceph.com/en/latest/radosgw/multitenancy/ > > > > From: Simon Pierre DESROSIERS > > Date: Monday, February 22, 2021 at 7:27 AM > > To: ceph-users@ceph.io > > Subject: [ceph-users] multiple-domain for S3 on rgws with same ceph > backend on one zone > > Hello, > > > > We have functional ceph swarm with a pair of S3 rgw in front that uses > > A.B.C.D domain to be accessed. > > > > Now a new client asks to have access using the domain : E.C.D, but to > > already existing buckets. This is not a scenario discussed in the docs. > > Apparently, looking at the code and by trying it, rgw does not support > > multiple domains for the variable rgw_dns_name. > > > > > > But reading through parts of the code, I am no dev, and my c++ is 25 > years > > rusty, I get the impression that maybe we could just add a second pair of > > rgw S3 servers that would give service to the same buckets, but using a > > different domain. > > > > Am I wrong ? Let's say this works, is this an unconscious behaviour that > > the ceph team would remove down the road ? > > > > Is there another solution that I might have missed ? We do not have > > multi-zone and there are no plans for it. And Cname (rgw_resolve_cname) > > seems to only be of use when using static sites (again, from my poor code > > reading abilities). > > > > Thank you > > > > -- > > **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont > > destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et > > peuvent contenir de l’information privilégiée ou confidentielle. Si vous > > avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, > veuillez > > le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi > que > > les pièces jointes, le cas échéant. La copie ou la redistribution non > > autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne > > peut être interprété qu’en conformité avec les lois et règlements qui > > régissent les pouvoirs des diverses instances décisionnelles compétentes > de > > la Ville de Montréal. > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et peuvent contenir de l’information privilégiée ou confidentielle. Si vous avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que les pièces jointes, le cas échéant. La copie ou la redistribution non autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne peut être interprété qu’en conformité avec les lois et règlements qui régissent les pouvoirs des diverses instances décisionnelles compétentes de la Ville de Montréal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
>>> Hello, >>> We have functional ceph swarm with a pair of S3 rgw in front that uses >>> A.B.C.D domain to be accessed. >>> >>> Now a new client asks to have access using the domain : E.C.D, but to >>> already existing buckets. This is not a scenario discussed in the docs. >>> Apparently, looking at the code and by trying it, rgw does not support >>> multiple domains for the variable rgw_dns_name. >>> >>> But reading through parts of the code, I am no dev, and my c++ is 25 years >>> rusty, I get the impression that maybe we could just add a second pair of >>> rgw S3 servers that would give service to the same buckets, but using a >>> different domain. >>> >>> Am I wrong ? Let's say this works, is this an unconscious behaviour that >>> the ceph team would remove down the road ? >> >> We run this, a LB sends to one pool for one DNS name and to another pool >> for a different DNS name, and both rgws serve the "same" buckets. > > > How can they serve the "same" buckets if they are in different ceph pools ? > Am I understanding you correctly ? To me, same bucket means same objects. I mean that a user can go via either one, and it works. And no, it is not different ceph pools, it is the same ceph pools underneath, only the rgw name in the conf differs. > So if I were to deploy a new pair of RGWS with the new domain, would it > create a bunch of new pools in ceph to store its objects or reuse the > preexisting ones ? It reuses the old pools. The pool names are not tied to the DNS name the rgw is using, so it starts looking for .rgw.root and from there divines which zones and zonegroups exist and (in our case) that the pools are default.rgw.buckets.index and so on, which is true for both sets of rgws. >> Since S3 auth v4 the dns name is very much a part of the hash to make your >> access work, so whatever the client thinks is the DNS name is what it will >> use to make the hash-of-hash-of-hash* combination to auth itself. >> >> We haven't made a huge attempt to break it by doing wacky parallel accesses >> from both directions, but it seems to work to move off clients from old name >> to new name and the stragglers that will never change will get the old small >> LB pool and the clients with a decent config get better service. > > I have a need for parallel access, have you tried it ? We have not tried since we see it as either you have moved to the new name or you haven't. I don't expect this to be a showstopper, since having N+1 rgws in all other cases is equally susceptible to races regardless of the DNS name the client used to reach an rgw. After auth is done, I expect it to be quite similar if your client and my client ends up on different rgw daemons. Since using N+1 rgw daemons is used in many many installations, I consider that use-case tested well enough. -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
Le lun. 22 févr. 2021, à 10 h 34, Janne Johansson a écrit : > Den mån 22 feb. 2021 kl 15:27 skrev Simon Pierre DESROSIERS < > simonpierre.desrosi...@montreal.ca>: > >> Hello, >> >> We have functional ceph swarm with a pair of S3 rgw in front that uses >> A.B.C.D domain to be accessed. >> >> Now a new client asks to have access using the domain : E.C.D, but to >> already existing buckets. This is not a scenario discussed in the docs. >> Apparently, looking at the code and by trying it, rgw does not support >> multiple domains for the variable rgw_dns_name. >> >> But reading through parts of the code, I am no dev, and my c++ is 25 years >> rusty, I get the impression that maybe we could just add a second pair of >> rgw S3 servers that would give service to the same buckets, but using a >> different domain. >> >> Am I wrong ? Let's say this works, is this an unconscious behaviour that >> the ceph team would remove down the road ? >> > > We run this, a LB sends to one pool for one DNS name and to another pool > for a different DNS name, and both rgws serve the "same" buckets. > How can they serve the "same" buckets if they are in different ceph pools ? Am I understanding you correctly ? To me, same bucket means same objects. So if I were to deploy a new pair of RGWS with the new domain, would it create a bunch of new pools in ceph to store its objects or reuse the preexisting ones ? > Since S3 auth v4 the dns name is very much a part of the hash to make your > access work, so whatever the client thinks is the DNS name is what it will > use to make the hash-of-hash-of-hash* combination to auth itself. > > We haven't made a huge attempt to break it by doing wacky parallel accesses > from both directions, but it seems to work to move off clients from old > name > to new name and the stragglers that will never change will get the old > small > LB pool and the clients with a decent config get better service. > I have a need for parallel access, have you tried it ? > In our case the domains are completely different, so not A.B.C.D vs B.C.D > but > rather F.G.H.I instead. > > *) SIGNATURE=$(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h > $(HMAC-SHA256s $AWS4SECRET $REQUEST_DATE ) $REQUEST_REGION) > $REQUEST_SERVICE) "aws4_request") $UPLOAD_REQUEST) > > -- > May the most significant bit of your life be positive. > -- **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et peuvent contenir de l’information privilégiée ou confidentielle. Si vous avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que les pièces jointes, le cas échéant. La copie ou la redistribution non autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne peut être interprété qu’en conformité avec les lois et règlements qui régissent les pouvoirs des diverses instances décisionnelles compétentes de la Ville de Montréal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
I'm not sure that the tenant solution is what the OP wants - my reading is that running under a different tenant allows you have different tenants use the same bucket and user names but still be distinct, which wasn't what I thought was meant. You can however get RGW to accept a list of host names regardless of domain. Assuming you are using the default zone (adjust if otherwise): Extract the default zonegroup settings: radosgw-admin zonegroup get > zg-default Edit zg-default looking for "hostnames" and adjust to be a list of the required ones (short and/or FQDN according to what the clients use). Then install the updated settings: radosgw-admin zonegroup set --infile zg-default and restart all the RGW instances. We do this when fronting RGW with HAProxy, to include the individual hostnames used by the HAProxy backend. Regards, Chris On 22/02/2021 14:29, Freddy Andersen wrote: You need to enable users with tenants … https://docs.ceph.com/en/latest/radosgw/multitenancy/ From: Simon Pierre DESROSIERS Date: Monday, February 22, 2021 at 7:27 AM To: ceph-users@ceph.io Subject: [ceph-users] multiple-domain for S3 on rgws with same ceph backend on one zone Hello, We have functional ceph swarm with a pair of S3 rgw in front that uses A.B.C.D domain to be accessed. Now a new client asks to have access using the domain : E.C.D, but to already existing buckets. This is not a scenario discussed in the docs. Apparently, looking at the code and by trying it, rgw does not support multiple domains for the variable rgw_dns_name. But reading through parts of the code, I am no dev, and my c++ is 25 years rusty, I get the impression that maybe we could just add a second pair of rgw S3 servers that would give service to the same buckets, but using a different domain. Am I wrong ? Let's say this works, is this an unconscious behaviour that the ceph team would remove down the road ? Is there another solution that I might have missed ? We do not have multi-zone and there are no plans for it. And Cname (rgw_resolve_cname) seems to only be of use when using static sites (again, from my poor code reading abilities). Thank you -- **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et peuvent contenir de l’information privilégiée ou confidentielle. Si vous avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que les pièces jointes, le cas échéant. La copie ou la redistribution non autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne peut être interprété qu’en conformité avec les lois et règlements qui régissent les pouvoirs des diverses instances décisionnelles compétentes de la Ville de Montréal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
Den mån 22 feb. 2021 kl 15:27 skrev Simon Pierre DESROSIERS < simonpierre.desrosi...@montreal.ca>: > Hello, > > We have functional ceph swarm with a pair of S3 rgw in front that uses > A.B.C.D domain to be accessed. > > Now a new client asks to have access using the domain : E.C.D, but to > already existing buckets. This is not a scenario discussed in the docs. > Apparently, looking at the code and by trying it, rgw does not support > multiple domains for the variable rgw_dns_name. > > But reading through parts of the code, I am no dev, and my c++ is 25 years > rusty, I get the impression that maybe we could just add a second pair of > rgw S3 servers that would give service to the same buckets, but using a > different domain. > > Am I wrong ? Let's say this works, is this an unconscious behaviour that > the ceph team would remove down the road ? > We run this, a LB sends to one pool for one DNS name and to another pool for a different DNS name, and both rgws serve the "same" buckets. Since S3 auth v4 the dns name is very much a part of the hash to make your access work, so whatever the client thinks is the DNS name is what it will use to make the hash-of-hash-of-hash* combination to auth itself. We haven't made a huge attempt to break it by doing wacky parallel accesses from both directions, but it seems to work to move off clients from old name to new name and the stragglers that will never change will get the old small LB pool and the clients with a decent config get better service. In our case the domains are completely different, so not A.B.C.D vs B.C.D but rather F.G.H.I instead. *) SIGNATURE=$(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256s $AWS4SECRET $REQUEST_DATE ) $REQUEST_REGION) $REQUEST_SERVICE) "aws4_request") $UPLOAD_REQUEST) -- May the most significant bit of your life be positive. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: multiple-domain for S3 on rgws with same ceph backend on one zone
You need to enable users with tenants … https://docs.ceph.com/en/latest/radosgw/multitenancy/ From: Simon Pierre DESROSIERS Date: Monday, February 22, 2021 at 7:27 AM To: ceph-users@ceph.io Subject: [ceph-users] multiple-domain for S3 on rgws with same ceph backend on one zone Hello, We have functional ceph swarm with a pair of S3 rgw in front that uses A.B.C.D domain to be accessed. Now a new client asks to have access using the domain : E.C.D, but to already existing buckets. This is not a scenario discussed in the docs. Apparently, looking at the code and by trying it, rgw does not support multiple domains for the variable rgw_dns_name. But reading through parts of the code, I am no dev, and my c++ is 25 years rusty, I get the impression that maybe we could just add a second pair of rgw S3 servers that would give service to the same buckets, but using a different domain. Am I wrong ? Let's say this works, is this an unconscious behaviour that the ceph team would remove down the road ? Is there another solution that I might have missed ? We do not have multi-zone and there are no plans for it. And Cname (rgw_resolve_cname) seems to only be of use when using static sites (again, from my poor code reading abilities). Thank you -- **AVERTISSEMENT** : Ce courriel et les pièces qui y sont jointes sont destinés exclusivement au(x) destinataire(s) mentionné(s) ci-dessus et peuvent contenir de l’information privilégiée ou confidentielle. Si vous avez reçu ce courriel par erreur, ou s’il ne vous est pas destiné, veuillez le mentionner immédiatement à l’expéditeur et effacer ce courriel ainsi que les pièces jointes, le cas échéant. La copie ou la redistribution non autorisée de ce courriel peut être illégale. Le contenu de ce courriel ne peut être interprété qu’en conformité avec les lois et règlements qui régissent les pouvoirs des diverses instances décisionnelles compétentes de la Ville de Montréal. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
