[ceph-users] RGW limiting requests/sec
Hello. I have a Ceph cluster (using Nautilus) in a lab environment on a smaller scale than the production environment. We had some problems with timeouts in production, so I started doing some benchmarking tests in this lab environment. The problem is that the performance of RGW (with beast) is very low, I'm only getting around 600 requests/s using "wrk" making HEAD requests. The size of the RGWs VMs are the same in both lab and production. Do you have any idea what could be causing this limit? I tried to increase the rgw thread pool size but all it did was decrease the number of requests/sec. My rgw client configuration: [client.rgw.ceph-rgw-1.rgw0] host = ceph-rgw-1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw-1.rgw0/keyring log_to_file = true log file = /var/log/ceph/ceph-rgw-ceph-rgw-1.rgw0.log rgw frontends = beast endpoint=10.79.35.245:8080 Another test I did was to switch to civitweb, and the number of requests/sec increased to 1800, which I found strange because I thought beast would be more efficient. To discard network problem, I started an nginx, and in that I managed to reach 16 requests/sec. Am I missing something here? Thank you very much, Marcelo. "Essa mensagem do SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO), empresa pública federal regida pelo disposto na Lei Federal nº 5.615, é enviada exclusivamente ao destinatário informado e pode conter dados pessoais, protegidos pela Lei Geral de Proteção de Dados (Lei 13.709/2018), assim como informações confidenciais, protegidas por sigilo profissional. O SERPRO ressalta seu comprometimento em assegurar a segurança e a proteção das informações contidas neste e-mail e informa que a sua utilização desautorizada é ilegal e sujeita o infrator às penas da lei. Se você o recebeu indevidamente, queira, por gentileza, reenviá-lo ao emitente, esclarecendo o equívoco." "This message from SERVIÇO FEDERAL DE PROCESSAMENTO DE DADOS (SERPRO) - a government company established under Brazilian law (5.615/70) - is directed exclusively to its addressee and may contain personal data protected by the General Data Protection Law (13.709/2018) as well as confidencial data, protected under professional secrecy rules. SERPRO highlights its commitment to ensuring the security and protection of the information contained in this email and its unauthorized use is illegal and may subject the transgressor to the law´s penalties. If you´re not the addressee, please send it back, elucidating the failure." ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW STS - MalformedPolicyDocument
I found the error, I was using a json with indentation (in the aws it worked).
When I put this format without spaces it worked. Maybe it would be interesting
to put this note on the page, about how json needs to be formatted. Thank you
very much, I was trying to make it work for days.
De: "Pritha Srivastava"
Para: "marcelo.miziara serpro"
Cc: "ceph-users"
Enviadas: Domingo, 5 de setembro de 2021 13:02:58
Assunto: Re: [ceph-users] Re: RGW STS - MalformedPolicyDocument
I tried the aws iam create role on master today and it worked for me. I've used
your policy file with "Main" corrected to "Principal", like below:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
Thanks,
Pritha
On Sun, Sep 5, 2021 at 9:11 PM Marcelo Mariano Miziara < [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=NDkyNzc5QURCNkUuQTlDNUI==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
| marcelo.mizi...@serpro.gov.br ] > wrote:
Hi Pritha, thanks for the answer.
Even changing to Principal I still get the MalformedPolicyDocument. I tested
with aws cli versions 1 and 2, both returning the error message. I put JSON in
several validators to see if there were any errors and it seems to be okay. I
don't know if I'm missing something in the aws cli configuration, my
credentials are:
[default]
aws_access_key_id = < admin-api-user key >
aws_secret_access_key = < admin-api-user secret >
And my config:
[default]
region = US
output = json
And I checked that no rule with the same same exists.
With the example python from the page [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=NDkyNzc5QURCNkUuQTlDNUI==b70b7db3446fb5c45272034c5174fcaa=aHR0cHM6Ly9kb2NzLmNlcGguY29tL2VuL2xhdGVzdC9yYWRvc2d3L1NUUy8=
| https://docs.ceph.com/en/latest/radosgw/STS/ ] it worked (it has an extra
"]" that needs to be removed in the policy_document variable).
Thanks again, Marcelo.
De: "Pritha Srivastava" < [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=NDkyNzc5QURCNkUuQTlDNUI==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOnByc3JpdmFzQHJlZGhhdC5jb20=
| prsri...@redhat.com ] >
Para: "marcelo.miziara serpro" < [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=NDkyNzc5QURCNkUuQTlDNUI==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
| marcelo.mizi...@serpro.gov.br ] >
Cc: "ceph-users" < [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=NDkyNzc5QURCNkUuQTlDNUI==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOmNlcGgtdXNlcnNAY2VwaC5pbw==
| ceph-users@ceph.io ] >
Enviadas: Domingo, 5 de setembro de 2021 7:07:14
Assunto: Re: [ceph-users] RGW STS - MalformedPolicyDocument
Hi Marcelo,
Your trust policy has an error:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
" Main ":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
In place of 'Main', use 'Principal' as you have done for the radosgw-admin role
create command.
Thanks,
Pritha
On Fri, Sep 3, 2021 at 9:30 PM Marcelo Mariano Miziara < [ [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=OTM1Q0I0OEYwODkuQTc4N0Q==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
|
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=OTM1Q0I0OEYwODkuQTc4N0Q==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
] | [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=NDkyNzc5QURCNkUuQTlDNUI==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
| marcelo.mizi...@serpro.gov.br ] ] > wrote:
Hello all!
I'm having a hard time trying to get the STS to work. I want to give a user
"someuser" the ability to assumerole. I don't know if I got it wrong how to do
it, or if my json is spelled wrong.
I've done tests on the latest versions of nautilus, octopus and pacific, and I
always get the same message.
In RGW I added the following settings:
-
rgw_s3_auth_use_sts = true
rgw_sts_key = "abcdefghijklmnop"
-
Then I create a user "admin-api-user", giving the following caps:
-
# radosgw-admin caps add --uid admin-api-user --caps
"users=*;buck
[ceph-users] Re: RGW STS - MalformedPolicyDocument
Hi Pritha, thanks for the answer.
Even changing to Principal I still get the MalformedPolicyDocument. I tested
with aws cli versions 1 and 2, both returning the error message. I put JSON in
several validators to see if there were any errors and it seems to be okay. I
don't know if I'm missing something in the aws cli configuration, my
credentials are:
[default]
aws_access_key_id = < admin-api-user key >
aws_secret_access_key = < admin-api-user secret >
And my config:
[default]
region = US
output = json
And I checked that no rule with the same same exists.
With the example python from the page
https://docs.ceph.com/en/latest/radosgw/STS/ it worked (it has an extra "]"
that needs to be removed in the policy_document variable).
Thanks again, Marcelo.
De: "Pritha Srivastava"
Para: "marcelo.miziara serpro"
Cc: "ceph-users"
Enviadas: Domingo, 5 de setembro de 2021 7:07:14
Assunto: Re: [ceph-users] RGW STS - MalformedPolicyDocument
Hi Marcelo,
Your trust policy has an error:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
" Main ":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
In place of 'Main', use 'Principal' as you have done for the radosgw-admin role
create command.
Thanks,
Pritha
On Fri, Sep 3, 2021 at 9:30 PM Marcelo Mariano Miziara < [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=OTM1Q0I0OEYwODkuQTc4N0Q==b70b7db3446fb5c45272034c5174fcaa=bWFpbHRvOm1hcmNlbG8ubWl6aWFyYUBzZXJwcm8uZ292LmJy
| marcelo.mizi...@serpro.gov.br ] > wrote:
Hello all!
I'm having a hard time trying to get the STS to work. I want to give a user
"someuser" the ability to assumerole. I don't know if I got it wrong how to do
it, or if my json is spelled wrong.
I've done tests on the latest versions of nautilus, octopus and pacific, and I
always get the same message.
In RGW I added the following settings:
-
rgw_s3_auth_use_sts = true
rgw_sts_key = "abcdefghijklmnop"
-
Then I create a user "admin-api-user", giving the following caps:
-
# radosgw-admin caps add --uid admin-api-user --caps
"users=*;buckets=*;metadata=*;usage=*;roles=*;user-policy=*"
-
But when I try to create a role using aws cli, I get an error message:
-
# aws --endpoint= [
https://mail-inspector.serpro.gov.br/mailinspector/tap/WarningUrlPage.php?HSCTYPE=0=4=OTM1Q0I0OEYwODkuQTc4N0Q==b70b7db3446fb5c45272034c5174fcaa=aHR0cDovLzEwLjc5LjM1LjI0NTo3NDgw
| http://10.79.35.245:7480 ] iam create-role --role-name=role1
--assume-role-policy-document file://policy_document.json
An error occurred (Unknown) when calling the CreateRole operation: Unknown
-
Running the above command with debug, on one of the lines comes the following
message:
-
2021-09-02 10:07:56,138 - MainThread - botocore.parsers - DEBUG - Response
body:
b'MalformedPolicyDocumenttx01-006130ccac-b3b82-defaultb3b82-default
-default'
-
My policy_document.json is like this:
-
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Main":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
-
If I run the the radosgw-admin command with the same JSON (but with escaped
characters), it works:
-
# radosgw-admin role create --role-name=role1
--assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"AWS\":\[\"arn:aws:iam:::user/someuser\"\]\},\"Action\":\[\"sts:AssumeRole\"\]\}\]\}
{
"RoleId": "007ed38e-a072-43a4-94f3-2958e5a19408",
"RoleName": "role1",
"Path": "/",
"Arn": "arn:aws:iam:::role/role1",
"CreateDate": "2021-09-02T13:19:39.721Z",
"MaxSessionDuration": 3600,
"AssumeRolePolicyDocument":
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/someuser\"]},\"Action\":[\"sts:AssumeRole\"]}]}"
}
-
Does anyone have any idea where I might be going wrong? I did a test on aws,
with the same JSON (adding my arn account) and it worked.
Thanks, Marcelo!
___
[ceph-users] RGW STS - MalformedPolicyDocument
Hello all!
I'm having a hard time trying to get the STS to work. I want to give a user
"someuser" the ability to assumerole. I don't know if I got it wrong how to do
it, or if my json is spelled wrong.
I've done tests on the latest versions of nautilus, octopus and pacific, and I
always get the same message.
In RGW I added the following settings:
-
rgw_s3_auth_use_sts = true
rgw_sts_key = "abcdefghijklmnop"
-
Then I create a user "admin-api-user", giving the following caps:
-
# radosgw-admin caps add --uid admin-api-user --caps
"users=*;buckets=*;metadata=*;usage=*;roles=*;user-policy=*"
-
But when I try to create a role using aws cli, I get an error message:
-
# aws --endpoint=http://10.79.35.245:7480 iam create-role --role-name=role1
--assume-role-policy-document file://policy_document.json
An error occurred (Unknown) when calling the CreateRole operation: Unknown
-
Running the above command with debug, on one of the lines comes the following
message:
-
2021-09-02 10:07:56,138 - MainThread - botocore.parsers - DEBUG - Response
body:
b'MalformedPolicyDocumenttx01-006130ccac-b3b82-defaultb3b82-default
-default'
-
My policy_document.json is like this:
-
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Main":{
"AWS":[
"arn:aws:iam:::user/someuser"
]
},
"Action":[
"sts:AssumeRole"
]
}
]
}
-
If I run the the radosgw-admin command with the same JSON (but with escaped
characters), it works:
-
# radosgw-admin role create --role-name=role1
--assume-role-policy-doc=\{\"Version\":\"2012-10-17\",\"Statement\":\[\{\"Effect\":\"Allow\",\"Principal\":\{\"AWS\":\[\"arn:aws:iam:::user/someuser\"\]\},\"Action\":\[\"sts:AssumeRole\"\]\}\]\}
{
"RoleId": "007ed38e-a072-43a4-94f3-2958e5a19408",
"RoleName": "role1",
"Path": "/",
"Arn": "arn:aws:iam:::role/role1",
"CreateDate": "2021-09-02T13:19:39.721Z",
"MaxSessionDuration": 3600,
"AssumeRolePolicyDocument":
"{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"arn:aws:iam:::user/someuser\"]},\"Action\":[\"sts:AssumeRole\"]}]}"
}
-
Does anyone have any idea where I might be going wrong? I did a test on aws,
with the same JSON (adding my arn account) and it worked.
Thanks, Marcelo!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW unable to delete a bucket
Hi, What version are you using? There seems to exist a bug (https://tracker.ceph.com/issues/42358), we had the same problem using redhat 12.2.12-84. Te only way to stop the rgw logs to crash the machine was to restart the service. To remove the bucket we had to upgrade to 12.2.12.-115 - Mensagem original - De: "Andrei Mikhailovsky" Para: "ceph-users" Enviadas: Terça-feira, 4 de agosto de 2020 13:16:28 Assunto: [ceph-users] RGW unable to delete a bucket Hi I am trying to delete a bucket using the following command: # radosgw-admin bucket rm --bucket= --purge-objects However, in console I get the following messages. About 100+ of those messages per second. 2020-08-04T17:11:06.411+0100 7fe64cacf080 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1 The command has been running for about 35 days days and it still hasn't finished. The size of the bucket is under 1TB for sure. Probably around 500GB. I have recently removed about a dozen of old buckets without any issues. It's this particular bucket that is being very stubborn. Anything I can do to remove it, including it's objects and any orphans it might have? Thanks Andrei ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
