Hi all, i was able to track down the problem:
our zone config contains a default-placement with a value for "data_extra_pool". This pool, e.g. "dev.rgw.buckets.non-ec" did not exists and could not be created automatically. Logs show: 2018-03-28 11:26:33.151533 7f569b30e700 1 -- 10.197.115.31:0/388946122 --> 10.197.115.21:6789/0 -- pool_op(create pool 0 auid 0 tid 33 name dev.rgw.buckets.non-ec v0) v4 -- 0x564e68394780 con 0 2018-03-28 11:26:33.151775 7f56f75cd700 1 -- 10.197.115.31:0/388946122 <== mon.0 10.197.115.21:6789/0 11 ==== pool_op_reply(tid 33 (1) Operation not permitted v1083) v1 ==== 43+0+0 (967791429 0 0) 0x564e68394780 con 0x564e66898000 I created the pool and that solved the problem. Could anybody tell me, which caps are needed, that this could happen automatically? Best regards, Ingo -----Ursprüngliche Nachricht----- Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von Ingo Reimann Gesendet: Freitag, 2. März 2018 14:15 An: ceph-users Betreff: [ceph-users] Multipart Upload - POST fails Hi, we discovered some problem with our installation - Multipart upload is not working. What we did: * tried upload with cyberduck as well as with script from http://tracker.ceph.com/issues/12790 * tried against jewel gateways and luminous gateways from old cluster * tried against 12.2.4 gateway with jewel-era cluster Surprisingly this is no signature problem as in the issue above, instead I get the following in the logs: 2018-03-02 13:59:04.927353 7fe2053ca700 1 ====== starting new request req=0x7fe2053c42c0 ===== 2018-03-02 13:59:04.927383 7fe2053ca700 2 req 61:0.000030::POST /luminous-12-2-4/Data128MB::initializing for trans_id = tx00000000000000000003d-005a994a98-10c84997-default 2018-03-02 13:59:04.927396 7fe2053ca700 10 rgw api priority: s3=5 s3website=4 2018-03-02 13:59:04.927399 7fe2053ca700 10 host=cephrgw01.dunkel.de 2018-03-02 13:59:04.927422 7fe2053ca700 20 subdomain= domain=cephrgw01.dunkel.de in_hosted_domain=1 in_hosted_domain_s3website=0 2018-03-02 13:59:04.927427 7fe2053ca700 20 final domain/bucket subdomain= domain=cephrgw01.dunkel.de in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=cephrgw01.dunkel.de s->info.request_uri=/luminous-12-2-4/Data128MB 2018-03-02 13:59:04.927447 7fe2053ca700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 2018-03-02 13:59:04.927454 7fe2053ca700 10 meta>> HTTP_X_AMZ_DATE 2018-03-02 13:59:04.927459 7fe2053ca700 10 x>> x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4 52b97453917 2018-03-02 13:59:04.927464 7fe2053ca700 10 x>> x-amz-date:20180302T125904Z 2018-03-02 13:59:04.927493 7fe2053ca700 20 get_handler handler=22RGWHandler_REST_Obj_S3 2018-03-02 13:59:04.927500 7fe2053ca700 10 handler=22RGWHandler_REST_Obj_S3 2018-03-02 13:59:04.927505 7fe2053ca700 2 req 61:0.000152:s3:POST /luminous-12-2-4/Data128MB::getting op 4 2018-03-02 13:59:04.927512 7fe2053ca700 10 op=28RGWInitMultipart_ObjStore_S3 2018-03-02 13:59:04.927514 7fe2053ca700 2 req 61:0.000161:s3:POST /luminous-12-2-4/Data128MB:init_multipart:verifying requester 2018-03-02 13:59:04.927519 7fe2053ca700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2018-03-02 13:59:04.927524 7fe2053ca700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2018-03-02 13:59:04.927531 7fe2053ca700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2018-03-02 13:59:04.927533 7fe2053ca700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2018-03-02 13:59:04.927569 7fe2053ca700 10 v4 signature format = 48cc8c61a70dde17932d925f65f843116199c1ca10094db83e7de05bfbd57dc4 2018-03-02 13:59:04.927584 7fe2053ca700 10 v4 credential format = 8DGDGA57XL9YPM8DGEQQ/20180302/us-east-1/s3/aws4_request 2018-03-02 13:59:04.927587 7fe2053ca700 10 access key id = 8DGDGA57XL9YPM8DGEQQ 2018-03-02 13:59:04.927589 7fe2053ca700 10 credential scope = 20180302/us-east-1/s3/aws4_request 2018-03-02 13:59:04.927620 7fe2053ca700 10 canonical headers format = content-type:application/octet-stream date:Fri, 02 Mar 2018 12:59:04 GMT host:cephrgw01.dunkel.de x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4 52b97453917 x-amz-date:20180302T125904Z 2018-03-02 13:59:04.927634 7fe2053ca700 10 payload request hash = 254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e452b97453917 2018-03-02 13:59:04.927690 7fe2053ca700 10 canonical request = POST /luminous-12-2-4/Data128MB uploads= content-type:application/octet-stream date:Fri, 02 Mar 2018 12:59:04 GMT host:cephrgw01.dunkel.de x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4 52b97453917 x-amz-date:20180302T125904Z content-type;date;host;x-amz-content-sha256;x-amz-date 254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e452b97453917 2018-03-02 13:59:04.927696 7fe2053ca700 10 canonical request hash = 54e9858263535b46a3c4e51b2ae5c1d0bf5e7a7690c5bba722eea749e7b936c4 2018-03-02 13:59:04.927716 7fe2053ca700 10 string to sign = AWS4-HMAC-SHA256 20180302T125904Z 20180302/us-east-1/s3/aws4_request 54e9858263535b46a3c4e51b2ae5c1d0bf5e7a7690c5bba722eea749e7b936c4 2018-03-02 13:59:04.927920 7fe2053ca700 10 date_k = dcef1f3be70873f1cb3240f7a56320e3c6763e7cf4bfae0e3182d2f9525292cd 2018-03-02 13:59:04.927954 7fe2053ca700 10 region_k = 3d83dd9161cf7ba15e6c8c28d264f6cfce9b848e927359f34364a6c8c98209b7 2018-03-02 13:59:04.927963 7fe2053ca700 10 service_k = e0708e00dc6b52aa1d889f45cd1dcced2bb1b2eee1b62e94ad9813c555e8eda9 2018-03-02 13:59:04.927972 7fe2053ca700 10 signing_k = 1ae362c4b2f1666786404fdb56c62d4f393635b2ce76d46ba325097fd3aa645e 2018-03-02 13:59:04.928021 7fe2053ca700 10 generated signature = 48cc8c61a70dde17932d925f65f843116199c1ca10094db83e7de05bfbd57dc4 2018-03-02 13:59:04.928031 7fe2053ca700 15 string_to_sign=AWS4-HMAC-SHA256 20180302T125904Z 20180302/us-east-1/s3/aws4_request 54e9858263535b46a3c4e51b2ae5c1d0bf5e7a7690c5bba722eea749e7b936c4 2018-03-02 13:59:04.928056 7fe2053ca700 15 server signature=48cc8c61a70dde17932d925f65f843116199c1ca10094db83e7de05bfbd57dc4 2018-03-02 13:59:04.928060 7fe2053ca700 15 client signature=48cc8c61a70dde17932d925f65f843116199c1ca10094db83e7de05bfbd57dc4 2018-03-02 13:59:04.928062 7fe2053ca700 15 compare=0 2018-03-02 13:59:04.928075 7fe2053ca700 20 rgw::auth::s3::LocalEngine granted access 2018-03-02 13:59:04.928078 7fe2053ca700 20 rgw::auth::s3::AWSAuthStrategy granted access 2018-03-02 13:59:04.928084 7fe2053ca700 2 req 61:0.000731:s3:POST /luminous-12-2-4/Data128MB:init_multipart:normalizing buckets and tenants 2018-03-02 13:59:04.928089 7fe2053ca700 10 s->object=Data128MB s->bucket=luminous-12-2-4 2018-03-02 13:59:04.928095 7fe2053ca700 2 req 61:0.000742:s3:POST /luminous-12-2-4/Data128MB:init_multipart:init permissions 2018-03-02 13:59:04.928134 7fe2053ca700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>Xa*****</ID><Di splayName>IngoReimann</DisplayName></Owner><AccessControlList><Grant><Gran tee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser"><ID>Xa*****</ID><DisplayName>IngoReimann</Display Name></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessContro lList></AccessControlPolicy> 2018-03-02 13:59:04.928207 7fe2053ca700 2 req 61:0.000852:s3:POST /luminous-12-2-4/Data128MB:init_multipart:recalculating target 2018-03-02 13:59:04.928221 7fe2053ca700 2 req 61:0.000868:s3:POST /luminous-12-2-4/Data128MB:init_multipart:reading permissions 2018-03-02 13:59:04.928227 7fe2053ca700 2 req 61:0.000874:s3:POST /luminous-12-2-4/Data128MB:init_multipart:init op 2018-03-02 13:59:04.928231 7fe2053ca700 2 req 61:0.000878:s3:POST /luminous-12-2-4/Data128MB:init_multipart:verifying op mask 2018-03-02 13:59:04.928234 7fe2053ca700 20 required_mask= 2 user.op_mask=7 2018-03-02 13:59:04.928237 7fe2053ca700 2 req 61:0.000884:s3:POST /luminous-12-2-4/Data128MB:init_multipart:verifying op permissions 2018-03-02 13:59:04.928242 7fe2053ca700 20 -- Getting permissions begin with perm_mask=50 2018-03-02 13:59:04.928244 7fe2053ca700 5 Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=Xa*****, acct_name=IngoReimann, subuser=, perm_mask=15, is_admin=0) mask=50 2018-03-02 13:59:04.928248 7fe2053ca700 5 Searching permissions for uid=Xa***** 2018-03-02 13:59:04.928253 7fe2053ca700 5 Found permission: 15 2018-03-02 13:59:04.928256 7fe2053ca700 5 Searching permissions for group=1 mask=50 2018-03-02 13:59:04.928258 7fe2053ca700 5 Permissions for group not found 2018-03-02 13:59:04.928261 7fe2053ca700 5 Searching permissions for group=2 mask=50 2018-03-02 13:59:04.928264 7fe2053ca700 5 Permissions for group not found 2018-03-02 13:59:04.928265 7fe2053ca700 5 -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=Xa*****, acct_name=IngoReimann, subuser=, perm_mask=15, is_admin=0), owner=Xa*****, perm=2 2018-03-02 13:59:04.928269 7fe2053ca700 10 identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=Xa*****, acct_name=IngoReimann, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 2018-03-02 13:59:04.928273 7fe2053ca700 2 req 61:0.000920:s3:POST /luminous-12-2-4/Data128MB:init_multipart:verifying op params 2018-03-02 13:59:04.928276 7fe2053ca700 2 req 61:0.000923:s3:POST /luminous-12-2-4/Data128MB:init_multipart:pre-executing 2018-03-02 13:59:04.928279 7fe2053ca700 2 req 61:0.000926:s3:POST /luminous-12-2-4/Data128MB:init_multipart:executing 2018-03-02 13:59:04.928358 7fe2053ca700 10 x>> x-amz-content-sha256:254bcc3fc4f27172636df4bf32de9f107f620d559b20d760197e4 52b97453917 2018-03-02 13:59:04.928386 7fe2053ca700 10 x>> x-amz-date:20180302T125904Z 2018-03-02 13:59:04.928473 7fe2053ca700 20 get_obj_state: rctx=0x7fe2053c2e80 obj=luminous-12-2-4:_multipart_Data128MB.2~F8MNbtuGGxv_tsWiC9lQC3r_M65EEdp .meta state=0x55c249dfa608 s->prefetch_data=0 2018-03-02 13:59:04.931750 7fe2053ca700 2 req 61:0.004395:s3:POST /luminous-12-2-4/Data128MB:init_multipart:completing 2018-03-02 13:59:04.932171 7fe2053ca700 2 req 61:0.004817:s3:POST /luminous-12-2-4/Data128MB:init_multipart:op status=-1 2018-03-02 13:59:04.932194 7fe2053ca700 2 req 61:0.004841:s3:POST /luminous-12-2-4/Data128MB:init_multipart:http status=403 Any hints, what could be the problem? Best regards, Ingo Reimann http://www.Dunkel.de/ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
