On Thursday 15 February 2018 04:38 AM, Yehuda Sadeh-Weinraub wrote:
This request goes from the elastisearch zone to the master zone,
trying to send an update of the period config. Not sure why it would
do it at this point. You can try bumping up the logs on the master
zone, see if there's any info there.
Yehuda
I have set log level to 20 on both RGW - Master Zone and Metadata Zone
with Elastic Tier.
The error response code is same.
*LOGS FROM MASTER ZONE*
2018-02-15 16:47:43.860937 7fd4080d4700 2
RGWDataChangesLog::ChangesRenewThread: start
2018-02-15 16:47:49.503699 7fd3b9e31700 20
cr:s=0x557bb8ac0a50:op=0x557bb88a7c00:14PushAndRetryCR: operate()
2018-02-15 16:47:49.503731 7fd3b9e31700 10 rgw period pusher: pushing
period 6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e to
13ee3cfa-10ba-45ef-aec0-e42d8f55e3b6
2018-02-15 16:47:49.503758 7fd3b9e31700 20
cr:s=0x557bb8ac0a50:op=0x557bb8d24900:21RGWPostRESTResourceCRI9RGWPeriodiE:
operate()
2018-02-15 16:47:49.503762 7fd3b9e31700 20
cr:s=0x557bb8ac0a50:op=0x557bb8d24900:21RGWPostRESTResourceCRI9RGWPeriodiE:
operate()
2018-02-15 16:47:49.503950 7fd3b9e31700 20 > HTTP_DATE -> Thu Feb 15
11:17:49 2018
2018-02-15 16:47:49.503963 7fd3b9e31700 10 get_canon_resource():
dest=/admin/realm/period
2018-02-15 16:47:49.503965 7fd3b9e31700 10 generated canonical header: POST
Thu Feb 15 16:47:49 2018
/admin/realm/period
2018-02-15 16:47:49.504031 7fd3b9e31700 15 generated auth header: AWS
HCTJCLF1F1E857N09X1Y:+k8NKAHCGCWduWIHASM30/bms7s=
2018-02-15 16:47:49.504045 7fd3b9e31700 20 sending request to
http://rgw2:8080/admin/realm/period?period=6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e&epoch=6&rgwx-zonegroup=0d9efa23-09d6-4adf-a486-0858f3261d7b
2018-02-15 16:47:49.504057 7fd3b9e31700 20 register_request
mgr=0x557bb8a55db0 req_data->id=3048, easy_handle=0x557bb9480000
2018-02-15 16:47:49.504089 7fd3b9e31700 20 run: stack=0x557bb8ac0a50 is
io blocked
2018-02-15 16:47:49.504364 7fd3ba632700 20 link_request
req_data=0x557bb89c4d80 req_data->id=3048, easy_handle=0x557bb9480000
2018-02-15 16:47:50.508567 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508592 7fd3ba632700 10 received header:HTTP/1.1 400
Bad Request
2018-02-15 16:47:50.508600 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508601 7fd3ba632700 10 received
header:Content-Length: 115
2018-02-15 16:47:50.508608 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508609 7fd3ba632700 10 received
header:x-amz-request-id: tx00000000000000000000b-005a856c5c-5e3a-uk-west
2018-02-15 16:47:50.508615 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508616 7fd3ba632700 10 received
header:Accept-Ranges: bytes
2018-02-15 16:47:50.508620 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508622 7fd3ba632700 10 received header:Content-Type:
application/json
2018-02-15 16:47:50.508625 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508626 7fd3ba632700 10 received header:Date: Thu, 15
Feb 2018 11:17:49 GMT
2018-02-15 16:47:50.508630 7fd3ba632700 10 receive_http_header
2018-02-15 16:47:50.508631 7fd3ba632700 10 received header:
2018-02-15 16:47:50.509282 7fd3b9e31700 20
cr:s=0x557bb8ac0a50:op=0x557bb8d24900:21RGWPostRESTResourceCRI9RGWPeriodiE:
operate()
2018-02-15 16:47:50.509344 7fd3b9e31700 5 failed to wait for op,
ret=-22: POST
http://rgw2:8080/admin/realm/period?period=6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e&epoch=6&rgwx-zonegroup=0d9efa23-09d6-4adf-a486-0858f3261d7b
2018-02-15 16:47:50.509441 7fd3b9e31700 20
cr:s=0x557bb8ac0a50:op=0x557bb8d24900:21RGWPostRESTResourceCRI9RGWPeriodiE:
*operate() returned r=-22*
2018-02-15 16:47:50.509466 7fd3b9e31700 20
cr:s=0x557bb8ac0a50:op=0x557bb88a7c00:14PushAndRetryCR: operate()
2018-02-15 16:47:50.509471 7fd3b9e31700 10 rgw period pusher: waiting
30.000000s for retry..
2018-02-15 16:47:50.509501 7fd3b9e31700 20 run: stack=0x557bb8ac0a50 is
io blocked
*LOGS ON METADATA ZONE*
2018-02-15 16:47:48.967635 7f12da464700 20 HTTP_ACCEPT=*/*
2018-02-15 16:47:48.967660 7f12da464700 20 HTTP_AUTHORIZATION=AWS
HCTJCLF1F1E857N09X1Y:+k8NKAHCGCWduWIHASM30/bms7s=
2018-02-15 16:47:48.967665 7f12da464700 20 HTTP_DATE=Thu Feb 15 11:17:49
2018
2018-02-15 16:47:48.967667 7f12da464700 20 HTTP_EXPECT=100-continue
2018-02-15 16:47:48.967669 7f12da464700 20 HTTP_HOST=rgw2:8080
2018-02-15 16:47:48.967673 7f12da464700 20 HTTP_TRANSFER_ENCODING=chunked
2018-02-15 16:47:48.967675 7f12da464700 20
QUERY_STRING=period=6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e&epoch=6&rgwx-zonegroup=0d9efa23-09d6-4adf-a486-0858f3261d7b
2018-02-15 16:47:48.967694 7f12da464700 20 REQUEST_METHOD=POST
2018-02-15 16:47:48.967695 7f12da464700 20 REQUEST_URI=/admin/realm/period
2018-02-15 16:47:48.967696 7f12da464700 20 SCRIPT_URI=/admin/realm/period
2018-02-15 16:47:48.967697 7f12da464700 20 SERVER_PORT=8080
2018-02-15 16:47:48.967699 7f12da464700 1 ====== starting new request
req=0x7f12da45e1f0 =====
2018-02-15 16:47:48.967723 7f12da464700 2 req 11:0.000023::POST
/admin/realm/period::initializing for trans_id =
tx00000000000000000000b-005a856c5c-5e3a-uk-west
2018-02-15 16:47:48.967734 7f12da464700 10 rgw api priority: s3=5
s3website=4
2018-02-15 16:47:48.967737 7f12da464700 10 host=rgw2
2018-02-15 16:47:48.967739 7f12da464700 20 subdomain= domain=
in_hosted_domain=0 in_hosted_domain_s3website=0
2018-02-15 16:47:48.967741 7f12da464700 20 final domain/bucket
subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
s->info.domain= s->info.request_uri=/admin/realm/period
2018-02-15 16:47:48.967783 7f12da464700 10 handler=17RGWHandler_Period
2018-02-15 16:47:48.967786 7f12da464700 2 req 11:0.000086::POST
/admin/realm/period::getting op 4
2018-02-15 16:47:48.967790 7f12da464700 10 op=17RGWOp_Period_Post
2018-02-15 16:47:48.967792 7f12da464700 2 req 11:0.000092::POST
/admin/realm/period:post_period:verifying requester
2018-02-15 16:47:48.967795 7f12da464700 20
rgw::auth::StrategyRegistry::s3_main_strategy_t: trying
rgw::auth::s3::AWSAuthStrategy
2018-02-15 16:47:48.967797 7f12da464700 20
rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2018-02-15 16:47:48.967801 7f12da464700 20
rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2018-02-15 16:47:48.967802 7f12da464700 20
rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2018-02-15 16:47:48.967823 7f12da464700 10 get_canon_resource():
dest=/admin/realm/period
2018-02-15 16:47:48.967826 7f12da464700 10 string_to_sign:
POST
Thu Feb 15 11:17:49 2018
/admin/realm/period
2018-02-15 16:47:48.967890 7f12da464700 15 string_to_sign=POST
Thu Feb 15 11:17:49 2018
/admin/realm/period
2018-02-15 16:47:48.967900 7f12da464700 15 server
signature=+k8NKAHCGCWduWIHASM30/bms7s=
2018-02-15 16:47:48.967901 7f12da464700 15 client
signature=+k8NKAHCGCWduWIHASM30/bms7s=
2018-02-15 16:47:48.967903 7f12da464700 15 compare=0
2018-02-15 16:47:48.967909 7f12da464700 20 rgw::auth::s3::LocalEngine
granted access
2018-02-15 16:47:48.967910 7f12da464700 20
rgw::auth::s3::AWSAuthStrategy granted access
2018-02-15 16:47:48.967917 7f12da464700 2 req 11:0.000217::POST
/admin/realm/period:post_period:normalizing buckets and tenants
2018-02-15 16:47:48.967920 7f12da464700 2 req 11:0.000220::POST
/admin/realm/period:post_period:init permissions
2018-02-15 16:47:48.967946 7f12da464700 2 req 11:0.000247::POST
/admin/realm/period:post_period:recalculating target
2018-02-15 16:47:48.967949 7f12da464700 2 req 11:0.000249::POST
/admin/realm/period:post_period:reading permissions
2018-02-15 16:47:48.967952 7f12da464700 2 req 11:0.000252::POST
/admin/realm/period:post_period:init op
2018-02-15 16:47:48.967954 7f12da464700 2 req 11:0.000254::POST
/admin/realm/period:post_period:verifying op mask
2018-02-15 16:47:48.967956 7f12da464700 20 required_mask= 0 user.op_mask=7
2018-02-15 16:47:48.967960 7f12da464700 2 req 11:0.000258::POST
/admin/realm/period:post_period:verifying op permissions
2018-02-15 16:47:48.967961 7f12da464700 2 req 11:0.000262::POST
/admin/realm/period:post_period:verifying op params
2018-02-15 16:47:48.967963 7f12da464700 2 req 11:0.000263::POST
/admin/realm/period:post_period:pre-executing
2018-02-15 16:47:48.967965 7f12da464700 2 req 11:0.000265::POST
/admin/realm/period:post_period:executing
2018-02-15 16:47:49.970893 7f12da464700 20 get_system_obj_state:
rctx=0x7f12da45bd00
obj=.rgw.root:realms.281ba7e8-3bd1-47de-981a-c94914bdf54f
state=0x56363b105ee0 s->prefetch_data=0
2018-02-15 16:47:49.970921 7f12da464700 10 cache get:
name=.rgw.root++realms.281ba7e8-3bd1-47de-981a-c94914bdf54f : hit
(requested=0x6, cached=0x7)
2018-02-15 16:47:49.970932 7f12da464700 20 get_system_obj_state:
s->obj_tag was set empty
2018-02-15 16:47:49.970934 7f12da464700 10 cache get:
name=.rgw.root++realms.281ba7e8-3bd1-47de-981a-c94914bdf54f : hit
(requested=0x1, cached=0x7)
2018-02-15 16:47:49.970954 7f12da464700 20 get_system_obj_state:
rctx=0x7f12da45baf0
obj=.rgw.root:realms.281ba7e8-3bd1-47de-981a-c94914bdf54f
state=0x56363b105ee0 s->prefetch_data=0
2018-02-15 16:47:49.970958 7f12da464700 10 cache get:
name=.rgw.root++realms.281ba7e8-3bd1-47de-981a-c94914bdf54f : hit
(requested=0x6, cached=0x7)
2018-02-15 16:47:49.970960 7f12da464700 20 get_system_obj_state:
s->obj_tag was set empty
2018-02-15 16:47:49.970962 7f12da464700 10 cache get:
name=.rgw.root++realms.281ba7e8-3bd1-47de-981a-c94914bdf54f : hit
(requested=0x1, cached=0x7)
2018-02-15 16:47:49.970970 7f12da464700 20 get_system_obj_state:
rctx=0x7f12da45bc60
obj=.rgw.root:periods.6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e.latest_epoch
state=0x56363b105ee0 s->prefetch_data=0
2018-02-15 16:47:49.970975 7f12da464700 10 cache get:
name=.rgw.root++periods.6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e.latest_epoch
: hit (requested=0x6, cached=0x7)
2018-02-15 16:47:49.970978 7f12da464700 20 get_system_obj_state:
s->obj_tag was set empty
2018-02-15 16:47:49.970980 7f12da464700 10 cache get:
name=.rgw.root++periods.6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e.latest_epoch
: hit (requested=0x1, cached=0x7)
2018-02-15 16:47:49.970991 7f12da464700 20 get_system_obj_state:
rctx=0x7f12da45bc80
obj=.rgw.root:periods.6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e.6
state=0x56363b105ee0 s->prefetch_data=0
2018-02-15 16:47:49.970994 7f12da464700 10 cache get:
name=.rgw.root++periods.6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e.6 : hit
(requested=0x6, cached=0x7)
2018-02-15 16:47:49.970996 7f12da464700 20 get_system_obj_state:
s->obj_tag was set empty
2018-02-15 16:47:49.970998 7f12da464700 10 cache get:
name=.rgw.root++periods.6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e.6 : hit
(requested=0x1, cached=0x7)
2018-02-15 16:47:49.971025 7f12da464700 10 master zone rejecting period
id=6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e epoch=6
2018-02-15 16:47:49.971033 7f12da464700 2 req 11:1.003331::POST
/admin/realm/period:post_period:completing
2018-02-15 16:47:49.971146 7f12da464700 2 req 11:1.003445::POST
/admin/realm/period:post_period:op status=0
2018-02-15 16:47:49.971154 7f12da464700 2 req 11:1.003454::POST
/admin/realm/period:post_period:http status=400
2018-02-15 16:47:49.971163 7f12da464700 1 ====== req done
req=0x7f12da45e1f0 op status=0 http_status=400 ======
2018-02-15 16:47:49.971181 7f12da464700 20 *process_request() returned -22*
2018-02-15 16:47:49.971246 7f12da464700 1 civetweb: 0x56363b42b000:
192.168.xx.xx - - [15/Feb/2018:16:46:15 +0530] "POST
/admin/realm/period?period=6a7cc9b5-c4b1-43cc-ac7e-ac3a63c8cb4e&epoch=6&rgwx-zonegroup=0d9efa23-09d6-4adf-a486-0858f3261d7b
HTTP/1.1" 1 0 - -
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
