On 1/19/16 4:00 PM, Yehuda Sadeh-Weinraub wrote:
On Fri, Jan 15, 2016 at 5:04 PM, seapasu...@uchicago.edu
<seapasu...@uchicago.edu> wrote:
I have looked all over and I do not see any explicit mention of
"NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959" in the logs nor do I
see a timestamp from November 4th although I do see log rotations dating
back to october 15th. I don't think it's possible it wasn't logged so I am
going through the bucket logs from the 'radosgw-admin log show --object'
side and I found the following::
4604932 {
4604933 "bucket": "noaa-nexrad-l2",
4604934 "time": "2015-11-04 21:29:27.346509Z",
4604935 "time_local": "2015-11-04 15:29:27.346509",
4604936 "remote_addr": "",
4604937 "object_owner": "b05f707271774dbd89674a0736c9406e",
4604938 "user": "b05f707271774dbd89674a0736c9406e",
4604939 "operation": "PUT",
I'd expect a multipart upload completion to be done with a POST, not a PUT.
Indeed it seems really weird.
4604940 "uri":
"\/noaa-nexrad-l2\/2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
4604941 "http_status": "200",
4604942 "error_code": "",
4604943 "bytes_sent": 19,
4604944 "bytes_received": 0,
4604945 "object_size": 0,
Do you see a zero object_size for other multipart uploads?
I think so. I still don't know how to tell for certain if a radosgw
object is a multipart object or not. I think all of the objects in
noaa-nexrad-l2 bucket are multipart::
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- {
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "bucket":
"noaa-nexrad-l2",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "time": "2015-10-16
19:49:30.579738Z",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "time_local":
"2015-10-16 14:49:30.579738",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "remote_addr": "",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "user":
"b05f707271774dbd89674a0736c9406e",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out: "operation": "POST",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "uri":
"\/noaa-nexrad-l2\/2015\/01\/13\/KGRK\/NWS_NEXRAD_NXL2DP_KGRK_20150113040000_20150113045959.tar",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "http_status": "200",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "error_code": "",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "bytes_sent": 331,
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "bytes_received": 152,
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "object_size": 0,
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "total_time": 0,
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "user_agent":
"Boto\/2.38.0 Python\/2.7.7 Linux\/2.6.32-573.7.1.el6.x86_64",
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- "referrer": ""
./2015-10-16-14-default.384153.1-noaa-nexrad-l2.out- }
The objects above
(NWS_NEXRAD_NXL2DP_KGRK_20150113040000_20150113045959.tar) pulls down
without an issue though. Below is a paste for object
"NWS_NEXRAD_NXL2DP_KVBX_20150225160000_20150225165959.tar" which 404's::
http://pastebin.com/Jtw8z7G4
I see two posts one recorded a minute before for this object both with 0
size though. Does this help at all?
Yehuda
4604946 "total_time": 1426404000000,
4604947 "user_agent": "Boto\/2.38.0 Python\/2.7.7
Linux\/2.6.32-573.7.1.el6.x86_64",
4604948 "referrer": ""
4604949 }
Does this help at all. The total time seems exceptionally high. Would it be
possible that there is a timeout issue where the put request started a
multipart upload with the correct header and then timed out but the radosgw
took the data anyway?
I am surprised the radosgw returned a 200 let alone placed the key in the
bucket listing.
That said here is another object (different object) that 404s:
1650873 {
1650874 "bucket": "noaa-nexrad-l2",
1650875 "time": "2015-11-05 04:50:42.606838Z",
1650876 "time_local": "2015-11-04 22:50:42.606838",
1650877 "remote_addr": "",
1650878 "object_owner": "b05f707271774dbd89674a0736c9406e",
1650879 "user": "b05f707271774dbd89674a0736c9406e",
1650880 "operation": "PUT",
1650881 "uri":
"\/noaa-nexrad-l2\/2015\/02\/25\/KVBX\/NWS_NEXRAD_NXL2DP_KVBX_20150225160000_20150225165959.tar",
1650882 "http_status": "200",
1650883 "error_code": "",
1650884 "bytes_sent": 19,
1650885 "bytes_received": 0,
1650886 "object_size": 0,
1650887 "total_time": 0,
1650888 "user_agent": "Boto\/2.38.0 Python\/2.7.7
Linux\/2.6.32-573.7.1.el6.x86_64",
1650889 "referrer": ""
1650890 }
And this one fails with a 404 as well. Does this help at all? Here is a
successful object (different object) log entry as well just in case::
17462367 {
17462368 "bucket": "noaa-nexrad-l2",
17462369 "time": "2015-11-04 21:16:44.148603Z",
17462370 "time_local": "2015-11-04 15:16:44.148603",
17462371 "remote_addr": "",
17462372 "object_owner": "b05f707271774dbd89674a0736c9406e",
17462373 "user": "b05f707271774dbd89674a0736c9406e",
17462374 "operation": "PUT",
17462375 "uri":
"\/noaa-nexrad-l2\/2015\/01\/01\/KAKQ\/NWS_NEXRAD_NXL2DP_KAKQ_20150101080000_20150101085959.tar",
17462376 "http_status": "200",
17462377 "error_code": "",
17462378 "bytes_sent": 19,
17462379 "bytes_received": 0,
17462380 "object_size": 0,
17462381 "total_time": 0,
17462382 "user_agent": "Boto\/2.38.0 Python\/2.7.7
Linux\/2.6.32-573.7.1.el6.x86_64",
17462383 "referrer": ""
17462384 }
So I am guessing these are not pertinent as they look nearly identical.
Unfortunately I do not have any client.radosgw.logs to show for the failed
files for some reason. Is there anything else I can do to troubleshoot this
issue. In the end the radosgw should have never list these files as they
never completed successfully, right?
On 1/15/16 4:36 PM, Yehuda Sadeh-Weinraub wrote:
The head object of a multipart object has 0 size, so it's expected.
What's missing is the tail of the object. I don't assume you have any
logs from when the object was uploaded?
Yehuda
On Fri, Jan 15, 2016 at 2:12 PM, seapasu...@uchicago.edu
<seapasu...@uchicago.edu> wrote:
Sorry for the confusion::
When I grepped for the prefix of the missing object::
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD"
I am not able to find any chunks of the object::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
lacadmin@kh28-10:~$
The only piece of the object that I can seem to find is the original one
I
posted::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep
'NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959'
default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar
And when we stat this object is is 0 bytes as shown earlier::
lacadmin@kh28-10:~$ rados -p .rgw.buckets stat
'default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar'
.rgw.buckets/default.384153.1_2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar
mtime 2015-11-04 15:29:30.000000, size 0
Sorry again for the confusion.
On 1/15/16 3:58 PM, Yehuda Sadeh-Weinraub wrote:
Ah, I see. Misread that and the object names were very similar. No,
don't copy it. You can try to grep for the specific object name and
see if there are pieces of it lying around under a different upload
id.
Yehuda
On Fri, Jan 15, 2016 at 1:44 PM, seapasu...@uchicago.edu
<seapasu...@uchicago.edu> wrote:
Sorry I am a bit confused. The successful list that I provided is from
a
different object of the same size to show that I could indeed get a
list.
Are you saying to copy the working object to the missing object? Sorry
for
the confusion.
On 1/15/16 3:20 PM, Yehuda Sadeh-Weinraub wrote:
That's interesting, and might point at the underlying issue that
caused it. Could be a racing upload that somehow ended up with the
wrong object head. The 'multipart' object should be 4M in size, and
the 'shadow' one should have the remainder of the data. You can run
'rados stat -p .rgw.buckets <oid>' to validate that. If that's the
case, you can copy these to the expected object names:
$ src_uploadid=wksHvto9gRgHUJbhm_TZPXJTZUPXLT2
$ dest_uploadid=pcu5Hz6foFXjlSxBat22D8YMcHlQOBD
$ rados -p .rgw.buckets cp
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${src_uploadid}.1
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${dest_uploadid}.1
$ rados -p .rgw.buckets cp
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${src_upload_id}.1_1
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~${dest_upload_id}.1_1
Yehuda
On Fri, Jan 15, 2016 at 1:02 PM, seapasu...@uchicago.edu
<seapasu...@uchicago.edu> wrote:
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6'
lacadmin@kh28-10:~$
Nothing was found. That said when I run the command with another
prefix
snippet::
lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'wksHvto'
default.384153.1__shadow_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1_1
default.384153.1__multipart_2015/01/01/KABR/NWS_NEXRAD_NXL2DP_KABR_20150101130000_20150101135959.tar.2~wksHvto9gRgHUJbhm_TZPXJTZUPXLT2.1
On 1/15/16 12:05 PM, Yehuda Sadeh-Weinraub wrote:
On Fri, Jan 15, 2016 at 9:36 AM, seapasu...@uchicago.edu
<seapasu...@uchicago.edu> wrote:
Hello Yehuda,
Here it is::
radosgw-admin object stat --bucket="noaa-nexrad-l2"
--object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar"
{
"name":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
"size": 7147520,
"policy": {
"acl": {
"acl_user_map": [
{
"user": "b05f707271774dbd89674a0736c9406e",
"acl": 15
}
],
"acl_group_map": [
{
"group": 1,
"acl": 1
}
],
"grant_map": [
{
"id": "",
"grant": {
"type": {
"type": 2
},
"id": "",
"email": "",
"permission": {
"flags": 1
},
"name": "",
"group": 1
}
},
{
"id": "b05f707271774dbd89674a0736c9406e",
"grant": {
"type": {
"type": 0
},
"id":
"b05f707271774dbd89674a0736c9406e",
"email": "",
"permission": {
"flags": 15
},
"name": "noaa-commons",
"group": 0
}
}
]
},
"owner": {
"id": "b05f707271774dbd89674a0736c9406e",
"display_name": "noaa-commons"
}
},
"etag": "b91b6f1650350965c5434c547b3c38ff-1\u0000",
"tag": "_cWrvEa914Gy1AeyzIhRlUdp1wJnek3E\u0000",
"manifest": {
"objs": [],
"obj_size": 7147520,
"explicit_objs": "false",
"head_obj": {
"bucket": {
"name": "noaa-nexrad-l2",
"pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "default.384153.1",
"bucket_id": "default.384153.1"
},
"key": "",
"ns": "",
"object":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar",
"instance": ""
},
"head_size": 0,
"max_head_size": 0,
"prefix":
"2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_20150101110000_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD",
Try running:
$ rados -p .rgw.buckets ls | grep pcu5Hz6
Yehuda
"tail_bucket": {
"name": "noaa-nexrad-l2",
"pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "default.384153.1",
"bucket_id": "default.384153.1"
},
"rules": [
{
"key": 0,
"val": {
"start_part_num": 1,
"start_ofs": 0,
"part_size": 0,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
]
},
"attrs": {}
}
On 1/15/16 11:17 AM, Yehuda Sadeh-Weinraub wrote:
radosgw-admin object stat --bucket=<bucket> --object=<object>'
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
