Hi Andrew,
I've got code that's uploading using the jclouds multi-part abstraction. Here
are the log trace messages for a portion of an upload of a 1G file:
2020-05-21 16:52:07.203 +0000,36570333402959 {} DEBUG o.j.r.i.InvokeHttpMethod
[cloud-normalpri-4] >> invoking PutObject
2020-05-21 16:52:07.205 +0000,36570335895211 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [cloud-normalpri-4] Sending request
-527754056: POST
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?uploads
HTTP/1.1
2020-05-21 16:52:07.206 +0000,36570336221046 {} DEBUG j.headers
[cloud-normalpri-4] >> POST
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?uploads
HTTP/1.1
2020-05-21 16:52:07.206 +0000,36570336410769 {} DEBUG j.headers
[cloud-normalpri-4] >> x-amz-meta-packedsize: 536870896
2020-05-21 16:52:07.206 +0000,36570336571487 {} DEBUG j.headers
[cloud-normalpri-4] >> x-amz-meta-checksum: 27B425F8F9450EF701D3161C7923064A
2020-05-21 16:52:07.206 +0000,36570336727425 {} DEBUG j.headers
[cloud-normalpri-4] >> Content-Disposition:
Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0
2020-05-21 16:52:07.206 +0000,36570336906947 {} DEBUG j.headers
[cloud-normalpri-4] >> Content-Type: application/octet-stream
2020-05-21 16:52:07.206 +0000,36570337050453 {} DEBUG j.headers
[cloud-normalpri-4] >> Date: Thu, 21 May 2020 16:52:07 GMT
2020-05-21 16:52:07.206 +0000,36570337147190 {} DEBUG j.headers
[cloud-normalpri-4] >> Authorization: AWS
14FP9F8ISZPXYCVYVLB1:F6tyKFeT79IUeXNmSFAVzs0NPT8=
2020-05-21 16:52:07.250 +0000,36570380776500 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [cloud-normalpri-4] Receiving
response -527754056: HTTP/1.1 200 OK
2020-05-21 16:52:07.251 +0000,36570381572643 {} DEBUG j.headers
[cloud-normalpri-4] << HTTP/1.1 200 OK
2020-05-21 16:52:07.251 +0000,36570382074919 {} DEBUG j.headers
[cloud-normalpri-4] << Transfer-Encoding: chunked
2020-05-21 16:52:07.251 +0000,36570382210690 {} DEBUG j.headers
[cloud-normalpri-4] << Accept-Ranges: bytes
2020-05-21 16:52:07.252 +0000,36570382315489 {} DEBUG j.headers
[cloud-normalpri-4] << Server: Minio/RELEASE.2018-08-02T23-11-36Z (linux; amd64)
2020-05-21 16:52:07.252 +0000,36570382436254 {} DEBUG j.headers
[cloud-normalpri-4] << X-Xss-Protection: 1; mode=block
2020-05-21 16:52:07.252 +0000,36570382531435 {} DEBUG j.headers
[cloud-normalpri-4] << Content-Security-Policy: block-all-mixed-content
2020-05-21 16:52:07.252 +0000,36570382623669 {} DEBUG j.headers
[cloud-normalpri-4] << Vary: Origin
2020-05-21 16:52:07.252 +0000,36570382716393 {} DEBUG j.headers
[cloud-normalpri-4] << Date: Thu, 21 May 2020 16:52:07 GMT
2020-05-21 16:52:07.252 +0000,36570383135581 {} DEBUG j.headers
[cloud-normalpri-4] << X-Amz-Request-Id: 16111944E9E38DFA
2020-05-21 16:52:07.253 +0000,36570383234336 {} DEBUG j.headers
[cloud-normalpri-4] << Content-Type: application/xml
2020-05-21 16:52:07.264 +0000,36570394322320 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 3] >> invoking PutObject
2020-05-21 16:52:07.265 +0000,36570395342332 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 3] Sending request
425611548: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=1&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.266 +0000,36570396738315 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 4] >> invoking PutObject
2020-05-21 16:52:07.267 +0000,36570397622955 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 4] Sending request
1704123656: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=2&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.277 +0000,36570407929930 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 5] >> invoking PutObject
2020-05-21 16:52:07.279 +0000,36570409298611 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 6] >> invoking PutObject
2020-05-21 16:52:07.281 +0000,36570411898907 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 6] Sending request
340915254: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=4&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.284 +0000,36570414474806 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 5] Sending request
-678166897: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=3&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.296 +0000,36570426258943 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 7] >> invoking PutObject
2020-05-21 16:52:07.297 +0000,36570427297262 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 7] Sending request
1947228191: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=5&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.300 +0000,36570431072679 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 8] >> invoking PutObject
2020-05-21 16:52:07.302 +0000,36570432756663 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 8] Sending request
1338175229: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=6&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.308 +0000,36570438251042 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 9] >> invoking PutObject
2020-05-21 16:52:07.309 +0000,36570439257615 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 9] Sending request
-170507915: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=7&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.326 +0000,36570456516784 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 10] >> invoking PutObject
2020-05-21 16:52:07.334 +0000,36570464872503 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 10] Sending request
1648226958: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=8&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.340 +0000,36570470751445 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 11] >> invoking PutObject
2020-05-21 16:52:07.344 +0000,36570474255437 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 11] Sending request
-318660758: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=9&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.356 +0000,36570486541555 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 12] >> invoking PutObject
2020-05-21 16:52:07.357 +0000,36570488032689 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 12] Sending request
-704157967: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=10&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.360 +0000,36570490561292 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 14] >> invoking PutObject
2020-05-21 16:52:07.360 +0000,36570491192156 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 13] >> invoking PutObject
2020-05-21 16:52:07.361 +0000,36570491949833 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 13] Sending request
-787144195: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=11&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.365 +0000,36570496147151 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 14] Sending request
245837237: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=12&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.377 +0000,36570508079079 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 15] >> invoking PutObject
2020-05-21 16:52:07.378 +0000,36570509031993 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 15] Sending request
-1250633896: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=13&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.386 +0000,36570516221740 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 16] >> invoking PutObject
2020-05-21 16:52:07.386 +0000,36570517146028 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 16] Sending request
-814695581: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=14&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.397 +0000,36570527331368 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 17] >> invoking PutObject
2020-05-21 16:52:07.398 +0000,36570528861611 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 17] Sending request
942770423: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=15&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:07.428 +0000,36570558288743 {} DEBUG o.j.r.i.InvokeHttpMethod
[user thread 18] >> invoking PutObject
2020-05-21 16:52:07.429 +0000,36570559287323 {} DEBUG
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 18] Sending request
1053643314: PUT
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD0980000000001FFFFFF0?partNumber=16&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
HTTP/1.1
2020-05-21 16:52:11.657 +0000,36574788135328 {} DEBUG o.j.h.i.HttpWire [user
thread 3] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:15.571 +0000,36578701711351 {} DEBUG o.j.h.i.HttpWire [user
thread 4] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:18.784 +0000,36581914617380 {} DEBUG o.j.h.i.HttpWire [user
thread 5] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:23.365 +0000,36586495218895 {} DEBUG o.j.h.i.HttpWire [user
thread 6] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:29.448 +0000,36592579053941 {} DEBUG o.j.h.i.HttpWire [user
thread 7] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:35.268 +0000,36598398683431 {} DEBUG o.j.h.i.HttpWire [user
thread 8] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:41.503 +0000,36604633555238 {} DEBUG o.j.h.i.HttpWire [user
thread 9] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:45.577 +0000,36608707438866 {} DEBUG o.j.h.i.HttpWire [user
thread 10] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:50.963 +0000,36614093381218 {} DEBUG o.j.h.i.HttpWire [user
thread 11] over limit 33554432/262144: wrote temp file
2020-05-21 16:52:55.288 +0000,36618418403439 {} DEBUG o.j.h.i.HttpWire [user
thread 12] over limit 33554432/262144: wrote temp file
2020-05-21 16:53:00.235 +0000,36623365314879 {} DEBUG o.j.h.i.HttpWire [user
thread 13] over limit 33554432/262144: wrote temp file
2020-05-21 16:53:02.495 +0000,36625625255182 {} DEBUG o.j.h.i.HttpWire [user
thread 14] over limit 33554432/262144: wrote temp file
2020-05-21 16:53:05.601 +0000,36628731966110 {} DEBUG o.j.h.i.HttpWire [user
thread 15] over limit 33554432/262144: wrote temp file
2020-05-21 16:53:08.236 +0000,36631366445393 {} DEBUG o.j.h.i.HttpWire [user
thread 16] over limit 33554432/262144: wrote temp file
2020-05-21 16:53:10.854 +0000,36633984956169 {} DEBUG o.j.h.i.HttpWire [user
thread 17] over limit 33554432/262144: wrote temp file
2020-05-21 16:53:12.020 +0000,36635150704660 {} DEBUG o.j.h.i.HttpWire [user
thread 18] over limit 33554416/262144: wrote temp file
I'm concerned about the trailing messages - "over limit ... wrote temp file" -
what does this mean?
Thanks,
John
