multi-part upload writing temp files

2020-05-21 Thread John Calcote 
Hi Andrew,

I've got code that's uploading using the jclouds multi-part abstraction. Here 
are the log trace messages for a portion of an upload of a 1G file:

2020-05-21 16:52:07.203 +,36570333402959 {} DEBUG o.j.r.i.InvokeHttpMethod 
[cloud-normalpri-4] >> invoking PutObject
2020-05-21 16:52:07.205 +,36570335895211 {} DEBUG 
o.j.h.i.JavaUrlHttpCommandExecutorService [cloud-normalpri-4] Sending request 
-527754056: POST 
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD09801FF0?uploads
 HTTP/1.1
2020-05-21 16:52:07.206 +,36570336221046 {} DEBUG j.headers 
[cloud-normalpri-4] >> POST 
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD09801FF0?uploads
 HTTP/1.1
2020-05-21 16:52:07.206 +,36570336410769 {} DEBUG j.headers 
[cloud-normalpri-4] >> x-amz-meta-packedsize: 536870896
2020-05-21 16:52:07.206 +,36570336571487 {} DEBUG j.headers 
[cloud-normalpri-4] >> x-amz-meta-checksum: 27B425F8F9450EF701D3161C7923064A
2020-05-21 16:52:07.206 +,36570336727425 {} DEBUG j.headers 
[cloud-normalpri-4] >> Content-Disposition: 
Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD09801FF0
2020-05-21 16:52:07.206 +,36570336906947 {} DEBUG j.headers 
[cloud-normalpri-4] >> Content-Type: application/octet-stream
2020-05-21 16:52:07.206 +,36570337050453 {} DEBUG j.headers 
[cloud-normalpri-4] >> Date: Thu, 21 May 2020 16:52:07 GMT
2020-05-21 16:52:07.206 +,36570337147190 {} DEBUG j.headers 
[cloud-normalpri-4] >> Authorization: AWS 
14FP9F8ISZPXYCVYVLB1:F6tyKFeT79IUeXNmSFAVzs0NPT8=
2020-05-21 16:52:07.250 +,36570380776500 {} DEBUG 
o.j.h.i.JavaUrlHttpCommandExecutorService [cloud-normalpri-4] Receiving 
response -527754056: HTTP/1.1 200 OK
2020-05-21 16:52:07.251 +,36570381572643 {} DEBUG j.headers 
[cloud-normalpri-4] << HTTP/1.1 200 OK
2020-05-21 16:52:07.251 +,36570382074919 {} DEBUG j.headers 
[cloud-normalpri-4] << Transfer-Encoding: chunked
2020-05-21 16:52:07.251 +,36570382210690 {} DEBUG j.headers 
[cloud-normalpri-4] << Accept-Ranges: bytes
2020-05-21 16:52:07.252 +,36570382315489 {} DEBUG j.headers 
[cloud-normalpri-4] << Server: Minio/RELEASE.2018-08-02T23-11-36Z (linux; amd64)
2020-05-21 16:52:07.252 +,36570382436254 {} DEBUG j.headers 
[cloud-normalpri-4] << X-Xss-Protection: 1; mode=block
2020-05-21 16:52:07.252 +,36570382531435 {} DEBUG j.headers 
[cloud-normalpri-4] << Content-Security-Policy: block-all-mixed-content
2020-05-21 16:52:07.252 +,36570382623669 {} DEBUG j.headers 
[cloud-normalpri-4] << Vary: Origin
2020-05-21 16:52:07.252 +,36570382716393 {} DEBUG j.headers 
[cloud-normalpri-4] << Date: Thu, 21 May 2020 16:52:07 GMT
2020-05-21 16:52:07.252 +,36570383135581 {} DEBUG j.headers 
[cloud-normalpri-4] << X-Amz-Request-Id: 16111944E9E38DFA
2020-05-21 16:52:07.253 +,36570383234336 {} DEBUG j.headers 
[cloud-normalpri-4] << Content-Type: application/xml
2020-05-21 16:52:07.264 +,36570394322320 {} DEBUG o.j.r.i.InvokeHttpMethod 
[user thread 3] >> invoking PutObject
2020-05-21 16:52:07.265 +,36570395342332 {} DEBUG 
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 3] Sending request 
425611548: PUT 
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD09801FF0?partNumber=1&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
 HTTP/1.1
2020-05-21 16:52:07.266 +,36570396738315 {} DEBUG o.j.r.i.InvokeHttpMethod 
[user thread 4] >> invoking PutObject
2020-05-21 16:52:07.267 +,36570397622955 {} DEBUG 
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 4] Sending request 
1704123656: PUT 
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD09801FF0?partNumber=2&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
 HTTP/1.1
2020-05-21 16:52:07.277 +,36570407929930 {} DEBUG o.j.r.i.InvokeHttpMethod 
[user thread 5] >> invoking PutObject
2020-05-21 16:52:07.279 +,36570409298611 {} DEBUG o.j.r.i.InvokeHttpMethod 
[user thread 6] >> invoking PutObject
2020-05-21 16:52:07.281 +,36570411898907 {} DEBUG 
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 6] Sending request 
340915254: PUT 
http://10.200.83.92:9000/minioee568a08-7ffb-41ed-be68-e8ab870d6d92-bucket/Hammerspace_v2/00022C/92/00022C921974CD850CCDA0E843D1094F20925DBA82CCA9DCBBCA27145DDD60CD09801FF0?partNumber=4&uploadId=f0ddb69b-62ab-4bec-9c0a-e3012e75cadf
 HTTP/1.1
2020-05-21 16:52:07.284 +,36570414474806 {} DEBUG 
o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 5] Sending request 
-678166897: P

Re: multi-part upload writing temp files

2020-05-21 Thread Andrew Gaul 
This is nothing to worry about.  jclouds creates temporary files when
HTTP payloads exceed 256 KB to avoid exhausting memory.  You can see the
source of this message at
core/src/main/java/org/jclouds/logging/internal/Wire.copy.  I believe
that this code is only exercised with debug logging; usually jclouds
streams payloads directly.

On Thu, May 21, 2020 at 04:56:17PM -, John Calcote wrote:
> Hi Andrew,
> 
> I've got code that's uploading using the jclouds multi-part abstraction. Here 
> are the log trace messages for a portion of an upload of a 1G file:
> 
> 2020-05-21 16:53:12.020 +,36635150704660 {} DEBUG o.j.h.i.HttpWire [user 
> thread 18] over limit 33554416/262144: wrote temp file
> 
> I'm concerned about the trailing messages - "over limit ... wrote temp file" 
> - what does this mean?
> 
> Thanks,
> John

-- 
Andrew Gaul
http://gaul.org/