Re: filesystem (LocalBlobStore) failing intermittently on initial "upload" in v2.2.1
Hi John, the code in LocalBlobStore.putBlob looks like this:
String eTag = storageStrategy.putBlob(containerName, blob);
setBlobAccess(containerName, blobKey, options.getBlobAccess());
I suspect that the object is being removed by another client in between
these calls. Instead FilesystemStorageStrategyImp.putBlob should allow
an options parameter to set the blobAccess on the temporary file which
then gets renamed to the real file name. Would you like to submit a PR
for this?
On Thu, Jan 07, 2021 at 10:09:29PM -, John Calcote wrote:
> We use the filesystem (LocalBlobStore) implementation for integration testing
> in our code.
>
> I'm seeing the following behavior in version 2.2.1 which I recently upgraded
> to:
>
> ```
> 2021-01-07 20:03:43.503,7964896021968234 {} DEBUG
> c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
> (data) chunk [0-1048575] - uploading
> 2021-01-07 20:03:43.575,7964896093429743 {} DEBUG
> o.j.f.s.i.FilesystemStorageStrategyImpl [cpu-normalpri-1] Creating container
> test-container
> 2021-01-07 20:03:43.575,7964896093478721 {} DEBUG
> o.j.f.s.i.FilesystemStorageStrategyImpl [cpu-normalpri-1] Creating directory
> //tmp/16100498223118761388361431415820/test-container
> 2021-01-07 20:03:43.575,7964896093881924 {} DEBUG
> c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
> (data) chunk [1048576-2097151] - checksummed
> checksum=D8ACDD71E006E090F4FBF32CAF0627A3
> 2021-01-07 20:03:43.575,7964896093909591 {} DEBUG
> c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
> (data) chunk [1048576-2097151] - uploading
> ..
> 2021-01-07 20:03:43.576,7964896094167857 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-0] Put blob with key [byHashV1/0001001020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.577,7964896095288960 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-0] Put blob with key [byHashV1/0001001020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.584,7964896102791478 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-1] Put blob with key [byHashV1/0001001020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.597,7964896115564074 {} DEBUG o.j.b.c.LocalBlobStore
> [cloud-normalpri-1] Put blob with key [byHashV1/0001001020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to
> container [test-container]
> ..
> 2021-01-07 20:03:43.605,7964896123355487 {} ERROR
> c.p.d.m.j.AbstractUploadCallable [cloud-normalpri-0] [task 1] Upload failed:
> chunk [1048576-2097151] - uploading
> java.lang.RuntimeException: java.nio.file.NoSuchFileException:
> /tmp/16100498223118761388361431415820/test-container/byHashV1/0001001020
> B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3
> at com.google.common.base.Throwables.propagate(Throwables.java:240)
> ~[guava-21.0.jar:?]
> at
> org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.setBlobAccess(FilesystemStorageStrategyImpl.java:694)
> ~[filesystem-2.2.1.jar:2.2.1]
> at
> org.jclouds.blobstore.config.LocalBlobStore.setBlobAccess(LocalBlobStore.java:409)
> ~[jclouds-blobstore-2.2.1.jar:2.2.1]
> at
> org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:794)
> ~[jclouds-blobstore-2.2.1.jar:2.2.1]
> at
> org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:533)
> ~[jclouds-blobstore-2.2.1.jar:2.2.1]
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> ~[guice-3.0.jar:?]
> at com.sun.proxy.$Proxy66.putBlob(Unknown Source) ~[?:?]
> at
> com.hammerspace.csp.rest.client.jclouds.JCloudsCspRestClient$1UploadRunnable.run(JCloudsCspRestClient.java:454)
> [csp-rest-client-4.5.1.jar:?]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_181]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> Caused by: java.nio.file.NoSuchFileException:
>
Re: filesystem (LocalBlobStore) failing intermittently on initial "upload" in v2.2.1
One more comment - it's an intermittent failure. Sometimes it works, sometimes not.
filesystem (LocalBlobStore) failing intermittently on initial "upload" in v2.2.1
We use the filesystem (LocalBlobStore) implementation for integration testing
in our code.
I'm seeing the following behavior in version 2.2.1 which I recently upgraded to:
```
2021-01-07 20:03:43.503,7964896021968234 {} DEBUG
c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
(data) chunk [0-1048575] - uploading
2021-01-07 20:03:43.575,7964896093429743 {} DEBUG
o.j.f.s.i.FilesystemStorageStrategyImpl [cpu-normalpri-1] Creating container
test-container
2021-01-07 20:03:43.575,7964896093478721 {} DEBUG
o.j.f.s.i.FilesystemStorageStrategyImpl [cpu-normalpri-1] Creating directory
//tmp/16100498223118761388361431415820/test-container
2021-01-07 20:03:43.575,7964896093881924 {} DEBUG
c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
(data) chunk [1048576-2097151] - checksummed
checksum=D8ACDD71E006E090F4FBF32CAF0627A3
2021-01-07 20:03:43.575,7964896093909591 {} DEBUG
c.p.d.m.j.AbstractUploadCallable [cpu-normalpri-1] [task 1] State change:
(data) chunk [1048576-2097151] - uploading
..
2021-01-07 20:03:43.576,7964896094167857 {} DEBUG o.j.b.c.LocalBlobStore
[cloud-normalpri-0] Put blob with key [byHashV1/0001001020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to container
[test-container]
..
2021-01-07 20:03:43.577,7964896095288960 {} DEBUG o.j.b.c.LocalBlobStore
[cloud-normalpri-0] Put blob with key [byHashV1/0001001020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to container
[test-container]
..
2021-01-07 20:03:43.584,7964896102791478 {} DEBUG o.j.b.c.LocalBlobStore
[cloud-normalpri-1] Put blob with key [byHashV1/0001001020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to container
[test-container]
..
2021-01-07 20:03:43.597,7964896115564074 {} DEBUG o.j.b.c.LocalBlobStore
[cloud-normalpri-1] Put blob with key [byHashV1/0001001020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3 ] to container
[test-container]
..
2021-01-07 20:03:43.605,7964896123355487 {} ERROR
c.p.d.m.j.AbstractUploadCallable [cloud-normalpri-0] [task 1] Upload failed:
chunk [1048576-2097151] - uploading
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/tmp/16100498223118761388361431415820/test-container/byHashV1/0001001020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3
at com.google.common.base.Throwables.propagate(Throwables.java:240)
~[guava-21.0.jar:?]
at
org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.setBlobAccess(FilesystemStorageStrategyImpl.java:694)
~[filesystem-2.2.1.jar:2.2.1]
at
org.jclouds.blobstore.config.LocalBlobStore.setBlobAccess(LocalBlobStore.java:409)
~[jclouds-blobstore-2.2.1.jar:2.2.1]
at
org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:794)
~[jclouds-blobstore-2.2.1.jar:2.2.1]
at
org.jclouds.blobstore.config.LocalBlobStore.putBlob(LocalBlobStore.java:533)
~[jclouds-blobstore-2.2.1.jar:2.2.1]
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
~[guice-3.0.jar:?]
at com.sun.proxy.$Proxy66.putBlob(Unknown Source) ~[?:?]
at
com.hammerspace.csp.rest.client.jclouds.JCloudsCspRestClient$1UploadRunnable.run(JCloudsCspRestClient.java:454)
[csp-rest-client-4.5.1.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.nio.file.NoSuchFileException:
/tmp/16100498223118761388361431415820/test-container/byHashV1/0001001020
B3A2D81C390E0531DBCF0DEC082C4CA96D26F26AA9A26A26E80B5F00FA9F48E3
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
~[?:1.8.0_181]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[?:1.8.0_181]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
~[?:1.8.0_181]
at
sun.nio.fs.UnixFileAttributeViews$Posix.readAttributes(UnixFileAttributeViews.java:218)
~[?:1.8.0_181]
at
sun.nio.fs.UnixFileAttributeViews$Posix.readAttributes(UnixFileAttributeViews.java:131)
~[?:1.8.0_181]
at
sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
~[?:1.8.0_181]
at
