[
https://issues.apache.org/jira/browse/JCLOUDS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Dimitrov updated JCLOUDS-1623:
-----------------------------------
Description:
Hello,
We are observing an issue with JClouds while transferring a file with an input
stream only.
We are opening an input stream to a remote file and we are passing the stream
to the JClouds library, but after some time it fails with the following error:
{noformat}
Exception in thread "main" java.lang.RuntimeException: java.io.EOFException:
reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
at com.google.common.base.Throwables.propagate(Throwables.java:241)
at
org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
at
org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
at
org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
at org.example.Main.main(Main.java:50)
Caused by: java.io.EOFException: reached end of stream after skipping 14933328
bytes; 67108864 bytes expected
at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
at
org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
... 5 more
{noformat}
Note: The issue occurs only for larger files (more than 32 mb).
Java version: sapmachine-jdk-11.0.15.0.1.jdk
IAAS: AWS-s3
If we pass a BufferedInputStream the code works, so I guess that there is some
issue with the processing of the stream. The stream returned by the Java Http
Client is HttpResponseInputStream.
Code:
{code:title=Main.java|borderStyle=solid}
package org.example;
import com.google.common.collect.ImmutableSet;
import com.google.inject.Module;
import org.jclouds.ContextBuilder;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.PutOptions;
import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
import java.io.BufferedInputStream;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.UUID;
public class Main {
public static void main(String[] args) throws Exception {
Iterable<Module> modules = ImmutableSet.<Module>of(
new SLF4JLoggingModule());
ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
.credentials("<IDENTITY>", "<CREDENTIAL>")
.modules(modules);
BlobStoreContext blobStoreContext =
contextBuilder.buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
String decodedUrl = "<URL>";
HttpClient client = buildHttpClient(decodedUrl);
HttpResponse<InputStream> response =
callRemoteEndpointWithRetry(client, decodedUrl);
long fileSize = response.headers()
.firstValueAsLong("Content-Length")
.orElseThrow(() -> new IllegalArgumentException("No
Content-Length"));
System.out.println("Bytes: " + fileSize);
InputStream inputStream = response.body();
String container = "<CONTAINER>";
Blob blob = blobStore.blobBuilder("TEST.zip")
// .payload(new BufferedInputStream(inputStream, 8 * 1024))
.payload(inputStream)
.contentDisposition(UUID.randomUUID().toString())
.contentType("application/octet-stream")
.contentLength(fileSize)
.build();
blobStore.putBlob(container, blob, PutOptions.Builder.multipart());
}
private static HttpClient buildHttpClient(String decodedUrl) {
return HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.connectTimeout(Duration.ofMinutes(60))
.followRedirects(HttpClient.Redirect.NORMAL)
.authenticator(buildPasswordAuthenticator(decodedUrl))
.build();
}
private static Authenticator buildPasswordAuthenticator(String decodedUrl) {
return new Authenticator() {
@Override
protected PasswordAuthentication getPasswordAuthentication() {
var uri = URI.create(decodedUrl);
var userInfo = uri.getUserInfo();
if (userInfo != null) {
var separatorIndex = userInfo.indexOf(':');
var username = userInfo.substring(0, separatorIndex);
var password = userInfo.substring(separatorIndex + 1);
return new PasswordAuthentication(username,
password.toCharArray());
}
return super.getPasswordAuthentication();
}
};
}
private static HttpResponse<InputStream>
callRemoteEndpointWithRetry(HttpClient client, String decodedUrl) throws
Exception {
var response = client.send(buildFetchFileRequest(decodedUrl),
HttpResponse.BodyHandlers.ofInputStream());
if (response.statusCode() / 100 != 2) {
throw new IllegalStateException("INVALID CODE " +
response.statusCode());
}
return response;
}
private static HttpRequest buildFetchFileRequest(String decodedUrl) {
var builder = HttpRequest.newBuilder()
.GET()
.expectContinue(true)
.headers("Content-Type", "multipart/form-data")
.timeout(Duration.ofMinutes(30));
var uri = URI.create(decodedUrl);
var userInfo = uri.getUserInfo();
if (userInfo != null) {
builder.uri(URI.create(decodedUrl.replace(userInfo + "@", "")));
} else {
builder.uri(uri);
}
return builder.build();
}
}
{code}
Could you please take a look? Any advice would be helpful. The one thing I see
differences between the HttpResponseInputStream and the BufferedInputStream is
that HttpResponseInputStream::available returns 0. But I'm not sure if that's
the issue because the skip of the stream is done by just reading it.
Note: Removed the binary content from jclouds-wire.log
Thanks!
was:
Hello,
We are observing an issue with JClouds while transferring a file with an input
stream only.
We are opening an input stream to a remote file and we are passing the stream
to the JClouds library, but after some time it fails with the following error:
{noformat}
Exception in thread "main" java.lang.RuntimeException: java.io.EOFException:
reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
at com.google.common.base.Throwables.propagate(Throwables.java:241)
at
org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
at
org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
at
org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
at org.example.Main.main(Main.java:50)
Caused by: java.io.EOFException: reached end of stream after skipping 14933328
bytes; 67108864 bytes expected
at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
at
org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
... 5 more
{noformat}
Note: The issue occurs only for larger files (more than 32 mb).
Java version: sapmachine-jdk-11.0.15.0.1.jdk
IAAS: AWS-s3
If we pass a BufferedInputStream the code works, so I guess that there is some
issue with the processing of the stream. The stream returned by the Java Http
Client is HttpResponseInputStream.
Code:
{code:title=Main.java|borderStyle=solid}
package org.example;
import com.google.common.collect.ImmutableSet;
import com.google.inject.Module;
import org.jclouds.ContextBuilder;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.PutOptions;
import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
import java.io.BufferedInputStream;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.UUID;
public class Main {
public static void main(String[] args) throws Exception {
Iterable<Module> modules = ImmutableSet.<Module>of(
new SLF4JLoggingModule());
ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
.credentials("<IDENTITY>", "<CREDENTIAL>")
.modules(modules);
BlobStoreContext blobStoreContext =
contextBuilder.buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
String decodedUrl = "<URL>";
HttpClient client = buildHttpClient(decodedUrl);
HttpResponse<InputStream> response =
callRemoteEndpointWithRetry(client, decodedUrl);
long fileSize = response.headers()
.firstValueAsLong("Content-Length")
.orElseThrow(() -> new IllegalArgumentException("No
Content-Length"));
System.out.println("Bytes: " + fileSize);
InputStream inputStream = response.body();
String container = "<CONTAINER>";
Blob blob = blobStore.blobBuilder("TEST.zip")
// .payload(new BufferedInputStream(inputStream, 8 * 1024))
.payload(inputStream)
.contentDisposition(UUID.randomUUID().toString())
.contentType("application/octet-stream")
.contentLength(fileSize)
.build();
blobStore.putBlob(container, blob, PutOptions.Builder.multipart());
}
private static HttpClient buildHttpClient(String decodedUrl) {
return HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.connectTimeout(Duration.ofMinutes(60))
.followRedirects(HttpClient.Redirect.NORMAL)
.authenticator(buildPasswordAuthenticator(decodedUrl))
.build();
}
private static Authenticator buildPasswordAuthenticator(String decodedUrl) {
return new Authenticator() {
@Override
protected PasswordAuthentication getPasswordAuthentication() {
var uri = URI.create(decodedUrl);
var userInfo = uri.getUserInfo();
if (userInfo != null) {
var separatorIndex = userInfo.indexOf(':');
var username = userInfo.substring(0, separatorIndex);
var password = userInfo.substring(separatorIndex + 1);
return new PasswordAuthentication(username,
password.toCharArray());
}
return super.getPasswordAuthentication();
}
};
}
private static HttpResponse<InputStream>
callRemoteEndpointWithRetry(HttpClient client, String decodedUrl) throws
Exception {
var response = client.send(buildFetchFileRequest(decodedUrl),
HttpResponse.BodyHandlers.ofInputStream());
if (response.statusCode() / 100 != 2) {
throw new IllegalStateException("INVALID CODE " +
response.statusCode());
}
return response;
}
private static HttpRequest buildFetchFileRequest(String decodedUrl) {
var builder = HttpRequest.newBuilder()
.GET()
.expectContinue(true)
.headers("Content-Type", "multipart/form-data")
.timeout(Duration.ofMinutes(30));
var uri = URI.create(decodedUrl);
var userInfo = uri.getUserInfo();
if (userInfo != null) {
builder.uri(URI.create(decodedUrl.replace(userInfo + "@", "")));
} else {
builder.uri(uri);
}
return builder.build();
}
}
{code}
Could you please take a look? Any advice would be helpful. The one thing I see
differences between the HttpResponseInputStream and the BufferedInputStream is
that HttpResponseInputStream::isAvailable returns 0. But I'm not sure if that's
the issue because the skip of the stream is done by just reading it.
Note: Removed the binary content from jclouds-wire.log
Thanks!
> Caused by: java.io.EOFException: reached end of stream after skipping
> 14933328 bytes; 67108864 bytes expected
> -------------------------------------------------------------------------------------------------------------
>
> Key: JCLOUDS-1623
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1623
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
> Affects Versions: 2.5.0
> Reporter: Ivan Dimitrov
> Assignee: Andrew Gaul
> Priority: Major
> Labels: aws-s3
> Attachments: jclouds-wire.log, jclouds.log
>
>
> Hello,
> We are observing an issue with JClouds while transferring a file with an
> input stream only.
> We are opening an input stream to a remote file and we are passing the stream
> to the JClouds library, but after some time it fails with the following error:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: java.io.EOFException:
> reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
> at com.google.common.base.Throwables.propagate(Throwables.java:241)
> at
> org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
> at
> org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
> at
> org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
> at org.example.Main.main(Main.java:50)
> Caused by: java.io.EOFException: reached end of stream after skipping
> 14933328 bytes; 67108864 bytes expected
> at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
> at
> org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
> ... 5 more
> {noformat}
> Note: The issue occurs only for larger files (more than 32 mb).
> Java version: sapmachine-jdk-11.0.15.0.1.jdk
> IAAS: AWS-s3
> If we pass a BufferedInputStream the code works, so I guess that there is
> some issue with the processing of the stream. The stream returned by the Java
> Http Client is HttpResponseInputStream.
> Code:
> {code:title=Main.java|borderStyle=solid}
> package org.example;
> import com.google.common.collect.ImmutableSet;
> import com.google.inject.Module;
> import org.jclouds.ContextBuilder;
> import org.jclouds.blobstore.BlobStore;
> import org.jclouds.blobstore.BlobStoreContext;
> import org.jclouds.blobstore.domain.Blob;
> import org.jclouds.blobstore.options.PutOptions;
> import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
> import java.io.BufferedInputStream;
> import java.io.InputStream;
> import java.net.Authenticator;
> import java.net.PasswordAuthentication;
> import java.net.URI;
> import java.net.http.HttpClient;
> import java.net.http.HttpRequest;
> import java.net.http.HttpResponse;
> import java.time.Duration;
> import java.util.UUID;
> public class Main {
> public static void main(String[] args) throws Exception {
> Iterable<Module> modules = ImmutableSet.<Module>of(
> new SLF4JLoggingModule());
> ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
> .credentials("<IDENTITY>", "<CREDENTIAL>")
> .modules(modules);
> BlobStoreContext blobStoreContext =
> contextBuilder.buildView(BlobStoreContext.class);
> BlobStore blobStore = blobStoreContext.getBlobStore();
> String decodedUrl = "<URL>";
> HttpClient client = buildHttpClient(decodedUrl);
> HttpResponse<InputStream> response =
> callRemoteEndpointWithRetry(client, decodedUrl);
> long fileSize = response.headers()
> .firstValueAsLong("Content-Length")
> .orElseThrow(() -> new IllegalArgumentException("No
> Content-Length"));
> System.out.println("Bytes: " + fileSize);
> InputStream inputStream = response.body();
> String container = "<CONTAINER>";
> Blob blob = blobStore.blobBuilder("TEST.zip")
> // .payload(new BufferedInputStream(inputStream, 8 * 1024))
> .payload(inputStream)
> .contentDisposition(UUID.randomUUID().toString())
> .contentType("application/octet-stream")
> .contentLength(fileSize)
> .build();
> blobStore.putBlob(container, blob,
> PutOptions.Builder.multipart());
> }
> private static HttpClient buildHttpClient(String decodedUrl) {
> return HttpClient.newBuilder()
> .version(HttpClient.Version.HTTP_1_1)
> .connectTimeout(Duration.ofMinutes(60))
> .followRedirects(HttpClient.Redirect.NORMAL)
> .authenticator(buildPasswordAuthenticator(decodedUrl))
> .build();
> }
> private static Authenticator buildPasswordAuthenticator(String
> decodedUrl) {
> return new Authenticator() {
> @Override
> protected PasswordAuthentication getPasswordAuthentication() {
> var uri = URI.create(decodedUrl);
> var userInfo = uri.getUserInfo();
> if (userInfo != null) {
> var separatorIndex = userInfo.indexOf(':');
> var username = userInfo.substring(0, separatorIndex);
> var password = userInfo.substring(separatorIndex + 1);
> return new PasswordAuthentication(username,
> password.toCharArray());
> }
> return super.getPasswordAuthentication();
> }
> };
> }
> private static HttpResponse<InputStream>
> callRemoteEndpointWithRetry(HttpClient client, String decodedUrl) throws
> Exception {
> var response = client.send(buildFetchFileRequest(decodedUrl),
> HttpResponse.BodyHandlers.ofInputStream());
> if (response.statusCode() / 100 != 2) {
> throw new IllegalStateException("INVALID CODE " +
> response.statusCode());
> }
> return response;
> }
> private static HttpRequest buildFetchFileRequest(String decodedUrl) {
> var builder = HttpRequest.newBuilder()
> .GET()
> .expectContinue(true)
> .headers("Content-Type", "multipart/form-data")
> .timeout(Duration.ofMinutes(30));
> var uri = URI.create(decodedUrl);
> var userInfo = uri.getUserInfo();
> if (userInfo != null) {
> builder.uri(URI.create(decodedUrl.replace(userInfo + "@", "")));
> } else {
> builder.uri(uri);
> }
> return builder.build();
> }
> }
> {code}
> Could you please take a look? Any advice would be helpful. The one thing I
> see differences between the HttpResponseInputStream and the
> BufferedInputStream is that HttpResponseInputStream::available returns 0. But
> I'm not sure if that's the issue because the skip of the stream is done by
> just reading it.
> Note: Removed the binary content from jclouds-wire.log
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
