[jira] [Commented] (JCLOUDS-1366) OutOfMemory when InputStream referencing to big file is used as payload
[
https://issues.apache.org/jira/browse/JCLOUDS-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290587#comment-16290587
]
Andrew Gaul commented on JCLOUDS-1366:
--
As a workaround can you try wrapping the {{File}} via Guava
{{Files.asByteSource}}? This will also make the payload repeatable, allowing
retries.
> OutOfMemory when InputStream referencing to big file is used as payload
> ---
>
> Key: JCLOUDS-1366
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1366
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.3
> Environment: Linux and Windows
>Reporter: Deyan
>Priority: Critical
>
> If I use InputStream which source is large file (lets say 3GB) I am getting
> OOE. This is with default java VM options.
> Here is the code I am using to construct the blob:
> {code:java}
> File bigFile = new File(file);
> try (InputStream inputStream = new FileInputStream(f)) {
> Blob b = blobStore.blobBuilder(blobName)
> .payload(inputStream).contentLength(f.length())
> .contentDisposition(blobName)
> .contentType(
> MediaType.OCTET_STREAM)
> .userMetadata(ImmutableMap.of("a", "b", "test",
> "beta"))
> .build();
> blobStore.putBlob("test", b, multipart());
> }
> {code}
> Stacktrace:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at
> org.jclouds.io.internal.BasePayloadSlicer$InputStreamPayloadIterator.getNextPayload(BasePayloadSlicer.java:101)
> at
> org.jclouds.io.internal.BasePayloadSlicer$InputStreamPayloadIterator.next(BasePayloadSlicer.java:90)
> at
> org.jclouds.io.internal.BasePayloadSlicer$InputStreamPayloadIterator.next(BasePayloadSlicer.java:63)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:363)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
> at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:262)
> {code}
> If 'bigFile' is used as payload the bug is not reproducible.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jclouds/jclouds-labs] Refactor DimensionDataCloudControlResponseUtils waitFor* methods so they are injectable (#427)
@nacx as requested I have done this piece of refactoring that was requested in https://github.com/jclouds/jclouds-labs/pull/426 and https://github.com/jclouds/jclouds-labs/pull/400#discussion_r127640050. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds-labs/pull/427 -- Commit Summary -- * Refactor DimensionDataCloudControlResponseUtils waitFor* methods so that they are injectable. -- File Changes -- M dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/DimensionDataCloudControlApiMetadata.java (4) M dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/compute/functions/CleanupServer.java (23) A dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/config/DimensionDataCloudControlComputeServiceContextModule.java (254) D dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/predicates/NetworkDomainState.java (50) D dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/predicates/ServerState.java (58) D dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/predicates/ServerStatus.java (58) D dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/predicates/VMToolsRunningStatus.java (51) D dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/predicates/VlanState.java (52) M dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/utils/DimensionDataCloudControlResponseUtils.java (49) M dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/compute/functions/CleanupServerTest.java (27) D dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/config/DimensionDataCloudControlComputeServiceContextModule.java (45) M dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/features/NetworkApiLiveTest.java (16) M dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/features/ServerApiLiveTest.java (26) M dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/internal/BaseDimensionDataCloudControlApiLiveTest.java (53) -- Patch Links -- https://github.com/jclouds/jclouds-labs/pull/427.patch https://github.com/jclouds/jclouds-labs/pull/427.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/427
Re: [jclouds/jclouds-labs] Refactor DimensionDataCloudControlResponseUtils waitFor* methods so they are injectable (#427)
trevorflanagan commented on this pull request.
> +import org.jclouds.dimensiondata.cloudcontrol.domain.NetworkDomain;
+import org.jclouds.dimensiondata.cloudcontrol.domain.Server;
+import org.jclouds.dimensiondata.cloudcontrol.domain.State;
+import org.jclouds.dimensiondata.cloudcontrol.domain.Vlan;
+import org.jclouds.dimensiondata.cloudcontrol.domain.VmTools;
+import org.jclouds.dimensiondata.cloudcontrol.features.NetworkApi;
+import org.jclouds.dimensiondata.cloudcontrol.features.ServerApi;
+import org.jclouds.logging.Logger;
+
+import javax.annotation.Resource;
+import java.text.MessageFormat;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.jclouds.util.Predicates2.retry;
+
+public class DimensionDataCloudControlComputeServiceContextModule extends
AbstractModule {
This was in the test package for some reason. I have moved it across to
production code. I have also simplified the Module so that it extends
`com.google.inject.AbstractModule`. Originally it extended
`ComputeServiceAdapterContextModule` but this causes problems in our live tests since not all of our
functions are complete yet.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/427#pullrequestreview-83581092
Re: [jclouds/jclouds-labs] Implement GetOrCreateNetworkDomainThenCreateNodes Strategy (#426)
@trevorflanagan pushed 1 commit. 07e8d0d Implement GetOrCreateNetworkDomainThenCreateNodes Strategy. PR updates. -- You are receiving this because you are subscribed to this thread. View it on GitHub: https://github.com/jclouds/jclouds-labs/pull/426/files/c7c31e073a1be4995272877f77b332fd09ea5e38..07e8d0d3fa5d86d815180d7c471895dae8c6d120
[jira] [Commented] (JCLOUDS-1302) Aliyun OSS blobstore
[ https://issues.apache.org/jira/browse/JCLOUDS-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291972#comment-16291972 ] zreal commented on JCLOUDS-1302: @Andrea Turli ,Hi Andrea, we use the old version of jclouds 1.9.1 in our products, so we can not be ensure that if the code can take effect in the 2.x version. > Aliyun OSS blobstore > > > Key: JCLOUDS-1302 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1302 > Project: jclouds > Issue Type: New Feature > Components: jclouds-blobstore >Affects Versions: 2.0.1 >Reporter: zreal > > The OSS of the Aliyun is widely used in China. So I want to konw whether the > jclouds can support the OSS? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [jclouds/jclouds-labs] Refactor DimensionDataCloudControlResponseUtils waitFor* methods so they are injectable (#427)
nacx commented on this pull request.
Some small comments about the predicates and timeout configuration. Overall
looks good. hanks @trevorflanagan!
> + public static final String SERVER_STARTED_PREDICATE =
> "SERVER_STARTED_PREDICATE";
+ public static final String SERVER_STOPPED_PREDICATE =
"SERVER_STOPPED_PREDICATE";
+ public static final String SERVER_DELETED_PREDICATE =
"SERVER_DELETED_PREDICATE";
+ public static final String SERVER_NORMAL_PREDICATE =
"SERVER_NORMAL_PREDICATE";
+ public static final String VM_TOOLS_RUNNING_PREDICATE =
"VM_TOOLS_RUNNING_PREDICATE";
+
+ @Override
+ protected void configure() {
+
+ }
+
+ @Provides
+ @Named(VLAN_DELETED_PREDICATE)
+ protected Predicate provideVlanDeletedPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new VlanState(api.getNetworkApi(), State.DELETED),
timeouts.imageAvailable,
The `imageAvailable` timeout is meant to be used by the `ImageExtension` when
waiting for a new image to be created. Instead of using it, just create a
property such as `operation-timeout` that can be configured as a generic
timeout for DimensionDataOperations, and get it injected here and in other
methods that don't have a matching timeout property in the
`ComputeServiceConstants.Timeouts` class.
> +pollPeriod.pollInitialPeriod, pollPeriod.pollMaxPeriod);
+ }
+
+ @Provides
+ @Named(NETWORK_DOMAIN_NORMAL_PREDICATE)
+ protected Predicate provideNetworkDomainNormalPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new NetworkDomainState(api.getNetworkApi(), State.NORMAL),
timeouts.imageAvailable,
+pollPeriod.pollInitialPeriod, pollPeriod.pollMaxPeriod);
+ }
+
+ @Provides
+ @Named(SERVER_STARTED_PREDICATE)
+ protected Predicate provideServerStartedPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new ServerStatus(api.getServerApi(), true, true),
timeouts.imageAvailable,
Use `timeouts.nodeRunning`.
> + }
+
+ @Provides
+ @Named(SERVER_STARTED_PREDICATE)
+ protected Predicate provideServerStartedPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new ServerStatus(api.getServerApi(), true, true),
timeouts.imageAvailable,
+pollPeriod.pollInitialPeriod, pollPeriod.pollMaxPeriod);
+ }
+
+ @Provides
+ @Named(SERVER_STOPPED_PREDICATE)
+ @VisibleForTesting
+ public Predicate provideServerStoppedPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new ServerStatus(api.getServerApi(), false, true),
timeouts.imageAvailable,
`timeouts.nodeSuspended`
> +
+ @Provides
+ @Named(SERVER_STOPPED_PREDICATE)
+ @VisibleForTesting
+ public Predicate provideServerStoppedPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new ServerStatus(api.getServerApi(), false, true),
timeouts.imageAvailable,
+pollPeriod.pollInitialPeriod, pollPeriod.pollMaxPeriod);
+ }
+
+ @Provides
+ @Named(SERVER_DELETED_PREDICATE)
+ @VisibleForTesting
+ public Predicate provideServerDeletedPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new ServerState(api.getServerApi(), State.DELETED),
timeouts.imageAvailable,
`timeouts.nodeTerminated`
> +
+ @Provides
+ @Named(SERVER_NORMAL_PREDICATE)
+ protected Predicate provideServerNormalPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new ServerState(api.getServerApi(), State.NORMAL),
timeouts.imageAvailable,
+pollPeriod.pollInitialPeriod, pollPeriod.pollMaxPeriod);
+ }
+
+ @Provides
+ @Named(VM_TOOLS_RUNNING_PREDICATE)
+ protected Predicate provideVMToolsRunningPredicate(final
DimensionDataCloudControlApi api,
+ final ComputeServiceConstants.Timeouts timeouts, final
ComputeServiceConstants.PollPeriod pollPeriod) {
+ return retry(new VMToolsRunningStatus(api.getServerApi()),
timeouts.imageAvailable, pollPeriod.pollInitialPeriod,
+pollPeriod.pollMaxPeriod);
+ }
In general, use the existing timeouts if they make sense, or use a
Dimens
