[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485435#comment-14485435
]
Loren Kratzke commented on IVY-1197:
------------------------------------
I created the project and attached to the Jira issue. You were right about my
patch, it does nothing. I confused the behavior during the OOME with the
behavior around detecting if httpclient is available. But I did learn a lot
from the sample project.
Note that when httpclient is available then the OOME is avoided. Without
httpclient, I do not have a fix but the project will reproduce the issue
reliably. Here is the section of readme_debug.txt that pinpoints the issue.
{quote}
6) Behavior
Assuming that you have connected and are waiting somehwere in
BasicURLHandler.upload() method,
the problem is simple to illustrate.
- An HttpURLConnection is obtained from the URL object.
- An OutputStream is obtained from the HttpURLConnection.
- The OutputStream is implemented by a sun.net.www.http.PosterOutputStream.
- Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream.
- The backing array for the ByteArrayOutputStream grows until all content has
been written.
That last point causes multiple problems for a 1GB+ file. The first is that
depending upon
your Java implementation you will run up against a hard limit on the maximum
array size
that can be allocated.
Another problem is that the backing array starts small and then doubles as
needed while
bytes are being written. So if you imagine that it is at 500MB and needs to
grow, then a
1GB array will be allocated and the 500MB array will be copied into it. So for
a short
period of time you are using 1.5GB of RAM to hold 500MB of data.
Between the array resizing and the hard limit on the array size, Ivy artifacts
in the 1GB range
are doomed unless httpclient is used. More on that in a bit.
There are methods available on the HttpURLConnection object to support
streaming the bytes
instead of buffering them, but then authentication is not performed.
See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l)
Indeed, if I call the above method then I fail to authenticate. I do not have a
work around
for streaming and preserving auth using HttpURLConnection.
{quote}
> OutOfMemoryError during ivy:publish
> -----------------------------------
>
> Key: IVY-1197
> URL: https://issues.apache.org/jira/browse/IVY-1197
> Project: Ivy
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0
> Reporter: Michael Rumpf
> Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt, ivylarge.zip,
> org.apache.ivy.util.url.HttpClientHandler.patch
>
>
> When publishing a large file, an OutOfMemoryError occurs.
> {code}
> [ivy:publish] published ppg to
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> BUILD FAILED
> /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
> The following error occurred while executing this line:
> /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
> at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
> at
> org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
> at
> org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
> at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
> at
> org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
> at
> org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
> at
> org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
> at
> org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
> at
> org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
> at
> org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
> at
> org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
> at org.apache.ivy.Ivy.publish(Ivy.java:600)
> at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
> at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
> at org.apache.tools.ant.Task.perform(Task.java:348)
> at org.apache.tools.ant.Target.execute(Target.java:390)
> at org.apache.tools.ant.Target.performTasks(Target.java:411)
> at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
> at
> org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
> at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
> at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
> at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> Total time: 14 minutes 24 seconds
> Finished: FAILURE
> {code}
> The size of the file that is being uploaded is: 687712714, so around
> 650-700MB.
> The publish task is part of a Hudson Ant build where the artefacts are
> published to an Artifactory repository at the end.
> I have given the Job 1300MB for the max heap size.
> It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
