[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485435#comment-14485435
]
Loren Kratzke edited comment on IVY-1197 at 4/8/15 4:21 PM:
I created the project and attached to the Jira issue. You were right about my
patch, it does nothing. I confused the behavior during the OOME with the
behavior around detecting if httpclient is available. But I did learn a lot
from the sample project.
Note that when httpclient is available then the OOME is avoided. Without
httpclient, I do not have a fix but the project will reproduce the issue
reliably. Here is the section of readme_debug.txt that pinpoints the issue.
{quote}
6) Behavior
Assuming that you have connected and are waiting somewhere in
BasicURLHandler.upload() method,
the problem is simple to illustrate.
- An HttpURLConnection is obtained from the URL object.
- An OutputStream is obtained from the HttpURLConnection.
- The OutputStream is implemented by a sun.net.www.http.PosterOutputStream.
- Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream.
- The backing array for the ByteArrayOutputStream grows until all content has
been written.
That last point causes multiple problems for a 1GB+ file. The first is that
depending upon
your Java implementation you will run up against a hard limit on the maximum
array size
that can be allocated.
Another problem is that the backing array starts small and then doubles as
needed while
bytes are being written. So if you imagine that it is at 500MB and needs to
grow, then a
1GB array will be allocated and the 500MB array will be copied into it. So for
a short
period of time you are using 1.5GB of RAM to hold 500MB of data.
Between the array resizing and the hard limit on the array size, Ivy artifacts
in the 1GB range
are doomed unless httpclient is used. More on that in a bit.
There are methods available on the HttpURLConnection object to support
streaming the bytes
instead of buffering them, but then authentication is not performed.
See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l)
http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode%28long%29
Indeed, if I call the above method then I fail to authenticate. I do not have a
work around
for streaming and preserving auth using HttpURLConnection.
{quote}
was (Author: qphase):
I created the project and attached to the Jira issue. You were right about my
patch, it does nothing. I confused the behavior during the OOME with the
behavior around detecting if httpclient is available. But I did learn a lot
from the sample project.
Note that when httpclient is available then the OOME is avoided. Without
httpclient, I do not have a fix but the project will reproduce the issue
reliably. Here is the section of readme_debug.txt that pinpoints the issue.
{quote}
6) Behavior
Assuming that you have connected and are waiting somehwere in
BasicURLHandler.upload() method,
the problem is simple to illustrate.
- An HttpURLConnection is obtained from the URL object.
- An OutputStream is obtained from the HttpURLConnection.
- The OutputStream is implemented by a sun.net.www.http.PosterOutputStream.
- Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream.
- The backing array for the ByteArrayOutputStream grows until all content has
been written.
That last point causes multiple problems for a 1GB+ file. The first is that
depending upon
your Java implementation you will run up against a hard limit on the maximum
array size
that can be allocated.
Another problem is that the backing array starts small and then doubles as
needed while
bytes are being written. So if you imagine that it is at 500MB and needs to
grow, then a
1GB array will be allocated and the 500MB array will be copied into it. So for
a short
period of time you are using 1.5GB of RAM to hold 500MB of data.
Between the array resizing and the hard limit on the array size, Ivy artifacts
in the 1GB range
are doomed unless httpclient is used. More on that in a bit.
There are methods available on the HttpURLConnection object to support
streaming the bytes
instead of buffering them, but then authentication is not performed.
See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l)
http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode%28long%29
Indeed, if I call the above method then I fail to authenticate. I do not have a
work around
for streaming and preserving auth using HttpURLConnection.
{quote}
OutOfMemoryError during ivy:publish
---
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components:
[jira] [Updated] (IVY-1197) OutOfMemoryError during ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loren Kratzke updated IVY-1197:
---
Attachment: ivylarge.zip
Attached sample project capable of reproducing OOME. See readme_debug.txt and
dist/readme.txt for detailed instructions.
OutOfMemoryError during ivy:publish
---
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt, ivylarge.zip,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485435#comment-14485435
]
Loren Kratzke edited comment on IVY-1197 at 4/8/15 4:08 PM:
I created the project and attached to the Jira issue. You were right about my
patch, it does nothing. I confused the behavior during the OOME with the
behavior around detecting if httpclient is available. But I did learn a lot
from the sample project.
Note that when httpclient is available then the OOME is avoided. Without
httpclient, I do not have a fix but the project will reproduce the issue
reliably. Here is the section of readme_debug.txt that pinpoints the issue.
{quote}
6) Behavior
Assuming that you have connected and are waiting somehwere in
BasicURLHandler.upload() method,
the problem is simple to illustrate.
- An HttpURLConnection is obtained from the URL object.
- An OutputStream is obtained from the HttpURLConnection.
- The OutputStream is implemented by a sun.net.www.http.PosterOutputStream.
- Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream.
- The backing array for the ByteArrayOutputStream grows until all content has
been written.
That last point causes multiple problems for a 1GB+ file. The first is that
depending upon
your Java implementation you will run up against a hard limit on the maximum
array size
that can be allocated.
Another problem is that the backing array starts small and then doubles as
needed while
bytes are being written. So if you imagine that it is at 500MB and needs to
grow, then a
1GB array will be allocated and the 500MB array will be copied into it. So for
a short
period of time you are using 1.5GB of RAM to hold 500MB of data.
Between the array resizing and the hard limit on the array size, Ivy artifacts
in the 1GB range
are doomed unless httpclient is used. More on that in a bit.
There are methods available on the HttpURLConnection object to support
streaming the bytes
instead of buffering them, but then authentication is not performed.
See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l)
http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode%28long%29
Indeed, if I call the above method then I fail to authenticate. I do not have a
work around
for streaming and preserving auth using HttpURLConnection.
{quote}
was (Author: qphase):
I created the project and attached to the Jira issue. You were right about my
patch, it does nothing. I confused the behavior during the OOME with the
behavior around detecting if httpclient is available. But I did learn a lot
from the sample project.
Note that when httpclient is available then the OOME is avoided. Without
httpclient, I do not have a fix but the project will reproduce the issue
reliably. Here is the section of readme_debug.txt that pinpoints the issue.
{quote}
6) Behavior
Assuming that you have connected and are waiting somehwere in
BasicURLHandler.upload() method,
the problem is simple to illustrate.
- An HttpURLConnection is obtained from the URL object.
- An OutputStream is obtained from the HttpURLConnection.
- The OutputStream is implemented by a sun.net.www.http.PosterOutputStream.
- Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream.
- The backing array for the ByteArrayOutputStream grows until all content has
been written.
That last point causes multiple problems for a 1GB+ file. The first is that
depending upon
your Java implementation you will run up against a hard limit on the maximum
array size
that can be allocated.
Another problem is that the backing array starts small and then doubles as
needed while
bytes are being written. So if you imagine that it is at 500MB and needs to
grow, then a
1GB array will be allocated and the 500MB array will be copied into it. So for
a short
period of time you are using 1.5GB of RAM to hold 500MB of data.
Between the array resizing and the hard limit on the array size, Ivy artifacts
in the 1GB range
are doomed unless httpclient is used. More on that in a bit.
There are methods available on the HttpURLConnection object to support
streaming the bytes
instead of buffering them, but then authentication is not performed.
See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l)
Indeed, if I call the above method then I fail to authenticate. I do not have a
work around
for streaming and preserving auth using HttpURLConnection.
{quote}
OutOfMemoryError during ivy:publish
---
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments:
[jira] [Commented] (IVY-1197) OutOfMemoryError during ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485435#comment-14485435
]
Loren Kratzke commented on IVY-1197:
I created the project and attached to the Jira issue. You were right about my
patch, it does nothing. I confused the behavior during the OOME with the
behavior around detecting if httpclient is available. But I did learn a lot
from the sample project.
Note that when httpclient is available then the OOME is avoided. Without
httpclient, I do not have a fix but the project will reproduce the issue
reliably. Here is the section of readme_debug.txt that pinpoints the issue.
{quote}
6) Behavior
Assuming that you have connected and are waiting somehwere in
BasicURLHandler.upload() method,
the problem is simple to illustrate.
- An HttpURLConnection is obtained from the URL object.
- An OutputStream is obtained from the HttpURLConnection.
- The OutputStream is implemented by a sun.net.www.http.PosterOutputStream.
- Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream.
- The backing array for the ByteArrayOutputStream grows until all content has
been written.
That last point causes multiple problems for a 1GB+ file. The first is that
depending upon
your Java implementation you will run up against a hard limit on the maximum
array size
that can be allocated.
Another problem is that the backing array starts small and then doubles as
needed while
bytes are being written. So if you imagine that it is at 500MB and needs to
grow, then a
1GB array will be allocated and the 500MB array will be copied into it. So for
a short
period of time you are using 1.5GB of RAM to hold 500MB of data.
Between the array resizing and the hard limit on the array size, Ivy artifacts
in the 1GB range
are doomed unless httpclient is used. More on that in a bit.
There are methods available on the HttpURLConnection object to support
streaming the bytes
instead of buffering them, but then authentication is not performed.
See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l)
Indeed, if I call the above method then I fail to authenticate. I do not have a
work around
for streaming and preserving auth using HttpURLConnection.
{quote}
OutOfMemoryError during ivy:publish
---
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt, ivylarge.zip,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483468#comment-14483468
]
Loren Kratzke edited comment on IVY-1197 at 4/7/15 4:45 PM:
I would be happy to provide you with a project that will reproduce the issue. I
can and will do that.
Generally speaking from a high level, the utility classes are calling
convenience methods and writing to streams that ultimately buffer the data
being written. There is buffering, then more buffering, and even more buffering
until you have multiple copies of the entire content of the stream stored in
over sized buffers (because they double in size when they fill up). Oddly, the
twist is that the JVM hits a limit no matter how much RAM you allocate. Once
the buffers total more than about ~1GB (which is what happens with a 100-200MB
upload) the JVM refuses to allocate more buffer space (even if you jack up the
RAM to 20GB, no cigar). Honestly, there is no benefit in buffering any of this
data to begin with, it is just a side effect of using high level copy methods.
There is no memory ballooning at all when the content is written directly to
the network.
I will provide a test project and note the break points where you can debug and
watch the process walk all the way down the isle to an OOME. I will have this
for you asap.
was (Author: qphase):
I would be happy to provide you with a project that will reproduce the issue. I
can and will do that.
Generally speaking from a high level, the utility classes are calling
convenience methods and writing to streams that ultimately buffer the data
being written. There is buffering, then more buffering, and even more buffering
until you have multiple copies of the entire content of the stream stored in
over sized buffers (because they double in size when they fill up). Oddly, the
twist is that the JVM hits a limit no matter how much RAM you allocate. Once
the buffers total more than about ~1GB (which is what happens with a 100-200MB
upload) the JVM refuses to allocate more buffer space (even is you jack up the
RAM to 20GB, no cigar). Honestly, there is no benefit in buffering any of this
data to begin with, it is just a side effect of using high level copy methods.
There is no memory ballooning at all when the content is written directly to
the network.
I will provide a test project and note the break points where you can debug and
watch the process walk all the way down the isle to an OOME. I will have this
for you asap.
OutOfMemoryError during ivy:publish
---
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at
[jira] [Commented] (IVY-1197) OutOfMemoryError during ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483468#comment-14483468
]
Loren Kratzke commented on IVY-1197:
I would be happy to provide you with a project that will reproduce the issue. I
can and will do that.
Generally speaking from a high level, the utility classes are calling
convenience methods and writing to streams that ultimately buffer the data
being written. There is buffering, then more buffering, and even more buffering
until you have multiple copies of the entire content of the stream stored in
over sized buffers (because they double in size when they fill up). Oddly, the
twist is that the JVM hits a limit no matter how much RAM you allocate. Once
the buffers total more than about ~1GB (which is what happens with a 100-200MB
upload) the JVM refuses to allocate more buffer space (even is you jack up the
RAM to 20GB, no cigar). Honestly, there is no benefit in buffering any of this
data to begin with, it is just a side effect of using high level copy methods.
There is no memory ballooning at all when the content is written directly to
the network.
I will provide a test project and note the break points where you can debug and
watch the process walk all the way down the isle to an OOME. I will have this
for you asap.
OutOfMemoryError during ivy:publish
---
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366203#comment-14366203
]
Loren Kratzke edited comment on IVY-1197 at 3/17/15 10:25 PM:
--
I finally found the time to recreate the requested patch file. This patch is
based upon HEAD as of 2015-03-17. Sorry for the delay.
was (Author: qphase):
This patch is based upon HEAD as of 2015-03-17.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loren Kratzke updated IVY-1197:
---
Attachment: (was: org.apache.ivy.util.url.HttpClientHandler.patch)
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loren Kratzke updated IVY-1197:
---
Attachment: org.apache.ivy.util.url.HttpClientHandler.patch
This patch is based upon HEAD as of 2015-03-17.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329151#comment-14329151
]
Loren Kratzke commented on IVY-1197:
No problem. Super busy at the moment but I will provide this asap.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318462#comment-14318462
]
Loren Kratzke commented on IVY-1197:
Seriously folks, my fix works. It has been sitting here for over a year now. Is
there anything I can do to help get this implemented?
I have been using my patched version of Ivy for the last 13 months and it has
performed flawlessly. It is tested and works. We regularly build and publish OS
images that are several GB in size with no issues.That is not possible using a
stock Ivy distro. We also use Ivy for about 80 other projects. In total I have
executed 10's of thousands of builds using my patched version without issues.
It would be nice to be able to use a released version of Ivy someday. The rest
of the Ivy world would really appreciate this fix.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072271#comment-14072271
]
Loren Kratzke commented on IVY-1197:
Just wanted to bump this bug. My fix works and this is a major issue. One could
argue that this fix alone could justify a bug fix release.
In the short term, at least allowing the users to work around the OOME using
the 3 jars seems like a good idea. My fix provides that.
In the long run, I would suggest trashing the FileUtil.copy(instream, out,
null, false). The code is quite problematic and it is not a good experience to
require users to add the 3 Apache extra jars to their Ant distro in order to
work around the OOME issue. I would be glad to help out. Just let me know what
you need.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020419#comment-14020419
]
Loren Kratzke commented on IVY-1197:
Just to clarify a few things, I was publishing to a Nexus server and the file
size limitation kicked in after a few hundred MB (not GB). In response to
Antoines comment above, it is unclear if 2.4.0-rc1 contains the fix or whether
the problem still reproduces when publishing to a target such as Nexus. I
imagine it would happen during any HTTP publish but can't be sure.
Regarding reproducing the problem to pinpoint the exact failure point that I
described, just attempt the publish during a debug session as many times as
necessary to zero in on the exception. I have not had time to go back and do
that again. It is a very tedious task after all. But I stand by my fix. It will
solve this issue for sure. I am absolutely certain about this. Here at my
organization we regularly publish 800MB and larger files. This was not possible
before I patched Ivy. This bug is a distant memory around here.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961930#comment-13961930
]
Loren Kratzke commented on IVY-1197:
It is not at all obvious when looking at source code, and off the top of my
head I do not have a quick answer to your question. Let me reproduce and
document the steps I used to debug this. In the process I can provide exact
details around the source files and methods that come together to make this bug
happen.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Loren Kratzke updated IVY-1197:
---
Attachment: org.apache.ivy.util.url.HttpClientHandler.patch
I have attached a patch for HttpClientHandler.java. We have been using this
patched version for a while now. It is working quite well for publishing 600MB+
files. We have experienced no issues so far.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13960027#comment-13960027
]
Loren Kratzke edited comment on IVY-1197 at 4/4/14 3:01 PM:
I have attached a patch for HttpClientHandler.java. We have been using this
patched version for a while now. It is working quite well for publishing 600MB+
files. We are using default heap and have experienced no issues so far.
was (Author: qphase):
I have attached a patch for HttpClientHandler.java. We have been using this
patched version for a while now. It is working quite well for publishing 600MB+
files. We have experienced no issues so far.
OutOfMemoryError duriong ivy:publish
Key: IVY-1197
URL: https://issues.apache.org/jira/browse/IVY-1197
Project: Ivy
Issue Type: Bug
Components: Core
Affects Versions: 2.0
Reporter: Michael Rumpf
Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt,
org.apache.ivy.util.url.HttpClientHandler.patch
When publishing a large file, an OutOfMemoryError occurs.
{code}
[ivy:publish] published ppg to
BUILD FAILED
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152:
The following error occurred while executing this line:
/export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168)
at
org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200)
at
org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82)
at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140)
at
org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85)
at
org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219)
at
org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261)
at
org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170)
at org.apache.ivy.Ivy.publish(Ivy.java:600)
at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299)
at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397)
at
org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38)
at org.apache.tools.ant.Project.executeTargets(Project.java:1249)
at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442)
at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Total time: 14 minutes 24 seconds
Finished: FAILURE
{code}
The size of the file that is being uploaded is: 687712714, so around
650-700MB.
The publish task is part of a Hudson Ant build where the artefacts are
published to an Artifactory repository at the end.
I have given the Job 1300MB for the max heap size.
It seems as if the whole file is loaded into memory for the upload.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875378#comment-13875378
]
Loren Kratzke edited comment on IVY-1197 at 1/17/14 11:12 PM:
--
I have identified the root cause of the issue and I have a solution.
Problem:
JVM experiences java.lang.OutOfMemoryError during ivy:publish of a file greater
than a few hundred MB.
Cause:
HttpURLConnection object as configured in
org.apache.ivy.util.url.BasicURLHandler buffers the data before sending so as
to calculate the content-length header value. This alone is bad however is
likely aggravated by the default array resizing algorythm of the
ByteArrayOutputStream used to buffer the data.
The buffer starts small having a 32 byte capacity. When written to, it fills up
and eventually it reaches maximum capacity. At this time a new buffer of twice
the size is allocated. The content of the original buffer is copied to the new
buffer and the old buffer is garbage collected. Lather, rinse, repeat.
This works fine for small amounts of data however this is a major problem when
talking about medium or large amounts of data. As the buffer size approaches
50% of the amount of free heap space, there will no longer be enough free RAM
to allocate the new buffer. For example, if a 512MB requires even one more
byte of capacity, and there is one byte less than 1024MB free, then an attempt
by ByteArrayOutputStream to allocate a new 1024MB will fail with an OOME. This
is a tragic waste of memory, especially if the content-length is already known
and buffering is not even required (which is always the case for Ivy).
Partial/Failed Solution Using HttpClient:
For one reason or another, possibly to fix this issue, a reflection invocation
is made by org.apache.ivy.util.url.URLHandlerRegistry for
org.apache.commons.httpclient.HttpClient. Comments about Ivy-1197 instruct
users to place HttpClient jar on the classpath (lib dir of Ant) to solve the
OOME issue. Indeed, HttpClient (even the ancient version from 2005 used by Ivy)
does not buffer the content and thus could avoid the OOME entirely, however
there are three major problems with this solution.
The first problem is that the trial invocation of HttpClient issued to detect
availablility on the classpath fails rather silently by logging a vague
message, and only if Ant is invoked in verbose mode. So somebody might drop the
jar into the lib, see nothing happen, and wonder why.
The second problem is that the trial invocation of HttpClient does indeed fail
if all one does is drop the jar into the lib dir. This is because HttpClient
requires two additional jars to in order to instantiate, let alone function:
commons-logging and commons-codec. Adding these jars successfully triggers the
Ivy code which substitutes the Apache HttpClient based
org.apache.ivy.util.url.HttpClientHandler for the problematic
URLConnectionHandler based org.apache.ivy.util.url.BasicURLConnectionHandler.
But there is one more problem.
The third (and most unusual) problem is as follows: Apache docs in the
HttpClient performance guide describe how to stream a request using a custom
RequestEntity object. This object is capable of restarting the stream in the
event of an interruption or an authentication request. They even provide sample
code. This code was copied into the Ivy HttpClientHandler however the block
that writes directly to the OutputStream (without buffering) has been replaced
by a call to the same method that URLConnectionHandler calls which buffers all
of the data. The net effect is that the OOME persists because nothing about the
upload has changed - the data is still buffered in a ByteArrayOutputStream
despite the fact that HttpClient is available and an OOME is still thrown.
Complete/Successful Solution:
In HttpClientHandler.FileRequestEntity.writeRequest(OutputStream) replace this
line:
FileUtil.copy(instream, out, null, false);
with the original Apache sample code (slightly refactored to match the class):
int length;
byte[] buffer = new byte[64*1024];
while ((length = instream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
Then recompile Ivy. I recompiled using JDK 1.6 because that is the oldest VM
around here.
This fix only works when all of the following libraries are located in
$ANT_HOME/lib directory:
commons-httpclient.jar
commons-logging.jar
commons-codec.jar
Those libraries are available in the apache-ivy-2.3.0-bin-with-deps
distribution.
was (Author: qphase):
I have identified the root cause of the issue and I have a solution.
Problem:
JVM experiences java.lang.OutOfMemoryError during ivy:publish of a file greater
than a few hundred MB.
Cause:
HttpURLConnection object as configured in
org.apache.ivy.util.url.BasicURLHandler buffers the data before sending so as
to
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError duriong ivy:publish
[
https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13875378#comment-13875378
]
Loren Kratzke edited comment on IVY-1197 at 1/18/14 4:38 AM:
-
I have identified the root cause of the issue and I have a solution.
Problem:
JVM experiences java.lang.OutOfMemoryError during ivy:publish of a file greater
than a few hundred MB.
Cause:
HttpURLConnection object as configured in
org.apache.ivy.util.url.BasicURLHandler buffers the data before sending so as
to calculate the content-length header value. This alone is bad however is
likely aggravated by the default array resizing algorythm of the
ByteArrayOutputStream used to buffer the data.
The buffer starts small having a 32 byte capacity. When written to, it fills up
and eventually it reaches maximum capacity. At this time a new buffer of twice
the size is allocated. The content of the original buffer is copied to the new
buffer and the old buffer is garbage collected. Lather, rinse, repeat.
This works fine for small amounts of data however this is a major problem when
talking about medium or large amounts of data. As the buffer size approaches
50% of the amount of free heap space, there will no longer be enough free RAM
to allocate the new buffer. For example, if a 512MB buffer requires even one
more byte of capacity, and there is one byte less than 1024MB free, then an
attempt by ByteArrayOutputStream to allocate a new 1024MB buffer will fail with
an OOME. This is a tragic waste of memory, especially if the content-length is
already known and buffering is not even required (which is always the case for
Ivy).
Partial/Failed Solution Using HttpClient:
For one reason or another, possibly to fix this issue, a reflection invocation
is made by org.apache.ivy.util.url.URLHandlerRegistry for
org.apache.commons.httpclient.HttpClient. Comments about Ivy-1197 instruct
users to place HttpClient jar on the classpath (lib dir of Ant) to solve the
OOME issue. Indeed, HttpClient (even the ancient version from 2005 used by Ivy)
does not buffer the content and thus could avoid the OOME entirely, however
there are three major problems with this solution.
The first problem is that the trial invocation of HttpClient issued to detect
availablility on the classpath fails rather silently by logging a vague
message, and only if Ant is invoked in verbose mode. So somebody might drop the
jar into the lib, see nothing happen, and wonder why.
The second problem is that the trial invocation of HttpClient does indeed fail
if all one does is drop the httpclient jar into the lib dir. This is because
HttpClient requires two additional jars to in order to instantiate:
commons-logging and commons-codec. Adding these jars successfully triggers the
Ivy code which substitutes the Apache HttpClient based
org.apache.ivy.util.url.HttpClientHandler for the problematic
URLConnectionHandler based org.apache.ivy.util.url.BasicURLConnectionHandler.
But there is one more problem.
The third problem is as follows: Apache docs in the HttpClient performance
guide describe how to stream a request using a custom RequestEntity object.
This object is capable of restarting the stream in the event of an interruption
or an authentication request. They provide sample code. This code was copied
into the Ivy HttpClientHandler however the block that writes directly to the
OutputStream (without buffering) has been replaced by a call to the same method
that URLConnectionHandler calls which buffers all of the data. The net effect
is that the OOME persists because nothing about the upload has changed - the
data is still buffered in a ByteArrayOutputStream.
Complete/Successful Solution:
In HttpClientHandler.FileRequestEntity.writeRequest(OutputStream) replace this
line:
FileUtil.copy(instream, out, null, false);
with the original Apache sample code (slightly refactored to match the class):
int length;
byte[] buffer = new byte[64*1024];
while ((length = instream.read(buffer)) != -1) {
out.write(buffer, 0, length);
}
Then recompile Ivy. I recompiled using JDK 1.6 because that is the oldest VM
around here.
This fix only works when all of the following libraries are located in
$ANT_HOME/lib directory:
commons-httpclient.jar
commons-logging.jar
commons-codec.jar
Those libraries are available in the apache-ivy-2.3.0-bin-with-deps
distribution.
was (Author: qphase):
I have identified the root cause of the issue and I have a solution.
Problem:
JVM experiences java.lang.OutOfMemoryError during ivy:publish of a file greater
than a few hundred MB.
Cause:
HttpURLConnection object as configured in
org.apache.ivy.util.url.BasicURLHandler buffers the data before sending so as
to calculate the content-length header value. This alone is bad however is
likely aggravated by the
