[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[ https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620530#comment-17620530 ] Andrew Norman edited comment on IVY-1197 at 10/19/22 7:39 PM: -- any chance this gets fixed soon? My use case is this out of memory failure when running an "sbt publish" of a large zip file >512M on jenkins was (Author: JIRAUSER297261): any chance this gets fixed soon? My use case is this out of memory failing when running an "sbt publish" of a large zip file >512M on jenkins > OutOfMemoryError during ivy:publish > --- > > Key: IVY-1197 > URL: https://issues.apache.org/jira/browse/IVY-1197 > Project: Ivy > Issue Type: Bug > Components: Core >Affects Versions: 2.0 >Reporter: Michael Rumpf >Priority: Major > Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt, ivylarge.zip, > org.apache.ivy.util.url.HttpClientHandler.patch > > > When publishing a large file, an OutOfMemoryError occurs. > {code} > [ivy:publish] published ppg to > > BUILD FAILED > /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152: > The following error occurred while executing this line: > /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277: > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2786) > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) > at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61) > at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168) > at > org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200) > at > org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82) > at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140) > at > org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85) > at > org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130) > at > org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219) > at > org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209) > at > org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282) > at > org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261) > at > org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170) > at org.apache.ivy.Ivy.publish(Ivy.java:600) > at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299) > at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277) > at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) > at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) > at org.apache.tools.ant.Task.perform(Task.java:348) > at org.apache.tools.ant.Target.execute(Target.java:390) > at org.apache.tools.ant.Target.performTasks(Target.java:411) > at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1397) > at > org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:38) > at org.apache.tools.ant.Project.executeTargets(Project.java:1249) > at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:442) > at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105) > at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > Total time: 14 minutes 24 seconds > Finished: FAILURE > {code} > The size of the file that is being uploaded is: 687712714, so around > 650-700MB. > The publish task is part of a Hudson Ant build where the artefacts are > published to an Artifactory repository at the end. > I have given the Job 1300MB for the max heap size. > It seems as if the whole file is loaded into memory for the upload. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[ https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485435#comment-14485435 ] Loren Kratzke edited comment on IVY-1197 at 4/8/15 4:21 PM: I created the project and attached to the Jira issue. You were right about my patch, it does nothing. I confused the behavior during the OOME with the behavior around detecting if httpclient is available. But I did learn a lot from the sample project. Note that when httpclient is available then the OOME is avoided. Without httpclient, I do not have a fix but the project will reproduce the issue reliably. Here is the section of readme_debug.txt that pinpoints the issue. {quote} 6) Behavior Assuming that you have connected and are waiting somewhere in BasicURLHandler.upload() method, the problem is simple to illustrate. - An HttpURLConnection is obtained from the URL object. - An OutputStream is obtained from the HttpURLConnection. - The OutputStream is implemented by a sun.net.www.http.PosterOutputStream. - Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream. - The backing array for the ByteArrayOutputStream grows until all content has been written. That last point causes multiple problems for a 1GB+ file. The first is that depending upon your Java implementation you will run up against a hard limit on the maximum array size that can be allocated. Another problem is that the backing array starts small and then doubles as needed while bytes are being written. So if you imagine that it is at 500MB and needs to grow, then a 1GB array will be allocated and the 500MB array will be copied into it. So for a short period of time you are using 1.5GB of RAM to hold 500MB of data. Between the array resizing and the hard limit on the array size, Ivy artifacts in the 1GB range are doomed unless httpclient is used. More on that in a bit. There are methods available on the HttpURLConnection object to support streaming the bytes instead of buffering them, but then authentication is not performed. See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l) http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode%28long%29 Indeed, if I call the above method then I fail to authenticate. I do not have a work around for streaming and preserving auth using HttpURLConnection. {quote} was (Author: qphase): I created the project and attached to the Jira issue. You were right about my patch, it does nothing. I confused the behavior during the OOME with the behavior around detecting if httpclient is available. But I did learn a lot from the sample project. Note that when httpclient is available then the OOME is avoided. Without httpclient, I do not have a fix but the project will reproduce the issue reliably. Here is the section of readme_debug.txt that pinpoints the issue. {quote} 6) Behavior Assuming that you have connected and are waiting somehwere in BasicURLHandler.upload() method, the problem is simple to illustrate. - An HttpURLConnection is obtained from the URL object. - An OutputStream is obtained from the HttpURLConnection. - The OutputStream is implemented by a sun.net.www.http.PosterOutputStream. - Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream. - The backing array for the ByteArrayOutputStream grows until all content has been written. That last point causes multiple problems for a 1GB+ file. The first is that depending upon your Java implementation you will run up against a hard limit on the maximum array size that can be allocated. Another problem is that the backing array starts small and then doubles as needed while bytes are being written. So if you imagine that it is at 500MB and needs to grow, then a 1GB array will be allocated and the 500MB array will be copied into it. So for a short period of time you are using 1.5GB of RAM to hold 500MB of data. Between the array resizing and the hard limit on the array size, Ivy artifacts in the 1GB range are doomed unless httpclient is used. More on that in a bit. There are methods available on the HttpURLConnection object to support streaming the bytes instead of buffering them, but then authentication is not performed. See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l) http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode%28long%29 Indeed, if I call the above method then I fail to authenticate. I do not have a work around for streaming and preserving auth using HttpURLConnection. {quote} OutOfMemoryError during ivy:publish --- Key: IVY-1197 URL: https://issues.apache.org/jira/browse/IVY-1197 Project: Ivy Issue Type: Bug Components:
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[ https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485435#comment-14485435 ] Loren Kratzke edited comment on IVY-1197 at 4/8/15 4:08 PM: I created the project and attached to the Jira issue. You were right about my patch, it does nothing. I confused the behavior during the OOME with the behavior around detecting if httpclient is available. But I did learn a lot from the sample project. Note that when httpclient is available then the OOME is avoided. Without httpclient, I do not have a fix but the project will reproduce the issue reliably. Here is the section of readme_debug.txt that pinpoints the issue. {quote} 6) Behavior Assuming that you have connected and are waiting somehwere in BasicURLHandler.upload() method, the problem is simple to illustrate. - An HttpURLConnection is obtained from the URL object. - An OutputStream is obtained from the HttpURLConnection. - The OutputStream is implemented by a sun.net.www.http.PosterOutputStream. - Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream. - The backing array for the ByteArrayOutputStream grows until all content has been written. That last point causes multiple problems for a 1GB+ file. The first is that depending upon your Java implementation you will run up against a hard limit on the maximum array size that can be allocated. Another problem is that the backing array starts small and then doubles as needed while bytes are being written. So if you imagine that it is at 500MB and needs to grow, then a 1GB array will be allocated and the 500MB array will be copied into it. So for a short period of time you are using 1.5GB of RAM to hold 500MB of data. Between the array resizing and the hard limit on the array size, Ivy artifacts in the 1GB range are doomed unless httpclient is used. More on that in a bit. There are methods available on the HttpURLConnection object to support streaming the bytes instead of buffering them, but then authentication is not performed. See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l) http://docs.oracle.com/javase/7/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode%28long%29 Indeed, if I call the above method then I fail to authenticate. I do not have a work around for streaming and preserving auth using HttpURLConnection. {quote} was (Author: qphase): I created the project and attached to the Jira issue. You were right about my patch, it does nothing. I confused the behavior during the OOME with the behavior around detecting if httpclient is available. But I did learn a lot from the sample project. Note that when httpclient is available then the OOME is avoided. Without httpclient, I do not have a fix but the project will reproduce the issue reliably. Here is the section of readme_debug.txt that pinpoints the issue. {quote} 6) Behavior Assuming that you have connected and are waiting somehwere in BasicURLHandler.upload() method, the problem is simple to illustrate. - An HttpURLConnection is obtained from the URL object. - An OutputStream is obtained from the HttpURLConnection. - The OutputStream is implemented by a sun.net.www.http.PosterOutputStream. - Writing to the OutputStream (in FileUtil) writes to a ByteArrayOutputStream. - The backing array for the ByteArrayOutputStream grows until all content has been written. That last point causes multiple problems for a 1GB+ file. The first is that depending upon your Java implementation you will run up against a hard limit on the maximum array size that can be allocated. Another problem is that the backing array starts small and then doubles as needed while bytes are being written. So if you imagine that it is at 500MB and needs to grow, then a 1GB array will be allocated and the 500MB array will be copied into it. So for a short period of time you are using 1.5GB of RAM to hold 500MB of data. Between the array resizing and the hard limit on the array size, Ivy artifacts in the 1GB range are doomed unless httpclient is used. More on that in a bit. There are methods available on the HttpURLConnection object to support streaming the bytes instead of buffering them, but then authentication is not performed. See javadoc for HttpURLConnection.setFixedLengthStreamingMode(long l) Indeed, if I call the above method then I fail to authenticate. I do not have a work around for streaming and preserving auth using HttpURLConnection. {quote} OutOfMemoryError during ivy:publish --- Key: IVY-1197 URL: https://issues.apache.org/jira/browse/IVY-1197 Project: Ivy Issue Type: Bug Components: Core Affects Versions: 2.0 Reporter: Michael Rumpf Attachments:
[jira] [Comment Edited] (IVY-1197) OutOfMemoryError during ivy:publish
[ https://issues.apache.org/jira/browse/IVY-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483468#comment-14483468 ] Loren Kratzke edited comment on IVY-1197 at 4/7/15 4:45 PM: I would be happy to provide you with a project that will reproduce the issue. I can and will do that. Generally speaking from a high level, the utility classes are calling convenience methods and writing to streams that ultimately buffer the data being written. There is buffering, then more buffering, and even more buffering until you have multiple copies of the entire content of the stream stored in over sized buffers (because they double in size when they fill up). Oddly, the twist is that the JVM hits a limit no matter how much RAM you allocate. Once the buffers total more than about ~1GB (which is what happens with a 100-200MB upload) the JVM refuses to allocate more buffer space (even if you jack up the RAM to 20GB, no cigar). Honestly, there is no benefit in buffering any of this data to begin with, it is just a side effect of using high level copy methods. There is no memory ballooning at all when the content is written directly to the network. I will provide a test project and note the break points where you can debug and watch the process walk all the way down the isle to an OOME. I will have this for you asap. was (Author: qphase): I would be happy to provide you with a project that will reproduce the issue. I can and will do that. Generally speaking from a high level, the utility classes are calling convenience methods and writing to streams that ultimately buffer the data being written. There is buffering, then more buffering, and even more buffering until you have multiple copies of the entire content of the stream stored in over sized buffers (because they double in size when they fill up). Oddly, the twist is that the JVM hits a limit no matter how much RAM you allocate. Once the buffers total more than about ~1GB (which is what happens with a 100-200MB upload) the JVM refuses to allocate more buffer space (even is you jack up the RAM to 20GB, no cigar). Honestly, there is no benefit in buffering any of this data to begin with, it is just a side effect of using high level copy methods. There is no memory ballooning at all when the content is written directly to the network. I will provide a test project and note the break points where you can debug and watch the process walk all the way down the isle to an OOME. I will have this for you asap. OutOfMemoryError during ivy:publish --- Key: IVY-1197 URL: https://issues.apache.org/jira/browse/IVY-1197 Project: Ivy Issue Type: Bug Components: Core Affects Versions: 2.0 Reporter: Michael Rumpf Attachments: ASF.LICENSE.NOT.GRANTED--clipboard.txt, org.apache.ivy.util.url.HttpClientHandler.patch When publishing a large file, an OutOfMemoryError occurs. {code} [ivy:publish] published ppg to BUILD FAILED /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:152: The following error occurred while executing this line: /export/build/hudson/jobs/ppg-rcp/workspace/ppg-rcp/com.daimler.ppg.rcp.builder/build-wrapper.xml:277: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:61) at org.apache.ivy.util.FileUtil.copy(FileUtil.java:168) at org.apache.ivy.util.url.BasicURLHandler.upload(BasicURLHandler.java:200) at org.apache.ivy.util.url.URLHandlerDispatcher.upload(URLHandlerDispatcher.java:82) at org.apache.ivy.util.FileUtil.copy(FileUtil.java:140) at org.apache.ivy.plugins.repository.url.URLRepository.put(URLRepository.java:85) at org.apache.ivy.plugins.repository.AbstractRepository.put(AbstractRepository.java:130) at org.apache.ivy.plugins.resolver.RepositoryResolver.put(RepositoryResolver.java:219) at org.apache.ivy.plugins.resolver.RepositoryResolver.publish(RepositoryResolver.java:209) at org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:282) at org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:261) at org.apache.ivy.core.publish.PublishEngine.publish(PublishEngine.java:170) at org.apache.ivy.Ivy.publish(Ivy.java:600) at org.apache.ivy.ant.IvyPublish.doExecute(IvyPublish.java:299) at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) at