[ https://issues.apache.org/jira/browse/BEAM-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017961#comment-16017961 ]
ASF GitHub Bot commented on BEAM-2334: -------------------------------------- GitHub user yonromai opened a pull request: https://github.com/apache/beam/pull/3189 [BEAM-2334] Fix OutOfMemoryError Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-<Jira issue #>] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/yonromai/beam patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3189.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3189 ---- commit c62c1da808dd8c14bea3c8a391923c7f38c17bd1 Author: Romain Yon <yonro...@users.noreply.github.com> Date: 2017-05-19T20:16:24Z [BEAM-2334] Fix OutOfMemoryError ---- > OutOfMemoryError in RandomAccessData.java:350 > --------------------------------------------- > > Key: BEAM-2334 > URL: https://issues.apache.org/jira/browse/BEAM-2334 > Project: Beam > Issue Type: Bug > Components: runner-core > Affects Versions: 2.0.0 > Reporter: Romain Yon > Assignee: Kenneth Knowles > > Got the following trace: > ``` > Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit > at java.util.Arrays.copyOf(Arrays.java:3236) > at > org.apache.beam.runners.dataflow.util.RandomAccessData.ensureCapacity(RandomAccessData.java:350) > at > org.apache.beam.runners.dataflow.util.RandomAccessData.access$300(RandomAccessData.java:51) > at > org.apache.beam.runners.dataflow.util.RandomAccessData$1.write(RandomAccessData.java:258) > at com.esotericsoftware.kryo.io.Output.flush(Output.java:185) > ``` > However the machine was not out of heap space. Seems like the problem is in > RandomAccessData.java:357: > Arrays.copyOf(buffer, Integer.MAX_VALUE) > Proof: > ``` > scala> Arrays.copyOf(v, Int.MaxValue) > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > at java.util.Arrays.copyOf(Arrays.java:3236) > ... 31 elided > scala> Arrays.copyOf(v, Int.MaxValue-2) > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3236) > ... 31 elided > scala> Arrays.copyOf(v, Int.MaxValue-1) > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > at java.util.Arrays.copyOf(Arrays.java:3236) > ... 31 elided > ``` > I'll submit a corresponding PR. -- This message was sent by Atlassian JIRA (v6.3.15#6346)