Kasia Kucharczyk created BEAM-6351:
--------------------------------------
Summary: OutOfMemoryError on DirectRunner while running load tests
Key: BEAM-6351
URL: https://issues.apache.org/jira/browse/BEAM-6351
Project: Beam
Issue Type: Bug
Components: testing
Reporter: Kasia Kucharczyk
Assignee: Jason Kuster
The GroupByKey Java load test with 1000000000 number of records is failing on
DirectRunner with OutOfMemory Error. Then the test is aborted after timeout.
This is [example failing run of the
job|https://builds.apache.org/job/beam_Java_LoadTests_GroupByKey_Direct_Small_PR/2/].
The stacktrace is following:
{code:java}
Exception in thread "main" java.lang.RuntimeException:
java.lang.OutOfMemoryError: GC overhead limit exceeded
18:02:15 at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:204)
18:02:15 at
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:64)
18:02:15 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
18:02:15 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
18:02:15 at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:75)
18:02:15 at
org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:58)
18:02:15 at
org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:130)
18:02:15 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
18:02:15 at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:97)
18:02:15 at
org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
18:02:15 at
org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init>(MutationDetectors.java:117)
18:02:15 at
org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:44)
18:02:15 at
org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
18:02:15 at
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:151)
18:02:15 at
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
18:02:15 at
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
18:02:15 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
18:02:15 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
18:02:15 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
18:02:15 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
18:02:15 at java.lang.Thread.run(Thread.java:748)
19:29:58 Build timed out (after 100 minutes). Marking the build as aborted.
{code}
The command to run this test is:
{code:java}
gradlew --info
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest
-Prunner=:beam-runners-direct-java '-PloadTest.args=--publishToBigQuery=false
--sourceOptions={"numRecords":1000000000,"splitPointFrequencyRecords":1,"keySizeBytes":1,"valueSizeBytes":9,"numHotKeys":0,"hotKeyFraction":0,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}}
--stepOptions={"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true,"perBundleDelay":10000,"perBundleDelayType":"MIXED","cpuUtilizationInMixedDelay":0.5}
--fanout=10 --iterations=1 --runner=DirectRunner'
:beam-sdks-java-load-tests:run
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
