[
https://issues.apache.org/jira/browse/BEAM-6351?focusedWorklogId=184888&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184888
]
ASF GitHub Bot logged work on BEAM-6351:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Jan/19 16:35
Start Date: 14/Jan/19 16:35
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #7497: [BEAM-6351]
Divide separate job for "smoke" load test suites
URL: https://github.com/apache/beam/pull/7497#discussion_r247564649
##########
File path: .test-infra/jenkins/job_LoadTests_Java.groovy
##########
@@ -69,6 +53,53 @@ for (testConfiguration in testsConfigurations) {
) {
description(testConfiguration.jobDescription)
commonJobProperties.setTopLevelMainJobProperties(delegate)
- loadTestsBuilder.buildTest(delegate, testConfiguration.jobDescription,
testConfiguration.runner, testConfiguration.jobProperties,
testConfiguration.itClass)
+ loadTestsBuilder.loadTest(delegate, testConfiguration.jobDescription,
testConfiguration.runner, testConfiguration.jobProperties,
testConfiguration.itClass)
+ }
+}
+
+def smokeTestConfigurations = [
+ [
+ title : 'GroupByKey load test Direct',
+ itClass :
'org.apache.beam.sdk.loadtests.GroupByKeyLoadTest',
+ runner : CommonTestProperties.Runner.DIRECT,
+ jobProperties: [
+ publishToBigQuery: true,
+ bigQueryDataset : 'load_test_SMOKE',
+ bigQueryTable : 'direct_gbk',
+ sourceOptions :
'{"numRecords":100000,"splitPointFrequencyRecords":1}',
+ stepOptions :
'{"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true}',
+ fanout : 10,
+ iterations : 1,
+ ]
+ ],
+ [
+ title : 'GroupByKey load test Dataflow',
Review comment:
I think adding smoke tests for other runners is probably useful. Most have
local runners so the test should be fairly quick. Dataflow doesn't have this
option and takes a few minutes to start up, so tests less than a few minutes
will be mostly startup time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 184888)
Time Spent: 10m
Remaining Estimate: 0h
> OutOfMemoryError on DirectRunner while running load tests
> ----------------------------------------------------------
>
> Key: BEAM-6351
> URL: https://issues.apache.org/jira/browse/BEAM-6351
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Kasia Kucharczyk
> Assignee: Lukasz Gajowy
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The GroupByKey Java load test with 1000000000 number of records is failing on
> DirectRunner with OutOfMemory Error. Then the test is aborted after timeout.
> This is [example failing run of the
> job|https://builds.apache.org/job/beam_Java_LoadTests_GroupByKey_Direct_Small_PR/2/].
> The stacktrace is following:
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 18:02:15 at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:204)
> 18:02:15 at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:64)
> 18:02:15 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
> 18:02:15 at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> 18:02:15 at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:75)
> 18:02:15 at
> org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.run(GroupByKeyLoadTest.java:58)
> 18:02:15 at
> org.apache.beam.sdk.loadtests.GroupByKeyLoadTest.main(GroupByKeyLoadTest.java:130)
> 18:02:15 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> 18:02:15 at
> org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:97)
> 18:02:15 at
> org.apache.beam.sdk.util.CoderUtils.decodeFromByteArray(CoderUtils.java:92)
> 18:02:15 at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.<init>(MutationDetectors.java:117)
> 18:02:15 at
> org.apache.beam.sdk.util.MutationDetectors.forValueWithCoder(MutationDetectors.java:44)
> 18:02:15 at
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.add(ImmutabilityCheckingBundleFactory.java:112)
> 18:02:15 at
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:151)
> 18:02:15 at
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
> 18:02:15 at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
> 18:02:15 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 18:02:15 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 18:02:15 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 18:02:15 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 18:02:15 at java.lang.Thread.run(Thread.java:748)
> 19:29:58 Build timed out (after 100 minutes). Marking the build as aborted.
> {code}
> The command to run this test is:
> {code:java}
> gradlew --info
> -PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest
> -Prunner=:beam-runners-direct-java '-PloadTest.args=--publishToBigQuery=false
> --sourceOptions={"numRecords":1000000000,"splitPointFrequencyRecords":1,"keySizeBytes":1,"valueSizeBytes":9,"numHotKeys":0,"hotKeyFraction":0,"seed":123456,"bundleSizeDistribution":{"type":"const","const":42},"forceNumInitialBundles":100,"progressShape":"LINEAR","initializeDelayDistribution":{"type":"const","const":42}}
>
> --stepOptions={"outputRecordsPerInputRecord":1,"preservesInputKeyDistribution":true,"perBundleDelay":10000,"perBundleDelayType":"MIXED","cpuUtilizationInMixedDelay":0.5}
> --fanout=10 --iterations=1 --runner=DirectRunner'
> :beam-sdks-java-load-tests:run
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
