[
https://issues.apache.org/jira/browse/BEAM-4122?focusedWorklogId=92667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-92667
]
ASF GitHub Bot logged work on BEAM-4122:
----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Apr/18 16:27
Start Date: 19/Apr/18 16:27
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #5171: [BEAM-4122] Tune
Gradle resource usage on Jenkins
URL: https://github.com/apache/beam/pull/5171
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/jenkins/common_job_properties.groovy
b/.test-infra/jenkins/common_job_properties.groovy
index beb6e2ac467..aba4ff51c8c 100644
--- a/.test-infra/jenkins/common_job_properties.groovy
+++ b/.test-infra/jenkins/common_job_properties.groovy
@@ -170,11 +170,10 @@ class common_job_properties {
"--info",
// Continue the build even if there is a failure to show as many potential
failures as possible.
'--continue',
- // Until we verify the build cache is working appropriately, force
rerunning all tasks
- '--rerun-tasks',
- // Disable daemon, which helps ensure hermetic environment at small
startup performance penalty.
- // This needs to be disabled if we move to incremental builds.
- "--no-daemon",
+ // Limit background number of workers to prevent exhausting machine memory.
+ // Jenkins machines have 15GB memory, and run 2 jobs in parallel; workers
are configured with
+ // JVM max heap size 3.5GB. So 2 jobs * 2 workers * 3.5GB heap = 14GB
+ '--max-workers=2',
]
static void setGradleSwitches(context) {
diff --git
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
index 31d3685f846..07f3b6ca6c3 100644
---
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
+++
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
@@ -48,12 +48,6 @@ job('beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle') {
rootBuildScriptDir(common_job_properties.checkoutDir)
tasks(':beam-runners-google-cloud-dataflow-java:validatesRunner')
common_job_properties.setGradleSwitches(delegate)
- // Increase parallel worker threads above processor limit since most
time is
- // spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are
slow
- // because each one launches a Dataflow job with about 3 mins of
overhead.
- // 3 x num_cores strikes a good balance between maxing out parallelism
without
- // overloading the machines.
- switches("--max-workers=${3 * Runtime.runtime.availableProcessors()}")
}
}
}
diff --git a/gradle.properties b/gradle.properties
index 4b34251cea1..c0034d2d0fd 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -15,6 +15,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
+org.gradle.caching=true
org.gradle.parallel=true
org.gradle.configureondemand=true
offlineRepositoryRoot=offline-repository
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 92667)
Time Spent: 2h (was: 1h 50m)
> Tune Gradle resource usage on Jenkins
> -------------------------------------
>
> Key: BEAM-4122
> URL: https://issues.apache.org/jira/browse/BEAM-4122
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
> Reporter: Scott Wegner
> Assignee: Scott Wegner
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The new Gradle build seems to be exhausting memory on the Jenkins machines,
> causing them to lose their connection and die. This has made our tests
> extremely flaky.
> We need to tune the Gradle build such that it does not exhaust Jenkins
> machine resources.
> A [Gradle build
> scan|https://scans.gradle.com/s/ehue5xqj36zeo/infrastructure] shows some
> information about the Jenkins VM:
> * Operating system Linux 3.19.0-79-generic
> * CPU cores 4 cores
> * Max Gradle workers 4 workers
> * Java runtime Oracle Java(TM) SE Runtime Environment 1.8.0_152-b16
> * Java VM Oracle Java HotSpot(TM) 64-Bit Server VM 25.152-b16 (mixed mode)
> * Max JVM memory heap size 3506 MB
> From the [Jenkins Load
> Statistics|https://builds.apache.org/computer/beam3/load-statistics] page, it
> appears that each machine has 2 executors.
> Some resources for tuning memory:
> * [https://wiki.jenkins.io/display/JENKINS/I%27m+getting+OutOfMemoryError]
> * [https://guides.gradle.org/performance/]
> *
> [https://docs.gradle.org/current/userguide/build_environment.html#sec:configuring_jvm_memory]
>
> *
> [https://docs.gradle.org/current/userguide/command_line_interface.html#sec:command_line_performance]
>
>
> We may also be able to improve performance by using Gradle features to remove
> redundant work:
> * [Build Cache|https://docs.gradle.org/current/userguide/build_cache.html]:
> re-use up-to-date outputs from previous builds
> * [Gradle
> Daemon|https://docs.gradle.org/current/userguide/gradle_daemon.html]: re-use
> worker process and cache output status
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
