[
https://issues.apache.org/jira/browse/BEAM-8256?focusedWorklogId=313616&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-313616
]
ASF GitHub Bot logged work on BEAM-8256:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Sep/19 10:51
Start Date: 17/Sep/19 10:51
Worklog Time Spent: 10m
Work Description: lgajowy commented on pull request #9596: [BEAM-8256]
Set fixed number of workers for Java IOITs
URL: https://github.com/apache/beam/pull/9596#discussion_r325101445
##########
File path: .test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
##########
@@ -28,7 +28,9 @@ def jobs = [
pipelineOptions : [
bigQueryDataset: 'beam_performance',
bigQueryTable : 'textioit_results',
- numberOfRecords: '1000000'
+ numberOfRecords: '1000000',
+ maxNumWorkers : '5',
Review comment:
Other (also important) thing: what is the reasoning behind setting 5 workers
everywhere? Maybe we're fine with 1 worker or 2 (to test parallelism as well)?
WDYT? What things should we consider when choosing the right amount?
We never really discussed this so I think this is the right moment and about
time. :)
@ibzib @chamikaramj @pabloem what is your opinion about this? Could you
share your insights?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 313616)
Time Spent: 40m (was: 0.5h)
> Set fixed number of workers for File-based IOITs
> ------------------------------------------------
>
> Key: BEAM-8256
> URL: https://issues.apache.org/jira/browse/BEAM-8256
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Michal Walenia
> Assignee: Michal Walenia
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Autoscaling is a feature of google cloud dataflow runner that adds/removes
> worker nodes dynamically as the job runs. It can behave in a different way
> creating different test (runtime) results in consequent runs. In integration
> tests (such as IOIT but others also apply) we don't need such nondeterminism
> and it's best to have a fixed number of workers for every test execution.
> IOITs use autoscaling but they shouldn't. This issue was created to disable
> it and set a fixed number of workers.
> Side note: autoscaling is already disabled in Nexmark and load tests of core
> operations.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
