[
https://issues.apache.org/jira/browse/BEAM-8256?focusedWorklogId=315198&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315198
]
ASF GitHub Bot logged work on BEAM-8256:
----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Sep/19 17:07
Start Date: 19/Sep/19 17:07
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #9596: [BEAM-8256]
Set fixed number of workers for Java IOITs
URL: https://github.com/apache/beam/pull/9596#discussion_r326285636
##########
File path: .test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy
##########
@@ -28,7 +28,9 @@ def jobs = [
pipelineOptions : [
bigQueryDataset: 'beam_performance',
bigQueryTable : 'textioit_results',
- numberOfRecords: '1000000'
+ numberOfRecords: '1000000',
+ maxNumWorkers : '5',
Review comment:
1. I think it makes sense to deactivate autoscaling, because the behavior
would be non-deterministic if we have autoscaling
2. I think the number of workers should be chosen to make the pipeline run
relatively quickly, and hopefully be more than one, so we measure any effects
from separate workers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 315198)
Time Spent: 50m (was: 40m)
> Set fixed number of workers for File-based IOITs
> ------------------------------------------------
>
> Key: BEAM-8256
> URL: https://issues.apache.org/jira/browse/BEAM-8256
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Michal Walenia
> Assignee: Michal Walenia
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Autoscaling is a feature of google cloud dataflow runner that adds/removes
> worker nodes dynamically as the job runs. It can behave in a different way
> creating different test (runtime) results in consequent runs. In integration
> tests (such as IOIT but others also apply) we don't need such nondeterminism
> and it's best to have a fixed number of workers for every test execution.
> IOITs use autoscaling but they shouldn't. This issue was created to disable
> it and set a fixed number of workers.
> Side note: autoscaling is already disabled in Nexmark and load tests of core
> operations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
