[jira] [Updated] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
[
https://issues.apache.org/jira/browse/BEAM-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels updated BEAM-3089:
-
Fix Version/s: 2.8.0
> Issue with setting the parallelism at client level using Flink runner
> -
>
> Key: BEAM-3089
> URL: https://issues.apache.org/jira/browse/BEAM-3089
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.0.0
> Environment: I am using Flink 1.2.1 running on Docker, with Task
> Managers distributed across different VMs as part of a Docker Swarm.
>Reporter: Thalita Vergilio
>Assignee: Grzegorz Kołakowski
>Priority: Major
> Labels: docker, flink, parallel-deployment
> Fix For: 2.8.0
>
> Attachments: flink-ui-parallelism.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When uploading an Apache Beam application using the Flink Web UI, the
> parallelism set at job submission doesn't get picked up. The same happens
> when submitting a job using the Flink CLI.
> In both cases, the parallelism ends up defaulting to 1.
> When I set the parallelism programmatically within the Apache Beam code, it
> works: {{flinkPipelineOptions.setParallelism(4);}}
> I suspect the root of the problem may be in the
> org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks
> for Flink's GlobalConfiguration, which may not pick up runtime values passed
> to Flink, then defaults to 1 if it doesn't find anything.
> Any ideas on how this could be fixed or worked around? I need to be able to
> change the parallelism dynamically, so the programmatic approach won't really
> work for me, nor will setting the Flink configuration at system level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
[
https://issues.apache.org/jira/browse/BEAM-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grzegorz Kołakowski updated BEAM-3089:
--
Attachment: flink-ui-parallelism.png
> Issue with setting the parallelism at client level using Flink runner
> -
>
> Key: BEAM-3089
> URL: https://issues.apache.org/jira/browse/BEAM-3089
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Affects Versions: 2.0.0
> Environment: I am using Flink 1.2.1 running on Docker, with Task
> Managers distributed across different VMs as part of a Docker Swarm.
>Reporter: Thalita Vergilio
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: docker, flink, parallel-deployment
> Attachments: flink-ui-parallelism.png
>
>
> When uploading an Apache Beam application using the Flink Web UI, the
> parallelism set at job submission doesn't get picked up. The same happens
> when submitting a job using the Flink CLI.
> In both cases, the parallelism ends up defaulting to 1.
> When I set the parallelism programmatically within the Apache Beam code, it
> works: {{flinkPipelineOptions.setParallelism(4);}}
> I suspect the root of the problem may be in the
> org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks
> for Flink's GlobalConfiguration, which may not pick up runtime values passed
> to Flink, then defaults to 1 if it doesn't find anything.
> Any ideas on how this could be fixed or worked around? I need to be able to
> change the parallelism dynamically, so the programmatic approach won't really
> work for me, nor will setting the Flink configuration at system level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
