Thalita Vergilio created BEAM-3089:
--------------------------------------
Summary: Issue with setting the parallelism at client level using
Flink runner
Key: BEAM-3089
URL: https://issues.apache.org/jira/browse/BEAM-3089
Project: Beam
Issue Type: Bug
Components: runner-flink
Affects Versions: 2.0.0
Environment: I am using Flink 1.2.1 running on Docker, with Task
Managers distributed across different VMs as part of a Docker Swarm.
Reporter: Thalita Vergilio
Assignee: Aljoscha Krettek
When uploading an Apache Beam application using the Flink Web UI, the
parallelism set at job submission doesn't get picked up. The same happens when
submitting a job using the Flink CLI.
In both cases, the parallelism ends up defaulting to 1.
When I set the parallelism programmatically within the Apache Beam code, it
works: {{flinkPipelineOptions.setParallelism(4);}}
I suspect the root of the problem may be in the
org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks for
Flink's GlobalConfiguration, which may not pick up runtime values passed to
Flink, then defaults to 1 if it doesn't find anything.
Any ideas on how this could be fixed or worked around? I need to be able to
change the parallelism dynamically, so the programmatic approach won't really
work for me, nor will setting the Flink configuration at system level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
