Hi there beam-team.
I've recently upgraded to 2.28.0 but I'm having problems running tests
which were previously working on 2.27.0.
Here's a description of the situation, and I'm wondering if we might be
able to suggest a way to move forward.
* The project is using python
* The project contains multiple pipelines, which run independent of each
other, but share some transforms.
* Each pipeline has its own set of options, which subclass
`PipelineOptions`, and use the .view_as pattern as described in the
PipelineOptions docs [1]
* Recently, the test runner changed to include a call to
options.get_all_options [2]
* The pipelines individually contain conflicting option names (eg.
log_level)
* When running tests, this now throws an error:
E argparse.ArgumentError: argument --log_level: conflicting
option string: --log_level
When running the individual pipelines, only one subclass is ever imported,
so the conflicting options problem doesn't present itself. I'm wondering if
this is an unexpected side effect of the fix for BEAM-11736 [3][4], or if I
should maybe:
* Create a "CommonOptions" class which is imported independently by both
pipelines
* Use independent names eg. --pipeline1_log_level, --pipeline2_log_level
Looking forward to people's input!
Regards,
Adam
[1]
https://beam.apache.org/releases/pydoc/2.17.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions
[2]
https://github.com/apache/beam/blame/master/sdks/python/apache_beam/runners/direct/direct_runner.py#L124
[3] https://issues.apache.org/jira/browse/BEAM-11736
[4] https://github.com/apache/beam/pull/13723
