/cc +Valentyn Tymofieiev <[email protected]> - might have an idea. On Sat, Feb 27, 2021 at 2:42 AM adam moore <[email protected]> wrote:
> Hi there beam-team. > > I've recently upgraded to 2.28.0 but I'm having problems running tests > which were previously working on 2.27.0. > > Here's a description of the situation, and I'm wondering if we might be > able to suggest a way to move forward. > > * The project is using python > * The project contains multiple pipelines, which run independent of each > other, but share some transforms. > * Each pipeline has its own set of options, which subclass > `PipelineOptions`, and use the .view_as pattern as described in the > PipelineOptions docs [1] > * Recently, the test runner changed to include a call to > options.get_all_options [2] > * The pipelines individually contain conflicting option names (eg. > log_level) > * When running tests, this now throws an error: > > E argparse.ArgumentError: argument --log_level: conflicting > option string: --log_level > > When running the individual pipelines, only one subclass is ever imported, > so the conflicting options problem doesn't present itself. I'm wondering if > this is an unexpected side effect of the fix for BEAM-11736 [3][4], or if I > should maybe: > > * Create a "CommonOptions" class which is imported independently by both > pipelines > * Use independent names eg. --pipeline1_log_level, --pipeline2_log_level > > Looking forward to people's input! > > Regards, > > Adam > > > [1] > https://beam.apache.org/releases/pydoc/2.17.0/apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions > [2] > https://github.com/apache/beam/blame/master/sdks/python/apache_beam/runners/direct/direct_runner.py#L124 > [3] https://issues.apache.org/jira/browse/BEAM-11736 > [4] https://github.com/apache/beam/pull/13723 >
