ODriscollBoys opened a new pull request, #40498:
URL: https://github.com/apache/beam/pull/40498
`get_all_options(retain_unknown_options=True)`, which `Pipeline.run()` calls
for every pipeline, registers each unknown flag with argparse as a boolean or a
string option and then parses the flags again. It decided which by looking at
the next entry of `unknown_args`, but known arguments are not in that list, so
two entries next to each other there need not be next to each other in the
flags. For example, with
```
--unknown_flag --runner=DirectRunner stray
```
`unknown_args` is `['--unknown_flag', 'stray']`, so `--unknown_flag` was
registered as taking a value and the second parse exited with `argument
--unknown_flag: expected one argument`. The pipeline never ran.
The arity of an unknown flag is now decided by the argument that follows it
in the original flags. A `--flag=value` flag is also always registered by its
name now; before, if a positional argument followed it, the whole
`--flag=value` string was registered as the option name.
Added tests for an unknown flag followed by a known option (with `=` and
with a separate value), a binary unknown flag after a known flag, and
`--flag=value` followed by a positional argument. Three of them fail on master.
All tests under `apache_beam/options/` pass, as do `pipeline_test.py`,
`apiclient_test.py`, `dataflow_runner_test.py` and `sdk_worker_main_test.py`. I
also ran a small DirectRunner pipeline with the flags above: it exits with the
argparse error on master and runs to completion with this change.
fixes #22434
------------------------
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable. This will automatically add a link to the
pull request in the issue. If you would like the issue to automatically close
on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
