I was sure that we ran some basic Dataflow integration tests in pre-commit, and that they should have caught this issue. But then I remembered that we only have those in Java SDK. I opened this bug to add end-to-end tests to Python pre-commits as well: https://issues.apache.org/jira/browse/BEAM-5058
The same goes for Flink: we should be running wordcount and wordcount_streaming integration tests as part of pre-commit tests. On Fri, Oct 5, 2018 at 1:37 PM Thomas Weise <t...@apache.org> wrote: > Fixed. Can someone please take a look at the usage of the --beam_plugins > flag in the Dataflow runner so that we can address the root cause? > > We can probably do more to avoid Friday Python post commit excitement. In > this case, extra checking was done pre-merge by running the Python VR tests > for Flink, but the failure occurred with the Dataflow runner. > > The changes were pipeline options related, so (pre-existing) test coverage > should have been better. > > But beyond that, we can probably make it easier for contributors and > reviewers to know what extra checks are available and possibly appropriate > to run pre-commit. Should we add some pointers to > https://beam.apache.org/contribute/testing/#pre-commit or is there a > better place? > > Thanks > > > > > On Fri, Oct 5, 2018 at 10:38 AM Udi Meiri <eh...@google.com> wrote: > >> More details in https://issues.apache.org/jira/browse/BEAM-5442 >> >> On Fri, Oct 5, 2018 at 10:26 AM Udi Meiri <eh...@google.com> wrote: >> >>> I'm seeing these errors at least in one test: >>> "Python sdk harness failed: >>> Traceback (most recent call last): >>> File >>> "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py", >>> line 133, in main >>> sdk_pipeline_options.get_all_options(drop_default=True)) >>> File >>> "/usr/local/lib/python2.7/dist-packages/apache_beam/options/pipeline_options.py", >>> line 224, in get_all_options >>> parser.add_argument(arg.split('=', 1)[0], nargs='?') >>> File "/usr/lib/python2.7/argparse.py", line 1308, in add_argument >>> return self._add_action(action) >>> File "/usr/lib/python2.7/argparse.py", line 1682, in _add_action >>> self._optionals._add_action(action) >>> File "/usr/lib/python2.7/argparse.py", line 1509, in _add_action >>> action = super(_ArgumentGroup, self)._add_action(action) >>> File "/usr/lib/python2.7/argparse.py", line 1322, in _add_action >>> self._check_conflict(action) >>> File "/usr/lib/python2.7/argparse.py", line 1460, in _check_conflict >>> conflict_handler(action, confl_optionals) >>> File "/usr/lib/python2.7/argparse.py", line 1467, in >>> _handle_conflict_error >>> raise ArgumentError(action, message % conflict_string) >>> ArgumentError: argument --beam_plugins: conflicting option string(s): >>> --beam_plugins" >>> >>> This looks like https://github.com/apache/beam/pull/6557 >>> >>> On Fri, Oct 5, 2018 at 9:41 AM Boyuan Zhang <boyu...@google.com> wrote: >>> >>>> Seems like tests failed: >>>> test_leader_board_it >>>> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) -> >>>> Bigquery table not found >>>> test_game_stats_it >>>> (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) -> >>>> Bigquery table not found >>>> streaming related tests -> Assertion errors >>>> >>>> On Fri, Oct 5, 2018 at 9:33 AM Udi Meiri <eh...@google.com> wrote: >>>> >>>>> I'm seeing post-commit failures in :beam-sdks-python:postCommitITTests: >>>>> >>>>> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/console >>>>> >>>>> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6182/console >>>>> >>>>> >>>>>
smime.p7s
Description: S/MIME Cryptographic Signature
