On Fri, Oct 5, 2018 at 1:51 PM, Udi Meiri <eh...@google.com> wrote: > I was sure that we ran some basic Dataflow integration tests in > pre-commit, and that they should have caught this issue. > But then I remembered that we only have those in Java SDK. > I opened this bug to add end-to-end tests to Python pre-commits as well: > https://issues.apache.org/jira/browse/BEAM-5058 >
+1 > > The same goes for Flink: we should be running wordcount and > wordcount_streaming integration tests as part of pre-commit tests. > > On Fri, Oct 5, 2018 at 1:37 PM Thomas Weise <t...@apache.org> wrote: > >> Fixed. Can someone please take a look at the usage of the --beam_plugins >> flag in the Dataflow runner so that we can address the root cause? >> >> We can probably do more to avoid Friday Python post commit excitement. In >> this case, extra checking was done pre-merge by running the Python VR tests >> for Flink, but the failure occurred with the Dataflow runner. >> > It would be good to have a list of what post commit test are available, what do they test and what is the keyword to trigger them from a PR. > >> The changes were pipeline options related, so (pre-existing) test >> coverage should have been better. >> >> But beyond that, we can probably make it easier for contributors and >> reviewers to know what extra checks are available and possibly appropriate >> to run pre-commit. Should we add some pointers to >> https://beam.apache.org/contribute/testing/#pre-commit or is there a >> better place? >> >> Thanks >> >> >> >> >> On Fri, Oct 5, 2018 at 10:38 AM Udi Meiri <eh...@google.com> wrote: >> >>> More details in https://issues.apache.org/jira/browse/BEAM-5442 >>> >>> On Fri, Oct 5, 2018 at 10:26 AM Udi Meiri <eh...@google.com> wrote: >>> >>>> I'm seeing these errors at least in one test: >>>> "Python sdk harness failed: >>>> Traceback (most recent call last): >>>> File "/usr/local/lib/python2.7/dist-packages/apache_beam/ >>>> runners/worker/sdk_worker_main.py", line 133, in main >>>> sdk_pipeline_options.get_all_options(drop_default=True)) >>>> File >>>> "/usr/local/lib/python2.7/dist-packages/apache_beam/options/pipeline_options.py", >>>> line 224, in get_all_options >>>> parser.add_argument(arg.split('=', 1)[0], nargs='?') >>>> File "/usr/lib/python2.7/argparse.py", line 1308, in add_argument >>>> return self._add_action(action) >>>> File "/usr/lib/python2.7/argparse.py", line 1682, in _add_action >>>> self._optionals._add_action(action) >>>> File "/usr/lib/python2.7/argparse.py", line 1509, in _add_action >>>> action = super(_ArgumentGroup, self)._add_action(action) >>>> File "/usr/lib/python2.7/argparse.py", line 1322, in _add_action >>>> self._check_conflict(action) >>>> File "/usr/lib/python2.7/argparse.py", line 1460, in _check_conflict >>>> conflict_handler(action, confl_optionals) >>>> File "/usr/lib/python2.7/argparse.py", line 1467, in >>>> _handle_conflict_error >>>> raise ArgumentError(action, message % conflict_string) >>>> ArgumentError: argument --beam_plugins: conflicting option string(s): >>>> --beam_plugins" >>>> >>>> This looks like https://github.com/apache/beam/pull/6557 >>>> >>>> On Fri, Oct 5, 2018 at 9:41 AM Boyuan Zhang <boyu...@google.com> wrote: >>>> >>>>> Seems like tests failed: >>>>> test_leader_board_it (apache_beam.examples. >>>>> complete.game.leader_board_it_test.LeaderBoardIT) -> Bigquery table >>>>> not found >>>>> test_game_stats_it >>>>> (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) >>>>> -> Bigquery table not found >>>>> streaming related tests -> Assertion errors >>>>> >>>>> On Fri, Oct 5, 2018 at 9:33 AM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> I'm seeing post-commit failures in :beam-sdks-python: >>>>>> postCommitITTests: >>>>>> https://builds.apache.org/job/beam_PostCommit_Python_Verify/ >>>>>> 6181/console >>>>>> https://builds.apache.org/job/beam_PostCommit_Python_Verify/ >>>>>> 6182/console >>>>>> >>>>>> >>>>>>
