Re: python post-commit failures

2018-10-08 Thread Mikhail Gryzykhin
Is it possible to add component/unit test for this case? I believe we should aim for all precommits to be executable on isolated singlebox. Regards, --Mikhail On Mon, Oct 8, 2018, 09:27 Udi Meiri It's the current status: I believe having a basic wordcount integration > test in precommit would h

Re: python post-commit failures

2018-10-08 Thread Udi Meiri
It's the current status: I believe having a basic wordcount integration test in precommit would have caught this issue, since it seems to have broken all tests using the Dataflow service. On Sun, Oct 7, 2018 at 9:06 PM Kenneth Knowles wrote: > Out of curiosity - is it a logical necessity, or jus

Re: python post-commit failures

2018-10-07 Thread Kenneth Knowles
Out of curiosity - is it a logical necessity, or just current status, that one needs to run a full job to catch this? On Sat, Oct 6, 2018 at 4:26 AM Maximilian Michels wrote: > My changes to the Python option parsing broke the PostCommit. PreCommit > passed, as well as the Portable Runner tests.

Re: python post-commit failures

2018-10-06 Thread Maximilian Michels
My changes to the Python option parsing broke the PostCommit. PreCommit passed, as well as the Portable Runner tests. Sorry about that. +1 It would be great to have some more basic integration tests in the PreCommit. That will give us more confidence before merge without always running the Pos

Re: python post-commit failures

2018-10-05 Thread Ahmet Altay
On Fri, Oct 5, 2018 at 1:51 PM, Udi Meiri wrote: > I was sure that we ran some basic Dataflow integration tests in > pre-commit, and that they should have caught this issue. > But then I remembered that we only have those in Java SDK. > I opened this bug to add end-to-end tests to Python pre-comm

Re: python post-commit failures

2018-10-05 Thread Udi Meiri
I was sure that we ran some basic Dataflow integration tests in pre-commit, and that they should have caught this issue. But then I remembered that we only have those in Java SDK. I opened this bug to add end-to-end tests to Python pre-commits as well: https://issues.apache.org/jira/browse/BEAM-505

Re: python post-commit failures

2018-10-05 Thread Thomas Weise
Fixed. Can someone please take a look at the usage of the --beam_plugins flag in the Dataflow runner so that we can address the root cause? We can probably do more to avoid Friday Python post commit excitement. In this case, extra checking was done pre-merge by running the Python VR tests for Flin

Re: python post-commit failures

2018-10-05 Thread Udi Meiri
More details in https://issues.apache.org/jira/browse/BEAM-5442 On Fri, Oct 5, 2018 at 10:26 AM Udi Meiri wrote: > I'm seeing these errors at least in one test: > "Python sdk harness failed: > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/dist-packages/apache_beam/runne

Re: python post-commit failures

2018-10-05 Thread Udi Meiri
I'm seeing these errors at least in one test: "Python sdk harness failed: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker_main.py", line 133, in main sdk_pipeline_options.get_all_options(drop_default=True)) File "/usr/loc

Re: python post-commit failures

2018-10-05 Thread Boyuan Zhang
Seems like tests failed: test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) -> Bigquery table not found test_game_stats_it (apache_beam.examples.complete.game.game_stats_it_test.GameStatsIT) -> Bigquery table not found streaming related tests -> Assertion e

python post-commit failures

2018-10-05 Thread Udi Meiri
I'm seeing post-commit failures in :beam-sdks-python:postCommitITTests: https://builds.apache.org/job/beam_PostCommit_Python_Verify/6181/console https://builds.apache.org/job/beam_PostCommit_Python_Verify/6182/console smime.p7s Description: S/MIME Cryptographic Signature