[beam] branch master updated (1386b94 -> 49d6efd)

2019-11-15 Thread pabloem
This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1386b94  Merge pull request #10124: [BEAM-8670] Manage environment 
parallelism in DefaultJobBundleFactory
 add 49d6efd  [BEAM-8661] Moving runners to have per-module logger (#10097)

No new revisions were added by this update.

Summary of changes:
 ...dataflow_exercise_streaming_metrics_pipeline.py |  4 ++-
 ...low_exercise_streaming_metrics_pipeline_test.py |  6 ++--
 .../runners/dataflow/dataflow_metrics.py   |  6 ++--
 .../runners/dataflow/dataflow_runner.py| 19 +++-
 .../runners/dataflow/internal/apiclient.py | 26 
 .../runners/dataflow/native_io/iobase.py   |  4 ++-
 .../runners/dataflow/test_dataflow_runner.py   |  4 ++-
 .../apache_beam/runners/direct/direct_runner.py|  7 +++--
 sdks/python/apache_beam/runners/direct/executor.py | 10 ---
 .../runners/direct/transform_evaluator.py  |  4 ++-
 .../runners/interactive/interactive_environment.py | 11 ---
 .../runners/interactive/interactive_runner.py  |  9 --
 sdks/python/apache_beam/runners/job/manager.py |  4 ++-
 .../runners/portability/abstract_job_service.py|  8 +++--
 .../runners/portability/expansion_service_test.py  |  6 ++--
 .../runners/portability/flink_runner.py|  4 ++-
 .../runners/portability/flink_runner_test.py   |  7 +++--
 .../portability/flink_uber_jar_job_server.py   |  8 +++--
 .../runners/portability/fn_api_runner.py   | 32 ++--
 .../runners/portability/fn_api_runner_test.py  |  4 ++-
 .../portability/fn_api_runner_transforms.py|  8 +++--
 .../runners/portability/local_job_service.py   | 12 
 .../runners/portability/local_job_service_main.py  |  4 ++-
 .../runners/portability/portable_metrics.py|  4 ++-
 .../runners/portability/portable_runner.py |  8 +++--
 .../runners/portability/portable_runner_test.py|  8 +++--
 .../apache_beam/runners/portability/stager.py  | 35 --
 .../apache_beam/runners/portability/stager_test.py |  8 +++--
 sdks/python/apache_beam/runners/runner.py  |  6 ++--
 .../apache_beam/runners/worker/bundle_processor.py | 12 
 .../apache_beam/runners/worker/data_plane.py   |  9 +++---
 .../apache_beam/runners/worker/log_handler_test.py |  8 +++--
 .../apache_beam/runners/worker/operations.py   | 13 
 .../apache_beam/runners/worker/sdk_worker.py   | 32 ++--
 .../apache_beam/runners/worker/sdk_worker_main.py  | 22 +++---
 .../apache_beam/runners/worker/sdk_worker_test.py  |  8 +++--
 .../apache_beam/runners/worker/sideinputs.py   |  5 +++-
 .../apache_beam/runners/worker/statecache.py   |  4 ++-
 .../runners/worker/statesampler_test.py|  6 ++--
 .../apache_beam/runners/worker/worker_pool_main.py |  8 +++--
 40 files changed, 245 insertions(+), 158 deletions(-)



[beam] branch master updated (7beb9ee -> 1386b94)

2019-11-15 Thread thw
This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7beb9ee  Merge pull request #10117 [BEAM-8335] Add service and tagged 
output capabilities to TestStream
 add 45dc280  [BEAM-8670] Manage environment parallelism in 
DefaultJobBundleFactory
 add 1386b94  Merge pull request #10124: [BEAM-8670] Manage environment 
parallelism in DefaultJobBundleFactory

No new revisions were added by this update.

Summary of changes:
 .../FlinkExecutableStageContextFactory.java|  21 ++--
 .../control/DefaultExecutableStageContext.java |  56 +--
 .../control/DefaultJobBundleFactory.java   | 107 ++---
 .../control/DefaultExecutableStageContextTest.java |  72 --
 .../control/DefaultJobBundleFactoryTest.java   |  72 ++
 .../SparkExecutableStageContextFactory.java|  24 ++---
 6 files changed, 158 insertions(+), 194 deletions(-)
 delete mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java



[beam] branch master updated (ec3515a -> 7beb9ee)

2019-11-15 Thread robertwb
This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ec3515a  Merge pull request #10069 Separate pydocs generation from py2 
precommit tests.
 add 7beb9ee  Merge pull request #10117 [BEAM-8335] Add service and tagged 
output capabilities to TestStream

No new revisions were added by this update.

Summary of changes:
 {sdks/java/javadoc => model/interactive}/OWNERS|  2 +-
 model/{fn-execution => interactive}/build.gradle   |  8 +--
 .../src/main/proto/beam_interactive_api.proto  | 59 ++
 .../pipeline/src/main/proto/beam_runner_api.proto  | 34 +
 sdks/python/gen_protos.py  |  1 +
 5 files changed, 99 insertions(+), 5 deletions(-)
 copy {sdks/java/javadoc => model/interactive}/OWNERS (86%)
 copy model/{fn-execution => interactive}/build.gradle (81%)
 create mode 100644 model/interactive/src/main/proto/beam_interactive_api.proto



[beam] branch master updated (76b9bdd -> ec3515a)

2019-11-15 Thread robertwb
This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 76b9bdd  [BEAM-8575] Test a customized window fn work as expected
 add 3330e1e  Separate pydocs generation from py2 precommit tests.
 add 6bcf166  Add settings file
 add ec3515a  Merge pull request #10069 Separate pydocs generation from py2 
precommit tests.

No new revisions were added by this update.

Summary of changes:
 build.gradle|  1 +
 sdks/python/test-suites/tox/py2/build.gradle|  5 -
 .../test-suites/tox/pycommon}/build.gradle  | 13 +++--
 settings.gradle |  1 +
 4 files changed, 13 insertions(+), 7 deletions(-)
 copy sdks/{java/build-tools => python/test-suites/tox/pycommon}/build.gradle 
(83%)



[beam] branch master updated (97093f4 -> 76b9bdd)

2019-11-15 Thread lcwik
This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 97093f4  Merge pull request #10131 from Ardagan/LowerExamplesTimeout
 add bda8f62  [BEAM-8575] Test a customized window fn work as expected
 add b41fe1a  fixup
 add 3358b37  fixup
 add c6c8fbd  Merge branch 'master' into addTest
 add 76b9bdd  [BEAM-8575] Test a customized window fn work as expected

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/transforms/window_test.py | 33 +++
 1 file changed, 33 insertions(+)



[beam] branch master updated: Reduce Java Examples Dataflow Precommit timeout

2019-11-15 Thread pabloem
This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 9a9f045  Reduce Java Examples Dataflow Precommit timeout
 new 97093f4  Merge pull request #10131 from Ardagan/LowerExamplesTimeout
9a9f045 is described below

commit 9a9f045a3e70db8f01ecc9f7fefbf51efd99168e
Author: Mikhail Gryzykhin 
AuthorDate: Fri Nov 15 13:11:08 2019 -0800

Reduce Java Examples Dataflow Precommit timeout
---
 .test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow.groovy | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/.test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow.groovy 
b/.test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow.groovy
index 7855085..e4c17ca 100644
--- a/.test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Java_Examples_Dataflow.groovy
@@ -30,7 +30,8 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
   '^examples/java/.*$',
   '^examples/kotlin/.*$',
   '^release/.*$',
-]
+],
+timeoutMins: 30,
 )
 builder.build {
   publishers {



[beam] branch master updated: [BEAM-8379] Cache Eviction

2019-11-15 Thread pabloem
This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 8de55cc  [BEAM-8379] Cache Eviction
 new 1bb581f  Merge pull request #10062 from [BEAM-8379] Cache Eviction for 
Interactive Beam
8de55cc is described below

commit 8de55cca86061a5724e647ffccd2300ce705d793
Author: Ning Kang 
AuthorDate: Wed Nov 13 10:16:57 2019 -0800

[BEAM-8379] Cache Eviction

1. Implemented cache eviction whenever Python interpreter exits.
2. Cache for PCollections is grouped by PCollections as the
Interactive Beam user flow is now data-centric. And cache including
its eviction is  managed by a global interactive environment instance
created/retrieved/reset implicitly by runners in the same main
thread/loop.
3. Unified Python version requirement to 3.6 for interactive features and 
their tests for simplicity. We don't have test suite between py35 and py36 
(both exclusive) anyway, so there is no need to check by 3.5.3 when some mock 
test features are only available in 3.6 and later.
---
 .../display/pcoll_visualization_test.py| 18 +
 .../runners/interactive/interactive_environment.py | 31 +++-
 .../interactive/interactive_environment_test.py| 93 +-
 3 files changed, 122 insertions(+), 20 deletions(-)

diff --git 
a/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
 
b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
index 4628c25..8eefec7 100644
--- 
a/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
+++ 
b/sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
@@ -42,6 +42,8 @@ except ImportError:
 
 @unittest.skipIf(not ie.current_env().is_interactive_ready,
  '[interactive] dependency is not installed.')
+@unittest.skipIf(sys.version_info < (3, 6),
+ 'The tests require at least Python 3.6 to work.')
 class PCollectionVisualizationTest(unittest.TestCase):
 
   def setUp(self):
@@ -56,8 +58,6 @@ class PCollectionVisualizationTest(unittest.TestCase):
 # pylint: disable=range-builtin-not-iterating
 self._pcoll = self._p | 'Create' >> beam.Create(range(1000))
 
-  @unittest.skipIf(sys.version_info < (3, 5, 3),
-   'PCollectionVisualization is supported on Python 3.5.3+.')
   def test_raise_error_for_non_pcoll_input(self):
 class Foo(object):
   pass
@@ -67,8 +67,6 @@ class PCollectionVisualizationTest(unittest.TestCase):
   self.assertTrue('pcoll should be apache_beam.pvalue.PCollection' in
   ctx.exception)
 
-  @unittest.skipIf(sys.version_info < (3, 5, 3),
-   'PCollectionVisualization is supported on Python 3.5.3+.')
   def test_pcoll_visualization_generate_unique_display_id(self):
 pv_1 = pv.PCollectionVisualization(self._pcoll)
 pv_2 = pv.PCollectionVisualization(self._pcoll)
@@ -76,8 +74,6 @@ class PCollectionVisualizationTest(unittest.TestCase):
 self.assertNotEqual(pv_1._overview_display_id, pv_2._overview_display_id)
 self.assertNotEqual(pv_1._df_display_id, pv_2._df_display_id)
 
-  @unittest.skipIf(sys.version_info < (3, 5, 3),
-   'PCollectionVisualization is supported on Python 3.5.3+.')
   @patch('apache_beam.runners.interactive.display.pcoll_visualization'
  '.PCollectionVisualization._to_element_list', lambda x: [1, 2, 3])
   def test_one_shot_visualization_not_return_handle(self):
@@ -91,8 +87,6 @@ class PCollectionVisualizationTest(unittest.TestCase):
 yield [1, 2, 3, 4, 5, 6, 7]
 yield [1, 2, 3, 4, 5, 6, 7, 8]
 
-  @unittest.skipIf(sys.version_info < (3, 5, 3),
-   'PCollectionVisualization is supported on Python 3.5.3+.')
   @patch('apache_beam.runners.interactive.display.pcoll_visualization'
  '.PCollectionVisualization._to_element_list', _mock_to_element_list)
   def test_dynamic_plotting_return_handle(self):
@@ -100,8 +94,6 @@ class PCollectionVisualizationTest(unittest.TestCase):
 self.assertIsInstance(h, timeloop.Timeloop)
 h.stop()
 
-  @unittest.skipIf(sys.version_info < (3, 5, 3),
-   'PCollectionVisualization is supported on Python 3.5.3+.')
   @patch('apache_beam.runners.interactive.display.pcoll_visualization'
  '.PCollectionVisualization._to_element_list', _mock_to_element_list)
   @patch('apache_beam.runners.interactive.display.pcoll_visualization'
@@ -126,10 +118,6 @@ class PCollectionVisualizationTest(unittest.TestCase):
   self.assertIs(kwargs['updating_pv'], updating_pv)
 h.stop()
 
-  # The code being tested supports 3.5.3+. This specific test has assertion
-  # feature that was introduced in 3.6.
-  @unittest.skipIf(sys.version_info < (3, 6),
-   'The test requires Python 3.6+.'

[beam] branch master updated (5de27d5 -> a2b0df9)

2019-11-15 Thread apilloud
This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5de27d5  [BEAM-8151] Swap to create SdkWorkers on demand when 
processing jobs
 add c430c27  Filter push-down for BigQuery (kind of) working.
 add 6314eb2  Added IT test for BigQuery. spotlesApply.
 add 14f3d58  review comments
 add a2b0df9  Merge pull request #10030: [BEAM-8583] [SQL] Big query filter 
push down

No new revisions were added by this update.

Summary of changes:
 .../provider/bigquery/BeamBigQuerySqlDialect.java  | 275 +
 .../sql/meta/provider/bigquery/BigQueryFilter.java | 144 +++
 .../sql/meta/provider/bigquery/BigQueryTable.java  |  63 -
 .../meta/provider/bigquery/BigQueryFilterTest.java | 121 +
 .../provider/bigquery/BigQueryReadWriteIT.java |  77 ++
 5 files changed, 674 insertions(+), 6 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQuerySqlDialect.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryFilter.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryFilterTest.java



[beam] branch release-2.17.0 updated: Revert "Merge pull request #9854 from [BEAM-8457] Label Dataflow jobs from Notebook"

2019-11-15 Thread mikhail
This is an automated email from the ASF dual-hosted git repository.

mikhail pushed a commit to branch release-2.17.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.17.0 by this push:
 new e17ddfd  Revert "Merge pull request #9854 from [BEAM-8457] Label 
Dataflow jobs from Notebook"
 new bd744a3  Merge pull request #9887 from pabloem/release-2.17.0
e17ddfd is described below

commit e17ddfdafe33a03c67977d1f9da49697f59a92f8
Author: Ahmet Altay 
AuthorDate: Thu Oct 24 17:25:22 2019 -0700

Revert "Merge pull request #9854 from [BEAM-8457] Label Dataflow jobs from 
Notebook"

This reverts commit 1a8391da9222ab8d0493b0007bd60bdbeeb5e275.
---
 sdks/python/apache_beam/pipeline.py| 48 --
 .../runners/dataflow/dataflow_runner.py| 10 -
 .../runners/interactive/interactive_runner.py  |  2 +-
 3 files changed, 9 insertions(+), 51 deletions(-)

diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 5574a82..a776d30 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -171,10 +171,6 @@ class Pipeline(object):
 # If a transform is applied and the full label is already in the set
 # then the transform will have to be cloned with a new label.
 self.applied_labels = set()
-# A boolean value indicating whether the pipeline is created in an
-# interactive environment such as interactive notebooks. Initialized as
-# None. The value is set ad hoc when `pipeline.run()` is invoked.
-self.interactive = None
 
   @property
   @deprecated(since='First stable release',
@@ -399,56 +395,28 @@ class Pipeline(object):
 for override in replacements:
   self._check_replacement(override)
 
-  def run(self, test_runner_api=True, runner=None, options=None,
-  interactive=None):
-"""Runs the pipeline. Returns whatever our runner returns after running.
-
-If another runner instance and options are provided, that runner will
-execute the pipeline with the given options. If either of them is not set,
-a ValueError is raised. The usage is similar to directly invoking
-`runner.run_pipeline(pipeline, options)`.
-Additionally, an interactive field can be set to override the pipeline's
-self.interactive field to mark current pipeline as being initiated from an
-interactive environment.
-"""
-from apache_beam.runners.interactive import interactive_runner
-if interactive:
-  self.interactive = interactive
-elif isinstance(self.runner, interactive_runner.InteractiveRunner):
-  self.interactive = True
-else:
-  self.interactive = False
-runner_in_use = self.runner
-options_in_use = self._options
-if runner and options:
-  runner_in_use = runner
-  options_in_use = options
-elif not runner and options:
-  raise ValueError('Parameter runner is not given when parameter options '
-   'is given.')
-elif not options and runner:
-  raise ValueError('Parameter options is not given when parameter runner '
-   'is given.')
+  def run(self, test_runner_api=True):
+"""Runs the pipeline. Returns whatever our runner returns after running."""
+
 # When possible, invoke a round trip through the runner API.
 if test_runner_api and self._verify_runner_api_compatible():
   return Pipeline.from_runner_api(
   self.to_runner_api(use_fake_coders=True),
-  runner_in_use,
-  options_in_use).run(test_runner_api=False,
-  interactive=self.interactive)
+  self.runner,
+  self._options).run(False)
 
-if options_in_use.view_as(TypeOptions).runtime_type_check:
+if self._options.view_as(TypeOptions).runtime_type_check:
   from apache_beam.typehints import typecheck
   self.visit(typecheck.TypeCheckVisitor())
 
-if options_in_use.view_as(SetupOptions).save_main_session:
+if self._options.view_as(SetupOptions).save_main_session:
   # If this option is chosen, verify we can pickle the main session early.
   tmpdir = tempfile.mkdtemp()
   try:
 pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))
   finally:
 shutil.rmtree(tmpdir)
-return runner_in_use.run_pipeline(self, options_in_use)
+return self.runner.run_pipeline(self, self._options)
 
   def __enter__(self):
 return self
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index f57be74..4928550 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -364,16 +364,6 @@ class DataflowRunner(PipelineRunner):
 
   def run_pipeline(self, pipeline, options):
 """Remotely executes entire pipeline or par

[beam] branch master updated (d16f2a9 -> 5de27d5)

2019-11-15 Thread lcwik
This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d16f2a9  Bump python precommit timeout
 add 33cc30e  [BEAM-8151] Swap to create SdkWorkers on demand when 
processing jobs
 add 5de27d5  [BEAM-8151] Swap to create SdkWorkers on demand when 
processing jobs

No new revisions were added by this update.

Summary of changes:
 .../runners/portability/fn_api_runner.py   |   8 +-
 .../runners/portability/fn_api_runner_test.py  |  12 +--
 .../runners/portability/portable_runner.py |   1 -
 .../apache_beam/runners/worker/sdk_worker.py   | 103 ++---
 .../apache_beam/runners/worker/sdk_worker_main.py  |  30 --
 .../runners/worker/sdk_worker_main_test.py |  47 +-
 .../apache_beam/runners/worker/sdk_worker_test.py  |   7 +-
 .../apache_beam/runners/worker/worker_pool_main.py |  16 +---
 sdks/python/apache_beam/transforms/environments.py |  24 ++---
 .../apache_beam/transforms/environments_test.py|   9 +-
 .../apache_beam/utils/thread_pool_executor_test.py |   8 ++
 11 files changed, 55 insertions(+), 210 deletions(-)



[beam] branch master updated (85248f3 -> d16f2a9)

2019-11-15 Thread lcwik
This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 85248f3  [BEAM-8557] Add log for the dropped unknown response
 add de9cd0b  Bump python precommit timeout
 add d16f2a9  Bump python precommit timeout

No new revisions were added by this update.

Summary of changes:
 .test-infra/jenkins/job_PreCommit_Python.groovy | 1 +
 1 file changed, 1 insertion(+)



[beam] branch master updated (8f54d07 -> 85248f3)

2019-11-15 Thread lcwik
This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8f54d07  Merge pull request #10052 from robertwb/no-gauge
 add 4013214  [BEAM-8557] Add log for the dropped unknown response
 add 85248f3  [BEAM-8557] Add log for the dropped unknown response

No new revisions were added by this update.

Summary of changes:
 .../fnexecution/control/FnApiControlClient.java| 22 --
 .../harness/state/BeamFnStateGrpcClientCache.java  | 14 --
 2 files changed, 20 insertions(+), 16 deletions(-)



[beam] branch master updated: Add option to test metrics on runners without gauge support.

2019-11-15 Thread robertwb
This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 3f7cf68  Add option to test metrics on runners without gauge support.
 new 8f54d07  Merge pull request #10052 from robertwb/no-gauge
3f7cf68 is described below

commit 3f7cf682afdcfc5912cad9359cc1ca8a756f2399
Author: Robert Bradshaw 
AuthorDate: Fri Nov 8 17:14:07 2019 -0800

Add option to test metrics on runners without gauge support.
---
 .../apache_beam/runners/portability/fn_api_runner_test.py | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py 
b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index 2204a24..5bd5492 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -608,7 +608,7 @@ class FnApiRunnerTest(unittest.TestCase):
   pcoll_b = p | 'b' >> beam.Create(['b'])
   assert_that((pcoll_a, pcoll_b) | First(), equal_to(['a']))
 
-  def test_metrics(self):
+  def test_metrics(self, check_gauge=True):
 p = self.create_pipeline()
 
 counter = beam.metrics.Metrics.counter('ns', 'counter')
@@ -630,14 +630,17 @@ class FnApiRunnerTest(unittest.TestCase):
 c2, = 
res.metrics().query(beam.metrics.MetricsFilter().with_step('count2'))[
 'counters']
 self.assertEqual(c2.committed, 4)
+
 dist, = 
res.metrics().query(beam.metrics.MetricsFilter().with_step('dist'))[
 'distributions']
-gaug, = res.metrics().query(
-beam.metrics.MetricsFilter().with_step('gauge'))['gauges']
 self.assertEqual(
 dist.committed.data, beam.metrics.cells.DistributionData(4, 2, 1, 3))
 self.assertEqual(dist.committed.mean, 2.0)
-self.assertEqual(gaug.committed.value, 3)
+
+if check_gauge:
+  gaug, = res.metrics().query(
+  beam.metrics.MetricsFilter().with_step('gauge'))['gauges']
+  self.assertEqual(gaug.committed.value, 3)
 
   def test_callbacks_with_exception(self):
 elements_list = ['1', '2']



[beam] branch master updated (780ef7a -> 2271fa0)

2019-11-15 Thread robertwb
This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 780ef7a  Merge pull request #10013: [BEAM-8554] Use 
WorkItemCommitRequest protobuf fields to signal that …
 add cea8d03  [BEAM-8655] Strengthen trigger transcript test to use 
multiple keys.
 add 2efca75  [BEAM-8655] Run trigger transcript tests with combiner as 
well as GroupByKey.
 add 63d8020  [BEAM-8655] Run the subset of trigger tests that make sense 
in batch mode.
 add 2271fa0  Merge pull request #10096 [BEAM-8655] Strengthen trigger 
transcript tests.

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/transforms/trigger_test.py | 187 -
 1 file changed, 149 insertions(+), 38 deletions(-)