[beam] branch master updated (c6cf20b -> f358024)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c6cf20b [BEAM-3719] Adds support for reading side-inputs from SDFs add c0039c3 Avoid warning in our default runner. new f358024 Merge pull request #4779 Avoid warning in our default runner. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/direct/direct_runner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4779 Avoid warning in our default runner.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit f3580248e0fd6e2f5c57e0eeabd9d3e7ba72b461 Merge: c6cf20b c0039c3 Author: Robert Bradshaw AuthorDate: Thu Mar 1 13:40:26 2018 -0800 Merge pull request #4779 Avoid warning in our default runner. Avoid warning in our default runner. sdks/python/apache_beam/runners/direct/direct_runner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4722 from Add MultiMap side inputs to Python SDK.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 67109452fed9bd48155c2858b9065d0ce6d1da43 Merge: a53968d 769351f Author: Robert Bradshaw AuthorDate: Thu Feb 22 13:10:00 2018 -0800 Merge pull request #4722 from Add MultiMap side inputs to Python SDK. Add MultiMap side inputs to Python SDK. sdks/python/apache_beam/coders/coder_impl.pxd | 2 + sdks/python/apache_beam/coders/coder_impl.py | 8 sdks/python/apache_beam/pvalue.py | 28 ++ .../runners/portability/fn_api_runner.py | 31 +++ .../runners/portability/fn_api_runner_test.py | 9 + .../apache_beam/runners/worker/bundle_processor.py | 44 +++--- 6 files changed, 109 insertions(+), 13 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (a53968d -> 6710945)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a53968d Merge pull request #4728: [BEAM-3632] Table partioning in DynamicDestination is lost with project is not set in Table Destination add 769351f Add MultiMap side inputs to Python SDK. new 6710945 Merge pull request #4722 from Add MultiMap side inputs to Python SDK. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/coders/coder_impl.pxd | 2 + sdks/python/apache_beam/coders/coder_impl.py | 8 sdks/python/apache_beam/pvalue.py | 28 ++ .../runners/portability/fn_api_runner.py | 31 +++ .../runners/portability/fn_api_runner_test.py | 9 + .../apache_beam/runners/worker/bundle_processor.py | 44 +++--- 6 files changed, 109 insertions(+), 13 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4721 from [BEAM-1866] Fixing nanosecond translation issue in Gauge Fn API translation.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ce0c92e0a15b9b44e53161827afe213eb18e18b4 Merge: 80fa9f8 6cf63af Author: Robert Bradshaw AuthorDate: Wed Feb 21 11:46:29 2018 -0800 Merge pull request #4721 from [BEAM-1866] Fixing nanosecond translation issue in Gauge Fn API translation. [BEAM-1866] Fixing nanosecond translation issue in Gauge Fn API translation. sdks/python/apache_beam/metrics/cells.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (80fa9f8 -> ce0c92e)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 80fa9f8 Merge pull request #4702: [BEAM-3715] Explicitly exclude some troublesome optional deps add 6cf63af Fixing nanosecond translation issue in Gauge Fn API translation. new ce0c92e Merge pull request #4721 from [BEAM-1866] Fixing nanosecond translation issue in Gauge Fn API translation. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/metrics/cells.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (5608168 -> 1b9d879)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5608168 Merge pull request #4696 from [BEAM-3644] Use the switching DirectRunner implementation add f6d8a93 long was renamed to int in Python 3 (en masse) new 1b9d879 Merge pull request #4691 from long was renamed to int in Python 3 (en masse) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/coders/coder_impl.py| 5 + sdks/python/apache_beam/io/vcfio.py | 6 ++ sdks/python/apache_beam/transforms/combiners.py | 5 + 3 files changed, 16 insertions(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4691 from long was renamed to int in Python 3 (en masse)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1b9d879c0534b4871dfe9f78de745d3969ce55dc Merge: 5608168 f6d8a93 Author: Robert Bradshaw AuthorDate: Fri Feb 16 15:04:28 2018 -0800 Merge pull request #4691 from long was renamed to int in Python 3 (en masse) long was renamed to int in Python 3 (en masse) sdks/python/apache_beam/coders/coder_impl.py| 5 + sdks/python/apache_beam/io/vcfio.py | 6 ++ sdks/python/apache_beam/transforms/combiners.py | 5 + 3 files changed, 16 insertions(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (dfbdcbf -> 5608168)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dfbdcbf Merge pull request #4666 from davidcavazos/game-streaming add 2c31ba3 Use the switching DirectRunner implementation new 5608168 Merge pull request #4696 from [BEAM-3644] Use the switching DirectRunner implementation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/direct/direct_runner.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4696 from [BEAM-3644] Use the switching DirectRunner implementation
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 56081686bf7926b65a18dc7c7d2c4e4a9fd265e9 Merge: dfbdcbf 2c31ba3 Author: Robert Bradshaw AuthorDate: Fri Feb 16 15:02:39 2018 -0800 Merge pull request #4696 from [BEAM-3644] Use the switching DirectRunner implementation [BEAM-3644] Use the switching DirectRunner implementation sdks/python/apache_beam/runners/direct/direct_runner.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4425 from [BEAM-3153] Add processing-time timers for the DirectRunner.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 77ecbc7d885ebd67f36022fb1dd8c7a17f6ebc3b Merge: a5d9fe1 4f681ee Author: Robert Bradshaw AuthorDate: Thu Feb 15 13:18:57 2018 -0800 Merge pull request #4425 from [BEAM-3153] Add processing-time timers for the DirectRunner. [BEAM-3153] Add processing-time timers for the DirectRunner. sdks/python/apache_beam/runners/direct/executor.py | 17 +-- .../runners/direct/transform_evaluator.py | 7 +- sdks/python/apache_beam/runners/direct/util.py | 5 + .../python/apache_beam/testing/test_stream_test.py | 102 - sdks/python/apache_beam/transforms/trigger.py | 123 - sdks/python/apache_beam/transforms/trigger_test.py | 3 +- 6 files changed, 215 insertions(+), 42 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (a5d9fe1 -> 77ecbc7)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a5d9fe1 Merge pull request #4515 from luke-zhu/python3 add 4f681ee Add test for processing-time timer new 77ecbc7 Merge pull request #4425 from [BEAM-3153] Add processing-time timers for the DirectRunner. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/direct/executor.py | 17 +-- .../runners/direct/transform_evaluator.py | 7 +- sdks/python/apache_beam/runners/direct/util.py | 5 + .../python/apache_beam/testing/test_stream_test.py | 102 - sdks/python/apache_beam/transforms/trigger.py | 123 - sdks/python/apache_beam/transforms/trigger_test.py | 3 +- 6 files changed, 215 insertions(+), 42 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (11ad97e -> a5d9fe1)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 11ad97e Merge pull request #4676: [BEAM-3182] [Nexmark][SQL] Implement query 5 add fa90b9c Support Python 3 in the metrics, internal, typehints, and utils modules. new a5d9fe1 Merge pull request #4515 from luke-zhu/python3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/internal/gcp/auth.py| 7 --- sdks/python/apache_beam/internal/gcp/json_value.py | 12 ++-- sdks/python/apache_beam/internal/gcp/json_value_test.py | 6 +++--- sdks/python/apache_beam/internal/util.py| 4 ++-- sdks/python/apache_beam/metrics/cells.py| 3 ++- sdks/python/apache_beam/metrics/cells_test.py | 8 ++-- sdks/python/apache_beam/metrics/execution_test.py | 8 sdks/python/apache_beam/typehints/decorators.py | 4 +++- .../apache_beam/typehints/native_type_compatibility.py | 2 +- sdks/python/apache_beam/typehints/opcodes.py| 2 +- sdks/python/apache_beam/typehints/trivial_inference.py | 1 - sdks/python/apache_beam/typehints/trivial_inference_test.py | 1 + sdks/python/apache_beam/typehints/typecheck.py | 10 ++ sdks/python/apache_beam/typehints/typed_pipeline_test.py| 4 ++-- sdks/python/apache_beam/typehints/typehints.py | 6 -- sdks/python/apache_beam/utils/profiler.py | 4 ++-- sdks/python/apache_beam/utils/retry.py | 6 -- sdks/python/apache_beam/utils/timestamp.py | 13 +++-- 18 files changed, 58 insertions(+), 43 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4515 from luke-zhu/python3
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a5d9fe1bd43a47fa4f65da50e1832d66642ddb8b Merge: 11ad97e fa90b9c Author: Robert Bradshaw AuthorDate: Thu Feb 15 13:03:19 2018 -0800 Merge pull request #4515 from luke-zhu/python3 [BEAM-3552] Support Python 3 in the metrics, internal, typehints, and utils modules. sdks/python/apache_beam/internal/gcp/auth.py| 7 --- sdks/python/apache_beam/internal/gcp/json_value.py | 12 ++-- sdks/python/apache_beam/internal/gcp/json_value_test.py | 6 +++--- sdks/python/apache_beam/internal/util.py| 4 ++-- sdks/python/apache_beam/metrics/cells.py| 3 ++- sdks/python/apache_beam/metrics/cells_test.py | 8 ++-- sdks/python/apache_beam/metrics/execution_test.py | 8 sdks/python/apache_beam/typehints/decorators.py | 4 +++- .../apache_beam/typehints/native_type_compatibility.py | 2 +- sdks/python/apache_beam/typehints/opcodes.py| 2 +- sdks/python/apache_beam/typehints/trivial_inference.py | 1 - sdks/python/apache_beam/typehints/trivial_inference_test.py | 1 + sdks/python/apache_beam/typehints/typecheck.py | 10 ++ sdks/python/apache_beam/typehints/typed_pipeline_test.py| 4 ++-- sdks/python/apache_beam/typehints/typehints.py | 6 -- sdks/python/apache_beam/utils/profiler.py | 4 ++-- sdks/python/apache_beam/utils/retry.py | 6 -- sdks/python/apache_beam/utils/timestamp.py | 13 +++-- 18 files changed, 58 insertions(+), 43 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (e668688 -> 33c349f)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e668688 Merge pull request #4646: [SQL] Refactor BeamSql interface add 41311f5 Use TestClock when TestStream is present in pipeline new 33c349f Merge pull request #4650 Use TestClock when TestStream is present in pipeline The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache_beam/runners/direct/direct_runner.py| 32 ++ 1 file changed, 21 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4650 Use TestClock when TestStream is present in pipeline
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 33c349fffb22478807034a6aec7f2a212116683b Merge: e668688 41311f5 Author: Robert Bradshaw AuthorDate: Wed Feb 14 16:29:13 2018 -0800 Merge pull request #4650 Use TestClock when TestStream is present in pipeline Use TestClock when TestStream is present in pipeline .../apache_beam/runners/direct/direct_runner.py| 32 ++ 1 file changed, 21 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4570 from [BEAM-1251] Migrate away from xrange()
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 08f3ffa67358edfead752acb9b0fc3bf7e7cbe4a
Merge: 814a80a c635c63
Author: Robert Bradshaw
AuthorDate: Tue Feb 13 16:02:27 2018 -0800
Merge pull request #4570 from [BEAM-1251] Migrate away from xrange()
[BEAM-1251] Migrate away from xrange()
sdks/python/apache_beam/examples/complete/estimate_pi.py | 2 +-
.../examples/complete/juliaset/juliaset/juliaset.py| 2 +-
.../apache_beam/examples/cookbook/bigquery_side_input.py | 2 +-
sdks/python/apache_beam/examples/snippets/snippets.py | 2 +-
sdks/python/apache_beam/io/filebasedsink.py| 8
sdks/python/apache_beam/io/filebasedsource_test.py | 10 +-
sdks/python/apache_beam/io/tfrecordio_test.py | 4 ++--
sdks/python/apache_beam/runners/worker/opcounters_test.py | 8
sdks/python/apache_beam/transforms/combiners_test.py | 4 ++--
sdks/python/apache_beam/transforms/trigger_test.py | 2 +-
sdks/python/apache_beam/typehints/native_type_compatibility.py | 2 +-
11 files changed, 23 insertions(+), 23 deletions(-)
diff --cc sdks/python/apache_beam/io/tfrecordio_test.py
index 5bc13ce,191c757..c540cfa
--- a/sdks/python/apache_beam/io/tfrecordio_test.py
+++ b/sdks/python/apache_beam/io/tfrecordio_test.py
@@@ -195,202 -203,127 +195,202 @@@ class TestTFRecordSink(unittest.TestCas
class TestWriteToTFRecord(TestTFRecordSink):
def test_write_record_gzip(self):
-file_path_prefix = os.path.join(self._new_tempdir(), 'result')
-with TestPipeline() as p:
- input_data = ['foo', 'bar']
- _ = p | beam.Create(input_data) | WriteToTFRecord(
- file_path_prefix, compression_type=CompressionTypes.GZIP)
-
-actual = []
-file_name = glob.glob(file_path_prefix + '-*')[0]
-for r in tf.python_io.tf_record_iterator(
-file_name, options=tf.python_io.TFRecordOptions(
-tf.python_io.TFRecordCompressionType.GZIP)):
- actual.append(r)
-self.assertEqual(actual, input_data)
+with TempDir() as temp_dir:
+ file_path_prefix = temp_dir.create_temp_file('result')
+ with TestPipeline() as p:
+input_data = ['foo', 'bar']
+_ = p | beam.Create(input_data) | WriteToTFRecord(
+file_path_prefix, compression_type=CompressionTypes.GZIP)
+
+ actual = []
+ file_name = glob.glob(file_path_prefix + '-*')[0]
+ for r in tf.python_io.tf_record_iterator(
+ file_name, options=tf.python_io.TFRecordOptions(
+ tf.python_io.TFRecordCompressionType.GZIP)):
+actual.append(r)
+ self.assertEqual(actual, input_data)
def test_write_record_auto(self):
-file_path_prefix = os.path.join(self._new_tempdir(), 'result')
-with TestPipeline() as p:
- input_data = ['foo', 'bar']
- _ = p | beam.Create(input_data) | WriteToTFRecord(
- file_path_prefix, file_name_suffix='.gz')
+with TempDir() as temp_dir:
+ file_path_prefix = temp_dir.create_temp_file('result')
+ with TestPipeline() as p:
+input_data = ['foo', 'bar']
+_ = p | beam.Create(input_data) | WriteToTFRecord(
+file_path_prefix, file_name_suffix='.gz')
-actual = []
-file_name = glob.glob(file_path_prefix + '-*.gz')[0]
-for r in tf.python_io.tf_record_iterator(
-file_name, options=tf.python_io.TFRecordOptions(
-tf.python_io.TFRecordCompressionType.GZIP)):
- actual.append(r)
-self.assertEqual(actual, input_data)
+ actual = []
+ file_name = glob.glob(file_path_prefix + '-*.gz')[0]
+ for r in tf.python_io.tf_record_iterator(
+ file_name, options=tf.python_io.TFRecordOptions(
+ tf.python_io.TFRecordCompressionType.GZIP)):
+actual.append(r)
+ self.assertEqual(actual, input_data)
-class TestTFRecordSource(_TestCaseWithTempDirCleanUp):
-
- def _write_file(self, path, base64_records):
-record = binascii.a2b_base64(base64_records)
-with open(path, 'wb') as f:
- f.write(record)
-
- def _write_file_gzip(self, path, base64_records):
-record = binascii.a2b_base64(base64_records)
-with gzip.GzipFile(path, 'wb') as f:
- f.write(record)
+class TestReadFromTFRecord(unittest.TestCase):
def test_process_single(self):
-path = os.path.join(self._new_tempdir(), 'result')
-self._write_file(path, FOO_RECORD_BASE64)
-with TestPipeline() as p:
- result = (p
-| beam.io.Read(
-
[beam] branch master updated (814a80a -> 08f3ffa)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 814a80a Merge pull request #4571 from [BEAM-1251] Use six.integer_types add c635c63 xrange() was removed in Python 3 (en masse) new 08f3ffa Merge pull request #4570 from [BEAM-1251] Migrate away from xrange() The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/examples/complete/estimate_pi.py | 2 +- .../examples/complete/juliaset/juliaset/juliaset.py| 2 +- .../apache_beam/examples/cookbook/bigquery_side_input.py | 2 +- sdks/python/apache_beam/examples/snippets/snippets.py | 2 +- sdks/python/apache_beam/io/filebasedsink.py| 8 sdks/python/apache_beam/io/filebasedsource_test.py | 10 +- sdks/python/apache_beam/io/tfrecordio_test.py | 4 ++-- sdks/python/apache_beam/runners/worker/opcounters_test.py | 8 sdks/python/apache_beam/transforms/combiners_test.py | 4 ++-- sdks/python/apache_beam/transforms/trigger_test.py | 2 +- sdks/python/apache_beam/typehints/native_type_compatibility.py | 2 +- 11 files changed, 23 insertions(+), 23 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (8207df6 -> 814a80a)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8207df6 Merge pull request #4564: [BEAM-3581] Make sure calcite gets an appropriate charset PRIOR to any static initializers add e6e875d from six import integer_types (en masse) new 814a80a Merge pull request #4571 from [BEAM-1251] Use six.integer_types The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/io/filebasedsource.py | 6 -- sdks/python/apache_beam/io/filesystem.py | 4 +++- sdks/python/apache_beam/io/range_trackers.py | 8 +--- sdks/python/apache_beam/io/range_trackers_test.py | 5 - sdks/python/apache_beam/io/textio.py | 4 +++- 5 files changed, 19 insertions(+), 8 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4571 from [BEAM-1251] Use six.integer_types
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 814a80a4aa6dcce69d0705bc1506a3c98d0b0817 Merge: 8207df6 e6e875d Author: Robert Bradshaw AuthorDate: Tue Feb 13 16:01:27 2018 -0800 Merge pull request #4571 from [BEAM-1251] Use six.integer_types from six import integer_types (en masse) sdks/python/apache_beam/io/filebasedsource.py | 6 -- sdks/python/apache_beam/io/filesystem.py | 4 +++- sdks/python/apache_beam/io/range_trackers.py | 8 +--- sdks/python/apache_beam/io/range_trackers_test.py | 5 - sdks/python/apache_beam/io/textio.py | 4 +++- 5 files changed, 19 insertions(+), 8 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4572 [BEAM-3074] Serialize DoFns by portable id in Dataflow runner.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit aec829a87541e3e6deac9363b7d7583fd317b1ca Merge: a0071ed 52cabfd Author: Robert Bradshaw AuthorDate: Mon Feb 12 10:05:46 2018 -0800 Merge pull request #4572 [BEAM-3074] Serialize DoFns by portable id in Dataflow runner. .../apache_beam/runners/dataflow/dataflow_runner.py | 17 ++--- sdks/python/apache_beam/runners/pipeline_context.py | 3 +++ 2 files changed, 17 insertions(+), 3 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (a0071ed -> aec829a)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a0071ed [BEAM-3629] Send the windowing strategy and whether its a merging window fn to Dataflow. add 52cabfd [BEAM-3074] Serialize DoFns by portable id in Dataflow runner. new aec829a Merge pull request #4572 [BEAM-3074] Serialize DoFns by portable id in Dataflow runner. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache_beam/runners/dataflow/dataflow_runner.py | 17 ++--- sdks/python/apache_beam/runners/pipeline_context.py | 3 +++ 2 files changed, 17 insertions(+), 3 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4634 [BEAM-3644] Add switchable DirectRunner which uses the fast FnApiRunner when possible
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 81932199e8da2a58be4c507f1b1a3f3abb35a92d Merge: 0feb813 46931b4b Author: Robert Bradshaw AuthorDate: Fri Feb 9 14:22:27 2018 -0800 Merge pull request #4634 [BEAM-3644] Add switchable DirectRunner which uses the fast FnApiRunner when possible [BEAM-3644] Add switchable DirectRunner which uses the fast FnApiRunner when possible sdks/python/apache_beam/pipeline_test.py | 16 ++-- .../runners/dataflow/native_io/iobase_test.py | 3 +- .../apache_beam/runners/direct/direct_runner.py| 89 +- .../runners/direct/direct_runner_test.py | 18 +++-- .../runners/direct/sdf_direct_runner_test.py | 6 +- sdks/python/apache_beam/runners/runner.py | 3 +- 6 files changed, 112 insertions(+), 23 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (0feb813 -> 8193219)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0feb813 Merge pull request #4608 from cclauss/patch-1 add 46931b4b Add switchable DirectRunner which uses the fast FnApiRunner when possible new 8193219 Merge pull request #4634 [BEAM-3644] Add switchable DirectRunner which uses the fast FnApiRunner when possible The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline_test.py | 16 ++-- .../runners/dataflow/native_io/iobase_test.py | 3 +- .../apache_beam/runners/direct/direct_runner.py| 89 +- .../runners/direct/direct_runner_test.py | 18 +++-- .../runners/direct/sdf_direct_runner_test.py | 6 +- sdks/python/apache_beam/runners/runner.py | 3 +- 6 files changed, 112 insertions(+), 23 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4641 [BEAM-3644] Disable verbose typecoder warnings.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8c5ebe4a4d158fa8c46640a0401376216a1d11c2 Merge: 3df248b 3b8b753 Author: Robert Bradshaw AuthorDate: Thu Feb 8 21:27:15 2018 -0800 Merge pull request #4641 [BEAM-3644] Disable verbose typecoder warnings. [BEAM-3644] Disable verbose typecoder warnings. sdks/python/apache_beam/coders/typecoders.py | 8 sdks/python/apache_beam/typehints/typehints.py | 5 + 2 files changed, 9 insertions(+), 4 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (3df248b -> 8c5ebe4)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3df248b Merge pull request #4600: [BEAM-410] Sort PriorityQueue with explicit comparator add 3b8b753 Disable verbose typecoder warnings. new 8c5ebe4 Merge pull request #4641 [BEAM-3644] Disable verbose typecoder warnings. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/coders/typecoders.py | 8 sdks/python/apache_beam/typehints/typehints.py | 5 + 2 files changed, 9 insertions(+), 4 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (6792846 -> f1909bc)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6792846 Merge pull request #4204: [BEAM-3157] Generate BeamRecord types from Pojos add 4e333f7 [BEAM-3625] Enable DoFn params in Map, Filter, etc. new f1909bc Merge pull request #4616 from robertwb/map-params The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline_test.py | 6 ++ sdks/python/apache_beam/transforms/core.py | 3 +++ 2 files changed, 9 insertions(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4616 from robertwb/map-params
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit f1909bceb80c0446d8dd025c8dd532a6ca9375aa Merge: 6792846 4e333f7 Author: Robert Bradshaw AuthorDate: Thu Feb 8 10:05:19 2018 -0800 Merge pull request #4616 from robertwb/map-params [BEAM-3625] Enable DoFn params in Map, Filter, etc. sdks/python/apache_beam/pipeline_test.py | 6 ++ sdks/python/apache_beam/transforms/core.py | 3 +++ 2 files changed, 9 insertions(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4602 from charlesccychen/snippets-pickling
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 37e1db7e33f99fb0a6147137841d682c7d4afbf9 Merge: 6745a61 8f9edbc Author: Robert Bradshaw AuthorDate: Wed Feb 7 17:11:08 2018 -0800 Merge pull request #4602 from charlesccychen/snippets-pickling Update snippets to fix pickling and clarify encoding issues .../apache_beam/examples/snippets/snippets.py | 208 +++-- .../apache_beam/examples/snippets/snippets_test.py | 16 +- 2 files changed, 115 insertions(+), 109 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (6745a61 -> 37e1db7)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6745a61 [BEAM-3639] Update to gradle 4.5.1 add 8f9edbc Update snippets to fix pickling and clarify encoding issues new 37e1db7 Merge pull request #4602 from charlesccychen/snippets-pickling The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache_beam/examples/snippets/snippets.py | 208 +++-- .../apache_beam/examples/snippets/snippets_test.py | 16 +- 2 files changed, 115 insertions(+), 109 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (06a4a15 -> 1ac8a24)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 06a4a15 Merge pull request #4593: [BEAM-2721] Refactor BeamRecordType and BeamRecord add f887ed0 [BEAM-3635] Infer type hints on PTransformOverrides new 1ac8a24 Merge pull request #4623 from charlesccychen/fix-python-streaming The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline.py| 57 +- sdks/python/apache_beam/pipeline_test.py | 40 +++ .../apache_beam/runners/direct/direct_runner.py| 25 +- .../runners/direct/transform_evaluator.py | 22 + 4 files changed, 100 insertions(+), 44 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4623 from charlesccychen/fix-python-streaming
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1ac8a24bbd952bd8a494ca7de5c89488b82d7a54 Merge: 06a4a15 f887ed0 Author: Robert Bradshaw AuthorDate: Tue Feb 6 17:39:16 2018 -0800 Merge pull request #4623 from charlesccychen/fix-python-streaming [BEAM-3635] Infer type hints on PTransformOverrides sdks/python/apache_beam/pipeline.py| 57 +- sdks/python/apache_beam/pipeline_test.py | 40 +++ .../apache_beam/runners/direct/direct_runner.py| 25 +- .../runners/direct/transform_evaluator.py | 22 + 4 files changed, 100 insertions(+), 44 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch go-sdk updated (52a598d -> a328e44)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch go-sdk in repository https://gitbox.apache.org/repos/asf/beam.git. from 52a598d [BEAM-3388] Remove reflection from varint codecs add 07e534b Changed core GBK to CoGBK add 66ecd91 Add CoGBK support to direct runner and Join example add 79fb7e1 [BEAM-3316] Translate bundle descriptors directly to execution plans in Go SDK add 1cf50dd Translate CoGBK into GBK for Dataflow and model pipeline runners add b7dce98 CR: [BEAM-3302] Support CoGBK in the Go SDK new a328e44 Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go SDK The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/go/examples/cookbook/join/join.go | 111 sdks/go/pkg/beam/coder.go | 8 +- sdks/go/pkg/beam/combine.go| 32 +- sdks/go/pkg/beam/core/graph/bind.go| 51 +- sdks/go/pkg/beam/core/graph/bind_test.go | 18 +- sdks/go/pkg/beam/core/graph/coder/coder.go | 47 +- sdks/go/pkg/beam/core/graph/edge.go| 160 +++-- sdks/go/pkg/beam/core/runtime/exec/coder.go| 28 +- sdks/go/pkg/beam/core/runtime/exec/cogbk.go| 171 + sdks/go/pkg/beam/core/runtime/exec/combine.go | 54 +- sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 4 +- sdks/go/pkg/beam/core/runtime/exec/datasink.go | 18 +- sdks/go/pkg/beam/core/runtime/exec/datasource.go | 20 +- sdks/go/pkg/beam/core/runtime/exec/fn.go | 4 +- sdks/go/pkg/beam/core/runtime/exec/fullvalue.go| 3 + .../pkg/beam/core/runtime/exec/optimized/inputs.go | 695 +++-- .../beam/core/runtime/exec/optimized/inputs.tmpl | 33 + sdks/go/pkg/beam/core/runtime/exec/pardo.go| 25 +- sdks/go/pkg/beam/core/runtime/exec/pardo_test.go | 2 +- sdks/go/pkg/beam/core/runtime/exec/plan.go | 2 + sdks/go/pkg/beam/core/runtime/exec/translate.go| 433 + sdks/go/pkg/beam/core/runtime/graphx/cobgk.go | 108 sdks/go/pkg/beam/core/runtime/graphx/coder.go | 68 +- sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 8 +- sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 87 ++- sdks/go/pkg/beam/core/runtime/graphx/translate.go | 493 --- .../pkg/beam/core/runtime/graphx/translate_test.go | 74 +-- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 194 -- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto | 19 +- sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 9 +- sdks/go/pkg/beam/core/runtime/harness/harness.go | 47 +- sdks/go/pkg/beam/core/typex/class.go | 4 +- sdks/go/pkg/beam/core/typex/class_test.go | 1 - sdks/go/pkg/beam/core/typex/fulltype.go| 42 +- sdks/go/pkg/beam/core/typex/fulltype_test.go | 28 +- sdks/go/pkg/beam/core/typex/special.go | 10 +- sdks/go/pkg/beam/gbk.go| 43 +- sdks/go/pkg/beam/impulse.go| 4 +- sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 7 + sdks/go/pkg/beam/io/textio/textio.go | 4 + sdks/go/pkg/beam/runners/dataflow/translate.go | 182 +- sdks/go/pkg/beam/runners/direct/direct.go | 48 +- sdks/go/pkg/beam/runners/direct/gbk.go | 79 ++- sdks/go/pkg/beam/testing/passert/passert.go| 4 +- sdks/go/pkg/beam/x/debug/print.go | 4 +- 45 files changed, 1967 insertions(+), 1519 deletions(-) create mode 100644 sdks/go/examples/cookbook/join/join.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/cogbk.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/translate.go create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/cobgk.go -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go SDK
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch go-sdk in repository https://gitbox.apache.org/repos/asf/beam.git commit a328e441dbe28f233929b7b3a4a5408663fe7468 Merge: 52a598d b7dce98 Author: Robert Bradshaw AuthorDate: Tue Feb 6 16:33:10 2018 -0800 Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go SDK sdks/go/examples/cookbook/join/join.go | 111 sdks/go/pkg/beam/coder.go | 8 +- sdks/go/pkg/beam/combine.go| 32 +- sdks/go/pkg/beam/core/graph/bind.go| 51 +- sdks/go/pkg/beam/core/graph/bind_test.go | 18 +- sdks/go/pkg/beam/core/graph/coder/coder.go | 47 +- sdks/go/pkg/beam/core/graph/edge.go| 160 +++-- sdks/go/pkg/beam/core/runtime/exec/coder.go| 28 +- sdks/go/pkg/beam/core/runtime/exec/cogbk.go| 171 + sdks/go/pkg/beam/core/runtime/exec/combine.go | 54 +- sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 4 +- sdks/go/pkg/beam/core/runtime/exec/datasink.go | 18 +- sdks/go/pkg/beam/core/runtime/exec/datasource.go | 20 +- sdks/go/pkg/beam/core/runtime/exec/fn.go | 4 +- sdks/go/pkg/beam/core/runtime/exec/fullvalue.go| 3 + .../pkg/beam/core/runtime/exec/optimized/inputs.go | 695 +++-- .../beam/core/runtime/exec/optimized/inputs.tmpl | 33 + sdks/go/pkg/beam/core/runtime/exec/pardo.go| 25 +- sdks/go/pkg/beam/core/runtime/exec/pardo_test.go | 2 +- sdks/go/pkg/beam/core/runtime/exec/plan.go | 2 + sdks/go/pkg/beam/core/runtime/exec/translate.go| 433 + sdks/go/pkg/beam/core/runtime/graphx/cobgk.go | 108 sdks/go/pkg/beam/core/runtime/graphx/coder.go | 68 +- sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 8 +- sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 87 ++- sdks/go/pkg/beam/core/runtime/graphx/translate.go | 493 --- .../pkg/beam/core/runtime/graphx/translate_test.go | 74 +-- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 194 -- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto | 19 +- sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 9 +- sdks/go/pkg/beam/core/runtime/harness/harness.go | 47 +- sdks/go/pkg/beam/core/typex/class.go | 4 +- sdks/go/pkg/beam/core/typex/class_test.go | 1 - sdks/go/pkg/beam/core/typex/fulltype.go| 42 +- sdks/go/pkg/beam/core/typex/fulltype_test.go | 28 +- sdks/go/pkg/beam/core/typex/special.go | 10 +- sdks/go/pkg/beam/gbk.go| 43 +- sdks/go/pkg/beam/impulse.go| 4 +- sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 7 + sdks/go/pkg/beam/io/textio/textio.go | 4 + sdks/go/pkg/beam/runners/dataflow/translate.go | 182 +- sdks/go/pkg/beam/runners/direct/direct.go | 48 +- sdks/go/pkg/beam/runners/direct/gbk.go | 79 ++- sdks/go/pkg/beam/testing/passert/passert.go| 4 +- sdks/go/pkg/beam/x/debug/print.go | 4 +- 45 files changed, 1967 insertions(+), 1519 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8e2092ba28ad31517d65b3085840e6e9a46c83bf Merge: 8edc18e e1a96eb Author: Robert Bradshaw AuthorDate: Mon Feb 5 15:32:09 2018 -0800 Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides sdks/python/apache_beam/io/gcp/pubsub_test.py | 58 - sdks/python/apache_beam/pipeline.py| 34 +-- sdks/python/apache_beam/pipeline_test.py | 9 +- .../runners/dataflow/dataflow_runner_test.py | 4 +- .../runners/dataflow/ptransform_overrides.py | 6 +- .../apache_beam/runners/direct/direct_runner.py| 234 - .../runners/direct/helper_transforms.py| 8 +- .../runners/direct/sdf_direct_runner.py| 9 +- sdks/python/apache_beam/runners/sdf_common.py | 15 +- sdks/python/apache_beam/transforms/combiners.py| 39 ++-- 10 files changed, 253 insertions(+), 163 deletions(-) diff --cc sdks/python/apache_beam/pipeline.py index 4c48428,c59a29a..f164bf5 --- a/sdks/python/apache_beam/pipeline.py +++ b/sdks/python/apache_beam/pipeline.py @@@ -61,8 -61,8 +61,9 @@@ from apache_beam.options.pipeline_optio from apache_beam.options.pipeline_options import StandardOptions from apache_beam.options.pipeline_options import TypeOptions from apache_beam.options.pipeline_options_validator import PipelineOptionsValidator +from apache_beam.portability import common_urns from apache_beam.pvalue import PCollection + from apache_beam.pvalue import PDone from apache_beam.runners import PipelineRunner from apache_beam.runners import create_runner from apache_beam.transforms import ptransform -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (8edc18e -> 8e2092b)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8edc18e Merge pull request #4595: [BEAM-3601] Switch to Java 8 futures add 16e5fef [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides add e1a96eb Address reviewer comments new 8e2092b Merge pull request #4529 from [BEAM-3566] Replace apply_* hooks in DirectRunner with PTransformOverrides The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/io/gcp/pubsub_test.py | 58 - sdks/python/apache_beam/pipeline.py| 34 +-- sdks/python/apache_beam/pipeline_test.py | 9 +- .../runners/dataflow/dataflow_runner_test.py | 4 +- .../runners/dataflow/ptransform_overrides.py | 6 +- .../apache_beam/runners/direct/direct_runner.py| 234 - .../runners/direct/helper_transforms.py| 8 +- .../runners/direct/sdf_direct_runner.py| 9 +- sdks/python/apache_beam/runners/sdf_common.py | 15 +- sdks/python/apache_beam/transforms/combiners.py| 39 ++-- 10 files changed, 253 insertions(+), 163 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (cb38392 -> 7904112)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cb38392 Match Maven Java SDK harness jar name format add 4d06e82 [BEAM-3183] Allow a callable as input to runner.run(). new 7904112 Merge pull request #4590 [BEAM-3183] Allow a callable as input to runner.run(). The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/runner.py | 9 +++-- sdks/python/apache_beam/runners/runner_test.py | 14 ++ 2 files changed, 21 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4590 [BEAM-3183] Allow a callable as input to runner.run().
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 79041126519ad64c4f357fb8ddcc7ae67a7b2203 Merge: cb38392 4d06e82 Author: Robert Bradshaw AuthorDate: Mon Feb 5 12:15:35 2018 -0800 Merge pull request #4590 [BEAM-3183] Allow a callable as input to runner.run(). sdks/python/apache_beam/runners/runner.py | 9 +++-- sdks/python/apache_beam/runners/runner_test.py | 14 ++ 2 files changed, 21 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4562 global INT64_MAX, INT64_MIN to placate linters
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1693e7d1affef6ebed83a64d46ef6f39b465d299 Merge: af864b8 fd07d72 Author: Robert Bradshaw AuthorDate: Fri Feb 2 17:14:38 2018 -0800 Merge pull request #4562 global INT64_MAX, INT64_MIN to placate linters global INT64_MAX, INT64_MIN to placate linters sdks/python/apache_beam/transforms/cy_combiners.py | 1 + 1 file changed, 1 insertion(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (af864b8 -> 1693e7d)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from af864b8 Merge pull request #4422: Implement a Local Artifact Retrieval service add fd07d72 global INT64_MAX, INT64_MIN to placate linters new 1693e7d Merge pull request #4562 global INT64_MAX, INT64_MIN to placate linters The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/transforms/cy_combiners.py | 1 + 1 file changed, 1 insertion(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4576 Reduce the flakiness of the state sampler progress metrics.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 537b9b78ace30304b419c859e67b6339fc16e92c Merge: 6831f2c ef12700 Author: Robert Bradshaw AuthorDate: Fri Feb 2 08:20:50 2018 -0800 Merge pull request #4576 Reduce the flakiness of the state sampler progress metrics. Reduce the flakiness of the state sampler progress metrics. sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (6831f2c -> 537b9b7)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6831f2c Merge pull request #4584: [BEAM-3551] Define compiler -parameters flag in the default options add ef12700 Reduce the flakiness of the state sampler progress metrics. new 537b9b7 Merge pull request #4576 Reduce the flakiness of the state sampler progress metrics. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (51da92c -> a2bf73f)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 51da92c Merge pull request #4559 from cclauss/patch-1 add a71042a import logging for line 1163 new a2bf73f Merge pull request #4560 [lint] import logging The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/transforms/trigger.py | 1 + 1 file changed, 1 insertion(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4560 [lint] import logging
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a2bf73f87599e4119130380366051dbd2fcc837f Merge: 51da92c a71042a Author: Robert Bradshaw AuthorDate: Thu Feb 1 10:24:23 2018 -0800 Merge pull request #4560 [lint] import logging sdks/python/apache_beam/transforms/trigger.py | 1 + 1 file changed, 1 insertion(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (5a5e71c -> bddcb0c)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5a5e71c Merge pull request #4552: [BEAM-3317] Sickbay flakey KinesisReaderTest add b52c385 [BEAM-3490] Make runtime type checking code runner agnostic. add de7bf0a Direct runner fixes. new bddcb0c Merge pull request #4534 [BEAM-3490] Make runtime type checking code runner agnostic. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline.py| 4 ++ .../runners/dataflow/dataflow_runner.py| 7 ++ .../apache_beam/runners/direct/direct_runner.py| 4 ++ .../runners/direct/transform_evaluator.py | 9 --- .../runners/portability/fn_api_runner.py | 5 ++ sdks/python/apache_beam/transforms/combiners.py| 2 +- sdks/python/apache_beam/transforms/core.py | 16 - .../apache_beam/transforms/ptransform_test.py | 41 ++-- sdks/python/apache_beam/typehints/typecheck.py | 75 +- 9 files changed, 129 insertions(+), 34 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4534 [BEAM-3490] Make runtime type checking code runner agnostic.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bddcb0c972a61063568a0c06d5e9b15a181e2119 Merge: 5a5e71c de7bf0a Author: Robert Bradshaw AuthorDate: Wed Jan 31 13:48:10 2018 -0800 Merge pull request #4534 [BEAM-3490] Make runtime type checking code runner agnostic. [BEAM-3490] Make runtime type checking code runner agnostic. sdks/python/apache_beam/pipeline.py| 4 ++ .../runners/dataflow/dataflow_runner.py| 7 ++ .../apache_beam/runners/direct/direct_runner.py| 4 ++ .../runners/direct/transform_evaluator.py | 9 --- .../runners/portability/fn_api_runner.py | 5 ++ sdks/python/apache_beam/transforms/combiners.py| 2 +- sdks/python/apache_beam/transforms/core.py | 16 - .../apache_beam/transforms/ptransform_test.py | 41 ++-- sdks/python/apache_beam/typehints/typecheck.py | 75 +- 9 files changed, 129 insertions(+), 34 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated: [BEAM-3537] Allow more general eager in-process pipeline execution (#4492)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d920644 [BEAM-3537] Allow more general eager in-process pipeline
execution (#4492)
d920644 is described below
commit d9206441a16fa39e171684c43ec724cdd80a6ca1
Author: Charles Chen
AuthorDate: Mon Jan 29 14:43:32 2018 -0800
[BEAM-3537] Allow more general eager in-process pipeline execution (#4492)
[BEAM-3537] Allow more general eager in-process pipeline execution
This change also removes the Python DirectRunner-specific PValue cache.
---
.../apache_beam/examples/snippets/snippets_test.py | 31 +++---
sdks/python/apache_beam/runners/__init__.py| 1 -
.../apache_beam/runners/direct/direct_runner.py| 55 --
.../runners/direct/evaluation_context.py | 14 ---
sdks/python/apache_beam/runners/direct/executor.py | 11 --
sdks/python/apache_beam/runners/runner.py | 2 +-
sdks/python/apache_beam/transforms/ptransform.py | 119 ++---
.../apache_beam/transforms/ptransform_test.py | 16 ++-
8 files changed, 136 insertions(+), 113 deletions(-)
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py
b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index f05dc39..e731236 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -223,24 +223,27 @@ class ParDoTest(unittest.TestCase):
self.assertEqual({'xyz'}, set(marked))
def test_pardo_with_undeclared_outputs(self):
-numbers = [1, 2, 3, 4, 5, 10, 20]
+# Note: the use of undeclared outputs is currently not supported in eager
+# execution mode.
+with TestPipeline() as p:
+ numbers = p | beam.Create([1, 2, 3, 4, 5, 10, 20])
-# [START model_pardo_with_undeclared_outputs]
-def even_odd(x):
- yield pvalue.TaggedOutput('odd' if x % 2 else 'even', x)
- if x % 10 == 0:
-yield x
+ # [START model_pardo_with_undeclared_outputs]
+ def even_odd(x):
+yield pvalue.TaggedOutput('odd' if x % 2 else 'even', x)
+if x % 10 == 0:
+ yield x
-results = numbers | beam.FlatMap(even_odd).with_outputs()
+ results = numbers | beam.FlatMap(even_odd).with_outputs()
-evens = results.even
-odds = results.odd
-tens = results[None] # the undeclared main output
-# [END model_pardo_with_undeclared_outputs]
+ evens = results.even
+ odds = results.odd
+ tens = results[None] # the undeclared main output
+ # [END model_pardo_with_undeclared_outputs]
-self.assertEqual({2, 4, 10, 20}, set(evens))
-self.assertEqual({1, 3, 5}, set(odds))
-self.assertEqual({10, 20}, set(tens))
+ assert_that(evens, equal_to([2, 4, 10, 20]), label='assert_even')
+ assert_that(odds, equal_to([1, 3, 5]), label='assert_odds')
+ assert_that(tens, equal_to([10, 20]), label='assert_tens')
class TypeHintsTest(unittest.TestCase):
diff --git a/sdks/python/apache_beam/runners/__init__.py
b/sdks/python/apache_beam/runners/__init__.py
index 2b93c30..863e67e 100644
--- a/sdks/python/apache_beam/runners/__init__.py
+++ b/sdks/python/apache_beam/runners/__init__.py
@@ -21,7 +21,6 @@ This package defines runners, which are used to execute a
pipeline.
"""
from apache_beam.runners.direct.direct_runner import DirectRunner
-from apache_beam.runners.direct.direct_runner import EagerRunner
from apache_beam.runners.runner import PipelineRunner
from apache_beam.runners.runner import PipelineState
from apache_beam.runners.runner import create_runner
diff --git a/sdks/python/apache_beam/runners/direct/direct_runner.py
b/sdks/python/apache_beam/runners/direct/direct_runner.py
index 2bd6b45..b18d492 100644
--- a/sdks/python/apache_beam/runners/direct/direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct/direct_runner.py
@@ -23,7 +23,6 @@ graph of transformations belonging to a pipeline on the local
machine.
from __future__ import absolute_import
-import collections
import logging
from google.protobuf import wrappers_pb2
@@ -41,7 +40,6 @@ from apache_beam.runners.direct.clock import TestClock
from apache_beam.runners.runner import PipelineResult
from apache_beam.runners.runner import PipelineRunner
from apache_beam.runners.runner import PipelineState
-from apache_beam.runners.runner import PValueCache
from apache_beam.transforms.core import _GroupAlsoByWindow
from apache_beam.transforms.core import _GroupByKeyOnly
from apache_beam.transforms.ptransform import PTransform
@@ -106,7 +104,6 @@ class DirectRunner(PipelineRunner):
"""Executes a single pipeline on the local machine."&qu
[beam] branch master updated: [BEAM-3126] Adding a new Flatten test to Python SDK. (#4463)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e70b068 [BEAM-3126] Adding a new Flatten test to Python SDK. (#4463)
e70b068 is described below
commit e70b0688fa57ad381a97630710c75233a4bb72f7
Author: Daniel Oliveira
AuthorDate: Wed Jan 24 09:58:30 2018 -0800
[BEAM-3126] Adding a new Flatten test to Python SDK. (#4463)
---
sdks/python/apache_beam/transforms/ptransform_test.py | 18 ++
1 file changed, 18 insertions(+)
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py
b/sdks/python/apache_beam/transforms/ptransform_test.py
index fb20d8c..09ac72b 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -535,6 +535,24 @@ class PTransformTest(unittest.TestCase):
with self.assertRaises(TypeError):
set([1, 2, 3]) | beam.Flatten()
+ @attr('ValidatesRunner')
+ def test_flatten_multiple_pcollections_having_multiple_consumers(self):
+pipeline = TestPipeline()
+input = pipeline | 'Start' >> beam.Create(['AA', 'BBB', 'CC'])
+
+def split_even_odd(element):
+ tag = 'even_length' if len(element) % 2 == 0 else 'odd_length'
+ return pvalue.TaggedOutput(tag, element)
+
+even_length, odd_length = (input | beam.Map(split_even_odd)
+ .with_outputs('even_length', 'odd_length'))
+merged = (even_length, odd_length) | 'Flatten' >> beam.Flatten()
+
+assert_that(merged, equal_to(['AA', 'BBB', 'CC']))
+assert_that(even_length, equal_to(['AA', 'CC']), label='assert:even')
+assert_that(odd_length, equal_to(['BBB']), label='assert:odd')
+pipeline.run()
+
def test_co_group_by_key_on_list(self):
pipeline = TestPipeline()
pcoll_1 = pipeline | 'Start 1' >> beam.Create(
--
To stop receiving notification emails like this one, please contact
rober...@apache.org.
[beam] branch master updated (3920cd7 -> d682042)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3920cd7 Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in non-portable tests. add 9b10739 Remove legacy windowfn translation. new d682042 Merge pull request #4212 Remove legacy windowfn translation. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/core/construction/WindowingStrategyTranslation.java| 2 -- 1 file changed, 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4212 Remove legacy windowfn translation.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d682042741079226af3ce2bf87a7a4452a2c9ba5 Merge: 3920cd7 9b10739 Author: Robert Bradshaw AuthorDate: Tue Jan 23 16:23:03 2018 -0800 Merge pull request #4212 Remove legacy windowfn translation. .../beam/runners/core/construction/WindowingStrategyTranslation.java| 2 -- 1 file changed, 2 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in non-portable tests.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 3920cd7b1270f658b79480b61ac15766d38f Merge: 19663de 4f92495 Author: Robert Bradshaw AuthorDate: Tue Jan 23 16:13:53 2018 -0800 Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in non-portable tests. sdks/python/apache_beam/pipeline_test.py | 15 +-- .../apache_beam/runners/dataflow/native_io/iobase_test.py | 3 ++- .../apache_beam/runners/direct/direct_runner_test.py | 2 +- .../apache_beam/runners/direct/sdf_direct_runner_test.py | 6 +++--- 4 files changed, 15 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (19663de -> 3920cd7)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 19663de Merge pull request #4278 from itamaro/patch-1 add 9641567 Manually specify direct runner for global-state modifying tests. add ade4fbd FakeSource requires direct runner. add 194c393 Explicitly use DirectRunner in DirectRunner tests. add 4f92495 Document DirectRunnerOnly tests. new 3920cd7 Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in non-portable tests. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline_test.py | 15 +-- .../apache_beam/runners/dataflow/native_io/iobase_test.py | 3 ++- .../apache_beam/runners/direct/direct_runner_test.py | 2 +- .../apache_beam/runners/direct/sdf_direct_runner_test.py | 6 +++--- 4 files changed, 15 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (55844fa -> 94d5c0b)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 55844fa Merge pull request #4266 add 4719468 Curry CombineFn arguments into runner API protos. new 94d5c0b Merge pull request #4437 from robertwb/runner-api-combine-fn The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/worker/bundle_processor.py | 1 - sdks/python/apache_beam/transforms/core.py | 14 -- 2 files changed, 12 insertions(+), 3 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4437 from robertwb/runner-api-combine-fn
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 94d5c0b71fa3017c855f59ced6c94fa5c7c735e0 Merge: 55844fa 4719468 Author: Robert Bradshaw AuthorDate: Mon Jan 22 16:10:54 2018 -0800 Merge pull request #4437 from robertwb/runner-api-combine-fn [BEAM-3490] Curry CombineFn arguments into runner API protos. sdks/python/apache_beam/runners/worker/bundle_processor.py | 1 - sdks/python/apache_beam/transforms/core.py | 14 -- 2 files changed, 12 insertions(+), 3 deletions(-) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] 01/01: Merge pull request #4451 from robertwb/fn-api-pickle-side
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a5398118a81874a4c33f7c43be87a6496d0ccf97 Merge: 53e4b93 916caaf Author: Robert Bradshaw AuthorDate: Mon Jan 22 10:44:43 2018 -0800 Merge pull request #4451 from robertwb/fn-api-pickle-side [BEAM-3490] Picklable side inputs for FnApi Workers. sdks/python/apache_beam/runners/worker/bundle_processor.py | 3 +++ 1 file changed, 3 insertions(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (53e4b93 -> a539811)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 53e4b93 Merge pull request #4459 from iemejia/BEAM-3492-spark-e2e-wordcount add 916caaf [BEAM-3490] Picklable side inputs for FnApi Workers. new a539811 Merge pull request #4451 from robertwb/fn-api-pickle-side The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/worker/bundle_processor.py | 3 +++ 1 file changed, 3 insertions(+) -- To stop receiving notification emails like this one, please contact rober...@apache.org.
[beam] branch master updated (49c839c -> 8982772)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 49c839c Merge pull request #4427 add 1f1904d [BEAM-3490] Wrap DistributionData in a DistributionResult for FnApiRunner. new 8982772 Merge pull request #4450 [BEAM-3490] Wrap DistributionData in a DistributionResult The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/portability/fn_api_runner.py | 5 +++-- sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4450 [BEAM-3490] Wrap DistributionData in a DistributionResult
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 898277297be7160900fdb48ed486a257a6bc92c0 Merge: 49c839c 1f1904d Author: Robert Bradshaw AuthorDate: Fri Jan 19 16:14:35 2018 -0800 Merge pull request #4450 [BEAM-3490] Wrap DistributionData in a DistributionResult sdks/python/apache_beam/runners/portability/fn_api_runner.py | 5 +++-- sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] 01/01: Merge pull request #4423 [BEAM-3486] Fixing Work Progress reporting
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1260a7da57530f100feb7773be737d2202c9d772 Merge: ca63b05 b0fc52e Author: Robert Bradshaw AuthorDate: Thu Jan 18 17:10:29 2018 -0800 Merge pull request #4423 [BEAM-3486] Fixing Work Progress reporting sdks/python/apache_beam/runners/worker/sdk_worker.py | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (ca63b05 -> 1260a7d)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ca63b05 Merge pull request #4435 add b0fc52e Fixing Work Progress reporting new 1260a7d Merge pull request #4423 [BEAM-3486] Fixing Work Progress reporting The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/worker/sdk_worker.py | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated: [BEAM-2732] Starting refactor of state tracking in Python (#4375)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 24e6bf8 [BEAM-2732] Starting refactor of state tracking in Python
(#4375)
24e6bf8 is described below
commit 24e6bf81790b97024fe1871575186d9db325bf1d
Author: Pablo
AuthorDate: Thu Jan 18 13:39:48 2018 -0800
[BEAM-2732] Starting refactor of state tracking in Python (#4375)
Also giving the Python-only state sampler full functionality.
---
.../runners/portability/fn_api_runner_test.py | 7 +-
.../runners/portability/maptask_executor_runner.py | 6 +-
.../apache_beam/runners/worker/bundle_processor.py | 7 +-
.../apache_beam/runners/worker/statesampler.py | 81
.../runners/worker/statesampler_fake.py| 51
.../{statesampler.pyx => statesampler_fast.pyx}| 136 ++---
.../runners/worker/statesampler_slow.py| 76
.../runners/worker/statesampler_test.py| 48 +---
sdks/python/apache_beam/utils/counters.py | 3 -
9 files changed, 225 insertions(+), 190 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index 6304f71..83bb83a 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -23,13 +23,14 @@ import apache_beam as beam
from apache_beam.runners.portability import fn_api_runner
from apache_beam.runners.portability import maptask_executor_runner_test
from apache_beam.runners.worker import sdk_worker
+from apache_beam.runners.worker import statesampler
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import window
-try:
- from apache_beam.runners.worker.statesampler import
DEFAULT_SAMPLING_PERIOD_MS
-except ImportError:
+if statesampler.FAST_SAMPLER:
+ DEFAULT_SAMPLING_PERIOD_MS = statesampler.DEFAULT_SAMPLING_PERIOD_MS
+else:
DEFAULT_SAMPLING_PERIOD_MS = 0
diff --git
a/sdks/python/apache_beam/runners/portability/maptask_executor_runner.py
b/sdks/python/apache_beam/runners/portability/maptask_executor_runner.py
index 913ccd5..74c6b03 100644
--- a/sdks/python/apache_beam/runners/portability/maptask_executor_runner.py
+++ b/sdks/python/apache_beam/runners/portability/maptask_executor_runner.py
@@ -36,15 +36,11 @@ from apache_beam.runners.runner import PipelineRunner
from apache_beam.runners.runner import PipelineState
from apache_beam.runners.worker import operation_specs
from apache_beam.runners.worker import operations
+from apache_beam.runners.worker import statesampler
from apache_beam.typehints import typehints
from apache_beam.utils import profiler
from apache_beam.utils.counters import CounterFactory
-try:
- from apache_beam.runners.worker import statesampler
-except ImportError:
- from apache_beam.runners.worker import statesampler_fake as statesampler
-
# This module is experimental. No backwards-compatibility guarantees.
diff --git a/sdks/python/apache_beam/runners/worker/bundle_processor.py
b/sdks/python/apache_beam/runners/worker/bundle_processor.py
index 97c318b..9bc9056 100644
--- a/sdks/python/apache_beam/runners/worker/bundle_processor.py
+++ b/sdks/python/apache_beam/runners/worker/bundle_processor.py
@@ -38,6 +38,7 @@ from apache_beam.portability.api import beam_runner_api_pb2
from apache_beam.runners import pipeline_context
from apache_beam.runners.worker import operation_specs
from apache_beam.runners.worker import operations
+from apache_beam.runners.worker import statesampler
from apache_beam.transforms import sideinputs
from apache_beam.utils import counters
from apache_beam.utils import proto_utils
@@ -46,12 +47,6 @@ from apache_beam.utils import urns
# This module is experimental. No backwards-compatibility guarantees.
-try:
- from apache_beam.runners.worker import statesampler
-except ImportError:
- from apache_beam.runners.worker import statesampler_fake as statesampler
-
-
DATA_INPUT_URN = 'urn:org.apache.beam:source:runner:0.1'
DATA_OUTPUT_URN = 'urn:org.apache.beam:sink:runner:0.1'
IDENTITY_DOFN_URN = 'urn:org.apache.beam:dofn:identity:0.1'
diff --git a/sdks/python/apache_beam/runners/worker/statesampler.py
b/sdks/python/apache_beam/runners/worker/statesampler.py
new file mode 100644
index 000..03af644
--- /dev/null
+++ b/sdks/python/apache_beam/runners/worker/statesampler.py
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this fil
[beam] 01/01: Merge pull request #4438 from robertwb/source-counters
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ed68f23eba0d8501dab04f086a38133fad1f3cd8 Merge: 1edea72 a1804f0 Author: Robert Bradshaw AuthorDate: Thu Jan 18 09:55:32 2018 -0800 Merge pull request #4438 from robertwb/source-counters Enable counters from sources on workers. sdks/python/apache_beam/metrics/execution.py | 4 ++-- sdks/python/apache_beam/runners/worker/operations.py | 20 ++-- .../python/apache_beam/transforms/ptransform_test.py | 4 +++- 3 files changed, 15 insertions(+), 13 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (1edea72 -> ed68f23)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1edea72 Merge pull request #4431 from robertwb/reshuffle add a1804f0 Enable counters from sources on workers. new ed68f23 Merge pull request #4438 from robertwb/source-counters The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/metrics/execution.py | 4 ++-- sdks/python/apache_beam/runners/worker/operations.py | 20 ++-- .../python/apache_beam/transforms/ptransform_test.py | 4 +++- 3 files changed, 15 insertions(+), 13 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated (6f12558 -> 1edea72)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6f12558 [BEAM-2863] Fix translation of side inputs due to missing method. add b65d4fa Fix reshuffle test on non-direct runners. add 2a972fc lint new 1edea72 Merge pull request #4431 from robertwb/reshuffle The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pvalue.py | 10 - sdks/python/apache_beam/testing/util.py | 3 ++ sdks/python/apache_beam/transforms/util.py | 8 +--- sdks/python/apache_beam/transforms/util_test.py | 50 +++-- 4 files changed, 34 insertions(+), 37 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4431 from robertwb/reshuffle
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1edea725389e2d3dc32201a7a416b4daa0bf5dc9 Merge: 6f12558 2a972fc Author: Robert Bradshaw AuthorDate: Thu Jan 18 09:54:44 2018 -0800 Merge pull request #4431 from robertwb/reshuffle Fix reshuffle test on non-direct runners. sdks/python/apache_beam/pvalue.py | 10 - sdks/python/apache_beam/testing/util.py | 3 ++ sdks/python/apache_beam/transforms/util.py | 8 +--- sdks/python/apache_beam/transforms/util_test.py | 50 +++-- 4 files changed, 34 insertions(+), 37 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] 01/01: Merge pull request #4408 from youngoli/beam3126
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 70a520ce9c0930742cf44efad6dbad1a5df29734 Merge: 441e30b bdcac3d Author: Robert Bradshaw AuthorDate: Tue Jan 16 17:09:37 2018 -0800 Merge pull request #4408 from youngoli/beam3126 [BEAM-3126] Creating flatten operation in Python SDK Harness sdks/python/apache_beam/runners/worker/bundle_processor.py | 14 ++ 1 file changed, 14 insertions(+) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (441e30b -> 70a520c)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 441e30b Merge pull request #4413 from DariuszAniszewski/use-virtualenv-for-perfkit-jobs add bdcac3d [BEAM-3126] Creating flatten operation for bundle processor. new 70a520c Merge pull request #4408 from youngoli/beam3126 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/worker/bundle_processor.py | 14 ++ 1 file changed, 14 insertions(+) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated (aff2a1e -> 5fd97c0)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aff2a1e Merge pull request #4366 from udim/baseexception add c2a0883 Return state from wait_until_finish. add 5fd97c0 Merge pull request #4369 from robertwb/return-state No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/portability/fn_api_runner.py | 2 +- sdks/python/apache_beam/runners/portability/universal_local_runner.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch go-sdk updated (f68f05e -> 2a0ba98)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch go-sdk in repository https://gitbox.apache.org/repos/asf/beam.git. from f68f05e Merge pull request #4350: Integrate master into go-sdk add d4b3624 Type-specialize custom decoders and encoders in Go SDK runtime add 7bdffd6 Type-specialize iterators and side input in the Go SDK add 9be0836 Add warnings if Go runtime registrations are overwritten add 0b43308 Add reusable element coders in Go SDK runtime new 2a0ba98 Merge pull request #4336 from herohde/runtime5 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/go/pkg/beam/core/runtime/exec/coder.go| 264 +- sdks/go/pkg/beam/core/runtime/exec/datasink.go | 7 +- sdks/go/pkg/beam/core/runtime/exec/datasource.go |52 +- sdks/go/pkg/beam/core/runtime/exec/decode.go |90 + sdks/go/pkg/beam/core/runtime/exec/decoders.go | 1347 ++ sdks/go/pkg/beam/core/runtime/exec/decoders.tmpl |89 + sdks/go/pkg/beam/core/runtime/exec/emit.go | 7 +- sdks/go/pkg/beam/core/runtime/exec/encode.go |91 + sdks/go/pkg/beam/core/runtime/exec/encoders.go | 1171 + sdks/go/pkg/beam/core/runtime/exec/encoders.tmpl |81 + sdks/go/pkg/beam/core/runtime/exec/fn.go | 138 +- .../pkg/beam/core/runtime/exec/fullvalue_test.go | 4 +- sdks/go/pkg/beam/core/runtime/exec/input.go| 190 + sdks/go/pkg/beam/core/runtime/exec/inputs.go | 21768 +++ sdks/go/pkg/beam/core/runtime/exec/inputs.tmpl | 144 + sdks/go/pkg/beam/core/runtime/exec/pardo.go| 2 +- sdks/go/pkg/beam/runners/direct/gbk.go | 6 +- sdks/go/pkg/beam/testing/passert/passert.go|16 +- 18 files changed, 25192 insertions(+), 275 deletions(-) create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decode.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decoders.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decoders.tmpl create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encode.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encoders.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encoders.tmpl create mode 100644 sdks/go/pkg/beam/core/runtime/exec/input.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/inputs.go create mode 100644 sdks/go/pkg/beam/core/runtime/exec/inputs.tmpl -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4336 from herohde/runtime5
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch go-sdk in repository https://gitbox.apache.org/repos/asf/beam.git commit 2a0ba9818c7f8468cdd1e2f8d28206882a5686bf Merge: f68f05e 0b43308 Author: Robert Bradshaw AuthorDate: Tue Jan 9 09:47:41 2018 -0800 Merge pull request #4336 from herohde/runtime5 [BEAM-3388] Avoid various reflection and re-processing in the Go SDK runtime sdks/go/pkg/beam/core/runtime/exec/coder.go| 264 +- sdks/go/pkg/beam/core/runtime/exec/datasink.go | 7 +- sdks/go/pkg/beam/core/runtime/exec/datasource.go |52 +- sdks/go/pkg/beam/core/runtime/exec/decode.go |90 + sdks/go/pkg/beam/core/runtime/exec/decoders.go | 1347 ++ sdks/go/pkg/beam/core/runtime/exec/decoders.tmpl |89 + sdks/go/pkg/beam/core/runtime/exec/emit.go | 7 +- sdks/go/pkg/beam/core/runtime/exec/encode.go |91 + sdks/go/pkg/beam/core/runtime/exec/encoders.go | 1171 + sdks/go/pkg/beam/core/runtime/exec/encoders.tmpl |81 + sdks/go/pkg/beam/core/runtime/exec/fn.go | 138 +- .../pkg/beam/core/runtime/exec/fullvalue_test.go | 4 +- sdks/go/pkg/beam/core/runtime/exec/input.go| 190 + sdks/go/pkg/beam/core/runtime/exec/inputs.go | 21768 +++ sdks/go/pkg/beam/core/runtime/exec/inputs.tmpl | 144 + sdks/go/pkg/beam/core/runtime/exec/pardo.go| 2 +- sdks/go/pkg/beam/runners/direct/gbk.go | 6 +- sdks/go/pkg/beam/testing/passert/passert.go|16 +- 18 files changed, 25192 insertions(+), 275 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] 01/01: Merge pull request #4344 from robertwb/user-metrics
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d2690fae0cd3c16b18973ed68f514a2f2319af2b Merge: 63e2696 a59b767 Author: Robert Bradshaw AuthorDate: Fri Jan 5 09:30:25 2018 -0800 Merge pull request #4344 from robertwb/user-metrics [BEAM-1866] Plumb user metrics through Fn API. .../fn-execution/src/main/proto/beam_fn_api.proto | 37 ++- sdks/python/apache_beam/metrics/cells.py | 13 +- sdks/python/apache_beam/metrics/execution.py | 24 ++ .../runners/portability/fn_api_runner.py | 365 ++--- .../runners/portability/fn_api_runner_test.py | 31 +- .../apache_beam/runners/worker/bundle_processor.py | 11 +- .../apache_beam/runners/worker/operations.py | 8 +- .../apache_beam/runners/worker/sdk_worker.py | 6 +- 8 files changed, 357 insertions(+), 138 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (63e2696 -> d2690fa)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 63e2696 Merge pull request #4342 from chamikaramj/beam_sdf_cython_fix add a59b767 Plumb user metrics through Fn API. new d2690fa Merge pull request #4344 from robertwb/user-metrics The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 37 ++- sdks/python/apache_beam/metrics/cells.py | 13 +- sdks/python/apache_beam/metrics/execution.py | 24 ++ .../runners/portability/fn_api_runner.py | 365 ++--- .../runners/portability/fn_api_runner_test.py | 31 +- .../apache_beam/runners/worker/bundle_processor.py | 11 +- .../apache_beam/runners/worker/operations.py | 8 +- .../apache_beam/runners/worker/sdk_worker.py | 6 +- 8 files changed, 357 insertions(+), 138 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated: Correctly handle pass-through "composites" in the Fn Api Runner. (#4326)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7d9bf07 Correctly handle pass-through "composites" in the Fn Api
Runner. (#4326)
7d9bf07 is described below
commit 7d9bf07f876a82ee5d10a4aca24197e470603f90
Author: Robert Bradshaw
AuthorDate: Tue Jan 2 17:24:03 2018 -0800
Correctly handle pass-through "composites" in the Fn Api Runner. (#4326)
---
.../python/apache_beam/runners/portability/fn_api_runner.py | 6 +-
.../apache_beam/runners/portability/fn_api_runner_test.py | 13 +++--
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index c5829a4..c921857 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -773,8 +773,12 @@ class FnApiRunner(runner.PipelineRunner):
def leaf_transforms(root_ids):
for root_id in root_ids:
root = pipeline_proto.components.transforms[root_id]
-if root.spec.urn in known_composites or not root.subtransforms:
+if root.spec.urn in known_composites:
yield root_id
+elif not root.subtransforms:
+ # Make sure its outputs are not a subset of its inputs.
+ if set(root.outputs.values()) - set(root.inputs.values()):
+yield root_id
else:
for leaf in leaf_transforms(root.subtransforms):
yield leaf
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index c15b329..1cffa26 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -84,7 +84,6 @@ class FnApiRunnerTest(
side1 = p | 'side1' >> beam.Create([('a', 1)])
side2 = p | 'side2' >> beam.Create([('b', 2)])
side = (side1, side2) | beam.Flatten()
- _ = main | 'Do' >> beam.Map(lambda a, b: (a, b),
beam.pvalue.AsDict(side))
assert_that(
main | beam.Map(lambda a, b: (a, b), beam.pvalue.AsDict(side)),
equal_to([(None, {'a': 1, 'b': 2})]))
@@ -93,7 +92,6 @@ class FnApiRunnerTest(
with self.create_pipeline() as p:
main = p | 'main' >> beam.Create([None])
side = p | 'side' >> beam.Create([('a', 1)]) | beam.GroupByKey()
- _ = main | 'Do' >> beam.Map(lambda a, b: (a, b),
beam.pvalue.AsDict(side))
assert_that(
main | beam.Map(lambda a, b: (a, b), beam.pvalue.AsDict(side)),
equal_to([(None, {'a': [1]})]))
@@ -105,6 +103,17 @@ class FnApiRunnerTest(
with self.create_pipeline() as p:
assert_that(p | beam.Create(['a', 'b']), equal_to(['a']))
+ def test_no_subtransform_composite(self):
+
+class First(beam.PTransform):
+ def expand(self, pcolls):
+return pcolls[0]
+
+with self.create_pipeline() as p:
+ pcoll_a = p | 'a' >> beam.Create(['a'])
+ pcoll_b = p | 'b' >> beam.Create(['b'])
+ assert_that((pcoll_a, pcoll_b) | First(), equal_to(['a']))
+
def test_progress_metrics(self):
p = self.create_pipeline()
if not isinstance(p.runner, fn_api_runner.FnApiRunner):
--
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].
[beam] branch master updated (180e19b -> 5a72437)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 180e19b Avoid importing testing hierarchy for Javadoc comments. add 694c050 Clean data receive queue new 5a72437 Merge pull request #4315 from angoenka/clean_receive_queue The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache_beam/runners/worker/data_plane.py | 39 -- 1 file changed, 28 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4315 from angoenka/clean_receive_queue
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5a72437f84c1d253f6451fac0eb90ae9360558f8 Merge: 180e19b 694c050 Author: Robert Bradshaw AuthorDate: Tue Jan 2 12:17:36 2018 -0800 Merge pull request #4315 from angoenka/clean_receive_queue [BEAM-3389] Clean data receive queue .../apache_beam/runners/worker/data_plane.py | 39 -- 1 file changed, 28 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] 01/01: Merge pull request #4313 from robertwb/robertwb-patch-1
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit aaa0ae6d9df55a4d22d9a817af37a08457dd4628 Merge: 95e1751 cc24c71 Author: Robert Bradshaw AuthorDate: Fri Dec 22 17:12:33 2017 -0800 Merge pull request #4313 from robertwb/robertwb-patch-1 Remove extraneous logging. .../java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java| 1 - 1 file changed, 1 deletion(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (95e1751 -> aaa0ae6)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 95e1751 Merge pull request #4325 from tvalentyn/all_released_sdks_use_containers_tagged_with_beam_version add cc24c71 Remove extraneous logging. new aaa0ae6 Merge pull request #4313 from robertwb/robertwb-patch-1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java| 1 - 1 file changed, 1 deletion(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated: Fix for side inputs following flatten/gbk. (#4281)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d468d4a Fix for side inputs following flatten/gbk. (#4281)
d468d4a is described below
commit d468d4a927031d9d181f9fd39d7c558abeeef8d1
Author: Robert Bradshaw
AuthorDate: Wed Dec 20 13:30:58 2017 -0800
Fix for side inputs following flatten/gbk. (#4281)
---
.../apache_beam/runners/portability/fn_api_runner.py | 4 ++--
.../runners/portability/fn_api_runner_test.py| 20
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index 683f52d..c5829a4 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -525,7 +525,7 @@ class FnApiRunner(runner.PipelineRunner):
spec=beam_runner_api_pb2.FunctionSpec(
urn=bundle_processor.DATA_INPUT_URN,
payload=param))],
- downstream_side_inputs=frozenset(),
+ downstream_side_inputs=stage.downstream_side_inputs,
must_follow=union(frozenset([gbk_write]), stage.must_follow))
else:
yield stage
@@ -594,7 +594,7 @@ class FnApiRunner(runner.PipelineRunner):
spec=beam_runner_api_pb2.FunctionSpec(
urn=bundle_processor.DATA_INPUT_URN,
payload=param))],
- downstream_side_inputs=frozenset(),
+ downstream_side_inputs=stage.downstream_side_inputs,
must_follow=union(frozenset(flatten_writes), stage.must_follow))
else:
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index eb297ab..c15b329 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -78,6 +78,26 @@ class FnApiRunnerTest(
(9, range(7, 10))]),
label='windowed')
+ def test_flattened_side_input(self):
+with self.create_pipeline() as p:
+ main = p | 'main' >> beam.Create([None])
+ side1 = p | 'side1' >> beam.Create([('a', 1)])
+ side2 = p | 'side2' >> beam.Create([('b', 2)])
+ side = (side1, side2) | beam.Flatten()
+ _ = main | 'Do' >> beam.Map(lambda a, b: (a, b),
beam.pvalue.AsDict(side))
+ assert_that(
+ main | beam.Map(lambda a, b: (a, b), beam.pvalue.AsDict(side)),
+ equal_to([(None, {'a': 1, 'b': 2})]))
+
+ def test_gbk_side_input(self):
+with self.create_pipeline() as p:
+ main = p | 'main' >> beam.Create([None])
+ side = p | 'side' >> beam.Create([('a', 1)]) | beam.GroupByKey()
+ _ = main | 'Do' >> beam.Map(lambda a, b: (a, b),
beam.pvalue.AsDict(side))
+ assert_that(
+ main | beam.Map(lambda a, b: (a, b), beam.pvalue.AsDict(side)),
+ equal_to([(None, {'a': [1]})]))
+
def test_assert_that(self):
# TODO: figure out a way for fn_api_runner to parse and raise the
# underlying exception.
--
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].
[beam] branch master updated: [BEAM-2937] Basic PGBK combiner lifting. (#4290)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e92f718 [BEAM-2937] Basic PGBK combiner lifting. (#4290)
e92f718 is described below
commit e92f71879c2bf5d3b3f383721e2b93c46f877b92
Author: Robert Bradshaw
AuthorDate: Wed Dec 20 13:30:19 2017 -0800
[BEAM-2937] Basic PGBK combiner lifting. (#4290)
---
.../runners/portability/fn_api_runner.py | 161 -
.../apache_beam/runners/worker/bundle_processor.py | 54 +++
.../apache_beam/runners/worker/operations.py | 1 +
sdks/python/apache_beam/utils/urns.py | 3 +
4 files changed, 216 insertions(+), 3 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index aa381f6..683f52d 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -171,6 +171,14 @@ class _WindowGroupingBuffer(object):
class FnApiRunner(runner.PipelineRunner):
def __init__(self, use_grpc=False, sdk_harness_factory=None):
+"""Creates a new Fn API Runner.
+
+Args:
+ use_grpc: whether to use grpc or simply make in-process calls
+ defaults to False
+ sdk_harness_factory: callable used to instantiate customized sdk
harnesses
+ typcially not set by users
+"""
super(FnApiRunner, self).__init__()
self._last_uid = -1
self._use_grpc = use_grpc
@@ -277,6 +285,150 @@ class FnApiRunner(runner.PipelineRunner):
safe_coders = {}
+def lift_combiners(stages):
+ """Expands CombinePerKey into pre- and post-grouping stages.
+
+ ... -> CombinePerKey -> ...
+
+ becomes
+
+ ... -> PreCombine -> GBK -> MergeAccumulators -> ExtractOutput -> ...
+ """
+ def add_or_get_coder_id(coder_proto):
+for coder_id, coder in pipeline_components.coders.items():
+ if coder == coder_proto:
+return coder_id
+new_coder_id = unique_name(pipeline_components.coders, 'coder')
+pipeline_components.coders[new_coder_id].CopyFrom(coder_proto)
+return new_coder_id
+
+ def windowed_coder_id(coder_id):
+proto = beam_runner_api_pb2.Coder(
+spec=beam_runner_api_pb2.SdkFunctionSpec(
+spec=beam_runner_api_pb2.FunctionSpec(
+urn=urns.WINDOWED_VALUE_CODER)),
+component_coder_ids=[coder_id, window_coder_id])
+return add_or_get_coder_id(proto)
+
+ for stage in stages:
+assert len(stage.transforms) == 1
+transform = stage.transforms[0]
+if transform.spec.urn == urns.COMBINE_PER_KEY_TRANSFORM:
+ combine_payload = proto_utils.parse_Bytes(
+ transform.spec.payload, beam_runner_api_pb2.CombinePayload)
+
+ input_pcoll = pipeline_components.pcollections[only_element(
+ transform.inputs.values())]
+ output_pcoll = pipeline_components.pcollections[only_element(
+ transform.outputs.values())]
+
+ windowed_input_coder = pipeline_components.coders[
+ input_pcoll.coder_id]
+ element_coder_id, window_coder_id = (
+ windowed_input_coder.component_coder_ids)
+ element_coder = pipeline_components.coders[element_coder_id]
+ key_coder_id, _ = element_coder.component_coder_ids
+ accumulator_coder_id = combine_payload.accumulator_coder_id
+
+ key_accumulator_coder = beam_runner_api_pb2.Coder(
+ spec=beam_runner_api_pb2.SdkFunctionSpec(
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=urns.KV_CODER)),
+ component_coder_ids=[key_coder_id, accumulator_coder_id])
+ key_accumulator_coder_id = add_or_get_coder_id(key_accumulator_coder)
+
+ accumulator_iter_coder = beam_runner_api_pb2.Coder(
+ spec=beam_runner_api_pb2.SdkFunctionSpec(
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=urns.ITERABLE_CODER)),
+ component_coder_ids=[accumulator_coder_id])
+ accumulator_iter_coder_id = add_or_get_coder_id(
+ accumulator_iter_coder)
+
+ key_accumulator_iter_coder = beam_runner_api_pb2.Coder(
+ spec=beam_runner_api_pb2.SdkFunctionSpec(
+ spec=beam_runner_api_pb2.FunctionSpec(
+ urn=urns.KV_CODER)),
+ component_coder_ids=[key_coder_id, accumulator_iter_coder_id])
+ key_accumulator_iter_coder_id = add_or_get_coder_id(
+ key_accumulator_iter_coder)
+
+ precom
[beam] 01/01: Merge pull request #4293 from robertwb/BEAM-3369
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 157787eceaf8e358e82508d0ea173164a6a97237 Merge: 0aa943d 550d6a5 Author: Robert Bradshaw AuthorDate: Tue Dec 19 14:49:55 2017 -0800 Merge pull request #4293 from robertwb/BEAM-3369 [BEAM-3369] Allow any coder to specify its deterministic variant. sdks/python/apache_beam/coders/coders.py | 40 sdks/python/apache_beam/coders/typecoders.py | 15 +-- 2 files changed, 41 insertions(+), 14 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (0aa943d -> 157787e)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0aa943d Merge pull request #4292 from robertwb/BEAM-3369 add 550d6a5 [BEAM-3369] Allow any coder to specify its deterministic variant. new 157787e Merge pull request #4293 from robertwb/BEAM-3369 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/coders/coders.py | 40 sdks/python/apache_beam/coders/typecoders.py | 15 +-- 2 files changed, 41 insertions(+), 14 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated (6709419 -> 0aa943d)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6709419 Merge pull request #3817 from jkff/fileio-write add 76d32e2 [BEAM-3369] Fix typehint errors introduced in PR #4282. new 0aa943d Merge pull request #4292 from robertwb/BEAM-3369 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/coders/typecoders.py | 4 sdks/python/apache_beam/typehints/opcodes.py | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4292 from robertwb/BEAM-3369
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0aa943df8e4a15669bd32672a587c69020161da7 Merge: 6709419 76d32e2 Author: Robert Bradshaw AuthorDate: Tue Dec 19 12:19:56 2017 -0800 Merge pull request #4292 from robertwb/BEAM-3369 [BEAM-3369] Fix typehint errors introduced in PR #4282. sdks/python/apache_beam/coders/typecoders.py | 4 sdks/python/apache_beam/typehints/opcodes.py | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] 01/01: Merge pull request #4247 from robertwb/runner-run
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1ff54bb17a45a532ffc1c9377da6e63760618180 Merge: 1acd1ae 8cf222d Author: Robert Bradshaw AuthorDate: Mon Dec 18 17:51:35 2017 -0800 Merge pull request #4247 from robertwb/runner-run [BEAM-3183] Add runner.run(transform) to Python SDK. sdks/python/apache_beam/pipeline.py | 2 +- .../apache_beam/runners/dataflow/dataflow_runner.py | 4 ++-- sdks/python/apache_beam/runners/direct/direct_runner.py | 2 +- .../apache_beam/runners/portability/fn_api_runner.py| 2 +- .../runners/portability/maptask_executor_runner.py | 4 ++-- .../runners/portability/universal_local_runner.py | 2 +- sdks/python/apache_beam/runners/runner.py | 17 +++-- sdks/python/apache_beam/runners/runner_test.py | 12 +++- 8 files changed, 34 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (1acd1ae -> 1ff54bb)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1acd1ae Merge pull request #4282 from robertwb/type-inference add 8cf222d [BEAM-3183] Add runner.run(transform) to Python SDK. new 1ff54bb Merge pull request #4247 from robertwb/runner-run The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline.py | 2 +- .../apache_beam/runners/dataflow/dataflow_runner.py | 4 ++-- sdks/python/apache_beam/runners/direct/direct_runner.py | 2 +- .../apache_beam/runners/portability/fn_api_runner.py| 2 +- .../runners/portability/maptask_executor_runner.py | 4 ++-- .../runners/portability/universal_local_runner.py | 2 +- sdks/python/apache_beam/runners/runner.py | 17 +++-- sdks/python/apache_beam/runners/runner_test.py | 12 +++- 8 files changed, 34 insertions(+), 11 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4282 from robertwb/type-inference
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1acd1ae901eefbcc8249d90e12ca82db0f91e41e Merge: 26560d0 8103010 Author: Robert Bradshaw AuthorDate: Mon Dec 18 17:49:12 2017 -0800 Merge pull request #4282 from robertwb/type-inference Fix getitem and list comprehension type inference. sdks/python/apache_beam/typehints/opcodes.py | 19 ++- .../apache_beam/typehints/trivial_inference_test.py | 20 2 files changed, 34 insertions(+), 5 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (26560d0 -> 1acd1ae)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 26560d0 Guard against closing data channel twice (#4283) add 8103010 Fix getitem and list comprehension type inference. new 1acd1ae Merge pull request #4282 from robertwb/type-inference The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/typehints/opcodes.py | 19 ++- .../apache_beam/typehints/trivial_inference_test.py | 20 2 files changed, 34 insertions(+), 5 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated: Guard against closing data channel twice (#4283)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 26560d0 Guard against closing data channel twice (#4283)
26560d0 is described below
commit 26560d0effcfd12bf9bfa40c1040e00f093e88ca
Author: Robert Bradshaw
AuthorDate: Mon Dec 18 17:48:30 2017 -0800
Guard against closing data channel twice (#4283)
---
.../fn/data/BeamFnDataBufferingOutboundObserver.java | 9 +
.../data/BeamFnDataBufferingOutboundObserverTest.java | 19 +++
2 files changed, 28 insertions(+)
diff --git
a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
index 9c5fd36..d2986c3 100644
---
a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
+++
b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java
@@ -71,6 +71,7 @@ public class BeamFnDataBufferingOutboundObserver
private long byteCounter;
private long counter;
+ private boolean closed;
private final int bufferLimit;
private final Coder> coder;
private final LogicalEndpoint outputLocation;
@@ -87,10 +88,15 @@ public class BeamFnDataBufferingOutboundObserver
this.coder = coder;
this.outboundObserver = outboundObserver;
this.bufferedElements = ByteString.newOutput();
+this.closed = false;
}
@Override
public void close() throws Exception {
+if (closed) {
+ throw new IllegalStateException("Already closed.");
+}
+closed = true;
BeamFnApi.Elements.Builder elements = convertBufferForTransmission();
// This will add an empty data block representing the end of stream.
elements.addDataBuilder()
@@ -108,6 +114,9 @@ public class BeamFnDataBufferingOutboundObserver
@Override
public void accept(WindowedValue t) throws IOException {
+if (closed) {
+ throw new IllegalStateException("Already closed.");
+}
coder.encode(t, bufferedElements);
counter += 1;
if (bufferedElements.size() >= bufferLimit) {
diff --git
a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
index d2f2cf9..30fae87 100644
---
a/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
+++
b/sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserverTest.java
@@ -21,6 +21,7 @@ import static
org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
import static org.hamcrest.Matchers.empty;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
+import static org.junit.Assert.fail;
import com.google.common.collect.Iterables;
import com.google.protobuf.ByteString;
@@ -98,6 +99,24 @@ public class BeamFnDataBufferingOutboundObserverTest {
consumer.close();
assertEquals(messageWithData(),
Iterables.get(values, 2));
+
+// Test that we can't write to a closed stream.
+try {
+ consumer.accept(
+ valueInGlobalWindow(
+ new
byte[BeamFnDataBufferingOutboundObserver.DEFAULT_BUFFER_LIMIT_BYTES - 50]));
+ fail("Writing after close should be prohibited.");
+} catch (IllegalStateException exn) {
+ // expected
+}
+
+// Test that we can't close a stream twice.
+try {
+ consumer.close();
+ fail("Closing twice should be prohibited.");
+} catch (IllegalStateException exn) {
+ // expected
+}
}
@Test
--
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].
[beam] branch go-sdk updated: [BEAM-3356] Add Go SDK int and varint custom coders (#4276)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/go-sdk by this push:
new 14eb2cd [BEAM-3356] Add Go SDK int and varint custom coders (#4276)
14eb2cd is described below
commit 14eb2cddcd11c73f79555bc8ddbc121e29b10c4d
Author: Henning Rohde
AuthorDate: Mon Dec 18 15:37:50 2017 -0800
[BEAM-3356] Add Go SDK int and varint custom coders (#4276)
* Add Go SDK int and varint custom coders
* CR: [BEAM-3356] Add Go SDK int and varint custom coders
---
sdks/go/pkg/beam/coder.go| 29 +++
sdks/go/pkg/beam/core/graph/coder/int.go | 66
sdks/go/pkg/beam/core/graph/coder/varint.go | 66 ++--
sdks/go/pkg/beam/core/graph/coder/varint_test.go | 64 +++
4 files changed, 211 insertions(+), 14 deletions(-)
diff --git a/sdks/go/pkg/beam/coder.go b/sdks/go/pkg/beam/coder.go
index 9405573..b81152a 100644
--- a/sdks/go/pkg/beam/coder.go
+++ b/sdks/go/pkg/beam/coder.go
@@ -85,18 +85,29 @@ func inferCoder(t FullType) (*coder.Coder, error) {
switch t.Class() {
case typex.Concrete, typex.Container:
switch t.Type() {
- // The type conversions here are very conservative. We handle
bytes/strings
- // equivalently because they are essentially equivalent in the
language.
- // Notably, we do not (currently) support equivalences in
numeric data types
- // due to risks around inadvertent widening or narrowing of
data.
+ case reflectx.Int, reflectx.Int8, reflectx.Int16,
reflectx.Int32, reflectx.Int64:
+ c, err := coder.NewVarIntZ(t.Type())
+ if err != nil {
+ return nil, err
+ }
+ return &coder.Coder{Kind: coder.Custom, T: t, Custom:
c}, nil
+ case reflectx.Uint, reflectx.Uint8, reflectx.Uint16,
reflectx.Uint32, reflectx.Uint64:
+ c, err := coder.NewVarUintZ(t.Type())
+ if err != nil {
+ return nil, err
+ }
+ return &coder.Coder{Kind: coder.Custom, T: t, Custom:
c}, nil
case reflectx.String, reflectx.ByteSlice:
+ // We handle bytes/strings equivalently because they
are essentially
+ // equivalent in the language. We generally infer exact
coders only.
return &coder.Coder{Kind: coder.Bytes}, nil
+ default:
+ c, err := newJSONCoder(t.Type())
+ if err != nil {
+ return nil, err
+ }
+ return &coder.Coder{Kind: coder.Custom, T: t, Custom:
c}, nil
}
- c, err := newJSONCoder(t.Type())
- if err != nil {
- return nil, err
- }
- return &coder.Coder{Kind: coder.Custom, T: t, Custom: c}, nil
case typex.Composite:
c, err := inferCoders(t.Components())
diff --git a/sdks/go/pkg/beam/core/graph/coder/int.go
b/sdks/go/pkg/beam/core/graph/coder/int.go
index 336e32b..77f2814 100644
--- a/sdks/go/pkg/beam/core/graph/coder/int.go
+++ b/sdks/go/pkg/beam/core/graph/coder/int.go
@@ -20,8 +20,74 @@ import (
"io"
"github.com/apache/beam/sdks/go/pkg/beam/core/util/ioutilx"
+ "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
)
+var (
+ // Fixed-sized custom coders for integers.
+
+ Uint32 *CustomCoder
+ Int32 *CustomCoder
+ Uint64 *CustomCoder
+ Int64 *CustomCoder
+)
+
+func init() {
+ var err error
+ Uint32, err = NewCustomCoder("uint32", reflectx.Uint32, encUint32,
decUint32)
+ if err != nil {
+ panic(err)
+ }
+ Int32, err = NewCustomCoder("int32", reflectx.Int32, encInt32, decInt32)
+ if err != nil {
+ panic(err)
+ }
+ Uint64, err = NewCustomCoder("uint64", reflectx.Uint64, encUint64,
decUint64)
+ if err != nil {
+ panic(err)
+ }
+ Int64, err = NewCustomCoder("int64", reflectx.Int64, encInt64, decInt64)
+ if err != nil {
+ panic(err)
+ }
+}
+
+func encUint32(v uint32) []byte {
+ ret := make([]byte, 4)
+ binary.BigEndian.PutUint32(ret, v)
+ return ret
+}
+
+func decUint32(data []byte) uint32 {
+ return binary.BigEndian.Uint32(data)
+}
+
+func encInt32(v int32) []byte {
+ return encUint32(uint32(v))
+}
+
+func decInt32(data []byte) int32 {
+ return
[beam] branch master updated: [BEAM-3143] Type Inference Python 3 Compatibility (#4183)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a424c64 [BEAM-3143] Type Inference Python 3 Compatibility (#4183) a424c64 is described below commit a424c64a3234a178a2a86161ee200c8be79d8f53 Author: Luke Zhu AuthorDate: Mon Dec 18 14:06:53 2017 -0500 [BEAM-3143] Type Inference Python 3 Compatibility (#4183) --- sdks/python/apache_beam/typehints/opcodes.py | 34 -- .../apache_beam/typehints/trivial_inference.py | 53 ++ .../typehints/trivial_inference_test.py| 18 +++- 3 files changed, 79 insertions(+), 26 deletions(-) diff --git a/sdks/python/apache_beam/typehints/opcodes.py b/sdks/python/apache_beam/typehints/opcodes.py index dcca6d0..ccf0195 100644 --- a/sdks/python/apache_beam/typehints/opcodes.py +++ b/sdks/python/apache_beam/typehints/opcodes.py @@ -28,9 +28,13 @@ For internal use only; no backwards-compatibility guarantees. """ from __future__ import absolute_import +import inspect +import sys import types from functools import reduce +import six + from . import typehints from .trivial_inference import BoundMethod from .trivial_inference import Const @@ -147,7 +151,7 @@ binary_subtract = inplace_subtract = symmetric_binary_op def binary_subscr(state, unused_arg): tos = state.stack.pop() - if tos in (str, unicode): + if tos in (str, six.text_type): out = tos else: out = element_type(tos) @@ -261,13 +265,22 @@ build_map = push_value(Dict[Any, Any]) def load_attr(state, arg): + """Replaces the top of the stack, TOS, with + getattr(TOS, co_names[arg]) + """ o = state.stack.pop() name = state.get_name(arg) if isinstance(o, Const) and hasattr(o.value, name): state.stack.append(Const(getattr(o.value, name))) - elif (isinstance(o, type) -and isinstance(getattr(o, name, None), types.MethodType)): -state.stack.append(Const(BoundMethod(getattr(o, name + elif (inspect.isclass(o) and +isinstance(getattr(o, name, None), + (types.MethodType, types.FunctionType))): +# TODO(luke-zhu): Support other callable objects +if sys.version_info[0] == 2: + func = getattr(o, name).__func__ +else: + func = getattr(o, name) # Python 3 has no unbound methods +state.stack.append(Const(BoundMethod(func, o))) else: state.stack.append(Any) @@ -327,7 +340,18 @@ def call_function(state, arg, has_var=False, has_kw=False): def make_function(state, arg): - state.stack[-arg - 1:] = [Any] # a callable + """Creates a function with the arguments at the top of the stack. + """ + # TODO(luke-zhu): Handle default argument types + globals = state.f.__globals__ # Inherits globals from the current frame + if sys.version_info[0] == 2: +func_code = state.stack[-1].value +func = types.FunctionType(func_code, globals) + else: +func_name = state.stack[-1].value +func_code = state.stack[-2].value +func = types.FunctionType(func_code, globals, name=func_name) + state.stack.append(Const(func)) def make_closure(state, arg): diff --git a/sdks/python/apache_beam/typehints/trivial_inference.py b/sdks/python/apache_beam/typehints/trivial_inference.py index a68bd18..28bf8f5 100644 --- a/sdks/python/apache_beam/typehints/trivial_inference.py +++ b/sdks/python/apache_beam/typehints/trivial_inference.py @@ -22,9 +22,9 @@ For internal use only; no backwards-compatibility guarantees. from __future__ import absolute_import from __future__ import print_function -import __builtin__ import collections import dis +import inspect import pprint import sys import types @@ -32,6 +32,8 @@ from functools import reduce from apache_beam.typehints import Any from apache_beam.typehints import typehints +from six.moves import builtins +from six.moves import zip class TypeInferenceError(ValueError): @@ -46,9 +48,9 @@ def instance_to_type(o): if o is None: return type(None) elif t not in typehints.DISALLOWED_PRIMITIVE_TYPES: -if t == types.InstanceType: +if sys.version_info[0] == 2 and t == types.InstanceType: return o.__class__ -elif t == BoundMethod: +if t == BoundMethod: return types.MethodType return t elif t == tuple: @@ -107,15 +109,12 @@ class FrameState(object): def __init__(self, f, local_vars=None, stack=()): self.f = f -if sys.version_info[0] >= 3: - self.co = f.__code__ -else: - self.co = f.func_code +self.co = f.__code__ self.vars = list(local_vars) self.stack = list(stack) def __eq__(self, other): -return self.__dict__ == other.__dict__ +return isinstance(other, FrameState) and self.__dict__ == other.__dict__ def copy(self):
[beam] branch master updated (02c766a -> e4a187b)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 02c766a Merge pull request #4257 add e4a187b Allow Java Harness to understand Fns the Java Runner produces. (#4214) No new revisions were added by this update. Summary of changes: .../beam/fn/harness/BeamFnDataReadRunner.java | 1 + .../beam/fn/harness/BeamFnDataWriteRunner.java | 1 + .../beam/fn/harness/BoundedSourceRunner.java | 28 -- .../apache/beam/fn/harness/FnApiDoFnRunner.java| 107 +++-- .../beam/fn/harness/PTransformRunnerFactory.java | 2 + .../fn/harness/control/ProcessBundleHandler.java | 8 +- .../fn/harness/logging/BeamFnLoggingClient.java| 2 +- .../beam/fn/harness/BeamFnDataReadRunnerTest.java | 1 + .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 1 + .../beam/fn/harness/BoundedSourceRunnerTest.java | 11 ++- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 11 ++- .../harness/control/ProcessBundleHandlerTest.java | 6 ++ 12 files changed, 154 insertions(+), 25 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4255 from robertwb/output-to-receive
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit aba5b0d056a555922680e3530ca826857ed208ba Merge: 511c91e 7011651 Author: Robert Bradshaw AuthorDate: Thu Dec 14 13:00:44 2017 -0800 Merge pull request #4255 from robertwb/output-to-receive Followup fix to pr/4195. sdks/python/apache_beam/runners/direct/transform_evaluator.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated (511c91e -> aba5b0d)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 511c91e Fix javadoc in FnDataReceiver add 7011651 Followup fix to pr/4195. new aba5b0d Merge pull request #4255 from robertwb/output-to-receive The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/direct/transform_evaluator.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] branch master updated: Minor changes to the FnApi Runner to allow Java to run. (#4220)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 60dd7c4 Minor changes to the FnApi Runner to allow Java to run.
(#4220)
60dd7c4 is described below
commit 60dd7c4035a8d320b01823cbd4c959d0ef09cf26
Author: Robert Bradshaw
AuthorDate: Tue Dec 12 17:30:59 2017 -0800
Minor changes to the FnApi Runner to allow Java to run. (#4220)
* Minor changes to the FnApi Runner to allow Java to run.
* Aligned the GBK urn.
* Started a logging service, which Java can't run without.
* Added some logging that was helpful for debugging.
---
.../runners/portability/fn_api_runner.py | 7 +-
.../runners/portability/universal_local_runner.py | 29 +-
.../portability/universal_local_runner_main.py | 1 +
sdks/python/apache_beam/utils/urns.py | 2 +-
4 files changed, 36 insertions(+), 3 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index e40faa5..9b143c6 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -745,10 +745,12 @@ class FnApiRunner(runner.PipelineRunner):
state_key, elements_data, process_bundle.instruction_id)
# Register and start running the bundle.
+logging.debug('Register and start running the bundle')
controller.control_handler.push(process_bundle_registration)
controller.control_handler.push(process_bundle)
# Wait for the bundle to finish.
+logging.debug('Wait for the bundle to finish.')
while True:
result = controller.control_handler.pull()
if result and result.instruction_id == process_bundle.instruction_id:
@@ -756,11 +758,14 @@ class FnApiRunner(runner.PipelineRunner):
raise RuntimeError(result.error)
break
-# Gather all output data.
expected_targets = [
beam_fn_api_pb2.Target(primitive_transform_reference=transform_id,
name=output_name)
for (transform_id, output_name), _ in data_output.items()]
+
+# Gather all output data.
+logging.debug('Gather all output data from %s.', expected_targets)
+
for output in controller.data_plane_handler.input_elements(
process_bundle.instruction_id, expected_targets):
target_tuple = (
diff --git
a/sdks/python/apache_beam/runners/portability/universal_local_runner.py
b/sdks/python/apache_beam/runners/portability/universal_local_runner.py
index b951194..365803d 100644
--- a/sdks/python/apache_beam/runners/portability/universal_local_runner.py
+++ b/sdks/python/apache_beam/runners/portability/universal_local_runner.py
@@ -31,6 +31,7 @@ from concurrent import futures
import grpc
from google.protobuf import text_format
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
from apache_beam.portability.api import beam_job_api_pb2
from apache_beam.portability.api import beam_job_api_pb2_grpc
from apache_beam.portability.api import endpoints_pb2
@@ -241,6 +242,7 @@ class BeamJob(threading.Thread):
use_grpc=self._use_grpc,
sdk_harness_factory=self._sdk_harness_factory
).run_via_runner_api(self._pipeline_proto)
+logging.info("Successfully completed job.")
self.state = beam_job_api_pb2.JobState.DONE
except: # pylint: disable=bare-except
logging.exception("Error running pipeline.")
@@ -271,10 +273,12 @@ class
JobServicer(beam_job_api_pb2_grpc.JobServiceServicer):
port = self._server.add_insecure_port('localhost:%d' % port)
beam_job_api_pb2_grpc.add_JobServiceServicer_to_server(self, self._server)
self._server.start()
+logging.info("Grpc server started on port %s", port)
return port
def Prepare(self, request, context=None):
# For now, just use the job name as the job id.
+logging.debug("Got Prepare request.")
preparation_id = "%s-%s" % (request.job_name, uuid.uuid4())
if self._worker_command_line:
sdk_harness_factory = functools.partial(
@@ -284,10 +288,12 @@ class
JobServicer(beam_job_api_pb2_grpc.JobServiceServicer):
self._jobs[preparation_id] = BeamJob(
preparation_id, request.pipeline_options, request.pipeline,
use_grpc=self._use_grpc, sdk_harness_factory=sdk_harness_factory)
+logging.debug("Prepared job '%s' as '%s'", request.job_name,
preparation_id)
return beam_job_api_pb2.PrepareJobResponse(preparation_id=preparation_id)
def Run(self, request, context=None):
job_id = request.preparation_id
+logging.debug("Runing job
[beam] branch master updated: [BEAM-1872] Add IdentityWindowFn for use in Reshuffle (#4040)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9c1eeb2 [BEAM-1872] Add IdentityWindowFn for use in Reshuffle (#4040) 9c1eeb2 is described below commit 9c1eeb2d7efc65978d189ebc259fad6fbe2176ce Author: Udi Meiri AuthorDate: Wed Dec 6 13:30:58 2017 -0800 [BEAM-1872] Add IdentityWindowFn for use in Reshuffle (#4040) * Implement Reshuffle for Python SDK. Two flavors of Reshuffle: ReshufflePerKey operates on key-value pairs, while Reshuffle adds a random key to each element (key-value or other). Add _IdentityWindowFn, for internal use in Reshuffle. Add and pass current window to WindowFn.AssignContext, for IdentityWindowFn implementation. testing/util.py: - Extend assert_that with reify_windows keyword, allowing verification of timestamp values and windowing functions. - Add contains_in_any_order matcher. --- .../apache_beam/runners/worker/bundle_processor.py | 5 +- sdks/python/apache_beam/testing/util.py| 41 +++- sdks/python/apache_beam/testing/util_test.py | 42 sdks/python/apache_beam/transforms/core.py | 6 +- sdks/python/apache_beam/transforms/util.py | 112 +- sdks/python/apache_beam/transforms/util_test.py| 230 + sdks/python/apache_beam/transforms/window.py | 12 +- sdks/python/apache_beam/utils/urns.py | 5 +- 8 files changed, 442 insertions(+), 11 deletions(-) diff --git a/sdks/python/apache_beam/runners/worker/bundle_processor.py b/sdks/python/apache_beam/runners/worker/bundle_processor.py index 0c46b81..94dca8b 100644 --- a/sdks/python/apache_beam/runners/worker/bundle_processor.py +++ b/sdks/python/apache_beam/runners/worker/bundle_processor.py @@ -529,9 +529,10 @@ def create(factory, transform_id, transform_proto, parameter, consumers): def __init__(self, windowing): self.windowing = windowing -def process(self, element, timestamp=beam.DoFn.TimestampParam): +def process(self, element, timestamp=beam.DoFn.TimestampParam, +window=beam.DoFn.WindowParam): new_windows = self.windowing.windowfn.assign( - WindowFn.AssignContext(timestamp, element=element)) + WindowFn.AssignContext(timestamp, element=element, window=window)) yield WindowedValue(element, timestamp, new_windows) from apache_beam.transforms.core import Windowing from apache_beam.transforms.window import WindowFn, WindowedValue diff --git a/sdks/python/apache_beam/testing/util.py b/sdks/python/apache_beam/testing/util.py index 34c15f9..2f18bde 100644 --- a/sdks/python/apache_beam/testing/util.py +++ b/sdks/python/apache_beam/testing/util.py @@ -19,13 +19,16 @@ from __future__ import absolute_import +import collections import glob import tempfile from apache_beam import pvalue from apache_beam.transforms import window from apache_beam.transforms.core import Create +from apache_beam.transforms.core import DoFn from apache_beam.transforms.core import Map +from apache_beam.transforms.core import ParDo from apache_beam.transforms.core import WindowInto from apache_beam.transforms.ptransform import PTransform from apache_beam.transforms.util import CoGroupByKey @@ -37,6 +40,7 @@ __all__ = [ 'is_empty', # open_shards is internal and has no backwards compatibility guarantees. 'open_shards', +'TestWindowedValue', ] @@ -46,11 +50,32 @@ class BeamAssertException(Exception): pass +# Used for reifying timestamps and windows for assert_that matchers. +TestWindowedValue = collections.namedtuple( +'TestWindowedValue', 'value timestamp windows') + + +def contains_in_any_order(iterable): + """Creates an object that matches another iterable if they both have the + same count of items. + + Arguments: +iterable: An iterable of hashable objects. + """ + class InAnyOrder(object): +def __init__(self, iterable): + self._counter = collections.Counter(iterable) + +def __eq__(self, other): + return self._counter == collections.Counter(other) + + return InAnyOrder(iterable) + + # Note that equal_to always sorts the expected and actual since what we # compare are PCollections for which there is no guaranteed order. # However the sorting does not go beyond top level therefore [1,2] and [2,1] # are considered equal and [[1,2]] and [[2,1]] are not. -# TODO(silviuc): Add contains_in_any_order-style matchers. def equal_to(expected): expected = list(expected) @@ -72,7 +97,7 @@ def is_empty(): return _empty -def assert_that(actual, matcher, label='assert_that'): +def assert_that(actual, matcher, label='assert_that', reify_windows=Fa
[beam] branch master updated (a26f124 -> fcdc49f)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a26f124 Merge pull request #4174 from iemejia/BEAM-3244-dofn-flink add c773586 Ensure producers are always set on unpickling from runner api protos. new fcdc49f Merge pull request #4071 from robertwb/interactive-context The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/pipeline.py | 23 --- 1 file changed, 16 insertions(+), 7 deletions(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4071 from robertwb/interactive-context
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit fcdc49f0f5b922c68f791f214da9cd91907a7092 Merge: a26f124 c773586 Author: Robert Bradshaw AuthorDate: Fri Dec 1 13:37:57 2017 -0700 Merge pull request #4071 from robertwb/interactive-context Ensure producers are always set on unpickling from runner api protos. sdks/python/apache_beam/pipeline.py | 23 --- 1 file changed, 16 insertions(+), 7 deletions(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
[beam] branch master updated: Various worker cleanups. (#4195)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8f7517d Various worker cleanups. (#4195)
8f7517d is described below
commit 8f7517d462b1c7a1c2a3a6e3e563898fd3f669b4
Author: Robert Bradshaw
AuthorDate: Thu Nov 30 18:36:02 2017 -0700
Various worker cleanups. (#4195)
Various worker cleanups.
---
sdks/python/apache_beam/runners/common.py | 79 +++---
.../apache_beam/runners/direct/bundle_factory.py | 6 +-
.../runners/direct/transform_evaluator.py | 5 +-
.../runners/portability/fn_api_runner.py | 60 +++-
.../apache_beam/runners/worker/bundle_processor.py | 42 +---
.../apache_beam/runners/worker/operations.py | 15 ++--
6 files changed, 50 insertions(+), 157 deletions(-)
diff --git a/sdks/python/apache_beam/runners/common.py
b/sdks/python/apache_beam/runners/common.py
index 64abe41..08ddf65 100644
--- a/sdks/python/apache_beam/runners/common.py
+++ b/sdks/python/apache_beam/runners/common.py
@@ -319,15 +319,9 @@ class DoFnRunner(Receiver):
kwargs,
side_inputs,
windowing,
- context=None,
tagged_receivers=None,
- logger=None,
step_name=None,
- # Preferred alternative to logger
- # TODO(robertwb): Remove once all runners are updated.
logging_context=None,
- # Preferred alternative to context
- # TODO(robertwb): Remove once all runners are updated.
state=None,
scoped_metrics_container=None):
"""Initializes a DoFnRunner.
@@ -338,45 +332,31 @@ class DoFnRunner(Receiver):
kwargs: keyword side input arguments (static and placeholder), if any
side_inputs: list of sideinput.SideInputMaps for deferred side inputs
windowing: windowing properties of the output PCollection(s)
- context: a DoFnContext to use (deprecated)
tagged_receivers: a dict of tag name to Receiver objects
- logger: a logging module (deprecated)
step_name: the name of this step
logging_context: a LoggingContext object
state: handle for accessing DoFn state
scoped_metrics_container: Context switcher for metrics container
"""
-self.scoped_metrics_container = (scoped_metrics_container
- or ScopedMetricsContainer())
-self.step_name = step_name
-
# Need to support multiple iterations.
side_inputs = list(side_inputs)
-if logging_context:
- self.logging_context = logging_context
-else:
- self.logging_context = get_logging_context(logger, step_name=step_name)
-
-# TODO(sourabh): Deprecate the use of context
-if state:
- assert context is None
- context = DoFnContext(step_name, state=state)
-else:
- assert context is not None
- context = context
-
-self.context = context
+self.scoped_metrics_container = (
+scoped_metrics_container or ScopedMetricsContainer())
+self.step_name = step_name
+self.logging_context = logging_context or LoggingContext()
+self.context = DoFnContext(step_name, state=state)
do_fn_signature = DoFnSignature(fn)
# Optimize for the common case.
-main_receivers = as_receiver(tagged_receivers[None])
+main_receivers = tagged_receivers[None]
output_processor = _OutputProcessor(
windowing.windowfn, main_receivers, tagged_receivers)
self.do_fn_invoker = DoFnInvoker.create_invoker(
-output_processor, do_fn_signature, context, side_inputs, args, kwargs)
+output_processor, do_fn_signature, self.context,
+side_inputs, args, kwargs)
def receive(self, windowed_value):
self.process(windowed_value)
@@ -479,7 +459,7 @@ class _OutputProcessor(object):
if tag is None:
self.main_receivers.receive(windowed_value)
else:
-self.tagged_receivers[tag].output(windowed_value)
+self.tagged_receivers[tag].receive(windowed_value)
def start_bundle_outputs(self, results):
"""Validate that start_bundle does not output any elements"""
@@ -514,7 +494,7 @@ class _OutputProcessor(object):
if tag is None:
self.main_receivers.receive(windowed_value)
else:
-self.tagged_receivers[tag].output(windowed_value)
+self.tagged_receivers[tag].receive(windowed_value)
class _NoContext(WindowFn.AssignContext):
@@ -586,42 +566,3 @@ class DoFnContext(object):
raise AttributeError('windows not accessible in this context')
else:
return self.windowed_value.windows
-
-
-# TODO(robertwb): Remove all these adapters once service is updated
[beam] branch master updated (ead5d43 -> fc56f09)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ead5d43 This closes #4177: [BEAM-2870] Strips partition decorators when creating/patching tables in batch add 3651bf9 Flush stderr on failures new fc56f09 Merge pull request #4126 from holdenk/BEAM-3164-capture-gen-prot-errors The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/gen_protos.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact ['"commits@beam.apache.org" '].
[beam] 01/01: Merge pull request #4126 from holdenk/BEAM-3164-capture-gen-prot-errors
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit fc56f094738e6678b47d3072da70804db016d1f1 Merge: ead5d43 3651bf9 Author: Robert Bradshaw AuthorDate: Wed Nov 29 16:41:12 2017 -0700 Merge pull request #4126 from holdenk/BEAM-3164-capture-gen-prot-errors [BEAM-3164] Flush stderr on failures sdks/python/gen_protos.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) -- To stop receiving notification emails like this one, please contact "commits@beam.apache.org" .
