[beam] 01/01: Add enum34 to manual_licenses

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch jkff/enum34-license
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b4348d2daf17a03aa7080c48d80ffebc3b41e9af
Author: Eugene Kirpichov 
AuthorDate: Tue Aug 18 16:46:57 2020 -0700

Add enum34 to manual_licenses
---
 .../container/license_scripts/dep_urls_py.yaml |  4 ++-
 .../license_scripts/manual_licenses/enum34/LICENSE | 32 ++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/sdks/python/container/license_scripts/dep_urls_py.yaml 
b/sdks/python/container/license_scripts/dep_urls_py.yaml
index d1e194f..13aa4cf 100644
--- a/sdks/python/container/license_scripts/dep_urls_py.yaml
+++ b/sdks/python/container/license_scripts/dep_urls_py.yaml
@@ -48,7 +48,9 @@ pip_dependencies:
   cython:
 license: 
"https://raw.githubusercontent.com/cython/cython/master/LICENSE.txt";
   enum34:
-license: 
"https://bitbucket.org/stoneleaf/enum34/raw/c208549a93b71b948ff7bbdfd29dce8f85527916/enum/LICENSE";
+# The original repo is down. This license taken from somebody's clone:
+# https://github.com/jamespharaoh/python-enum34/blob/master/enum/LICENSE
+license: "file:///tmp/license_scripts/manual_licenses/enum34/LICENSE"
   fastavro:
 license: 
"https://raw.githubusercontent.com/fastavro/fastavro/master/LICENSE";
 notice: 
"https://raw.githubusercontent.com/fastavro/fastavro/master/NOTICE.txt";
diff --git 
a/sdks/python/container/license_scripts/manual_licenses/enum34/LICENSE 
b/sdks/python/container/license_scripts/manual_licenses/enum34/LICENSE
new file mode 100644
index 000..9003b88
--- /dev/null
+++ b/sdks/python/container/license_scripts/manual_licenses/enum34/LICENSE
@@ -0,0 +1,32 @@
+Copyright (c) 2013, Ethan Furman.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+Redistributions of source code must retain the above
+copyright notice, this list of conditions and the
+following disclaimer.
+
+Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following
+disclaimer in the documentation and/or other materials
+provided with the distribution.
+
+Neither the name Ethan Furman nor the names of any
+contributors may be used to endorse or promote products
+derived from this software without specific prior written
+permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

[beam] branch jkff/enum34-license created (now b4348d2)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch jkff/enum34-license
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at b4348d2  Add enum34 to manual_licenses

This branch includes the following new commits:

 new b4348d2  Add enum34 to manual_licenses

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[beam] branch jkff-cherry-12571 updated (d85ba7d -> 9ea783d)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch jkff-cherry-12571
in repository https://gitbox.apache.org/repos/asf/beam.git.


 discard d85ba7d  Add enum34 to manual_licenses
 add 9ea783d  Add enum34 to manual_licenses

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (d85ba7d)
\
 N -- N -- N   refs/heads/jkff-cherry-12571 (9ea783d)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../license_scripts/manual_licenses/enum34/LICENSE | 32 ++
 1 file changed, 32 insertions(+)
 create mode 100644 
sdks/python/container/license_scripts/manual_licenses/enum34/LICENSE

[beam] branch jkff-cherry-12571 updated (d582acf -> d85ba7d)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch jkff-cherry-12571
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d582acf  [BEAM-10702] Do not implicitly decompress artifacts
 add d85ba7d  Add enum34 to manual_licenses

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/license_scripts/dep_urls_py.yaml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

[beam] branch jkff-cherry-12571 updated (f3c3d2f -> d582acf)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch jkff-cherry-12571
in repository https://gitbox.apache.org/repos/asf/beam.git.


 discard f3c3d2f  [BEAM-10702] Do not implicitly decompress artifacts
 new d582acf  [BEAM-10702] Do not implicitly decompress artifacts

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (f3c3d2f)
\
 N -- N -- N   refs/heads/jkff-cherry-12571 (d582acf)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.md | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

[beam] 01/01: [BEAM-10702] Do not implicitly decompress artifacts

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch jkff-cherry-12571
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d582acf51955539c365e454f60868a2abba7ab57
Author: Eugene Kirpichov 
AuthorDate: Thu Aug 13 12:06:55 2020 -0700

[BEAM-10702] Do not implicitly decompress artifacts

Also adds a Julia set test on portable local runner, which uses a
setup.py and hence exercises the artifact staging codepath.

This is a squashed cherrypick of https://github.com/apache/beam/pull/12571
---
 CHANGES.md |  3 +--
 .../runners/portability/artifact_service.py|  4 +++-
 .../portability/fn_api_runner/worker_handlers.py   |  7 +-
 sdks/python/test-suites/portable/common.gradle | 25 ++
 4 files changed, 35 insertions(+), 4 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index a4882b3..45e29c9 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -63,16 +63,15 @@
   you may use `--experiments=use_legacy_bq_sink`.
 * Add cross-language support to Java's JdbcIO, now available in the Python 
module `apache_beam.io.external.jdbc` 
([BEAM-10135](https://issues.apache.org/jira/browse/BEAM-10135), 
[BEAM-10136](https://issues.apache.org/jira/browse/BEAM-10136)).
 * Add support of AWS SDK v2 for KinesisIO.Read (Java) 
([BEAM-9702](https://issues.apache.org/jira/browse/BEAM-9702)).
-* Support for X source added (Java/Python) 
([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
 * Add streaming support to SnowflakeIO in Java SDK 
([BEAM-9896](https://issues.apache.org/jira/browse/BEAM-9896  ))
 * Support reading and writing to Google Healthcare DICOM APIs in Python SDK 
([BEAM-10601](https://issues.apache.org/jira/browse/BEAM-10601))
 
 ## New Features / Improvements
 
 * Shared library for simplifying management of large shared objects added to 
Python SDK. Example use case is sharing a large TF model object across threads 
([BEAM-10417](https://issues.apache.org/jira/browse/BEAM-10417)).
-* X feature added (Java/Python) 
([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
 * Dataflow streaming timers are not strictly time ordered when set earlier 
mid-bundle ([BEAM-8543](https://issues.apache.org/jira/browse/BEAM-8543)).
 * OnTimerContext should not create a new one when processing each 
element/timer in FnApiDoFnRunner 
([BEAM-9839](https://issues.apache.org/jira/browse/BEAM-9839))
+* Fixed BEAM-10702 (Python) - Python portable runner with embedded job 
endpoint was unable to run pipelines which had dependencies in setup.py or 
requirements.txt 
([BEAM-10702](https://issues.apache.org/jira/browse/BEAM-10702)).
 
 ## Breaking Changes
 
diff --git a/sdks/python/apache_beam/runners/portability/artifact_service.py 
b/sdks/python/apache_beam/runners/portability/artifact_service.py
index f2bbf53..1f3ec1c 100644
--- a/sdks/python/apache_beam/runners/portability/artifact_service.py
+++ b/sdks/python/apache_beam/runners/portability/artifact_service.py
@@ -41,6 +41,7 @@ import grpc
 from future.moves.urllib.request import urlopen
 
 from apache_beam.io import filesystems
+from apache_beam.io.filesystems import CompressionTypes
 from apache_beam.portability import common_urns
 from apache_beam.portability.api import beam_artifact_api_pb2
 from apache_beam.portability.api import beam_artifact_api_pb2_grpc
@@ -263,7 +264,8 @@ class BeamFilesystemHandler(object):
 self._root = root
 
   def file_reader(self, path):
-return filesystems.FileSystems.open(path)
+return filesystems.FileSystems.open(
+path, compression_type=CompressionTypes.UNCOMPRESSED)
 
   def file_writer(self, name=None):
 full_path = filesystems.FileSystems.join(self._root, name)
diff --git 
a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py 
b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
index 9d27549..1f1d483 100644
--- 
a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
+++ 
b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
@@ -45,6 +45,7 @@ from typing import overload
 import grpc
 
 from apache_beam.io import filesystems
+from apache_beam.io.filesystems import CompressionTypes
 from apache_beam.portability import common_urns
 from apache_beam.portability import python_urns
 from apache_beam.portability.api import beam_artifact_api_pb2_grpc
@@ -464,9 +465,13 @@ class GrpcServer(object):
 self.provision_info.provision_info, worker_manager),
 self.control_server)
 
+  def open_uncompressed(f):
+return filesystems.FileSystems.open(
+f, compression_type=CompressionTypes.UNCOMPRESSED)
+
   
beam_artifact_api_pb2_grpc.add_ArtifactRetrievalServiceServicer_to_server(
   artifact_service.ArtifactRetrievalService(
-  file_reader=filesystems.FileSystems.open),
+  file_r

[beam] branch jkff-cherry-12571 created (now f3c3d2f)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch jkff-cherry-12571
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at f3c3d2f  [BEAM-10702] Do not implicitly decompress artifacts

This branch includes the following new commits:

 new f3c3d2f  [BEAM-10702] Do not implicitly decompress artifacts

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[beam] 01/01: [BEAM-10702] Do not implicitly decompress artifacts

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch jkff-cherry-12571
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f3c3d2f06ceba2b4379187278ae7b8b50f0d528f
Author: Eugene Kirpichov 
AuthorDate: Thu Aug 13 12:06:55 2020 -0700

[BEAM-10702] Do not implicitly decompress artifacts

Also adds a Julia set test on portable local runner, which uses a
setup.py and hence exercises the artifact staging codepath.

This is a squashed cherrypick of https://github.com/apache/beam/pull/12571
---
 .../runners/portability/artifact_service.py|  4 +++-
 .../portability/fn_api_runner/worker_handlers.py   |  7 +-
 sdks/python/test-suites/portable/common.gradle | 25 ++
 3 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/sdks/python/apache_beam/runners/portability/artifact_service.py 
b/sdks/python/apache_beam/runners/portability/artifact_service.py
index f2bbf53..1f3ec1c 100644
--- a/sdks/python/apache_beam/runners/portability/artifact_service.py
+++ b/sdks/python/apache_beam/runners/portability/artifact_service.py
@@ -41,6 +41,7 @@ import grpc
 from future.moves.urllib.request import urlopen
 
 from apache_beam.io import filesystems
+from apache_beam.io.filesystems import CompressionTypes
 from apache_beam.portability import common_urns
 from apache_beam.portability.api import beam_artifact_api_pb2
 from apache_beam.portability.api import beam_artifact_api_pb2_grpc
@@ -263,7 +264,8 @@ class BeamFilesystemHandler(object):
 self._root = root
 
   def file_reader(self, path):
-return filesystems.FileSystems.open(path)
+return filesystems.FileSystems.open(
+path, compression_type=CompressionTypes.UNCOMPRESSED)
 
   def file_writer(self, name=None):
 full_path = filesystems.FileSystems.join(self._root, name)
diff --git 
a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py 
b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
index 9d27549..1f1d483 100644
--- 
a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
+++ 
b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
@@ -45,6 +45,7 @@ from typing import overload
 import grpc
 
 from apache_beam.io import filesystems
+from apache_beam.io.filesystems import CompressionTypes
 from apache_beam.portability import common_urns
 from apache_beam.portability import python_urns
 from apache_beam.portability.api import beam_artifact_api_pb2_grpc
@@ -464,9 +465,13 @@ class GrpcServer(object):
 self.provision_info.provision_info, worker_manager),
 self.control_server)
 
+  def open_uncompressed(f):
+return filesystems.FileSystems.open(
+f, compression_type=CompressionTypes.UNCOMPRESSED)
+
   
beam_artifact_api_pb2_grpc.add_ArtifactRetrievalServiceServicer_to_server(
   artifact_service.ArtifactRetrievalService(
-  file_reader=filesystems.FileSystems.open),
+  file_reader=open_uncompressed),
   self.control_server)
 
 self.data_plane_handler = data_plane.BeamFnDataServicer(
diff --git a/sdks/python/test-suites/portable/common.gradle 
b/sdks/python/test-suites/portable/common.gradle
index 48312a6..60f825c 100644
--- a/sdks/python/test-suites/portable/common.gradle
+++ b/sdks/python/test-suites/portable/common.gradle
@@ -101,6 +101,30 @@ task flinkTriggerTranscript() {
   }
 }
 
+// Verifies BEAM-10702.
+task portableLocalRunnerJuliaSetWithSetupPy {
+  dependsOn 'setupVirtualenv'
+  dependsOn ":sdks:python:container:py${pythonVersionSuffix}:docker"
+
+  doLast {
+exec {
+  executable 'sh'
+  args '-c', """
+  . ${envdir}/bin/activate \\
+  && cd ${pythonRootDir} \\
+  && pip install -e . \\
+  && cd apache_beam/examples/complete/juliaset \\
+  && python juliaset_main.py \\
+  --runner=PortableRunner \\
+  --job_endpoint=embed \\
+  --setup_file=./setup.py \\
+  --coordinate_output=/tmp/juliaset \\
+  --grid_size=1
+  """
+}
+  }
+}
+
 task createProcessWorker {
   dependsOn ':sdks:python:container:build'
   dependsOn 'setupVirtualenv'
@@ -181,6 +205,7 @@ project.task("postCommitPy${pythonVersionSuffix}") {
   dependsOn = ['setupVirtualenv',
"postCommitPy${pythonVersionSuffix}IT",
':runners:spark:job-server:shadowJar',
+   'portableLocalRunnerJuliaSetWithSetupPy',
'portableWordCountSparkRunnerBatch']
 }
 

[beam] branch master updated (bf85a7a -> caabb8b)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bf85a7a  Merge pull request #12460 from [BEAM-10545] HtmlView module
 add b37f1f6  [BEAM-10702] Do not implicitly decompress artifacts
 add b214d1b  Adds a Julia Set test on portable local runner
 add c5da208  Address review comments
 add caabb8b  Merge pull request #12571 from apache/beam-10702: 
[BEAM-10702] Do not implicitly decompress artifacts

No new revisions were added by this update.

Summary of changes:
 .../runners/portability/artifact_service.py|  4 +++-
 .../portability/fn_api_runner/worker_handlers.py   |  7 +-
 sdks/python/test-suites/portable/common.gradle | 25 ++
 3 files changed, 34 insertions(+), 2 deletions(-)

[beam] branch beam-10702 updated (8e57e98 -> c5da208)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch beam-10702
in repository https://gitbox.apache.org/repos/asf/beam.git.


 discard 8e57e98  Adds a Julia Set test on portable local runner
 discard 6c4dc26  [BEAM-10702] Do not implicitly decompress artifacts
 add 198a583  [BEAM-9547] Lift associative aggregations. (#12469)
 add a098130  Merge pull request #12427 from [BEAM-2855] nexmark python 
suite implement queries 0, 1, 2 and 9
 add de4464b  Update nexmark dashboard links.
 add e028256  Merge pull request #12475 from tysonjh/www-nexmark
 add 1b985bb  [BEAM-10500] Make KeyedTimerDataCoder encode output timestamp 
(#12535)
 add 72a9093  [BEAM-10648] Remove unused BigQuery queryTempDataset value
 add a6907c0  Merge pull request #12478 from regadas/bigquery_remove_unused
 add 23f3b49  Import WordExtractingDoFn from wordcount_with_metrics
 add 70fcd57  Merge pull request #12569 from apache/aaltay-patch-1
 add f1e780d  [BEAM-10646] Remove SparkPortableExecutionTest.testExecution.
 add 1709588  [BEAM-10646] Don't wait for test to time out if pipeline 
fails.
 add 228c621  Merge pull request #12477 from ibzib/BEAM-10646
 add 699f872  Extending ApproximateQuantiles functionality to deal with 
non-uniform weights. (#12420)
 add b423fe3  fix logic issue in metric name namespace filtering (#12570)
 add 0f04de7  Better error on BQ schema parse (#12549)
 add 5ae71f9  [BEAM-10545] HtmlView module
 add 213c2c7  Use primitive string[] to replace Array type
 add bf85a7a  Merge pull request #12460 from [BEAM-10545] HtmlView module
 add b37f1f6  [BEAM-10702] Do not implicitly decompress artifacts
 add b214d1b  Adds a Julia Set test on portable local runner
 add c5da208  Address review comments

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (8e57e98)
\
 N -- N -- N   refs/heads/beam-10702 (c5da208)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../beam/runners/jobsubmission/JobInvocation.java  |   2 +-
 .../beam/runners/samza/runtime/KeyedTimerData.java |   6 +-
 .../runners/samza/runtime/KeyedTimerDataTest.java  |   8 +-
 .../runners/spark/SparkPortableExecutionTest.java  |  96 +---
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java   |   2 -
 sdks/python/apache_beam/dataframe/expressions.py   |  11 +-
 sdks/python/apache_beam/dataframe/frame_base.py|   5 -
 sdks/python/apache_beam/dataframe/frames.py| 136 ---
 sdks/python/apache_beam/dataframe/frames_test.py   |  23 +-
 sdks/python/apache_beam/dataframe/transforms.py|  22 ++
 .../apache_beam/examples/streaming_wordcount.py|   2 +-
 sdks/python/apache_beam/io/gcp/bigquery_tools.py   |  12 +-
 sdks/python/apache_beam/metrics/metric.py  |  11 +-
 .../apache-beam-jupyterlab-sidepanel/.eslintrc.js  |  12 +-
 .../.prettierrc.json   |   3 +-
 .../apache-beam-jupyterlab-sidepanel/package.json  |   8 +-
 .../src/__tests__/common/HtmlView.test.tsx | 111 +
 .../src/__tests__/kernel/KernelModel.test.ts   |  21 +-
 .../src/common/HtmlView.tsx| 119 +
 .../apache-beam-jupyterlab-sidepanel/tsconfig.json |   1 +
 .../apache-beam-jupyterlab-sidepanel/yarn.lock | 146 ++-
 .../benchmarks/nexmark/models/auction_bid.py   |  57 +
 .../benchmarks/nexmark/models/auction_price.py |  56 +
 .../benchmarks/nexmark/models/field_name.py}   |  27 ++-
 .../benchmarks/nexmark/models/nexmark_model.py | 153 ++--
 .../testing/benchmarks/nexmark/nexmark_launcher.py |  24 +-
 .../testing/benchmarks/nexmark/nexmark_util.py | 130 +-
 .../queries/{query2.py => nexmark_query_util.py}   |  59 +++--
 .../testing/benchmarks/nexmark/queries/query0.py   |  15 +-
 .../testing/benchmarks/nexmark/queries/query1.py   |  21 +-
 .../testing/benchmarks/nexmark/queries/query2.py   |  29 +--
 .../benchmarks/nexmark/queries/query9.py}  |  19 +-
 .../benchmarks/nexmark/queries/winning_bids.py | 198 +++
 sdks/python/apache_beam/transforms/stats.py| 266 +++--
 sdks/python/apache_beam/transforms/stats_test.py   |  62 -
 sdks/python/scripts/generate_pydoc.sh  |   1 +
 sdks/python/test-suites/portable/common.gradle |   2 +-
 .../en/doc

[beam] branch beam-10702 updated (6c4dc26 -> 8e57e98)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch beam-10702
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6c4dc26  [BEAM-10702] Do not implicitly decompress artifacts
 add 8e57e98  Adds a Julia Set test on portable local runner

No new revisions were added by this update.

Summary of changes:
 sdks/python/test-suites/portable/common.gradle | 25 +
 1 file changed, 25 insertions(+)

[beam] 01/01: [BEAM-10702] Do not implicitly decompress artifacts

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch beam-10702
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6c4dc26fb2474d7e8278ac10a7c53f2bb053c19a
Author: Eugene Kirpichov 
AuthorDate: Thu Aug 13 12:06:55 2020 -0700

[BEAM-10702] Do not implicitly decompress artifacts
---
 sdks/python/apache_beam/runners/portability/artifact_service.py| 4 +++-
 .../runners/portability/fn_api_runner/worker_handlers.py   | 7 ++-
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/sdks/python/apache_beam/runners/portability/artifact_service.py 
b/sdks/python/apache_beam/runners/portability/artifact_service.py
index f2bbf53..1f3ec1c 100644
--- a/sdks/python/apache_beam/runners/portability/artifact_service.py
+++ b/sdks/python/apache_beam/runners/portability/artifact_service.py
@@ -41,6 +41,7 @@ import grpc
 from future.moves.urllib.request import urlopen
 
 from apache_beam.io import filesystems
+from apache_beam.io.filesystems import CompressionTypes
 from apache_beam.portability import common_urns
 from apache_beam.portability.api import beam_artifact_api_pb2
 from apache_beam.portability.api import beam_artifact_api_pb2_grpc
@@ -263,7 +264,8 @@ class BeamFilesystemHandler(object):
 self._root = root
 
   def file_reader(self, path):
-return filesystems.FileSystems.open(path)
+return filesystems.FileSystems.open(
+path, compression_type=CompressionTypes.UNCOMPRESSED)
 
   def file_writer(self, name=None):
 full_path = filesystems.FileSystems.join(self._root, name)
diff --git 
a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py 
b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
index 9d27549..1f1d483 100644
--- 
a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
+++ 
b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py
@@ -45,6 +45,7 @@ from typing import overload
 import grpc
 
 from apache_beam.io import filesystems
+from apache_beam.io.filesystems import CompressionTypes
 from apache_beam.portability import common_urns
 from apache_beam.portability import python_urns
 from apache_beam.portability.api import beam_artifact_api_pb2_grpc
@@ -464,9 +465,13 @@ class GrpcServer(object):
 self.provision_info.provision_info, worker_manager),
 self.control_server)
 
+  def open_uncompressed(f):
+return filesystems.FileSystems.open(
+f, compression_type=CompressionTypes.UNCOMPRESSED)
+
   
beam_artifact_api_pb2_grpc.add_ArtifactRetrievalServiceServicer_to_server(
   artifact_service.ArtifactRetrievalService(
-  file_reader=filesystems.FileSystems.open),
+  file_reader=open_uncompressed),
   self.control_server)
 
 self.data_plane_handler = data_plane.BeamFnDataServicer(

[beam] branch beam-10702 created (now 6c4dc26)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch beam-10702
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at 6c4dc26  [BEAM-10702] Do not implicitly decompress artifacts

This branch includes the following new commits:

 new 6c4dc26  [BEAM-10702] Do not implicitly decompress artifacts

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[beam] branch master updated (c14c975 -> 21979d7)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c14c975  [BEAM-4752] Add support for newer dill dependency (#5931)
 add 728591b  FileSystems should throw exception if filesystem for schema 
is not found
 add 3fff58c  Make schema pattern stricter
 add ad4b057  Adding tet cases for matchResouces
 new 21979d7  Merge pull request #5808: FileSystems should throw exception 
if filesystem for schema is not found

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/io/FileSystems.java  |  8 
 .../java/org/apache/beam/sdk/io/FileSystemsTest.java   | 18 ++
 2 files changed, 22 insertions(+), 4 deletions(-)

[beam] 01/01: Merge pull request #5808: FileSystems should throw exception if filesystem for schema is not found

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 21979d7dd21f2840145a751af34f99ee06386c25
Merge: c14c975 ad4b057
Author: Eugene Kirpichov 
AuthorDate: Fri Jul 13 15:20:35 2018 -0700

Merge pull request #5808: FileSystems should throw exception if filesystem 
for schema is not found

FileSystems should throw exception if filesystem for schema is not found

 .../main/java/org/apache/beam/sdk/io/FileSystems.java  |  8 
 .../java/org/apache/beam/sdk/io/FileSystemsTest.java   | 18 ++
 2 files changed, 22 insertions(+), 4 deletions(-)

[beam] branch master updated (b911d29 -> 3c1191a)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b911d29  Merge pull request #5933: [BEAM-4773] Warn on docker image 
pull fail
 add 53d63e2  Adds license header to CODEOWNERS
 new 3c1191a  Merge pull request #5943: Adds license header to CODEOWNERS

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CODEOWNERS | 17 +
 1 file changed, 17 insertions(+)

[beam] 01/01: Merge pull request #5943: Adds license header to CODEOWNERS

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3c1191ab21337d5ae12abad686c0d03699f6e188
Merge: b911d29 53d63e2
Author: Eugene Kirpichov 
AuthorDate: Thu Jul 12 18:03:29 2018 -0700

Merge pull request #5943: Adds license header to CODEOWNERS

Adds license header to CODEOWNERS

 CODEOWNERS | 17 +
 1 file changed, 17 insertions(+)

[beam] branch master updated (c114445 -> 8104095)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c114445  [BEAM-4644] Use SLF4J in ExecutableStageDoFnOperator
 add 316a996  Pull the docker image before starting docker container.
 new 8104095  Merge pull request #5897: Pull the docker image before 
starting docker container.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/fnexecution/environment/DockerCommand.java | 3 +++
 1 file changed, 3 insertions(+)

[beam] 01/01: Merge pull request #5897: Pull the docker image before starting docker container.

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8104095ddf974b9b39b7dde9fcbb99b86c225526
Merge: c114445 316a996
Author: Eugene Kirpichov 
AuthorDate: Fri Jul 6 17:38:10 2018 -0700

Merge pull request #5897: Pull the docker image before starting docker 
container.

Pull the docker image before starting docker container.

 .../org/apache/beam/runners/fnexecution/environment/DockerCommand.java | 3 +++
 1 file changed, 3 insertions(+)

[beam] 01/01: Merge pull request #5874: Fix artifact retrival file system

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b439ebe92b6033aa0d93ed0fbd67facd88401909
Merge: 92275d6 cff51fb
Author: Eugene Kirpichov 
AuthorDate: Fri Jul 6 16:47:52 2018 -0700

Merge pull request #5874: Fix artifact retrival file system

Fix artifact retrival file system

 .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 6 ++
 .../flink/translation/functions/FlinkExecutableStageFunction.java   | 5 +
 2 files changed, 7 insertions(+), 4 deletions(-)

[beam] branch master updated (92275d6 -> b439ebe)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 92275d6  Merge pull request #5837 from mariapython/query1_2_pr
 add 6bcaeaa  Fix artifact retrival file system
 add cff51fb  Initialization BeamFileSystem in FlinkExecutableStageFunction
 new b439ebe  Merge pull request #5874: Fix artifact retrival file system

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 6 ++
 .../flink/translation/functions/FlinkExecutableStageFunction.java   | 5 +
 2 files changed, 7 insertions(+), 4 deletions(-)

[beam] 01/01: Merge pull request #5838: Read worker harness image name from pipeline option for portable runners

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4a92d28967a1c3fbe4719d8599bdde7b9ff2e31f
Merge: d667097 19c92df
Author: Eugene Kirpichov 
AuthorDate: Tue Jul 3 15:01:31 2018 -0700

Merge pull request #5838: Read worker harness image name from pipeline 
option for portable runners

Read worker harness image name from pipeline option for portable runners

 .../core/construction/CombineTranslation.java|  5 ++---
 .../CreatePCollectionViewTranslation.java|  5 -
 .../beam/runners/core/construction/Environments.java |  7 +++
 .../construction/ExecutableStageTranslation.java |  3 ++-
 .../runners/core/construction/ParDoTranslation.java  | 14 --
 .../core/construction/PipelineTranslation.java   |  2 +-
 .../runners/core/construction/ReadTranslation.java   | 11 ++-
 .../runners/core/construction/SdkComponents.java | 16 
 .../core/construction/TestStreamTranslation.java |  2 +-
 .../core/construction/WindowIntoTranslation.java |  4 ++--
 .../construction/WindowingStrategyTranslation.java   | 20 +++-
 .../core/construction/WriteFilesTranslation.java |  5 +++--
 .../core/construction/CoderTranslationTest.java  |  8 +---
 .../core/construction/CombineTranslationTest.java|  4 
 .../CreatePCollectionViewTranslationTest.java|  3 +++
 .../runners/core/construction/EnvironmentsTest.java  |  6 ++
 .../construction/PCollectionTranslationTest.java |  3 +++
 .../construction/PCollectionViewTranslationTest.java |  9 +++--
 .../core/construction/PTransformTranslationTest.java |  2 +-
 .../core/construction/ParDoTranslationTest.java  |  4 
 .../core/construction/PipelineTranslationTest.java   |  2 +-
 .../core/construction/ReadTranslationTest.java   |  9 +++--
 .../core/construction/RehydratedComponentsTest.java  |  4 
 .../runners/core/construction/SdkComponentsTest.java |  9 -
 .../core/construction/TestStreamTranslationTest.java |  3 +++
 .../core/construction/WindowIntoTranslationTest.java |  2 ++
 .../WindowingStrategyTranslationTest.java|  6 +-
 .../core/construction/WriteFilesTranslationTest.java |  5 -
 .../BundleFactoryOutputReceiverFactoryTest.java  |  2 ++
 .../portable/GroupByKeyOnlyEvaluatorFactoryTest.java |  2 ++
 .../runners/dataflow/DataflowPipelineTranslator.java |  7 ++-
 .../wire/LengthPrefixUnknownCodersTest.java  |  2 ++
 .../src/main/java/org/apache/beam/sdk/Pipeline.java  |  5 +
 .../beam/sdk/options/PortablePipelineOptions.java|  7 +++
 .../org/apache/beam/sdk/testing/TestPipeline.java|  1 +
 .../beam/fn/harness/AssignWindowsRunnerTest.java | 14 ++
 .../apache/beam/fn/harness/CombineRunnersTest.java   |  2 +-
 .../apache/beam/fn/harness/FnApiDoFnRunnerTest.java  |  6 +++---
 .../beam/fn/harness/WindowMappingFnRunnerTest.java   |  5 -
 39 files changed, 169 insertions(+), 57 deletions(-)

[beam] branch master updated (d667097 -> 4a92d28)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d667097  [BEAM-4726] Add a simple benchmark to the ParDo execution 
node.
 add 3f7ed91  Read worker harness image name from pipeline option for 
portable runners
 add 68d0457  Spotless
 add 45cfda34 Support only a single environment in SdkComponents
 add 7772868  Renaming createEnvironment to createOrGetDefaultEnvironment.
 add 19c92df  Create SDKComponents with pipeline options
 new 4a92d28  Merge pull request #5838: Read worker harness image name from 
pipeline option for portable runners

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/CombineTranslation.java|  5 ++---
 .../CreatePCollectionViewTranslation.java|  5 -
 .../beam/runners/core/construction/Environments.java |  7 +++
 .../construction/ExecutableStageTranslation.java |  3 ++-
 .../runners/core/construction/ParDoTranslation.java  | 14 --
 .../core/construction/PipelineTranslation.java   |  2 +-
 .../runners/core/construction/ReadTranslation.java   | 11 ++-
 .../runners/core/construction/SdkComponents.java | 16 
 .../core/construction/TestStreamTranslation.java |  2 +-
 .../core/construction/WindowIntoTranslation.java |  4 ++--
 .../construction/WindowingStrategyTranslation.java   | 20 +++-
 .../core/construction/WriteFilesTranslation.java |  5 +++--
 .../core/construction/CoderTranslationTest.java  |  8 +---
 .../core/construction/CombineTranslationTest.java|  4 
 .../CreatePCollectionViewTranslationTest.java|  3 +++
 .../runners/core/construction/EnvironmentsTest.java  |  6 ++
 .../construction/PCollectionTranslationTest.java |  3 +++
 .../construction/PCollectionViewTranslationTest.java |  9 +++--
 .../core/construction/PTransformTranslationTest.java |  2 +-
 .../core/construction/ParDoTranslationTest.java  |  4 
 .../core/construction/PipelineTranslationTest.java   |  2 +-
 .../core/construction/ReadTranslationTest.java   |  9 +++--
 .../core/construction/RehydratedComponentsTest.java  |  4 
 .../runners/core/construction/SdkComponentsTest.java |  9 -
 .../core/construction/TestStreamTranslationTest.java |  3 +++
 .../core/construction/WindowIntoTranslationTest.java |  2 ++
 .../WindowingStrategyTranslationTest.java|  6 +-
 .../core/construction/WriteFilesTranslationTest.java |  5 -
 .../BundleFactoryOutputReceiverFactoryTest.java  |  2 ++
 .../portable/GroupByKeyOnlyEvaluatorFactoryTest.java |  2 ++
 .../runners/dataflow/DataflowPipelineTranslator.java |  7 ++-
 .../wire/LengthPrefixUnknownCodersTest.java  |  2 ++
 .../src/main/java/org/apache/beam/sdk/Pipeline.java  |  5 +
 .../beam/sdk/options/PortablePipelineOptions.java|  7 +++
 .../org/apache/beam/sdk/testing/TestPipeline.java|  1 +
 .../beam/fn/harness/AssignWindowsRunnerTest.java | 14 ++
 .../apache/beam/fn/harness/CombineRunnersTest.java   |  2 +-
 .../apache/beam/fn/harness/FnApiDoFnRunnerTest.java  |  6 +++---
 .../beam/fn/harness/WindowMappingFnRunnerTest.java   |  5 -
 39 files changed, 169 insertions(+), 57 deletions(-)

[beam] branch master updated (cb72b48 -> e963dbb)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cb72b48  Merge pull request #5847: Remove more extraneous printlns 
from build
 add 6daf403  Makes FileIO.match watermark advance even without new files
 new e963dbb  Merge pull request #5836: Makes FileIO.match watermark 
advance even without new files

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5836: Makes FileIO.match watermark advance even without new files

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e963dbb476b911298cba93f8a38c6366b8293506
Merge: cb72b48 6daf403
Author: Eugene Kirpichov 
AuthorDate: Mon Jul 2 14:02:28 2018 -0700

Merge pull request #5836: Makes FileIO.match watermark advance even without 
new files

Makes FileIO.match watermark advance even without new files

 sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

[beam] branch master updated (c5c20b6 -> 2820bab)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c5c20b6  Merge pull request #5785: [BEAM-4641] Samza runner postcommit 
status in PR template
 add 34edbd5  [BEAM-4689] Reverts change of SDF key type
 new 2820bab  Merge pull request #5834: [BEAM-4689] Reverts change of SDF 
key type

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 examples/java/build.gradle |  1 +
 .../org/apache/beam/examples/MinimalWordCount.java | 88 +++---
 .../translation/operators/ApexParDoOperator.java   | 12 +--
 .../runners/core/construction/SplittableParDo.java | 13 ++--
 .../apache/beam/runners/core/ProcessFnRunner.java  | 16 ++--
 .../core/SplittableParDoViaKeyedWorkItems.java | 32 
 .../runners/core/SplittableParDoProcessFnTest.java |  9 +--
 .../SplittableProcessElementsEvaluatorFactory.java | 14 ++--
 .../SplittableRemoteStageEvaluatorFactory.java | 10 +--
 .../flink/FlinkStreamingTransformTranslators.java  |  6 +-
 .../wrappers/streaming/SplittableDoFnOperator.java | 18 ++---
 11 files changed, 90 insertions(+), 129 deletions(-)

[beam] 01/01: Merge pull request #5834: [BEAM-4689] Reverts change of SDF key type

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2820bab7c38fd62e17472d9354c0c40199eb8c93
Merge: c5c20b6 34edbd5
Author: Eugene Kirpichov 
AuthorDate: Fri Jun 29 16:54:35 2018 -0700

Merge pull request #5834: [BEAM-4689] Reverts change of SDF key type

[BEAM-4689] Reverts change of SDF key type

 examples/java/build.gradle |  1 +
 .../org/apache/beam/examples/MinimalWordCount.java | 88 +++---
 .../translation/operators/ApexParDoOperator.java   | 12 +--
 .../runners/core/construction/SplittableParDo.java | 13 ++--
 .../apache/beam/runners/core/ProcessFnRunner.java  | 16 ++--
 .../core/SplittableParDoViaKeyedWorkItems.java | 32 
 .../runners/core/SplittableParDoProcessFnTest.java |  9 +--
 .../SplittableProcessElementsEvaluatorFactory.java | 14 ++--
 .../SplittableRemoteStageEvaluatorFactory.java | 10 +--
 .../flink/FlinkStreamingTransformTranslators.java  |  6 +-
 .../wrappers/streaming/SplittableDoFnOperator.java | 18 ++---
 11 files changed, 90 insertions(+), 129 deletions(-)

[beam] branch master updated (684216a -> 9cb27de)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 684216a  Merge pull request #5807: Update pre-commit filter to include 
migrated build rules
 add b10f52d  Adding non standard side input urn to flink
 new 9cb27de  Merge pull request #5823: Adding non standard side input urn 
to flink

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../translation/functions/FlinkBatchSideInputHandlerFactory.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5823: Adding non standard side input urn to flink

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9cb27de9a76d4df624532dac6387516d5c696c30
Merge: 684216a b10f52d
Author: Eugene Kirpichov 
AuthorDate: Fri Jun 29 11:22:31 2018 -0700

Merge pull request #5823: Adding non standard side input urn to flink

Adding non standard side input urn to flink

 .../translation/functions/FlinkBatchSideInputHandlerFactory.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

[beam] branch master updated (f9ecef5 -> 6fb17a5)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f9ecef5  Merge pull request #5824: Changes portableWordCount gradle 
task to be runnable at master
 add dc93042  Fixes spotless errors at master
 new 6fb17a5  Merge pull request #5828: Fixes spotless errors at master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java| 2 +-
 .../beam/runners/fnexecution/control/FnApiControlClientPoolService.java | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

[beam] 01/01: Merge pull request #5828: Fixes spotless errors at master

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6fb17a50a81ed1f37a85b1d3dc607a366e5acf44
Merge: f9ecef5 dc93042
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 18:19:46 2018 -0700

Merge pull request #5828: Fixes spotless errors at master

Fixes spotless errors at master

 .../fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java| 2 +-
 .../beam/runners/fnexecution/control/FnApiControlClientPoolService.java | 2 --
 2 files changed, 1 insertion(+), 3 deletions(-)

[beam] branch master updated (ab1e09f -> f9ecef5)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ab1e09f  Merge pull request #5817 from [BEAM-3883] Stage files in the 
portability runner.
 add a86a78b  Changes portableWordCount gradle task to be runnable at master
 new f9ecef5  Merge pull request #5824: Changes portableWordCount gradle 
task to be runnable at master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/build.gradle | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5824: Changes portableWordCount gradle task to be runnable at master

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f9ecef5f90212026313492744f32156498713c73
Merge: ab1e09f a86a78b
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 17:57:08 2018 -0700

Merge pull request #5824: Changes portableWordCount gradle task to be 
runnable at master

Changes portableWordCount gradle task to be runnable at master

 sdks/python/build.gradle | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

[beam] branch master updated (4bb10f3 -> 9cbbe91)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4bb10f3  Merge pull request #5784: Do not email individuals about 
failures in Jenkins
 add 9cef1a4  Job service additionally need google cloud dependency to use 
GCS
 add 0d199dc  TODO to add more file systems
 new 9cbbe91  Merge pull request #5806: Flink Job Server setup Beam File 
Systems

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/flink/job-server/build.gradle| 2 ++
 .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java | 9 -
 2 files changed, 10 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5806: Flink Job Server setup Beam File Systems

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9cbbe917edd39087dd02cef1659ed3cde01acaef
Merge: 4bb10f3 0d199dc
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 14:54:36 2018 -0700

Merge pull request #5806: Flink Job Server setup Beam File Systems

Flink Job Server setup Beam File Systems

 runners/flink/job-server/build.gradle| 2 ++
 .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java | 9 -
 2 files changed, 10 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5805: Makes Python PortableRunner usable via PipelineOptions

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 385faa713951813371dffaf654b5dc8d96e27aa1
Merge: efe0aaf b0bc32a
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 13:24:56 2018 -0700

Merge pull request #5805: Makes Python PortableRunner usable via 
PipelineOptions

Makes Python PortableRunner usable via PipelineOptions

 .../core/construction/graph/PipelineValidator.java | 12 ++---
 .../python/apache_beam/options/pipeline_options.py | 16 ++
 .../runners/portability/portable_runner.py | 62 +++---
 .../runners/portability/portable_runner_test.py| 20 ---
 sdks/python/apache_beam/runners/runner.py  |  7 ++-
 sdks/python/build.gradle   | 10 
 6 files changed, 80 insertions(+), 47 deletions(-)

[beam] branch master updated (efe0aaf -> 385faa7)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from efe0aaf  [BEAM-4659] Add well known timer coder to Java SDK.
 add b0bc32a  Makes Python PortableRunner usable via PipelineOptions
 new 385faa7  Merge pull request #5805: Makes Python PortableRunner usable 
via PipelineOptions

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/graph/PipelineValidator.java | 12 ++---
 .../python/apache_beam/options/pipeline_options.py | 16 ++
 .../runners/portability/portable_runner.py | 62 +++---
 .../runners/portability/portable_runner_test.py| 20 ---
 sdks/python/apache_beam/runners/runner.py  |  7 ++-
 sdks/python/build.gradle   | 10 
 6 files changed, 80 insertions(+), 47 deletions(-)

[beam] branch master updated (b28fe91 -> 06626df)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b28fe91  Merge pull request #5759: [BEAM-4642] Pipeline options in 
JDBC URI; default userAgent=BeamSQL
 add 25a1362  Uploading job server jar when running on remote flink 
environment
 add 1d32a9d  Passing files to stage on flink explicitly while creating the 
ExecutionEnvironment
 add dc1ccfa  Spotless
 new 06626df  Merge pull request #5803: Flink upload jobserver jar to flink 
remote environment

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../flink/FlinkBatchPortablePipelineTranslator.java |  5 +++--
 .../beam/runners/flink/FlinkExecutionEnvironments.java  | 11 +--
 .../apache/beam/runners/flink/FlinkJobInvocation.java   | 17 +++--
 .../org/apache/beam/runners/flink/FlinkJobInvoker.java  |  8 +++-
 .../flink/FlinkPipelineExecutionEnvironment.java|  8 ++--
 .../flink/FlinkStreamingPortablePipelineTranslator.java |  6 --
 6 files changed, 36 insertions(+), 19 deletions(-)

[beam] 01/01: Merge pull request #5803: Flink upload jobserver jar to flink remote environment

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 06626dfc2a74e257a2d198e2c4a2dfde820e7706
Merge: b28fe91 dc1ccfa
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 12:45:29 2018 -0700

Merge pull request #5803: Flink upload jobserver jar to flink remote 
environment

Flink upload jobserver jar to flink remote environment

 .../flink/FlinkBatchPortablePipelineTranslator.java |  5 +++--
 .../beam/runners/flink/FlinkExecutionEnvironments.java  | 11 +--
 .../apache/beam/runners/flink/FlinkJobInvocation.java   | 17 +++--
 .../org/apache/beam/runners/flink/FlinkJobInvoker.java  |  8 +++-
 .../flink/FlinkPipelineExecutionEnvironment.java|  8 ++--
 .../flink/FlinkStreamingPortablePipelineTranslator.java |  6 --
 6 files changed, 36 insertions(+), 19 deletions(-)

[beam] branch master updated (56eab2c -> e9a874f)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 56eab2c  Merge pull request #5799 from robertwb/root-name
 add 6436c0c  Makes fuser materialize PCollections used as side inputs
 new e9a874f  Merge pull request #5804: Makes fuser materialize 
PCollections used as side inputs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/graph/ExecutableStage.java   |  3 +-
 .../construction/graph/GreedyPipelineFuser.java| 11 ++--
 .../core/construction/graph/GreedyStageFuser.java  |  4 ++
 .../core/construction/graph/QueryablePipeline.java | 18 ++
 .../graph/GreedyPipelineFuserTest.java |  2 +-
 .../construction/graph/GreedyStageFuserTest.java   | 68 ++
 6 files changed, 100 insertions(+), 6 deletions(-)

[beam] 01/01: Merge pull request #5804: Makes fuser materialize PCollections used as side inputs

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e9a874fa7b19c750adba7fc77826685c2b134a6f
Merge: 56eab2c 6436c0c
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 12:38:57 2018 -0700

Merge pull request #5804: Makes fuser materialize PCollections used as side 
inputs

Makes fuser materialize PCollections used as side inputs

 .../core/construction/graph/ExecutableStage.java   |  3 +-
 .../construction/graph/GreedyPipelineFuser.java| 11 ++--
 .../core/construction/graph/GreedyStageFuser.java  |  4 ++
 .../core/construction/graph/QueryablePipeline.java | 18 ++
 .../graph/GreedyPipelineFuserTest.java |  2 +-
 .../construction/graph/GreedyStageFuserTest.java   | 68 ++
 6 files changed, 100 insertions(+), 6 deletions(-)

[beam] 01/01: Merge pull request #5814: [BEAM-4285] Extend side input handlers to handle multiple access patterns.

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 06887741135cfbf93e82bd7f22670d22fa938d1b
Merge: 279a056 1207c28
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 28 11:28:46 2018 -0700

Merge pull request #5814:  [BEAM-4285] Extend side input handlers to handle 
multiple access patterns.

 [BEAM-4285] Extend side input handlers to handle multiple access patterns.

 .../core/construction/PTransformTranslation.java   |   5 +
 .../BatchFlinkExecutableStageContext.java  |   8 +-
 .../FlinkBatchSideInputHandlerFactory.java | 106 +---
 .../FlinkBatchSideInputHandlerFactoryTest.java |  93 +
 .../control/ProcessBundleDescriptors.java  |  79 ---
 .../fnexecution/state/StateRequestHandlers.java| 111 +++--
 .../fnexecution/control/RemoteExecutionTest.java   |  25 +++--
 7 files changed, 289 insertions(+), 138 deletions(-)

[beam] branch master updated (279a056 -> 0688774)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 279a056  Merge pull request #5797: [BEAM-4394] Spotless Java everywhere
 add f95ca80  [BEAM-4285] Extend side input handlers to handle multiple 
access patterns.
 add 1207c28  Rename several MultimapSideInput* classes to more generic 
SideInput*.
 new 0688774  Merge pull request #5814:  [BEAM-4285] Extend side input 
handlers to handle multiple access patterns.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/PTransformTranslation.java   |   5 +
 .../BatchFlinkExecutableStageContext.java  |   8 +-
 .../FlinkBatchSideInputHandlerFactory.java | 106 +---
 .../FlinkBatchSideInputHandlerFactoryTest.java |  93 +
 .../control/ProcessBundleDescriptors.java  |  79 ---
 .../fnexecution/state/StateRequestHandlers.java| 111 +++--
 .../fnexecution/control/RemoteExecutionTest.java   |  25 +++--
 7 files changed, 289 insertions(+), 138 deletions(-)

[beam] branch master updated (34b071c -> a14791e)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 34b071c  Merge pull request #5790: Rename refactored Precommit source 
files to remove GradleBuild suffix.
 add cff90f5  Adding flinkMarterUrl as an option
 new a14791e  Merge pull request #5798: Adding flinkMasterUrl as an option

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/flink/build.gradle   |  1 +
 .../org/apache/beam/runners/flink/FlinkJobInvoker.java   | 16 +---
 .../apache/beam/runners/flink/FlinkJobServerDriver.java  |  8 +++-
 3 files changed, 17 insertions(+), 8 deletions(-)

[beam] 01/01: Merge pull request #5798: Adding flinkMasterUrl as an option

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a14791e64d087be266ed54808143383d8c60beab
Merge: 34b071c cff90f5
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 27 17:22:11 2018 -0700

Merge pull request #5798: Adding flinkMasterUrl as an option

Adding flinkMasterUrl as an option

 runners/flink/build.gradle   |  1 +
 .../org/apache/beam/runners/flink/FlinkJobInvoker.java   | 16 +---
 .../apache/beam/runners/flink/FlinkJobServerDriver.java  |  8 +++-
 3 files changed, 17 insertions(+), 8 deletions(-)

[beam] branch master updated (17e6a62 -> 3550517)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 17e6a62  Merge pull request #5768 from aaltay/comove
 add db531ae  Fixes Flink ValidatesRunner and a compile error
 new 3550517  Merge pull request #5769: Fixes Flink ValidatesRunner and a 
compile error

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/apex/translation/operators/ApexParDoOperator.java  | 6 +++---
 .../beam/runners/flink/FlinkStreamingTransformTranslators.java  | 6 +++---
 .../translation/wrappers/streaming/SplittableDoFnOperator.java  | 4 ++--
 .../src/main/java/org/apache/beam/fn/harness/CombineRunners.java| 2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)

[beam] 01/01: Merge pull request #5769: Fixes Flink ValidatesRunner and a compile error

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3550517a7e92815b0da792c0a1eba66902657d8b
Merge: 17e6a62 db531ae
Author: Eugene Kirpichov 
AuthorDate: Tue Jun 26 12:44:42 2018 -0700

Merge pull request #5769: Fixes Flink ValidatesRunner and a compile error

Fixes Flink ValidatesRunner and a compile error

 .../beam/runners/apex/translation/operators/ApexParDoOperator.java  | 6 +++---
 .../beam/runners/flink/FlinkStreamingTransformTranslators.java  | 6 +++---
 .../translation/wrappers/streaming/SplittableDoFnOperator.java  | 4 ++--
 .../src/main/java/org/apache/beam/fn/harness/CombineRunners.java| 2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)

[beam] branch master updated (7b4a32b -> f55e80f)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7b4a32b  Merge pull request #5709: Simplifies API of WireCoders and 
LengthPrefixUnknownCoders.
 add 45d509c  [BEAM-4615] Flink job server wrapper and shadow jar
 new f55e80f  Merge pull request #5726: [BEAM-4615] Flink job server 
wrapper and shadow jar

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/flink/job-server/build.gradle | 46 +++
 settings.gradle   |  2 ++
 2 files changed, 48 insertions(+)
 create mode 100644 runners/flink/job-server/build.gradle

[beam] 01/01: Merge pull request #5726: [BEAM-4615] Flink job server wrapper and shadow jar

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f55e80ff3016790df49185581850c78476b5465c
Merge: 7b4a32b 45d509c
Author: Eugene Kirpichov 
AuthorDate: Mon Jun 25 17:24:55 2018 -0700

Merge pull request #5726: [BEAM-4615] Flink job server wrapper and shadow 
jar

[BEAM-4615] Flink job server wrapper and shadow jar

 runners/flink/job-server/build.gradle | 46 +++
 settings.gradle   |  2 ++
 2 files changed, 48 insertions(+)

[beam] branch master updated (999d03f -> 7b4a32b)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 999d03f  Add PortableRunner service registrars
 add d2bd995  Simplifies API of WireCoders and LengthPrefixUnknownCoders.
 add eaab053  Address comments
 new 7b4a32b  Merge pull request #5709: Simplifies API of WireCoders and 
LengthPrefixUnknownCoders.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../direct/GroupByKeyOnlyEvaluatorFactory.java |   3 +-
 .../portable/GroupByKeyOnlyEvaluatorFactory.java   |  12 +-
 .../runners/direct/portable/ReferenceRunner.java   |  10 +-
 .../BundleFactoryOutputReceiverFactoryTest.java|  57 +++
 .../GroupByKeyOnlyEvaluatorFactoryTest.java|  35 ++---
 .../BatchFlinkExecutableStageContext.java  |  18 +--
 .../control/ProcessBundleDescriptors.java  | 159 ---
 .../fnexecution/state/StateRequestHandlers.java|  17 +-
 .../wire/LengthPrefixUnknownCoders.java| 172 -
 .../beam/runners/fnexecution/wire/WireCoders.java  |  45 +++---
 .../fnexecution/control/RemoteExecutionTest.java   |   2 +-
 .../wire/LengthPrefixUnknownCodersTest.java|  20 +--
 12 files changed, 224 insertions(+), 326 deletions(-)

[beam] 01/01: Merge pull request #5709: Simplifies API of WireCoders and LengthPrefixUnknownCoders.

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7b4a32b13579bbf6f893e4c3f44694cd691c9eaa
Merge: 999d03f eaab053
Author: Eugene Kirpichov 
AuthorDate: Mon Jun 25 17:06:28 2018 -0700

Merge pull request #5709: Simplifies API of WireCoders and 
LengthPrefixUnknownCoders.

Simplifies API of WireCoders and LengthPrefixUnknownCoders.

 .../direct/GroupByKeyOnlyEvaluatorFactory.java |   3 +-
 .../portable/GroupByKeyOnlyEvaluatorFactory.java   |  12 +-
 .../runners/direct/portable/ReferenceRunner.java   |  10 +-
 .../BundleFactoryOutputReceiverFactoryTest.java|  57 +++
 .../GroupByKeyOnlyEvaluatorFactoryTest.java|  35 ++---
 .../BatchFlinkExecutableStageContext.java  |  18 +--
 .../control/ProcessBundleDescriptors.java  | 159 ---
 .../fnexecution/state/StateRequestHandlers.java|  17 +-
 .../wire/LengthPrefixUnknownCoders.java| 172 -
 .../beam/runners/fnexecution/wire/WireCoders.java  |  45 +++---
 .../fnexecution/control/RemoteExecutionTest.java   |   2 +-
 .../wire/LengthPrefixUnknownCodersTest.java|  20 +--
 12 files changed, 224 insertions(+), 326 deletions(-)

[beam] branch master updated (5f8c615 -> 34e6936)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5f8c615  Merge pull request #5752: Merge Samza runner to master
 add dcb8148  Adds support for SDF in ULR and the Java SDK.
 add 59d066a  Address easy review comments
 add 78c4c2f  Handles memoization of encoded key/window in StateAccessor
 new 34e6936  Merge pull request #5566: [BEAM-2939, BEAM-3743, BEAM-3833] 
Supports SDF in Reference Runner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../fn-execution/src/main/proto/beam_fn_api.proto  |  11 +-
 .../pipeline/src/main/proto/beam_runner_api.proto  |  18 +
 .../translation/operators/ApexParDoOperator.java   |   6 +-
 .../runners/core/construction/Environments.java|   1 +
 .../core/construction/PTransformTranslation.java   |   2 +
 .../core/construction/RehydratedComponents.java|   4 +
 .../runners/core/construction/SplittableParDo.java |  17 +-
 .../core/construction/graph/PipelineValidator.java |   2 +-
 .../apache/beam/runners/core/ProcessFnRunner.java  |  16 +-
 .../core/SplittableParDoViaKeyedWorkItems.java |  32 +-
 .../runners/core/SplittableParDoProcessFnTest.java |   9 +-
 .../runners/direct/ParDoMultiOverrideFactory.java  |   4 +-
 .../SplittableProcessElementsEvaluatorFactory.java |  14 +-
 .../CopyOnAccessInMemoryStateInternals.java|   2 +-
 .../portable/ImmutableListBundleFactory.java   |   8 +
 .../runners/direct/portable/ReferenceRunner.java   | 236 +-
 .../SplittableRemoteStageEvaluatorFactory.java | 179 +
 .../portable/TransformEvaluatorRegistry.java   |   4 +
 .../direct/portable/ReferenceRunnerTest.java   |  91 ++-
 .../wrappers/streaming/SplittableDoFnOperator.java |  14 +-
 .../splittabledofn/SDFFeederViaStateAndTimers.java | 178 +
 .../fnexecution/splittabledofn}/package-info.java  |   6 +-
 .../beam/fn/harness/BeamFnDataReadRunner.java  |   4 +-
 .../beam/fn/harness/BeamFnDataWriteRunner.java |   4 +-
 .../beam/fn/harness/BoundedSourceRunner.java   |   4 +-
 .../fn/harness/DoFnPTransformRunnerFactory.java| 229 ++
 .../org/apache/beam/fn/harness/FlattenRunner.java  |   4 +-
 .../apache/beam/fn/harness/FnApiDoFnRunner.java| 865 +++--
 .../org/apache/beam/fn/harness/MapFnRunners.java   |   4 +-
 .../beam/fn/harness/PTransformRunnerFactory.java   |   6 +-
 .../harness/SplittableProcessElementsRunner.java   | 268 +++
 .../fn/harness/control/BundleSplitListener.java|  41 +
 .../fn/harness/control/ProcessBundleHandler.java   |  51 +-
 .../beam/fn/harness/state/FnApiStateAccessor.java  | 459 +++
 .../beam/fn/harness/state/SideInputSpec.java}  |  40 +-
 .../beam/fn/harness/AssignWindowsRunnerTest.java   |   3 +-
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  |   3 +-
 .../beam/fn/harness/BeamFnDataWriteRunnerTest.java |   3 +-
 .../beam/fn/harness/BoundedSourceRunnerTest.java   |   3 +-
 .../apache/beam/fn/harness/FlattenRunnerTest.java  |   6 +-
 .../beam/fn/harness/FnApiDoFnRunnerTest.java   | 304 ++--
 .../apache/beam/fn/harness/MapFnRunnersTest.java   |   9 +-
 .../harness/control/ProcessBundleHandlerTest.java  |  23 +-
 43 files changed, 2064 insertions(+), 1123 deletions(-)
 create mode 100644 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SplittableRemoteStageEvaluatorFactory.java
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java
 copy runners/{apex/src/main/java/org/apache/beam/runners/apex/translation => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/splittabledofn}/package-info.java
 (84%)
 create mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/DoFnPTransformRunnerFactory.java
 create mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableProcessElementsRunner.java
 create mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/BundleSplitListener.java
 create mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/FnApiStateAccessor.java
 copy 
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/AggregateWindowField.java
 => harness/src/main/java/org/apache/beam/fn/harness/state/SideInputSpec.java} 
(52%)

[beam] 01/01: Merge pull request #5566: [BEAM-2939, BEAM-3743, BEAM-3833] Supports SDF in Reference Runner

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 34e6936c097330e39a9f1bda57eaad7c10d5be69
Merge: 5f8c615 78c4c2f
Author: Eugene Kirpichov 
AuthorDate: Mon Jun 25 14:52:18 2018 -0700

Merge pull request #5566: [BEAM-2939, BEAM-3743, BEAM-3833] Supports SDF in 
Reference Runner

[BEAM-2939, BEAM-3743, BEAM-3833] Supports SDF in Reference Runner

 .../fn-execution/src/main/proto/beam_fn_api.proto  |  11 +-
 .../pipeline/src/main/proto/beam_runner_api.proto  |  18 +
 .../translation/operators/ApexParDoOperator.java   |   6 +-
 .../runners/core/construction/Environments.java|   1 +
 .../core/construction/PTransformTranslation.java   |   2 +
 .../core/construction/RehydratedComponents.java|   4 +
 .../runners/core/construction/SplittableParDo.java |  17 +-
 .../core/construction/graph/PipelineValidator.java |   2 +-
 .../apache/beam/runners/core/ProcessFnRunner.java  |  16 +-
 .../core/SplittableParDoViaKeyedWorkItems.java |  32 +-
 .../runners/core/SplittableParDoProcessFnTest.java |   9 +-
 .../runners/direct/ParDoMultiOverrideFactory.java  |   4 +-
 .../SplittableProcessElementsEvaluatorFactory.java |  14 +-
 .../CopyOnAccessInMemoryStateInternals.java|   2 +-
 .../portable/ImmutableListBundleFactory.java   |   8 +
 .../runners/direct/portable/ReferenceRunner.java   | 236 +-
 .../SplittableRemoteStageEvaluatorFactory.java | 179 +
 .../portable/TransformEvaluatorRegistry.java   |   4 +
 .../direct/portable/ReferenceRunnerTest.java   |  91 ++-
 .../wrappers/streaming/SplittableDoFnOperator.java |  14 +-
 .../splittabledofn/SDFFeederViaStateAndTimers.java | 178 +
 .../fnexecution/splittabledofn/package-info.java   |  20 +
 .../beam/fn/harness/BeamFnDataReadRunner.java  |   4 +-
 .../beam/fn/harness/BeamFnDataWriteRunner.java |   4 +-
 .../beam/fn/harness/BoundedSourceRunner.java   |   4 +-
 .../fn/harness/DoFnPTransformRunnerFactory.java| 229 ++
 .../org/apache/beam/fn/harness/FlattenRunner.java  |   4 +-
 .../apache/beam/fn/harness/FnApiDoFnRunner.java| 865 +++--
 .../org/apache/beam/fn/harness/MapFnRunners.java   |   4 +-
 .../beam/fn/harness/PTransformRunnerFactory.java   |   6 +-
 .../harness/SplittableProcessElementsRunner.java   | 268 +++
 .../fn/harness/control/BundleSplitListener.java|  41 +
 .../fn/harness/control/ProcessBundleHandler.java   |  51 +-
 .../beam/fn/harness/state/FnApiStateAccessor.java  | 459 +++
 .../beam/fn/harness/state/SideInputSpec.java   |  51 ++
 .../beam/fn/harness/AssignWindowsRunnerTest.java   |   3 +-
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  |   3 +-
 .../beam/fn/harness/BeamFnDataWriteRunnerTest.java |   3 +-
 .../beam/fn/harness/BoundedSourceRunnerTest.java   |   3 +-
 .../apache/beam/fn/harness/FlattenRunnerTest.java  |   6 +-
 .../beam/fn/harness/FnApiDoFnRunnerTest.java   | 304 ++--
 .../apache/beam/fn/harness/MapFnRunnersTest.java   |   9 +-
 .../harness/control/ProcessBundleHandlerTest.java  |  23 +-
 43 files changed, 2113 insertions(+), 1099 deletions(-)

[beam] branch master updated (809fcf9 -> 77e93d9)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 809fcf9  Merge pull request #5702 from [BEAM-4605] Remove subtransform 
of GBK.
 add f0da541  Add an AddHarnessIdInterceptor
 add c097034  Remove the runners/reference InProcessManagedChannelFactory
 add 33616cc  Allow ManagedChannelFactories to attach Interceptors
 add 04e77aa  Populate and Use Harness IDs in the Java Harness
 add 56f1ecc  Fixes checkstyle
 new 77e93d9  Merge pull request #5680: [BEAM-4145] Populate the worker_id 
metadata in the Java SDK Harness

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../environment/InProcessEnvironmentFactory.java   |  1 +
 .../fnexecution/control/RemoteExecutionTest.java   |  1 +
 .../testing/InProcessManagedChannelFactory.java| 36 ---
 sdks/go/pkg/beam/util/grpcx/metadata.go|  2 +-
 sdks/java/container/boot.go|  1 +
 .../beam/sdk/fn/channel/ManagedChannelFactory.java | 63 ++
 .../fn/test/InProcessManagedChannelFactory.java|  6 +-
 .../java/org/apache/beam/fn/harness/FnHarness.java | 75 --
 .../harness/control/AddHarnessIdInterceptor.java}  | 33 +-
 .../fn/harness/control/BeamFnControlClient.java| 17 +++--
 .../org/apache/beam/fn/harness/FnHarnessTest.java  |  2 +-
 .../harness/control/BeamFnControlClientTest.java   | 15 ++---
 12 files changed, 134 insertions(+), 118 deletions(-)
 delete mode 100644 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/testing/InProcessManagedChannelFactory.java
 copy 
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlBinaryOperator.java
 => 
harness/src/main/java/org/apache/beam/fn/harness/control/AddHarnessIdInterceptor.java}
 (52%)

[beam] 01/01: Merge pull request #5680: [BEAM-4145] Populate the worker_id metadata in the Java SDK Harness

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 77e93d925b6aaf6c0f764b521c60a1d35976a821
Merge: 809fcf9 56f1ecc
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 21 12:39:27 2018 -0700

Merge pull request #5680: [BEAM-4145] Populate the worker_id metadata in 
the Java SDK Harness

[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness

 .../environment/InProcessEnvironmentFactory.java   |  1 +
 .../fnexecution/control/RemoteExecutionTest.java   |  1 +
 .../testing/InProcessManagedChannelFactory.java| 36 ---
 sdks/go/pkg/beam/util/grpcx/metadata.go|  2 +-
 sdks/java/container/boot.go|  1 +
 .../beam/sdk/fn/channel/ManagedChannelFactory.java | 63 ++
 .../fn/test/InProcessManagedChannelFactory.java|  6 +-
 .../java/org/apache/beam/fn/harness/FnHarness.java | 75 --
 .../harness/control/AddHarnessIdInterceptor.java   | 41 
 .../fn/harness/control/BeamFnControlClient.java| 17 +++--
 .../org/apache/beam/fn/harness/FnHarnessTest.java  |  2 +-
 .../harness/control/BeamFnControlClientTest.java   | 15 ++---
 12 files changed, 158 insertions(+), 102 deletions(-)

diff --cc 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index e969e5f,cd9f8b0..e2e9bcb
--- 
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++ 
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@@ -132,10 -129,10 +132,11 @@@ public class RemoteExecutionTest implem
  
  // Create the SDK harness, and wait until it connects
  sdkHarnessExecutor = Executors.newSingleThreadExecutor(threadFactory);
 -sdkHarnessExecutor.submit(
 -() ->
 +sdkHarnessExecutorFuture = sdkHarnessExecutor.submit(
 +() -> {
 +  try {
  FnHarness.main(
+ "id",
  PipelineOptionsFactory.create(),
  loggingServer.getApiServiceDescriptor(),
  controlServer.getApiServiceDescriptor(),
diff --cc 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
index 5bac1ac,aa6a6fb..d13d42a
--- a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
+++ b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java
@@@ -120,12 -131,13 +131,13 @@@ public class FnHarness 
Endpoints.ApiServiceDescriptor loggingApiServiceDescriptor,
Endpoints.ApiServiceDescriptor controlApiServiceDescriptor,
ManagedChannelFactory channelFactory,
 -  OutboundObserverFactory outboundObserverFactory) {
 +  OutboundObserverFactory outboundObserverFactory) throws Exception {
  IdGenerator idGenerator = IdGenerators.decrementingLongs();
- try (BeamFnLoggingClient logging = new BeamFnLoggingClient(
- options,
- loggingApiServiceDescriptor,
- channelFactory::forDescriptor)) {
+ // The logging client variable is not used per se, but during its 
lifetime (until close()) it
+ // intercepts logging and sends it to the logging service.
+ try (BeamFnLoggingClient logging =
+ new BeamFnLoggingClient(
+ options, loggingApiServiceDescriptor, 
channelFactory::forDescriptor)) {
  
LOG.info("Fn Harness started");
// Register standard file systems.

[beam] 01/01: Merge pull request #5712: Fixing key lookup in FlinkBatchSideInputHandlerFactory.SideInputHandler

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b26e143e490c00d511554b88854739027800e35a
Merge: f5f2c16 def972c
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 21 10:56:14 2018 -0700

Merge pull request #5712: Fixing key lookup in 
FlinkBatchSideInputHandlerFactory.SideInputHandler

Fixing key lookup in FlinkBatchSideInputHandlerFactory.SideInputHandler

 .../FlinkBatchSideInputHandlerFactory.java | 30 ++
 1 file changed, 19 insertions(+), 11 deletions(-)

[beam] branch master updated (f5f2c16 -> b26e143)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f5f2c16  Merge pull request #5721: [BEAM-4611] Point gearpump 
post-commit tests to master
 add def972c  Fixing key lookup in 
FlinkBatchSideInputHandlerFactory.SideInputHandler
 new b26e143  Merge pull request #5712: Fixing key lookup in 
FlinkBatchSideInputHandlerFactory.SideInputHandler

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../FlinkBatchSideInputHandlerFactory.java | 30 ++
 1 file changed, 19 insertions(+), 11 deletions(-)

[beam] branch master updated (9d5cbc4 -> db1057c)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9d5cbc4  Merge pull request #5705: [BEAM-4559] Fix typo in _ghprb 
suffix
 add e0c377a  Introduces PipelineValidator that checks the well-formedness 
of a pipeline
 new db1057c  Merge pull request #5703: Introduces PipelineValidator that 
checks the well-formedness of a pipeline

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/PipelineTranslation.java |   7 +-
 .../core/construction/graph/ExecutableStage.java   |   6 +-
 .../core/construction/graph/FusedPipeline.java |   9 +-
 .../construction/graph/GreedyPipelineFuser.java|   2 +
 .../construction/graph/OutputDeduplicator.java |   9 +-
 .../core/construction/graph/PipelineValidator.java | 287 +
 .../construction/graph/ExecutableStageTest.java|   4 +-
 .../graph/GreedyPipelineFuserTest.java | 264 +++
 .../graph/ImmutableExecutableStageTest.java|   2 +-
 .../runners/direct/portable/ReferenceRunner.java   |  16 +-
 .../jobsubmission/InMemoryJobService.java  |   6 +
 ...gleEnvironmentInstanceJobBundleFactoryTest.java |   2 +-
 .../beam/runners/reference/ReferenceRunner.java|  35 ---
 13 files changed, 488 insertions(+), 161 deletions(-)
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/PipelineValidator.java
 delete mode 100644 
runners/reference/java/src/main/java/org/apache/beam/runners/reference/ReferenceRunner.java

[beam] 01/01: Merge pull request #5703: Introduces PipelineValidator that checks the well-formedness of a pipeline

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit db1057c37e0249029bdd6302567e1e2ce124e269
Merge: 9d5cbc4 e0c377a
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 21:41:37 2018 -0700

Merge pull request #5703: Introduces PipelineValidator that checks the 
well-formedness of a pipeline

Introduces PipelineValidator that checks the well-formedness of a pipeline

 .../core/construction/PipelineTranslation.java |   7 +-
 .../core/construction/graph/ExecutableStage.java   |   6 +-
 .../core/construction/graph/FusedPipeline.java |   9 +-
 .../construction/graph/GreedyPipelineFuser.java|   2 +
 .../construction/graph/OutputDeduplicator.java |   9 +-
 .../core/construction/graph/PipelineValidator.java | 287 +
 .../construction/graph/ExecutableStageTest.java|   4 +-
 .../graph/GreedyPipelineFuserTest.java | 264 +++
 .../graph/ImmutableExecutableStageTest.java|   2 +-
 .../runners/direct/portable/ReferenceRunner.java   |  16 +-
 .../jobsubmission/InMemoryJobService.java  |   6 +
 ...gleEnvironmentInstanceJobBundleFactoryTest.java |   2 +-
 .../beam/runners/reference/ReferenceRunner.java|  35 ---
 13 files changed, 488 insertions(+), 161 deletions(-)

[beam] 03/03: This closes #5688: [BEAM-4285] Implement Flink batch side input handler

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1c42e90ee46314a3f606d088a8f48334d7ac
Merge: d7cc0bd 0471713
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 17:18:13 2018 -0700

This closes #5688: [BEAM-4285] Implement Flink batch side input handler

 .../graph/ImmutableExecutableStage.java|   4 +-
 .../BatchFlinkExecutableStageContext.java  |  28 ++-
 .../FlinkBatchSideInputHandlerFactory.java | 131 ++
 .../functions/FlinkBatchStateRequestHandler.java   |  42 -
 .../FlinkBatchSideInputHandlerFactoryTest.java | 199 +
 .../control/DockerJobBundleFactory.java|   3 +-
 6 files changed, 361 insertions(+), 46 deletions(-)

[beam] branch master updated (d7cc0bd -> 1c42e90)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d7cc0bd  Merge pull request #5707: Remove unused import
 new 72d3c7b  Make ImmutableExecutableStage constructors public
 new 0471713  [BEAM-4285] Implement Flink batch side input handler
 new 1c42e90  This closes #5688: [BEAM-4285] Implement Flink batch side 
input handler

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../graph/ImmutableExecutableStage.java|   4 +-
 .../BatchFlinkExecutableStageContext.java  |  28 ++-
 .../FlinkBatchSideInputHandlerFactory.java | 131 ++
 .../functions/FlinkBatchStateRequestHandler.java   |  42 -
 .../FlinkBatchSideInputHandlerFactoryTest.java | 199 +
 .../control/DockerJobBundleFactory.java|   3 +-
 6 files changed, 361 insertions(+), 46 deletions(-)
 create mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchSideInputHandlerFactory.java
 delete mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchStateRequestHandler.java
 create mode 100644 
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchSideInputHandlerFactoryTest.java

[beam] 02/03: [BEAM-4285] Implement Flink batch side input handler

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 047171392595007964c4f8ed1ae37204ac301c93
Author: Ben Sidhom 
AuthorDate: Wed May 2 17:55:19 2018 -0700

[BEAM-4285] Implement Flink batch side input handler

This implements a state request handler for multimap side inputs.
---
 .../BatchFlinkExecutableStageContext.java  |  28 ++-
 .../FlinkBatchSideInputHandlerFactory.java | 131 ++
 .../functions/FlinkBatchStateRequestHandler.java   |  42 -
 .../FlinkBatchSideInputHandlerFactoryTest.java | 199 +
 .../control/DockerJobBundleFactory.java|   3 +-
 5 files changed, 359 insertions(+), 44 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/BatchFlinkExecutableStageContext.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/BatchFlinkExecutableStageContext.java
index 4410476..519a63b 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/BatchFlinkExecutableStageContext.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/BatchFlinkExecutableStageContext.java
@@ -20,16 +20,23 @@ package org.apache.beam.runners.flink.translation.functions;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
 import com.google.common.cache.LoadingCache;
+import java.io.IOException;
+import org.apache.beam.model.pipeline.v1.Endpoints;
 import org.apache.beam.runners.core.construction.graph.ExecutableStage;
 import org.apache.beam.runners.fnexecution.control.DockerJobBundleFactory;
 import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
 import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
 import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
 import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
+import 
org.apache.beam.runners.fnexecution.state.StateRequestHandlers.MultimapSideInputHandlerFactory;
 import org.apache.flink.api.common.functions.RuntimeContext;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+// TODO: Rename this to FlinkBatchExecutableStageContext for consistency.
 /** Implementation of a {@link FlinkExecutableStageContext} for batch jobs. */
 class BatchFlinkExecutableStageContext implements FlinkExecutableStageContext {
   private static final Logger LOG = 
LoggerFactory.getLogger(BatchFlinkExecutableStageContext.class);
@@ -53,7 +60,26 @@ class BatchFlinkExecutableStageContext implements 
FlinkExecutableStageContext {
   @Override
   public StateRequestHandler getStateRequestHandler(
   ExecutableStage executableStage, RuntimeContext runtimeContext) {
-return FlinkBatchStateRequestHandler.forStage(executableStage, 
runtimeContext);
+MultimapSideInputHandlerFactory sideInputHandlerFactory =
+FlinkBatchSideInputHandlerFactory.forStage(executableStage, 
runtimeContext);
+ExecutableProcessBundleDescriptor processBundleDescriptor;
+try {
+  // NOTE: We require an executable bundle descriptor for the 
StateRequestHandlers construction
+  // below. This only uses the bundle descriptor for side input specs and 
effectively ignores
+  // data and state endpoints. We rely on the fact that PCollections and 
coders are structurally
+  // identical between instantiations here to prevent having to wire the 
original executable
+  // bundle descriptor here. The correct long-term fix is to move side 
input logic out of
+  // ExecutableProcessBundleDescriptor and into ExecutableStage.
+  processBundleDescriptor =
+  ProcessBundleDescriptors.fromExecutableStage(
+  "id", executableStage, 
Endpoints.ApiServiceDescriptor.getDefaultInstance());
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+StateRequestHandler stateRequestHandler =
+StateRequestHandlers.forMultimapSideInputHandlerFactory(
+processBundleDescriptor, sideInputHandlerFactory);
+return stateRequestHandler;
   }
 
   @Override
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchSideInputHandlerFactory.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchSideInputHandlerFactory.java
new file mode 100644
index 000..c07647c
--- /dev/null
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchSideInputHandlerFactory.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to 

[beam] 01/03: Make ImmutableExecutableStage constructors public

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 72d3c7b5cdf3aa4bb50b38171028dcf0b6efdbd3
Author: Ben Sidhom 
AuthorDate: Tue Jun 19 14:16:12 2018 -0700

Make ImmutableExecutableStage constructors public
---
 .../runners/core/construction/graph/ImmutableExecutableStage.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
index 278e5c6..ee77a87 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ImmutableExecutableStage.java
@@ -30,7 +30,7 @@ import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNo
 /** An {@link ExecutableStage} which is constructed with all of its initial 
state. */
 @AutoValue
 public abstract class ImmutableExecutableStage implements ExecutableStage {
-  static ImmutableExecutableStage ofFullComponents(
+  public static ImmutableExecutableStage ofFullComponents(
   Components components,
   Environment environment,
   PCollectionNode input,
@@ -49,7 +49,7 @@ public abstract class ImmutableExecutableStage implements 
ExecutableStage {
 return of(prunedComponents, environment, input, sideInputs, transforms, 
outputs);
   }
 
-  static ImmutableExecutableStage of(
+  public static ImmutableExecutableStage of(
   Components components,
   Environment environment,
   PCollectionNode input,

[beam] 01/01: Merge pull request #5707: Remove unused import

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d7cc0bd27ac39a586a8589c7b9514bc72cde599e
Merge: 32f72b1 7175b80
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 16:58:12 2018 -0700

Merge pull request #5707: Remove unused import

Remove unused import

 .../flink/translation/functions/BatchFlinkExecutableStageContext.java| 1 -
 1 file changed, 1 deletion(-)

[beam] branch master updated (32f72b1 -> d7cc0bd)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 32f72b1  Merge pull request #5706 from charlesccychen/fix-build
 add 7175b80  Remove unused import
 new d7cc0bd  Merge pull request #5707: Remove unused import

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../flink/translation/functions/BatchFlinkExecutableStageContext.java| 1 -
 1 file changed, 1 deletion(-)

[beam] 01/01: Merge pull request #5701: Remove submodule

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit edf558cef85f041ae4aaded93d18dd3d24f53503
Merge: 908c63c fca5220
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 15:31:02 2018 -0700

Merge pull request #5701: Remove submodule

Remove submodule

 beam | 1 -
 1 file changed, 1 deletion(-)

[beam] branch master updated (908c63c -> edf558c)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 908c63c  Merge pull request #5434: [BEAM-3949] IOIT's setup() and 
teardown() db connection attempt sometimes fail resulting in test flakiness
 add fca5220  Remove submodule
 new edf558c  Merge pull request #5701: Remove submodule

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 beam | 1 -
 1 file changed, 1 deletion(-)
 delete mode 16 beam

[beam] branch master updated (b9a94dd -> 3596534)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b9a94dd  Merge pull request #5692: Fix cache references in portable 
flink runner
 add 5e0fd68  A couple more trivial changes extracted from #5662
 new 3596534  Merge pull request #5690: A couple more trivial changes 
extracted from #5662

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../BeamFileSystemArtifactRetrievalService.java| 26 +-
 .../BeamFileSystemArtifactServicesTest.java|  4 ++--
 2 files changed, 17 insertions(+), 13 deletions(-)

[beam] 01/01: Merge pull request #5690: A couple more trivial changes extracted from #5662

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 359653495686cfc391b5c097d0e36cc27f96c6bc
Merge: b9a94dd 5e0fd68
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 12:50:47 2018 -0700

Merge pull request #5690: A couple more trivial changes extracted from #5662

A couple more trivial changes extracted from #5662

 .../BeamFileSystemArtifactRetrievalService.java| 26 +-
 .../BeamFileSystemArtifactServicesTest.java|  4 ++--
 2 files changed, 17 insertions(+), 13 deletions(-)

[beam] 01/01: Merge pull request #5692: Fix cache references in portable flink runner

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b9a94dda07d9a37d0c6a6bd932eb79de0d67ee42
Merge: 43a4044 fbe3063
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 12:37:59 2018 -0700

Merge pull request #5692: Fix cache references in portable flink runner

Fix cache references in portable flink runner

 .../functions/BatchFlinkExecutableStageContext.java | 13 ++---
 .../translation/functions/FlinkExecutableStageFunction.java |  5 -
 .../wrappers/streaming/ExecutableStageDoFnOperator.java |  5 -
 .../runners/fnexecution/control/DockerJobBundleFactory.java |  1 -
 4 files changed, 10 insertions(+), 14 deletions(-)

[beam] branch master updated (43a4044 -> b9a94dd)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 43a4044  Merge pull request #5652 from Fematich/examples
 add fbe3063  Fix cache references in portable flink runner
 new b9a94dd  Merge pull request #5692: Fix cache references in portable 
flink runner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../functions/BatchFlinkExecutableStageContext.java | 13 ++---
 .../translation/functions/FlinkExecutableStageFunction.java |  5 -
 .../wrappers/streaming/ExecutableStageDoFnOperator.java |  5 -
 .../runners/fnexecution/control/DockerJobBundleFactory.java |  1 -
 4 files changed, 10 insertions(+), 14 deletions(-)

[beam] 01/01: Merge pull request #5696: Fix build break from PR 5676, which modified method signatures.

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8ce1dca15fe473f4cd134aa5008c92109bbb4262
Merge: 3dc9617 921a3c2
Author: Eugene Kirpichov 
AuthorDate: Wed Jun 20 11:39:13 2018 -0700

Merge pull request #5696: Fix build break from PR 5676, which modified 
method signatures.

Fix build break from PR 5676, which modified method signatures.

 beam|  1 +
 sdks/go/cmd/beamctl/cmd/artifact.go |  7 +--
 sdks/go/pkg/beam/runners/universal/runnerlib/execute.go |  6 +++---
 sdks/go/pkg/beam/runners/universal/runnerlib/job.go | 14 +++---
 sdks/go/pkg/beam/runners/universal/runnerlib/stage.go   |  8 
 5 files changed, 20 insertions(+), 16 deletions(-)

[beam] branch master updated (3dc9617 -> 8ce1dca)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3dc9617  Merge pull request #5654 from udim/jupyter
 add 921a3c2  Fix build break from PR 5676, which modified method 
signatures.
 new 8ce1dca  Merge pull request #5696: Fix build break from PR 5676, which 
modified method signatures.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 beam|  1 +
 sdks/go/cmd/beamctl/cmd/artifact.go |  7 +--
 sdks/go/pkg/beam/runners/universal/runnerlib/execute.go |  6 +++---
 sdks/go/pkg/beam/runners/universal/runnerlib/job.go | 14 +++---
 sdks/go/pkg/beam/runners/universal/runnerlib/stage.go   |  8 
 5 files changed, 20 insertions(+), 16 deletions(-)
 create mode 16 beam

[beam] branch master updated (ebdf6ba -> 3ae2a78)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ebdf6ba  Fixes null pointer exception on empty pubsub message data 
(#5016)
 add 721357e  Propagates artifact retrieval token in Flink runner and to 
the Java harness
 new 3ae2a78  Merge pull request #5676: [BEAM-4291] Propagates artifact 
retrieval token in Flink runner and to the Java harness

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/proto/beam_provision_api.proto|   4 +
 .../beam/runners/flink/FlinkJobInvocation.java |  29 +-
 .../apache/beam/runners/flink/FlinkJobInvoker.java |   3 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   3 +-
 .../FlinkExecutableStageFunctionTest.java  |   3 +-
 .../fnexecution/jobsubmission/JobInvoker.java  |  10 +-
 .../runners/fnexecution/provisioning/JobInfo.java  |  10 +-
 sdks/go/BUILD.md   |   3 +
 sdks/go/container/boot.go  |   2 +-
 sdks/go/pkg/beam/artifact/materialize.go   |  18 +-
 sdks/go/pkg/beam/artifact/materialize_test.go  |  63 +--
 sdks/go/pkg/beam/artifact/server_test.go   |  49 ++-
 sdks/go/pkg/beam/artifact/stage.go |  16 +-
 sdks/go/pkg/beam/artifact/stage_test.go|  22 +-
 .../go/pkg/beam/model/PROTOBUF.md  |  14 +-
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 370 +-
 .../model/fnexecution_v1/beam_provision_api.pb.go  | 127 +++---
 .../model/jobmanagement_v1/beam_artifact_api.pb.go |  68 ++--
 .../beam/model/jobmanagement_v1/beam_job_api.pb.go |  72 ++--
 .../beam/model/pipeline_v1/beam_runner_api.pb.go   | 432 ++---
 sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go |  14 +-
 .../model/pipeline_v1/standard_window_fns.pb.go|  32 +-
 sdks/java/container/boot.go|   2 +-
 sdks/python/container/boot.go  |   2 +-
 24 files changed, 687 insertions(+), 681 deletions(-)
 copy CONTRIBUTING.md => sdks/go/pkg/beam/model/PROTOBUF.md (63%)

[beam] 01/01: Merge pull request #5676: [BEAM-4291] Propagates artifact retrieval token in Flink runner and to the Java harness

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3ae2a78c2da5f5e60b672dcf374939b7c036cf47
Merge: ebdf6ba 721357e
Author: Eugene Kirpichov 
AuthorDate: Tue Jun 19 18:09:33 2018 -0700

Merge pull request #5676: [BEAM-4291] Propagates artifact retrieval token 
in Flink runner and to the Java harness

[BEAM-4291] Propagates artifact retrieval token in Flink runner and to the 
Java harness

 .../src/main/proto/beam_provision_api.proto|   4 +
 .../beam/runners/flink/FlinkJobInvocation.java |  29 +-
 .../apache/beam/runners/flink/FlinkJobInvoker.java |   3 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   3 +-
 .../FlinkExecutableStageFunctionTest.java  |   3 +-
 .../fnexecution/jobsubmission/JobInvoker.java  |  10 +-
 .../runners/fnexecution/provisioning/JobInfo.java  |  10 +-
 sdks/go/BUILD.md   |   3 +
 sdks/go/container/boot.go  |   2 +-
 sdks/go/pkg/beam/artifact/materialize.go   |  18 +-
 sdks/go/pkg/beam/artifact/materialize_test.go  |  63 +--
 sdks/go/pkg/beam/artifact/server_test.go   |  49 ++-
 sdks/go/pkg/beam/artifact/stage.go |  16 +-
 sdks/go/pkg/beam/artifact/stage_test.go|  22 +-
 sdks/go/pkg/beam/model/PROTOBUF.md |  30 ++
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 370 +-
 .../model/fnexecution_v1/beam_provision_api.pb.go  | 127 +++---
 .../model/jobmanagement_v1/beam_artifact_api.pb.go |  68 ++--
 .../beam/model/jobmanagement_v1/beam_job_api.pb.go |  72 ++--
 .../beam/model/pipeline_v1/beam_runner_api.pb.go   | 432 ++---
 sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go |  14 +-
 .../model/pipeline_v1/standard_window_fns.pb.go|  32 +-
 sdks/java/container/boot.go|   2 +-
 sdks/python/container/boot.go  |   2 +-
 24 files changed, 707 insertions(+), 677 deletions(-)

[beam] 01/01: Merge pull request #5678: [BEAM-4519] Java SDK passes the staging session token artifact staging service.

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 04389728d3b813b8ffc128901a70c7ee587fdc9a
Merge: a926318 d17c5f1
Author: Eugene Kirpichov 
AuthorDate: Tue Jun 19 12:39:39 2018 -0700

Merge pull request #5678: [BEAM-4519] Java SDK passes the staging session 
token artifact staging service.

[BEAM-4519] Java SDK passes the staging session token artifact staging 
service.

 .../core/construction/ArtifactServiceStager.java   | 24 ++
 .../construction/ArtifactServiceStagerTest.java|  6 ++-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |  3 +-
 .../job/ReferenceRunnerJobServiceTest.java |  2 +
 .../BeamFileSystemArtifactStagingService.java  | 56 ++
 .../jobsubmission/InMemoryJobService.java  |  5 --
 .../jobsubmission/InMemoryJobServiceTest.java  |  2 +-
 .../beam/runners/reference/PortableRunner.java |  3 +-
 8 files changed, 64 insertions(+), 37 deletions(-)

[beam] branch master updated (a926318 -> 0438972)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a926318  Merge pull request #5679: load beam fileSystems in FnHarness
 add d17c5f1  Implement new staging token contract in java SDK
 new 0438972  Merge pull request #5678: [BEAM-4519] Java SDK passes the 
staging session token artifact staging service.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/ArtifactServiceStager.java   | 24 ++
 .../construction/ArtifactServiceStagerTest.java|  6 ++-
 ...ocalFileSystemArtifactRetrievalServiceTest.java |  3 +-
 .../job/ReferenceRunnerJobServiceTest.java |  2 +
 .../BeamFileSystemArtifactStagingService.java  | 56 ++
 .../jobsubmission/InMemoryJobService.java  |  5 --
 .../jobsubmission/InMemoryJobServiceTest.java  |  2 +-
 .../beam/runners/reference/PortableRunner.java |  3 +-
 8 files changed, 64 insertions(+), 37 deletions(-)

[beam] 01/01: Merge pull request #5679: load beam fileSystems in FnHarness

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a9263189ff8da702ae3a52d5acfa857640609666
Merge: c7f18c7 47547c8
Author: Eugene Kirpichov 
AuthorDate: Tue Jun 19 12:38:21 2018 -0700

Merge pull request #5679: load beam fileSystems in FnHarness

load beam fileSystems in FnHarness

 .../harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java| 3 +++
 1 file changed, 3 insertions(+)

[beam] branch master updated (c7f18c7 -> a926318)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c7f18c7  Merge pull request #5481: [BEAM-4388] Support optimized 
logical plan
 add 47547c8  load beam fileSystems in FnHarness
 new a926318  Merge pull request #5679: load beam fileSystems in FnHarness

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../harness/src/main/java/org/apache/beam/fn/harness/FnHarness.java| 3 +++
 1 file changed, 3 insertions(+)

[beam] branch master updated (3df1861 -> 6889792)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3df1861  Merge pull request #5670: Fix case sensitivity in 
JdbcDriverTest
 add a2210f2  Fixing concurrency bug in BeamFileSystemArtifactServicesTest
 new 6889792  Merge pull request #5663: Fix flaky test 
BeamFileSystemArtifactServicesTest

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../fnexecution/artifact/BeamFileSystemArtifactServicesTest.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5663: Fix flaky test BeamFileSystemArtifactServicesTest

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6889792b994e1418ae73940198eb8767950287b5
Merge: 3df1861 a2210f2
Author: Eugene Kirpichov 
AuthorDate: Mon Jun 18 10:42:14 2018 -0700

Merge pull request #5663: Fix flaky test BeamFileSystemArtifactServicesTest

Fix flaky test BeamFileSystemArtifactServicesTest

 .../fnexecution/artifact/BeamFileSystemArtifactServicesTest.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (ba62679 -> 7c3fba0)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ba62679  Merge pull request #5393 Improve the TaggedOutput error
 add 0be1690  [BEAM-4291] Implements distributed artifact retrieval
 add 194ac6d  Deletes ArtifactSource
 new 7c3fba0  Merge pull request #5651: [BEAM-4291] Implements distributed 
artifact retrieval

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/flink/ArtifactSourcePool.java |  67 ---
 .../BatchFlinkExecutableStageContext.java  |  16 +-
 .../functions/FlinkExecutableStageContext.java |   3 -
 .../functions/FlinkExecutableStageFunction.java|   9 +-
 .../streaming/ExecutableStageDoFnOperator.java |   9 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   3 -
 .../FlinkExecutableStageFunctionTest.java  |   3 -
 .../fnexecution/artifact/ArtifactSource.java   |  41 --
 .../BeamFileSystemArtifactRetrievalService.java| 201 +
 .../artifact/BeamFileSystemArtifactSource.java |  88 
 .../BeamFileSystemArtifactStagingService.java  |  22 +-
 .../control/DockerJobBundleFactory.java|   7 +-
 .../BeamFileSystemArtifactServicesTest.java| 486 +
 .../artifact/BeamFileSystemArtifactSourceTest.java | 132 --
 .../BeamFileSystemArtifactStagingServiceTest.java  | 382 
 .../org/apache/beam/sdk/coders/ByteArrayCoder.java |   2 +-
 .../apache/beam/sdk/coders/StringUtf8Coder.java|   2 +-
 .../main/java/org/apache/beam/sdk/io/FileIO.java   |   5 +-
 .../java/org/apache/beam/sdk/util/StreamUtils.java |   2 +-
 .../org/apache/beam/sdk/util/StreamUtilsTest.java  |   6 +-
 20 files changed, 725 insertions(+), 761 deletions(-)
 delete mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/ArtifactSourcePool.java
 delete mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactSource.java
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
 delete mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactSource.java
 create mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactServicesTest.java
 delete mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactSourceTest.java
 delete mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingServiceTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5651: [BEAM-4291] Implements distributed artifact retrieval

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7c3fba01c9a69aca2a75bec52572d097a6f9c777
Merge: ba62679 194ac6d
Author: Eugene Kirpichov 
AuthorDate: Thu Jun 14 19:03:47 2018 -0700

Merge pull request #5651: [BEAM-4291] Implements distributed artifact 
retrieval

[BEAM-4291] Implements distributed artifact retrieval

 .../beam/runners/flink/ArtifactSourcePool.java |  67 ---
 .../BatchFlinkExecutableStageContext.java  |  16 +-
 .../functions/FlinkExecutableStageContext.java |   3 -
 .../functions/FlinkExecutableStageFunction.java|   9 +-
 .../streaming/ExecutableStageDoFnOperator.java |   9 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   3 -
 .../FlinkExecutableStageFunctionTest.java  |   3 -
 .../fnexecution/artifact/ArtifactSource.java   |  41 --
 .../BeamFileSystemArtifactRetrievalService.java| 201 +
 .../artifact/BeamFileSystemArtifactSource.java |  88 
 .../BeamFileSystemArtifactStagingService.java  |  22 +-
 .../control/DockerJobBundleFactory.java|   7 +-
 .../BeamFileSystemArtifactServicesTest.java| 486 +
 .../artifact/BeamFileSystemArtifactSourceTest.java | 132 --
 .../BeamFileSystemArtifactStagingServiceTest.java  | 382 
 .../org/apache/beam/sdk/coders/ByteArrayCoder.java |   2 +-
 .../apache/beam/sdk/coders/StringUtf8Coder.java|   2 +-
 .../main/java/org/apache/beam/sdk/io/FileIO.java   |   5 +-
 .../java/org/apache/beam/sdk/util/StreamUtils.java |   2 +-
 .../org/apache/beam/sdk/util/StreamUtilsTest.java  |   6 +-
 20 files changed, 725 insertions(+), 761 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (a5b2382 -> 6411964)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a5b2382  Merge pull request #5650 from robertwb/import-2018-06-14
 add 943b8f0  Removed some dead code, fixed some warnings
 add 436fceb  Adds tracing of watermarks in direct runner
 add 01368f9  Fixes typo in class name
 add af83aa3  Simplifies GBK expansion code a bit
 add 990d2ec  Fixes bug: Processes values arriving for a GBK key 
sequentially
 add e25243a  Introduces BundleProgressHandler.
 add b057fb8  [BEAM-4473,BEAM-4281] Makes ReferenceRunner use a Direct 
stream observer factory for synchronization
 add 6411964  Merge pull request #5585: Various smaller-scale changes 
extracted while working on SDF

No new revisions were added by this update.

Summary of changes:
 .../core/SplittableParDoViaKeyedWorkItems.java |   7 -
 .../beam/runners/direct/EvaluationContext.java |   2 +-
 .../beam/runners/direct/WatermarkManager.java  |  98 ++-
 ...ava => BundleFactoryOutputReceiverFactory.java} |   6 +-
 .../runners/direct/portable/EvaluationContext.java |   2 +-
 .../runners/direct/portable/ReferenceRunner.java   | 149 ++
 .../portable/RemoteStageEvaluatorFactory.java  |   6 +-
 .../beam/runners/direct/WatermarkManagerTest.java  |   6 +-
 ...=> BundleFactoryOutputReceiverFactoryTest.java} |   6 +-
 .../direct/portable/ReferenceRunnerTest.java   |  35 +++
 .../portable/RemoteStageEvaluatorFactoryTest.java  |   5 +-
 .../functions/FlinkExecutableStageFunction.java|   5 +-
 .../streaming/ExecutableStageDoFnOperator.java |   9 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   8 +-
 .../FlinkExecutableStageFunctionTest.java  |   8 +-
 .../fnexecution/control/BundleProgressHandler.java |  54 
 .../control/DockerJobBundleFactory.java|  10 +-
 .../runners/fnexecution/control/RemoteBundle.java  |   6 +-
 .../fnexecution/control/SdkHarnessClient.java  |  27 +-
 .../SingleEnvironmentInstanceJobBundleFactory.java |   5 +-
 .../fnexecution/control/StageBundleFactory.java|   3 +-
 .../runners/fnexecution/data/GrpcDataService.java  |  14 +-
 .../environment/InProcessEnvironmentFactory.java   |   4 +-
 .../runners/fnexecution/InProcessSdkHarness.java   |   5 +-
 .../fnexecution/control/RemoteExecutionTest.java   |  16 +-
 .../fnexecution/control/SdkHarnessClientTest.java  |  63 +++--
 ...gleEnvironmentInstanceJobBundleFactoryTest.java |   5 +-
 .../fnexecution/data/GrpcDataServiceTest.java  |  17 +-
 .../sdk/fn/data/BeamFnDataGrpcMultiplexer.java |   9 +-
 .../sdk/fn/stream/OutboundObserverFactory.java | 146 ++
 .../beam/sdk/fn/stream/StreamObserverFactory.java  | 115 
 .../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java |   9 +-
 ...yTest.java => OutboundObserverFactoryTest.java} |  18 +-
 .../apache/beam/fn/harness/FnApiDoFnRunner.java| 313 -
 .../java/org/apache/beam/fn/harness/FnHarness.java |  22 +-
 .../fn/harness/control/BeamFnControlClient.java|  12 +-
 .../beam/fn/harness/data/BeamFnDataGrpcClient.java |  23 +-
 .../harness/state/BeamFnStateGrpcClientCache.java  |  22 +-
 .../stream/HarnessStreamObserverFactories.java |  10 +-
 .../harness/control/BeamFnControlClientTest.java   |  11 +-
 .../fn/harness/data/BeamFnDataGrpcClientTest.java  |  31 +-
 .../state/BeamFnStateGrpcClientCacheTest.java  |  11 +-
 .../stream/HarnessStreamObserverFactoriesTest.java |   6 +-
 43 files changed, 691 insertions(+), 648 deletions(-)
 rename 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/{BundleFactoryOutputRecieverFactory.java
 => BundleFactoryOutputReceiverFactory.java} (94%)
 rename 
runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/{BundleFactoryOutputRecieverFactoryTest.java
 => BundleFactoryOutputReceiverFactoryTest.java} (98%)
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleProgressHandler.java
 create mode 100644 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
 delete mode 100644 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/StreamObserverFactory.java
 rename 
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/{StreamObserverFactoryTest.java
 => OutboundObserverFactoryTest.java} (78%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (bb3e3af -> 2d5f2ae)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bb3e3af  Merge pull request #5605: [BEAM-4536] Remove with_attributes 
keyword from ReadFromPubSub.
 add 43ff709  Beam File System based Artifact Staging Service
 add 9fa07ff  BeamFileSystemArtifactStagingService Tests
 add 4ed710a  Enhancing test case and error handling.
 add 911fd13  Enhancing testcases and incorporating review comments.
 add b19bdff  Review comments Fix
 add cdf433c  Rebasing
 new 2d5f2ae  Merge pull request #5591: [BEAM-4290] Beam File System based 
ArtifactStagingService

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../BeamFileSystemArtifactStagingService.java  | 302 +++
 .../BeamFileSystemArtifactStagingServiceTest.java  | 406 +
 2 files changed, 708 insertions(+)
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingService.java
 create mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactStagingServiceTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5591: [BEAM-4290] Beam File System based ArtifactStagingService

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2d5f2aedf78eb113ac091c52a07bdac9e26463d7
Merge: bb3e3af cdf433c
Author: Eugene Kirpichov 
AuthorDate: Tue Jun 12 11:37:22 2018 -0700

Merge pull request #5591: [BEAM-4290] Beam File System based 
ArtifactStagingService

[BEAM-4290] Beam File System based ArtifactStagingService

 .../BeamFileSystemArtifactStagingService.java  | 302 +++
 .../BeamFileSystemArtifactStagingServiceTest.java  | 406 +
 2 files changed, 708 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5588: [BEAM-4523] Implement batch flink executable stage context

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dd14be7341443a0708e57f1a55515c53d08ea94b
Merge: 69fb5c9 8e1b3bd
Author: Eugene Kirpichov 
AuthorDate: Fri Jun 8 14:09:11 2018 -0700

Merge pull request #5588: [BEAM-4523] Implement batch flink executable 
stage context

[BEAM-4523] Implement batch flink executable stage context

 .../beam/runners/flink/ArtifactSourcePool.java |   6 ++
 .../BatchFlinkExecutableStageContext.java  | 107 +
 ...ext.java => FlinkBatchStateRequestHandler.java} |  37 +++
 .../functions/FlinkExecutableStageContext.java |  22 ++---
 .../functions/FlinkExecutableStageFunction.java|   7 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   2 +-
 .../FlinkExecutableStageFunctionTest.java  |  20 ++--
 7 files changed, 146 insertions(+), 55 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (69fb5c9 -> dd14be7)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 69fb5c9  Merge pull request #5576: [BEAM-4414] Create more specific 
namespace for each IOIT in FileBasedIOIT
 add 96006ed  Add reference to ArtifactSourcePool implementation
 add e38f7aa  Clean up FlinkExecutableStageFunction and tests
 add 8e1b3bd  [BEAM-4523] Implement batch flink executable stage context
 new dd14be7  Merge pull request #5588: [BEAM-4523] Implement batch flink 
executable stage context

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/flink/ArtifactSourcePool.java |   6 ++
 .../BatchFlinkExecutableStageContext.java  | 107 +
 ...ext.java => FlinkBatchStateRequestHandler.java} |  37 +++
 .../functions/FlinkExecutableStageContext.java |  22 ++---
 .../functions/FlinkExecutableStageFunction.java|   7 +-
 .../streaming/ExecutableStageDoFnOperatorTest.java |   2 +-
 .../FlinkExecutableStageFunctionTest.java  |  20 ++--
 7 files changed, 146 insertions(+), 55 deletions(-)
 create mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/BatchFlinkExecutableStageContext.java
 copy 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/{FlinkExecutableStageContext.java
 => FlinkBatchStateRequestHandler.java} (51%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (72cbd99 -> 1f712bc)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 72cbd99  Moving to 2.6.0-SNAPSHOT on master branch
 add 8975cbd  Fix ProvisionInfo in DockerJobBundleFactory
 add 1f712bc  Merge pull request #5563: [BEAM-4491] Fix ProvisionInfo in 
DockerJobBundleFactory

No new revisions were added by this update.

Summary of changes:
 .../runners/fnexecution/control/DockerJobBundleFactory.java|  7 ---
 .../apache/beam/runners/fnexecution/provisioning/JobInfo.java  | 10 ++
 2 files changed, 14 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (be6185f -> abe3f3e)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from be6185f  Merge pull request #5489: [BEAM-4290] proto changes to 
support staging_session_token
 add 3b303ec  Fix build break due to artifact staging change
 new abe3f3e  Merge pull request #5558: Fix build break due to artifact 
staging change

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5558: Fix build break due to artifact staging change

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit abe3f3edd5e34245f0a5fef63ce48fef57204b41
Merge: be6185f 3b303ec
Author: Eugene Kirpichov 
AuthorDate: Mon Jun 4 19:41:10 2018 -0700

Merge pull request #5558: Fix build break due to artifact staging change

Fix build break due to artifact staging change

 sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5489: [BEAM-4290] proto changes to support staging_session_token

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit be6185fd5c2b471f371d7591108360974954ec12
Merge: 697a1d1 6f23949
Author: Eugene Kirpichov 
AuthorDate: Mon Jun 4 15:22:16 2018 -0700

Merge pull request #5489: [BEAM-4290] proto changes to support 
staging_session_token

[BEAM-4290] proto changes to support staging_session_token

 .../src/main/proto/beam_artifact_api.proto |   19 +-
 .../src/main/proto/beam_job_api.proto  |6 +-
 .../core/construction/ArtifactServiceStager.java   |6 +-
 .../InMemoryArtifactStagerService.java |2 +-
 .../LocalFileSystemArtifactStagerService.java  |2 +-
 .../portable/job/ReferenceRunnerJobService.java|3 +
 .../LocalFileSystemArtifactStagerServiceTest.java  |   18 +-
 .../jobsubmission/InMemoryJobService.java  |3 +
 .../runners/reference/testing/TestJobService.java  |1 +
 sdks/go/gogradle.lock  |2 +-
 sdks/go/pkg/beam/artifact/materialize_test.go  |6 +-
 sdks/go/pkg/beam/artifact/server_test.go   |2 +-
 sdks/go/pkg/beam/artifact/stage.go |7 +-
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 1572 ---
 .../model/fnexecution_v1/beam_provision_api.pb.go  |  255 +-
 .../model/jobmanagement_v1/beam_artifact_api.pb.go |  567 +++-
 .../beam/model/jobmanagement_v1/beam_job_api.pb.go |  604 +++--
 .../beam/model/pipeline_v1/beam_runner_api.pb.go   | 2750 +++-
 sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go |   80 +-
 .../model/pipeline_v1/standard_window_fns.pb.go|  174 +-
 .../runners/portability/local_job_service.py   |5 +-
 .../runners/portability/portable_stager.py |   20 +-
 .../runners/portability/portable_stager_test.py|   18 +-
 23 files changed, 4706 insertions(+), 1416 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (697a1d1 -> be6185f)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 697a1d1  Merge pull request #5508: [SQL] Add integration tests for 
BigQuery writes
 add 6118222  proto changes to support artifact_staging_id
 add 5c2c092  Doc changes
 add 845e8c5  Renaming artifact_staging_id to staging_session_token
 add d0955c2  Fixing python code for staging_session_token proto changes
 add 9072088  Fixing java code for staging_session_token proto changes
 add 6df43d7  Go proto update
 add 6f23949  Code review comments fixes
 new be6185f  Merge pull request #5489: [BEAM-4290] proto changes to 
support staging_session_token

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/proto/beam_artifact_api.proto |   19 +-
 .../src/main/proto/beam_job_api.proto  |6 +-
 .../core/construction/ArtifactServiceStager.java   |6 +-
 .../InMemoryArtifactStagerService.java |2 +-
 .../LocalFileSystemArtifactStagerService.java  |2 +-
 .../portable/job/ReferenceRunnerJobService.java|3 +
 .../LocalFileSystemArtifactStagerServiceTest.java  |   18 +-
 .../jobsubmission/InMemoryJobService.java  |3 +
 .../runners/reference/testing/TestJobService.java  |1 +
 sdks/go/gogradle.lock  |2 +-
 sdks/go/pkg/beam/artifact/materialize_test.go  |6 +-
 sdks/go/pkg/beam/artifact/server_test.go   |2 +-
 sdks/go/pkg/beam/artifact/stage.go |7 +-
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 1572 ---
 .../model/fnexecution_v1/beam_provision_api.pb.go  |  255 +-
 .../model/jobmanagement_v1/beam_artifact_api.pb.go |  567 +++-
 .../beam/model/jobmanagement_v1/beam_job_api.pb.go |  604 +++--
 .../beam/model/pipeline_v1/beam_runner_api.pb.go   | 2750 +++-
 sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go |   80 +-
 .../model/pipeline_v1/standard_window_fns.pb.go|  174 +-
 .../runners/portability/local_job_service.py   |5 +-
 .../runners/portability/portable_stager.py |   20 +-
 .../runners/portability/portable_stager_test.py|   18 +-
 23 files changed, 4706 insertions(+), 1416 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (5e96d5f -> fe73369)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5e96d5f  Merge pull request #5454: [BEAM-4354] Enforce ErrorProne 
analysis in tika IO
 add fd67c87  [BEAM-214] Add ParquetIO
 add c9badee  [BEAM-214] Add gradle build in ParquetIO
 add 825da2d  [BEAM-214] Refactore the reads to leverage FileIO
 add 7c48802  [BEAM-214] Use Parquet 1.10.0 version in ParquetIO.Sink
 add 030b714  [BEAM-214] Add ParquetIOIT
 add c144cfb  [BEAM-214] Update ReadFn to an non deprecated, newest Parquet 
API
 add 37720bf  [BEAM-214] Simplify the IO by removing redundant PTransforms
 add eb00d20  [BEAM-214] Implement writing slices of bytes in 
BeamOutputStream
 add a3a69ec  [BEAM-214] Apply reviewer's suggestions
 add fe73369  Merge pull request #5242: [BEAM-214] ParquetIO

No new revisions were added by this update.

Summary of changes:
 .../job_PerformanceTests_FileBasedIO_IT.groovy |  12 +
 sdks/java/io/file-based-io-tests/build.gradle  |   1 +
 sdks/java/io/file-based-io-tests/pom.xml   |   5 +
 .../beam/sdk/io/common/FileBasedIOITHelper.java|   1 +
 .../AvroIOIT.java => parquet/ParquetIOIT.java} |  97 +++---
 sdks/java/io/{kafka => parquet}/build.gradle   |  19 +-
 sdks/java/io/{amqp => parquet}/pom.xml |  98 +++---
 .../org/apache/beam/sdk/io/parquet/ParquetIO.java  | 386 +
 .../apache/beam/sdk/io/parquet}/package-info.java  |   4 +-
 .../apache/beam/sdk/io/parquet/ParquetIOTest.java  | 136 
 sdks/java/io/pom.xml   |   1 +
 settings.gradle|   2 +
 12 files changed, 646 insertions(+), 116 deletions(-)
 copy 
sdks/java/io/file-based-io-tests/src/test/java/org/apache/beam/sdk/io/{avro/AvroIOIT.java
 => parquet/ParquetIOIT.java} (57%)
 copy sdks/java/io/{kafka => parquet}/build.gradle (78%)
 copy sdks/java/io/{amqp => parquet}/pom.xml (67%)
 create mode 100644 
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java
 copy 
{runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators => 
sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet}/package-info.java
 (89%)
 create mode 100644 
sdks/java/io/parquet/src/test/java/org/apache/beam/sdk/io/parquet/ParquetIOTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5273: [BEAM-3883] Adding Client to push artifacts to artifact staging service

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dd8dcc65f39f60004eb5df0c43f083a7d78e19c6
Merge: 15600d6 a0ec87c
Author: Eugene Kirpichov 
AuthorDate: Tue May 22 16:41:33 2018 -0700

Merge pull request #5273: [BEAM-3883] Adding Client to push artifacts to 
artifact staging service

[BEAM-3883] Adding Client to push artifacts to artifact staging service

 .../runners/portability/portable_stager.py |  86 +++
 .../runners/portability/portable_stager_test.py| 162 +
 2 files changed, 248 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (15600d6 -> dd8dcc6)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 15600d6  Merge pull request #5334 from RobbeSneyders/internal
 add 67c97c9  Adding ArtifactStagingFileHandler to push files to staging 
server
 add 6a8650a  Adding filesystem based artifact service to ULR job service
 add acfa20d  Adding context manager to ArtifactStagingFileHandler
 add ea525b0  Converting ArtifactServiceFileHandler to PortableStager based 
on discussion in parent PR
 add 0d60ae4  Simplifying numeric constants
 add 7c33a70  Logging and naming changes
 add 4bc2006  Enhancing test cases
 add a0ec87c  Updating the dostring
 new dd8dcc6  Merge pull request #5273: [BEAM-3883] Adding Client to push 
artifacts to artifact staging service

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/portability/portable_stager.py |  86 +++
 .../runners/portability/portable_stager_test.py| 162 +
 2 files changed, 248 insertions(+)
 create mode 100644 
sdks/python/apache_beam/runners/portability/portable_stager.py
 create mode 100644 
sdks/python/apache_beam/runners/portability/portable_stager_test.py

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5392: [BEAM-4267] JobBundleFactory that uses Docker-backed environments

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a753d21e605444dc1f7d3fa5e350ad38a6610ed3
Merge: 81ff55f b1526f8
Author: Eugene Kirpichov 
AuthorDate: Tue May 22 12:54:23 2018 -0700

Merge pull request #5392: [BEAM-4267] JobBundleFactory that uses 
Docker-backed environments

[BEAM-4267] JobBundleFactory that uses Docker-backed environments

 runners/java-fn-execution/build.gradle |   1 +
 .../beam/runners/fnexecution/ServerFactory.java|  29 +-
 .../control/DockerJobBundleFactory.java| 363 +
 .../control/ProcessBundleDescriptors.java  |   2 +-
 .../environment/DockerEnvironmentFactory.java  |   8 +-
 .../runners/fnexecution/ServerFactoryTest.java |  17 +-
 .../control/DockerJobBundleFactoryTest.java| 195 +++
 7 files changed, 605 insertions(+), 10 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (81ff55f -> a753d21)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 81ff55f  Merge pull request #5451: [BEAM-4351] Enforce ErrorProne 
analysis in mqtt IO
 add d3b223d  Require explicit IdGenerator for all DockerEnvironmentFactory 
constructors
 add 2dcb0dc  [BEAM-4148] Add pluggable UrlFactory to ServerFactory
 add b1526f8  [BEAM-4267] Docker-based JobBundleFactory
 new a753d21  Merge pull request #5392: [BEAM-4267] JobBundleFactory that 
uses Docker-backed environments

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/java-fn-execution/build.gradle |   1 +
 .../beam/runners/fnexecution/ServerFactory.java|  29 +-
 .../control/DockerJobBundleFactory.java| 363 +
 .../control/ProcessBundleDescriptors.java  |   2 +-
 .../environment/DockerEnvironmentFactory.java  |   8 +-
 .../runners/fnexecution/ServerFactoryTest.java |  17 +-
 .../control/DockerJobBundleFactoryTest.java| 195 +++
 7 files changed, 605 insertions(+), 10 deletions(-)
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 create mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactoryTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #5440: [BEAM-4375] Replaces usage of JUnit runners that do not work well with Gradle

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 67f6cf2c7786e339758854efb1cf1230909e4f1d
Merge: 2105c25 e051544
Author: Eugene Kirpichov 
AuthorDate: Mon May 21 20:55:14 2018 -0700

Merge pull request #5440: [BEAM-4375] Replaces usage of JUnit runners that 
do not work well with Gradle

[BEAM-4375] Replaces usage of JUnit runners that do not work well with 
Gradle

 .../translation/utils/ApexStateInternalsTest.java  |   6 -
 .../core/construction/CoderTranslationTest.java|   2 -
 .../core/construction/CombineTranslationTest.java  |   2 -
 .../CreatePCollectionViewTranslationTest.java  | 160 ++---
 .../core/construction/ParDoTranslationTest.java|   6 -
 .../PipelineOptionsTranslationTest.java|   2 -
 .../construction/TestStreamTranslationTest.java| 118 +++
 .../construction/WriteFilesTranslationTest.java| 109 +++---
 .../runners/core/InMemoryStateInternalsTest.java   |   6 -
 .../runners/core/metrics/MetricsPusherTest.java|   3 +
 runners/direct-java/build.gradle   |  17 ++-
 .../streaming/FlinkKeyGroupStateInternalsTest.java |   6 -
 .../streaming/UnboundedSourceWrapperTest.java  |   4 +-
 .../runners/dataflow/util/CloudObjectsTest.java|   2 -
 .../org/apache/beam/sdk/io/TextIOReadTest.java |   2 -
 .../java/org/apache/beam/sdk/io/TextIOTest.java|  31 
 .../org/apache/beam/sdk/io/TextIOWriteTest.java|   3 +
 .../org/apache/beam/sdk/metrics/MetricsTest.java   |   2 -
 .../apache/beam/sdk/testing/TestPipelineTest.java  |   8 --
 .../sdk/transforms/ApproximateQuantilesTest.java   |   2 -
 .../beam/sdk/transforms/ApproximateUniqueTest.java |   8 --
 .../apache/beam/sdk/transforms/CombineTest.java|   2 -
 .../apache/beam/sdk/transforms/GroupByKeyTest.java |   2 -
 .../org/apache/beam/sdk/transforms/ParDoTest.java  |   3 -
 .../org/apache/beam/sdk/transforms/SampleTest.java |   6 -
 .../sdk/extensions/gcp/options/GcpOptionsTest.java |   2 -
 .../apache/beam/sdk/fn/stream/DataStreamsTest.java |   2 -
 .../harness/state/StateFetchingIteratorsTest.java  |   2 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIOTest.java   |  33 -
 29 files changed, 197 insertions(+), 354 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (2105c25 -> 67f6cf2)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2105c25  Merge pull request #5439: Fixes MetricsTest
 add e051544  [BEAM-4375] Replaces usage of JUnit runners that do not work 
well with Gradle
 new 67f6cf2  Merge pull request #5440: [BEAM-4375] Replaces usage of JUnit 
runners that do not work well with Gradle

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../translation/utils/ApexStateInternalsTest.java  |   6 -
 .../core/construction/CoderTranslationTest.java|   2 -
 .../core/construction/CombineTranslationTest.java  |   2 -
 .../CreatePCollectionViewTranslationTest.java  | 160 ++---
 .../core/construction/ParDoTranslationTest.java|   6 -
 .../PipelineOptionsTranslationTest.java|   2 -
 .../construction/TestStreamTranslationTest.java| 118 +++
 .../construction/WriteFilesTranslationTest.java| 109 +++---
 .../runners/core/InMemoryStateInternalsTest.java   |   6 -
 .../runners/core/metrics/MetricsPusherTest.java|   3 +
 runners/direct-java/build.gradle   |  17 ++-
 .../streaming/FlinkKeyGroupStateInternalsTest.java |   6 -
 .../streaming/UnboundedSourceWrapperTest.java  |   4 +-
 .../runners/dataflow/util/CloudObjectsTest.java|   2 -
 .../org/apache/beam/sdk/io/TextIOReadTest.java |   2 -
 .../java/org/apache/beam/sdk/io/TextIOTest.java|  31 
 .../org/apache/beam/sdk/io/TextIOWriteTest.java|   3 +
 .../org/apache/beam/sdk/metrics/MetricsTest.java   |   2 -
 .../apache/beam/sdk/testing/TestPipelineTest.java  |   8 --
 .../sdk/transforms/ApproximateQuantilesTest.java   |   2 -
 .../beam/sdk/transforms/ApproximateUniqueTest.java |   8 --
 .../apache/beam/sdk/transforms/CombineTest.java|   2 -
 .../apache/beam/sdk/transforms/GroupByKeyTest.java |   2 -
 .../org/apache/beam/sdk/transforms/ParDoTest.java  |   3 -
 .../org/apache/beam/sdk/transforms/SampleTest.java |   6 -
 .../sdk/extensions/gcp/options/GcpOptionsTest.java |   2 -
 .../apache/beam/sdk/fn/stream/DataStreamsTest.java |   2 -
 .../harness/state/StateFetchingIteratorsTest.java  |   2 -
 .../beam/sdk/io/gcp/bigquery/BigQueryIOTest.java   |  33 -
 29 files changed, 197 insertions(+), 354 deletions(-)
 delete mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java
 delete mode 100644 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.