[beam] branch master updated: Remove misleading unsupported features mention for Python streaming (#12140)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 23d0c47 Remove misleading unsupported features mention for Python
streaming (#12140)
23d0c47 is described below
commit 23d0c474576e5e58a62a17443587bbd1e9d7e5c3
Author: Thomas Weise
AuthorDate: Tue Jun 30 17:28:40 2020 -0700
Remove misleading unsupported features mention for Python streaming (#12140)
---
.../en/documentation/sdks/python-streaming.md | 21 +
1 file changed, 5 insertions(+), 16 deletions(-)
diff --git a/website/www/site/content/en/documentation/sdks/python-streaming.md
b/website/www/site/content/en/documentation/sdks/python-streaming.md
index 9c4a688..5fb0aa1 100644
--- a/website/www/site/content/en/documentation/sdks/python-streaming.md
+++ b/website/www/site/content/en/documentation/sdks/python-streaming.md
@@ -126,16 +126,12 @@ python -m apache_beam.examples.streaming_wordcount \
--streaming
{{< /highlight >}}
-{{< highlight class="runner-flink-local" >}}
-See https://beam.apache.org/roadmap/portability/#python-on-flink for more
information.
-{{< /highlight >}}
-
-{{< highlight class="runner-flink-cluster" >}}
+{{< highlight class="runner-flink" >}}
See https://beam.apache.org/documentation/runners/flink/ for more information.
{{< /highlight >}}
{{< highlight class="runner-spark" >}}
-See https://beam.apache.org/roadmap/portability/#python-on-spark for more
information.
+See https://beam.apache.org/documentation/runners/spark/ for more information.
{{< /highlight >}}
{{< highlight class="runner-dataflow" >}}
@@ -158,19 +154,12 @@ about executing streaming pipelines:
- [DirectRunner streaming
execution](/documentation/runners/direct/#streaming-execution)
- [DataflowRunner streaming
execution](/documentation/runners/dataflow/#streaming-execution)
-
+- [Portable Flink runner](/documentation/runners/flink/)
## Unsupported features
-Python streaming execution does not currently support the following features.
+Python streaming execution does not currently support the following features:
-### General Beam features
-
-These unsupported Beam features apply to all runners.
-
-- `State` and `Timers` APIs
- Custom source API
-- Splittable `DoFn` API
-- Handling of late data
- User-defined custom merging `WindowFn` (with fnapi)
-
+- For portable runners, see [portability support
table](https://s.apache.org/apache-beam-portability-support-table).
[beam] 01/01: Merge pull request #11722: Bump Flink 1.10 version
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch release-2.21.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 89b33f21f4c7d8aadc49a17aa3a6b42230912a00 Merge: e859735 27634a2 Author: Thomas Weise AuthorDate: Wed May 20 22:10:41 2020 -0700 Merge pull request #11722: Bump Flink 1.10 version .test-infra/dataproc/flink_cluster.sh | 2 +- .test-infra/jenkins/Flink.groovy | 2 +- runners/flink/1.10/build.gradle | 2 +- .../src/test/java/org/apache/beam/runners/flink/FlinkRunnerTest.java | 4 +++- 4 files changed, 6 insertions(+), 4 deletions(-)
[beam] branch release-2.21.0 updated (e859735 -> 89b33f2)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch release-2.21.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from e859735 Merge pull request #11703 from ibzib/BEAM-9001 add 27634a2 Bump Flink 1.10 version new 89b33f2 Merge pull request #11722: Bump Flink 1.10 version The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/dataproc/flink_cluster.sh | 2 +- .test-infra/jenkins/Flink.groovy | 2 +- runners/flink/1.10/build.gradle | 2 +- .../src/test/java/org/apache/beam/runners/flink/FlinkRunnerTest.java | 4 +++- 4 files changed, 6 insertions(+), 4 deletions(-)
[beam] branch master updated (a698ae3 -> 34d6b83)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a698ae3 Merge pull request #11696 from ibzib/BEAM-9975 add ac2a677 Bump Flink 1.10 version add 34d6b83 Merge pull request #11683: Bump Flink 1.10 version No new revisions were added by this update. Summary of changes: .test-infra/dataproc/flink_cluster.sh | 2 +- .test-infra/jenkins/Flink.groovy | 2 +- runners/flink/1.10/build.gradle | 2 +- .../src/test/java/org/apache/beam/runners/flink/FlinkRunnerTest.java | 4 +++- 4 files changed, 6 insertions(+), 4 deletions(-)
[beam] branch master updated (022fe5e -> 6d29fb8)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 022fe5e Merge pull request #11519: [BEAM-9761] Disable auto builds for tests that are currently failing. add 6d29fb8 [BEAM-9811] Copy pom for all publishMavenJavaPublication tasks No new revisions were added by this update. Summary of changes: sdks/java/bom/build.gradle | 6 +- 1 file changed, 1 insertion(+), 5 deletions(-)
[beam] branch master updated (28fd597 -> d8519dd)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 28fd597 Merge pull request #11454 from boyuanzz/restriction_tracker add d8519dd [BEAM-9811] Nightly snapshot publish error No new revisions were added by this update. Summary of changes: sdks/java/bom/build.gradle | 2 ++ 1 file changed, 2 insertions(+)
[beam] 01/01: Merge pull request #11399: Interpolate publish credentials as expected for Maven settings.xml
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1658d9054bcbdf1437295ac434825be53acce4fd Merge: 1db7186 7438265 Author: Thomas Weise AuthorDate: Wed Apr 15 14:19:04 2020 -0700 Merge pull request #11399: Interpolate publish credentials as expected for Maven settings.xml buildSrc/build.gradle | 1 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 16 +++- .../groovy/org/apache/beam/gradle/Repositories.groovy | 4 .../org/apache/beam/gradle/VendorJavaPlugin.groovy | 17 +++-- 4 files changed, 11 insertions(+), 27 deletions(-)
[beam] branch master updated (1db7186 -> 1658d90)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1db7186 Merge pull request #11404 from Fixing type names for BQ Avro Tools add a562f1e Fix py37-lint add 7438265 Maven compatible publish repository authentication via settings.xml new 1658d90 Merge pull request #11399: Interpolate publish credentials as expected for Maven settings.xml The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/build.gradle | 1 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 16 +++- .../groovy/org/apache/beam/gradle/Repositories.groovy | 4 .../org/apache/beam/gradle/VendorJavaPlugin.groovy | 17 +++-- 4 files changed, 11 insertions(+), 27 deletions(-)
[beam] branch master updated (0e88219 -> d62521f)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0e88219 [BEAM-7815] update MemoryReporter comments about using guppy3 (#11073) add cf219ba [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment add d62521f Merge pull request #11084: [BEAM-9474] Improve robustness of BundleFactory and ProcessEnvironment No new revisions were added by this update. Summary of changes: .../control/DefaultJobBundleFactory.java | 101 + .../environment/ProcessEnvironment.java| 28 -- .../fnexecution/environment/ProcessManager.java| 50 +- 3 files changed, 128 insertions(+), 51 deletions(-)
[beam] branch master updated (4b77225 -> 4f84653)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 4b77225 Merge pull request #9677 from xubii/time-output-timestamp
add 59f10d0 Rename FlinkClassloading to Workarounds
add 021e7b3 [BEAM-9060] Restore stdout/stderr in case Flink's
OptimizedPlanEnvironment is used
add 4f84653 Merge pull request #10517: [BEAM-9060] Restore stdout/stderr
in case Flink's OptimizedPlanEnvironment is used
No new revisions were added by this update.
Summary of changes:
.../apache/beam/runners/flink/FlinkJobInvoker.java | 7 +-
.../org/apache/beam/runners/flink/FlinkRunner.java | 4 +-
.../translation/functions/FlinkDoFnFunction.java | 4 +-
.../functions/FlinkStatefulDoFnFunction.java | 4 +-
.../{FlinkClassloading.java => Workarounds.java} | 22 +-
.../wrappers/streaming/DoFnOperator.java | 4 +-
.../streaming/io/UnboundedSourceWrapper.java | 4 +-
.../beam/runners/flink/FlinkJobInvokerTest.java| 73 ++
.../apache/beam/runners/flink/FlinkRunnerTest.java | 89 ++
9 files changed, 197 insertions(+), 14 deletions(-)
rename
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/utils/{FlinkClassloading.java
=> Workarounds.java} (52%)
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkJobInvokerTest.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkRunnerTest.java
[beam] branch master updated (fb0353e -> a861278)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fb0353e Merge pull request #10216 from y1chi/allowed_lateness add ea2a3f6 [BEAM-8962] Report Flink metric accumulator only when pipeline ends new a861278 Merge pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/flink/FlinkPipelineOptions.java | 9 -- .../flink/metrics/FlinkMetricContainer.java| 49 +- .../runners/flink/metrics/MetricsAccumulator.java | 6 +- .../translation/functions/FlinkDoFnFunction.java | 10 +- .../functions/FlinkExecutableStageFunction.java| 9 +- .../functions/FlinkStatefulDoFnFunction.java | 11 +-- .../translation/wrappers/SourceInputFormat.java| 8 +- .../wrappers/streaming/DoFnOperator.java | 4 +- .../streaming/io/UnboundedSourceWrapper.java | 9 +- .../runners/flink/FlinkPipelineOptionsTest.java| 1 - .../flink/metrics/FlinkMetricContainerTest.java| 14 +-- .../functions/FlinkDoFnFunctionTest.java | 106 + .../FlinkExecutableStageFunctionTest.java | 16 .../functions/FlinkStatefulDoFnFunctionTest.java | 106 + .../wrappers/SourceInputFormatTest.java| 64 + .../wrappers/streaming/DoFnOperatorTest.java | 77 +-- .../streaming/io/UnboundedSourceWrapperTest.java | 30 ++ .../src/_includes/flink_java_pipeline_options.html | 5 - .../_includes/flink_python_pipeline_options.html | 5 - 19 files changed, 426 insertions(+), 113 deletions(-) create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkDoFnFunctionTest.java create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunctionTest.java create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/SourceInputFormatTest.java
[beam] 01/01: Merge pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a8612788cb77a8b9d79bfbe44321c941d38322e7 Merge: fb0353e ea2a3f6 Author: Thomas Weise AuthorDate: Fri Dec 27 19:08:19 2019 -0800 Merge pull request #10415: [BEAM-8962] Report Flink metric accumulator only when pipeline ends .../beam/runners/flink/FlinkPipelineOptions.java | 9 -- .../flink/metrics/FlinkMetricContainer.java| 49 +- .../runners/flink/metrics/MetricsAccumulator.java | 6 +- .../translation/functions/FlinkDoFnFunction.java | 10 +- .../functions/FlinkExecutableStageFunction.java| 9 +- .../functions/FlinkStatefulDoFnFunction.java | 11 +-- .../translation/wrappers/SourceInputFormat.java| 8 +- .../wrappers/streaming/DoFnOperator.java | 4 +- .../streaming/io/UnboundedSourceWrapper.java | 9 +- .../runners/flink/FlinkPipelineOptionsTest.java| 1 - .../flink/metrics/FlinkMetricContainerTest.java| 14 +-- .../functions/FlinkDoFnFunctionTest.java | 106 + .../FlinkExecutableStageFunctionTest.java | 16 .../functions/FlinkStatefulDoFnFunctionTest.java | 106 + .../wrappers/SourceInputFormatTest.java| 64 + .../wrappers/streaming/DoFnOperatorTest.java | 77 +-- .../streaming/io/UnboundedSourceWrapperTest.java | 30 ++ .../src/_includes/flink_java_pipeline_options.html | 5 - .../_includes/flink_python_pipeline_options.html | 5 - 19 files changed, 426 insertions(+), 113 deletions(-)
[beam] branch master updated (b816ae3 -> e486202)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b816ae3 [BEAM-8575] Added two unit tests in CombineTest class to test AccumulatingCombine (#10173) add c8595ec [BEAM-8816] Option to load balance bundle processing w/ multiple SDK workers add e486202 Merge pull request #10313: [BEAM-8816] Option to load balance bundle processing w/ multiple SDK workers No new revisions were added by this update. Summary of changes: .../control/DefaultJobBundleFactory.java | 198 +++-- .../control/DefaultJobBundleFactoryTest.java | 79 +++- .../beam/sdk/options/ExperimentalOptions.java | 21 +++ .../beam/sdk/options/PortablePipelineOptions.java | 7 + .../beam/sdk/options/ExperimentalOptionsTest.java | 11 ++ 5 files changed, 258 insertions(+), 58 deletions(-)
[beam] branch master updated (c7f7da6 -> bc18874)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c7f7da6 [GoSDK] Improve StateChannel resilience. (#10363) add bc18874 [BEAM-8273] Expand portability environment documentation (#10116) No new revisions were added by this update. Summary of changes: .../src/_includes/section-menu/documentation.html | 3 +- website/src/documentation/runtime/environments.md | 4 +- .../documentation/runtime/sdk-harness-config.md| 57 ++ website/src/roadmap/portability.md | 22 ++--- 4 files changed, 64 insertions(+), 22 deletions(-) create mode 100644 website/src/documentation/runtime/sdk-harness-config.md
[beam] branch master updated (dfc814a -> 6c6647c)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dfc814a Merge pull request #10174 from davidcavazos/sample-code add 4e748a6 [BEAM-8815] Define the no artifacts retrieval token in proto add 6c6647c Merge pull request #10285: [BEAM-8815] Define the no artifacts retrieval token in proto No new revisions were added by this update. Summary of changes: model/job-management/src/main/proto/beam_artifact_api.proto | 6 ++ .../fnexecution/artifact/AbstractArtifactStagingService.java | 8 +++- 2 files changed, 13 insertions(+), 1 deletion(-)
[beam] branch master updated (066b7b1 -> 6a921b8)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 066b7b1 Merge pull request #9462 from KangZhiDong/master add 6a1b88a [BEAM-8815] Skip manifest when no artifacts are staged add 6a921b8 Merge pull request #10205: [BEAM-8815] Skip manifest when no artifacts are staged No new revisions were added by this update. Summary of changes: .../artifact/AbstractArtifactRetrievalService.java | 20 ++- .../artifact/AbstractArtifactStagingService.java | 40 +- .../BeamFileSystemArtifactServicesTest.java| 18 ++ 3 files changed, 53 insertions(+), 25 deletions(-)
[beam] branch master updated (a8f3195 -> a42fd74)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a8f3195 [BEAM-8805] Remove obsolete worker_threads experiment in tests add a42fd74 Documenting how to use Beam twitter handle (#10082) No new revisions were added by this update. Summary of changes: website/src/_includes/section-menu/community.html | 1 + website/src/community/twitter-handle.md | 41 +++ 2 files changed, 42 insertions(+) create mode 100644 website/src/community/twitter-handle.md
[beam] 01/01: Merge pull request #10139: [BEAM-8471] Flink native job submission for portable pipelines
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch release-2.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 60da80c4daa7221c025065abb182b9fa6ac2f3e0 Merge: bd744a3 16e85b0 Author: Thomas Weise AuthorDate: Mon Nov 18 09:29:18 2019 -0800 Merge pull request #10139: [BEAM-8471] Flink native job submission for portable pipelines .../apache/beam/runners/flink/FlinkJobInvoker.java | 4 +- .../beam/runners/flink/FlinkJobServerDriver.java | 39 ++-- .../flink/FlinkPortableClientEntryPoint.java | 257 + .../runners/flink/FlinkJobServerDriverTest.java| 9 +- .../beam/runners/flink/FlinkSavepointTest.java | 15 +- .../beam/runners/flink/PortableExecutionTest.java | 17 +- .../runners/flink/PortableStateExecutionTest.java | 17 +- .../runners/flink/PortableTimersExecutionTest.java | 17 +- .../beam/runners/flink/ReadSourcePortableTest.java | 17 +- .../fnexecution/environment/ProcessManager.java| 39 +++- .../fnexecution/jobsubmission/JobServerDriver.java | 16 +- .../environment/ProcessManagerTest.java| 58 + .../beam/runners/spark/SparkJobServerDriver.java | 12 +- 13 files changed, 437 insertions(+), 80 deletions(-)
[beam] branch release-2.17.0 updated (bd744a3 -> 60da80c)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch release-2.17.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from bd744a3 Merge pull request #9887 from pabloem/release-2.17.0 add 1796139 [BEAM-8471] Flink native job submission for portable pipelines add 16e85b0 [BEAM-8471] Put back FlinkJobServerDriver.fromParams to fix post commit new 60da80c Merge pull request #10139: [BEAM-8471] Flink native job submission for portable pipelines The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/runners/flink/FlinkJobInvoker.java | 4 +- .../beam/runners/flink/FlinkJobServerDriver.java | 39 ++-- .../flink/FlinkPortableClientEntryPoint.java | 257 + .../runners/flink/FlinkJobServerDriverTest.java| 9 +- .../beam/runners/flink/FlinkSavepointTest.java | 15 +- .../beam/runners/flink/PortableExecutionTest.java | 17 +- .../runners/flink/PortableStateExecutionTest.java | 17 +- .../runners/flink/PortableTimersExecutionTest.java | 17 +- .../beam/runners/flink/ReadSourcePortableTest.java | 17 +- .../fnexecution/environment/ProcessManager.java| 39 +++- .../fnexecution/jobsubmission/JobServerDriver.java | 16 +- .../environment/ProcessManagerTest.java| 58 + .../beam/runners/spark/SparkJobServerDriver.java | 12 +- 13 files changed, 437 insertions(+), 80 deletions(-) create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableClientEntryPoint.java
[beam] branch master updated (7beb9ee -> 1386b94)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7beb9ee Merge pull request #10117 [BEAM-8335] Add service and tagged output capabilities to TestStream add 45dc280 [BEAM-8670] Manage environment parallelism in DefaultJobBundleFactory add 1386b94 Merge pull request #10124: [BEAM-8670] Manage environment parallelism in DefaultJobBundleFactory No new revisions were added by this update. Summary of changes: .../FlinkExecutableStageContextFactory.java| 21 ++-- .../control/DefaultExecutableStageContext.java | 56 +-- .../control/DefaultJobBundleFactory.java | 107 ++--- .../control/DefaultExecutableStageContextTest.java | 72 -- .../control/DefaultJobBundleFactoryTest.java | 72 ++ .../SparkExecutableStageContextFactory.java| 24 ++--- 6 files changed, 158 insertions(+), 194 deletions(-) delete mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java
[beam] branch master updated (7937317 -> 9c76dbf)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7937317 Merge pull request #9885 from [BEAM-8457] Label Dataflow jobs from Notebook add 9c76dbf Add cross-language efforts to roadmap index page No new revisions were added by this update. Summary of changes: website/src/roadmap/index.md | 16 +--- 1 file changed, 9 insertions(+), 7 deletions(-)
[beam] branch master updated (ccc98f0 -> 66a67e6)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ccc98f0 Merge pull request #10022: Isolate sql precommit add 66a67e6 Add portable schemas to the roadmap No new revisions were added by this update. Summary of changes: website/src/roadmap/index.md | 9 + 1 file changed, 9 insertions(+)
[beam] branch master updated (474dcd9 -> 18b63e4)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 474dcd9 [BEAM-8474] A microbenchmark for Python FnApiRunner: (#9891) add d85f64b Revert "Remove --flink-master-url alias." add 18b63e4 Merge pull request #9947: Revert "Remove --flink-master-url alias." No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java| 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (d664592 -> 7b3a3fa)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d664592 [BEAM-8398] Upgrade Dataflow Java Client API add 7b3a3fa [BEAM-8507] Augment 'flink_master' pipeline option for Python FlinkRunner (#9918) No new revisions were added by this update. Summary of changes: .../runners/flink/FlinkExecutionEnvironments.java | 24 ++ .../flink/FlinkExecutionEnvironmentsTest.java | 38 +- .../runners/portability/flink_runner.py| 37 +++-- .../runners/portability/portable_runner.py | 5 +++ 4 files changed, 87 insertions(+), 17 deletions(-)
[beam] branch master updated (1ceabf0 -> 6b6dd20)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1ceabf0 Merge pull request #9893 Move use_sdf_bounded_source tests to iobase. add 6b6dd20 [BEAM-8471] Put back FlinkJobServerDriver.fromParams to fix post commit No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/flink/FlinkJobServerDriver.java | 7 ++- .../org/apache/beam/runners/flink/FlinkJobServerDriverTest.java| 5 ++--- 2 files changed, 8 insertions(+), 4 deletions(-)
[beam] branch master updated (f6adc23 -> 0645404)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f6adc23 Merge pull request #9908 Increasing timeout due to still constant timeouts. add f89c1b9 [BEAM-8471] Flink native job submission for portable pipelines add 0645404 Merge pull request #9872: [BEAM-8471] Flink native job submission for portable pipelines No new revisions were added by this update. Summary of changes: .../apache/beam/runners/flink/FlinkJobInvoker.java | 4 +- .../beam/runners/flink/FlinkJobServerDriver.java | 38 +-- .../flink/FlinkPortableClientEntryPoint.java | 257 + .../runners/flink/FlinkJobServerDriverTest.java| 14 +- .../beam/runners/flink/FlinkSavepointTest.java | 15 +- .../beam/runners/flink/PortableExecutionTest.java | 17 +- .../runners/flink/PortableStateExecutionTest.java | 17 +- .../runners/flink/PortableTimersExecutionTest.java | 17 +- .../beam/runners/flink/ReadSourcePortableTest.java | 17 +- .../fnexecution/environment/ProcessManager.java| 39 +++- .../fnexecution/jobsubmission/JobServerDriver.java | 16 +- .../environment/ProcessManagerTest.java| 58 + .../beam/runners/spark/SparkJobServerDriver.java | 12 +- 13 files changed, 437 insertions(+), 84 deletions(-) create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableClientEntryPoint.java
[beam] branch master updated (d9add56 -> 274f73b)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d9add56 Merge pull request #9819 from lgajowy/BEAM-8424 add d6bcb03 [BEAM-8417] Expose ExternalWorkerHandler hostname add 274f73b Merge pull request #9814: [BEAM-8417] Expose ExternalWorkerHandler hostname No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/portability/fn_api_runner.py | 12 1 file changed, 8 insertions(+), 4 deletions(-)
[beam] branch master updated (0a00a8f -> 6ab60dc)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0a00a8f Merge pull request #9786 from davidcavazos/remove-old-element-wise-snippets add 71b9292 [BEAM-8387] Remove sdk-worker-parallelism option from JobServerDriver add 6ab60dc Merge pull request #9785: [BEAM-8387] Remove sdk-worker-parallelism option from JobServerDriver No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/flink/FlinkJobInvoker.java | 3 --- .../apache/beam/runners/flink/FlinkJobServerDriverTest.java | 3 --- .../runners/fnexecution/jobsubmission/JobServerDriver.java | 13 - 3 files changed, 19 deletions(-)
[beam] branch master updated (2e548c5 -> 2ff462d)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2e548c5 Merge pull request #9687 from rakeshcusat/BEAM-7739-modify-idl-definition add 4f4c7c6 [BEAM-8183] restructure Flink portable jars to support multiple pipelines add 2ff462d Merge pull request #9752: [BEAM-8183] restructure Flink portable jars to support multiple pipelines No new revisions were added by this update. Summary of changes: .../beam/runners/flink/FlinkPipelineRunner.java| 21 - .../jobsubmission/PortablePipelineJarCreator.java | 35 + .../jobsubmission/PortablePipelineJarUtils.java| 89 +- .../PortablePipelineJarCreatorTest.java| 10 +-- 4 files changed, 108 insertions(+), 47 deletions(-)
[beam] branch master updated (7587aa4 -> 3f705bb)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7587aa4 Merge pull request: [BEAM-8309] Document SDK 2.16.0 Python dependencies (#9653) add 3f705bb [BEAM-6829] Use transform/pcollection name for metric namespace if none provided (#8585) No new revisions were added by this update. Summary of changes: .../core/metrics/MonitoringInfoMetricName.java | 7 + .../core/metrics/MonitoringInfoMetricNameTest.java | 20 +- .../FlinkBatchPortablePipelineTranslator.java | 1 + .../flink/metrics/DoFnRunnerWithMetricsUpdate.java | 5 ++-- .../translation/functions/FlinkDoFnFunction.java | 5 +++- .../functions/FlinkExecutableStageFunction.java| 11 .../functions/FlinkStatefulDoFnFunction.java | 5 +++- .../wrappers/streaming/DoFnOperator.java | 7 - .../streaming/ExecutableStageDoFnOperator.java | 11 .../flink/metrics/FlinkMetricContainerTest.java| 32 +++--- .../FlinkExecutableStageFunctionTest.java | 1 + 11 files changed, 78 insertions(+), 27 deletions(-)
[beam] branch master updated (8e75935 -> 6a3fdfd)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8e75935 Merge pull request #9669 from davidcavazos/include-buttons add 0216d53 [BEAM-5707] Modify Flink streaming impulse function to include a counter add 6a3fdfd Merge pull request #9728: [BEAM-5707] Modify Flink streaming impulse function to include a counter No new revisions were added by this update. Summary of changes: .../wrappers/streaming/io/StreamingImpulseSource.java | 19 +++ 1 file changed, 11 insertions(+), 8 deletions(-)
[beam] branch master updated (e78943c -> 469618c)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e78943c Merge pull request #9672: [BEAM-8317] Add (skipped) test for aggregating after a filter add 42cae0a [BEAM-8157] Increase parallelism for Python PVR tests add 469618c Merge pull request #9649: [BEAM-8157] Increase parallelism for Python PVR tests No new revisions were added by this update. Summary of changes: .../runners/portability/flink_runner_test.py | 26 +++--- 1 file changed, 13 insertions(+), 13 deletions(-)
[beam] branch master updated (032201c -> a141c4e)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 032201c [BEAM-8114] Fix failing Chicago Taxi Example job on Jenkins (#9456) add 9eda188 [BEAM-7980] Exactly once artifact retrieval for Python SDK worker pool add a141c4e Merge pull request #9398: [BEAM-7980] Exactly once artifact retrieval for Python SDK worker pool No new revisions were added by this update. Summary of changes: sdks/go/gogradle.lock | 9 - sdks/python/container/boot.go | 76 ++- 2 files changed, 75 insertions(+), 10 deletions(-)
[beam] branch master updated (1efa67a -> 9d45346)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1efa67a Merge pull request #9395 from rahul8383/beamsql-join-rules add 9d45346 [BEAM-8038] Fix worker pool exit hook No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/worker/worker_pool_main.py | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam] branch master updated (098c11f -> 3561100)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 098c11f Merge pull request #9392 from kamilwu/seed-job-from-local-git add 9266887 [BEAM-7980] Containerized Python worker pool for external environment add 3561100 Merge pull request #9371: [BEAM-7980] External environment with containerized worker pool No new revisions were added by this update. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 18 ++- .../environment/DockerEnvironmentFactory.java | 4 +- .../environment/ExternalEnvironmentFactory.java| 40 - .../runners/reference/ExternalWorkerService.java | 11 +- .../runners/portability/fn_api_runner.py | 4 +- .../runners/portability/portable_runner.py | 60 +-- .../runners/portability/portable_runner_test.py| 3 +- .../apache_beam/runners/worker/worker_pool_main.py | 177 + sdks/python/container/boot.go | 13 ++ 9 files changed, 251 insertions(+), 79 deletions(-) create mode 100644 sdks/python/apache_beam/runners/worker/worker_pool_main.py
[beam] branch master updated (0fd2cd3 -> 94260de)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0fd2cd3 Merge pull request #8909 [BEAM-7602] Add link to gRPC.io on website portability page add 6915114 [BEAM7597] Fix Typo on website new 94260de Merge pull request #8911: [BEAM-7597] Fix Typo on website The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/documentation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #8911: [BEAM-7597] Fix Typo on website
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 94260de0b20b361b589dd7507e6b41eccd7efda7 Merge: 0fd2cd3 6915114 Author: Thomas Weise AuthorDate: Thu Jun 20 15:03:15 2019 +0200 Merge pull request #8911: [BEAM-7597] Fix Typo on website website/src/documentation/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 211c7886e3f7cc7b8a98a60535f3908328212da1 Merge: 6d093f5 f4cf92d Author: Thomas Weise AuthorDate: Fri Jun 14 10:59:37 2019 +0200 Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization .../streaming/ExecutableStageDoFnOperator.java | 21 +++-- .../state/InMemoryBagUserStateFactory.java | 17 - .../fnexecution/state/StateRequestHandlers.java | 10 +- .../fnexecution/control/RemoteExecutionTest.java| 19 ++- 4 files changed, 34 insertions(+), 33 deletions(-)
[beam] branch master updated (6d093f5 -> 211c788)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6d093f5 Merge pull request #8825: [BEAM-7533] Fix CoderRegistry for Float add f4cf92d [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization new 211c788 Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../streaming/ExecutableStageDoFnOperator.java | 21 +++-- .../state/InMemoryBagUserStateFactory.java | 17 - .../fnexecution/state/StateRequestHandlers.java | 10 +- .../fnexecution/control/RemoteExecutionTest.java| 19 ++- 4 files changed, 34 insertions(+), 33 deletions(-)
[beam] branch master updated (5f83637 -> 23115ce)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5f83637 Merge pull request #8701 from akedin/dc-float64 add 23115ce [cleanup] Remove unncecessary Flink timestamp from UnboundedSourceWrapper No new revisions were added by this update. Summary of changes: .../flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (4e20419 -> 8e0e83f)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4e20419 Merge pull request #8655 from youngoli/go-errors add 8b379b4 [BEAM-7348] Support environment expiration add 8e0e83f Merge pull request #8599: [BEAM-7348] Support environment expiration No new revisions were added by this update. Summary of changes: .../control/DefaultJobBundleFactory.java | 218 + .../fnexecution/control/SdkHarnessClient.java | 4 +- .../control/DefaultJobBundleFactoryTest.java | 117 +++ .../beam/sdk/options/PortablePipelineOptions.java | 6 + 4 files changed, 216 insertions(+), 129 deletions(-)
[beam] branch master updated: Beam community update blogpost (#8559)
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new b0b2ac2 Beam community update blogpost (#8559) b0b2ac2 is described below commit b0b2ac26d1659b48eb12ee40723ea5d29dd4e9e6 Author: matthiasa4 AuthorDate: Tue May 14 04:04:55 2019 +0100 Beam community update blogpost (#8559) --- .../_posts/2019-05-11-beam-summit-europe-2019.md | 88 ++ 1 file changed, 88 insertions(+) diff --git a/website/src/_posts/2019-05-11-beam-summit-europe-2019.md b/website/src/_posts/2019-05-11-beam-summit-europe-2019.md new file mode 100644 index 000..75913e2 --- /dev/null +++ b/website/src/_posts/2019-05-11-beam-summit-europe-2019.md @@ -0,0 +1,88 @@ +--- +layout: post +title: "Beam community update!" +date: 2019-05-11 00:00:01 -0800 +excerpt_separator: +categories: blog +authors: + - mbaetens +--- + + +# The Apache Beam community in 2019 + + +2019 has already been a busy time for the Apache Beam community. The ASF blog featured [our way of community building](https://blogs.apache.org/comdev/date/20190222) and we've had [more Beam meetups](https://www.meetup.com/San-Francisco-Apache-Beam/events/257482350) around the world. Apache Beam also received the [Technology of the Year Award](https://www.infoworld.com/article/3336072/infoworlds-2019-technology-of-the-year-award-winners.html) from InfoWorld. + +As these events happened, we were building up to the [20th anniversary of the Apache Software Foundation](https://opensource.googleblog.com/2019/03/celebrating-20-years-of-apache.html). The contributions of the Beam community were a part of Maximilian Michels blog post on the success of the ASF's open source development model: + +Success at Apache: What You Need to Know by Maximilian Michels https://t.co/XjtVYgPAHX";>https://t.co/XjtVYgPAHX https://twitter.com/hashtag/Apache?src=hash&ref_src=twsrc%5Etfw";>#Apache https://twitter.com/hashtag/Open?src=hash&ref_src=twsrc%5Etfw";>#Open https://twitter.com/hashtag/Innovation?src=hash&ref_src=twsrc%5Etfw";>#Innovation https://twit [...] + +https://platform.twitter.com/widgets.js"</a>; charset="utf-8"> + +In that spirit, let's have an overview of the things that have happened, what the next few months look like, and how we can foster even more community growth. + + +## Meetups +We've had a flurry of activity, with several meetups in the planning process and more popping up globally over time. As diversity of contributors is a core ASF value, this geographic spread is exciting for the community. Here's a picture from the latest Apache Beam meetup organized at Lyft in San Francisco: + +https://secure.meetupstatic.com/photos/event/8/0/1/2/600_481292786.jpeg"; }}" alt="Beam Meetup Bay Area" > + +We have more [Bay Area meetups](https://www.meetup.com/San-Francisco-Apache-Beam) coming soon, and the community is looking into kicking off a meetup in Toronto! + +[London](https://www.meetup.com/London-Apache-Beam-Meetup) had its first meetup of 2019 at the start of April: + +https://secure.meetupstatic.com/photos/event/4/7/0/e/600_480318190.jpeg"; }}" alt="Beam Meetup London" height="360" width="640" > + +and [Stockholm](https://www.meetup.com/Apache-Beam-Stockholm/events/260634514) had its second meetup at the start of May: + +Big audience for the second https://twitter.com/ApacheBeam?ref_src=twsrc%5Etfw";>@ApacheBeam meetup in Stockholm! Gleb, https://twitter.com/kanterov?ref_src=twsrc%5Etfw";>@kanterov from https://twitter.com/SpotifyEng?ref_src=twsrc%5Etfw";>@SpotifyEng kicking off the first talk with Beam SQL.https://twitter.com/hashtag/ApacheBeamStockholm?src=hash&ref_src=twsrc%5Etfw [...] +https://platform.twitter.com/widgets.js"</a>; charset="utf-8"> + +Keep an eye out for a meetup in [Paris](https://www.meetup.com/Paris-Apache-Beam-Meetup). + +If you are interested in starting your own meetup, feel free [to reach out](https://beam.apache.org/community/contact-us)! Good places to start include our Slack channel, the dev and user mailing lists, or the Apache Beam Twitter. + +Even if you can’t travel to these meetups, you can stay informed on the happenings of the community. The talks and sessions from previous conferences and meetups are archived on the [Apache Beam YouTube channel](https://www.youtube.com/c/ApacheBeamYT). If you want your session added to the channel, don’t hesitate to get in touch! And in case you want to attend the next Beam event in style, you can also order your swag on the [Beam swag store](https://store-beam.myshopify.com) + +## Summits +The first summit of the year wi
[beam] branch master updated: [BEAM-7015] Remove duplicate standard_coders.yaml
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1840b43 [BEAM-7015] Remove duplicate standard_coders.yaml
new a1680b5 Merge pull request #8319: [BEAM-7015] Remove duplicate
standard_coders.yaml
1840b43 is described below
commit 1840b4339c44391fd2057244d09bb9cac60c812d
Author: Thomas Weise
AuthorDate: Mon Apr 15 22:16:27 2019 -0700
[BEAM-7015] Remove duplicate standard_coders.yaml
---
.gitignore | 1 +
.../apache_beam/coders/standard_coders_test.py | 6 +-
.../apache_beam/testing/data/standard_coders.yaml | 235 -
sdks/python/gen_protos.py | 9 +
sdks/python/setup.py | 3 +-
5 files changed, 15 insertions(+), 239 deletions(-)
diff --git a/.gitignore b/.gitignore
index 6a5feac..385b99f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -42,6 +42,7 @@ sdks/python/LICENSE
sdks/python/NOTICE
sdks/python/README.md
sdks/python/apache_beam/portability/api/*pb2*.*
+sdks/python/apache_beam/portability/api/*.yaml
sdks/python/nosetests.xml
sdks/python/postcommit_requirements.txt
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py
b/sdks/python/apache_beam/coders/standard_coders_test.py
index b4d8443..437d2be 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -37,8 +37,8 @@ from apache_beam.transforms.window import IntervalWindow
from apache_beam.utils import windowed_value
from apache_beam.utils.timestamp import Timestamp
-STANDARD_CODERS_YAML = os.path.join(
-os.path.dirname(__file__), '..', 'testing', 'data', 'standard_coders.yaml')
+STANDARD_CODERS_YAML = os.path.normpath(os.path.join(
+os.path.dirname(__file__), '../portability/api/standard_coders.yaml'))
def _load_test_cases(test_yaml):
@@ -76,7 +76,7 @@ class StandardCodersTest(unittest.TestCase):
'beam:coder:timer:v1':
lambda x, payload_parser: dict(
payload=payload_parser(x['payload']),
- timestamp=Timestamp(micros=x['timestamp'])),
+ timestamp=Timestamp(micros=x['timestamp'] * 1000)),
'beam:coder:double:v1': lambda x: float(x),
}
diff --git a/sdks/python/apache_beam/testing/data/standard_coders.yaml
b/sdks/python/apache_beam/testing/data/standard_coders.yaml
deleted file mode 100644
index e3927ab..000
--- a/sdks/python/apache_beam/testing/data/standard_coders.yaml
+++ /dev/null
@@ -1,235 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This file is broken into multiple sections delimited by ---. Each section
specifies a set of
-# reference encodings for a single standardized coder used in a specific
context.
-#
-# Each section contains up to 3 properties:
-#
-# coder: a common coder spec. Currently, a URN and URNs for component coders
as necessary.
-# nested: a boolean meaning whether the coder was used in the nested
context. Missing means to
-# test both contexts, a shorthand for when the coder is invariant
across context.
-# examples: a map of {encoded bytes: original JSON object} encoded with the
coder in the context.
-# The LHS (key) is a byte array encoded as a JSON-escaped string.
The RHS (value) is
-# one of a few standard JSON types such as numbers, strings, dicts
that map naturally
-# to the type encoded by the coder.
-#
-# These choices were made to strike a balance between portability, ease of
use, and simple
-# legibility of this file itself.
-#
-# It is expected that future work will move the `coder` field into a format
that it would be
-# represented by the Runner API, so that it can be understood by all SDKs and
harnesses.
-#
-# If a coder is marked non-deterministic in the coder spec, then only the
decoding should be validated.
-
-
-coder:
- urn: "beam:coder:bytes:v1"
[beam] branch master updated: [BEAM-7112] Defer state cleanup timers to avoid interference with user timers
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b23ef64 [BEAM-7112] Defer state cleanup timers to avoid interference
with user timers
new ab82768 Merge pull request #8351: [BEAM-7112] [flink] Defer state
cleanup till bundle completion
b23ef64 is described below
commit b23ef6432935970d6a568a632903ac0d70eb2ec0
Author: Thomas Weise
AuthorDate: Thu Apr 18 18:39:44 2019 -0700
[BEAM-7112] Defer state cleanup timers to avoid interference with user
timers
---
.../streaming/ExecutableStageDoFnOperator.java | 58 +-
.../wrappers/streaming/FlinkKeyUtils.java | 7 ++-
.../streaming/ExecutableStageDoFnOperatorTest.java | 2 +-
3 files changed, 53 insertions(+), 14 deletions(-)
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
index 2288b3f..064b87a 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
@@ -21,6 +21,7 @@ import static
org.apache.flink.util.Preconditions.checkNotNull;
import java.io.IOException;
import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
@@ -65,7 +66,6 @@ import
org.apache.beam.runners.fnexecution.provisioning.JobInfo;
import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.CoderException;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.state.BagState;
@@ -75,7 +75,6 @@ import org.apache.beam.sdk.transforms.DoFnSchemaInformation;
import org.apache.beam.sdk.transforms.join.RawUnionValue;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.CoderUtils;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
@@ -119,6 +118,7 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator sdkHarnessRunner;
private transient FlinkMetricContainer flinkMetricContainer;
private transient long backupWatermarkHold = Long.MIN_VALUE;
+ private transient ArrayDeque>
cleanupTimers;
/** Constructor. */
public ExecutableStageDoFnOperator(
@@ -189,6 +189,7 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator();
// This will call {@code createWrappingDoFnRunner} which needs the above
dependencies.
super.open();
@@ -330,15 +331,12 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator keyCoder) {
- final ByteBuffer encodedKey;
- try {
-// We need to have NESTED context here with the ByteStringCoder.
-// See StateRequestHandlers.
-encodedKey =
-ByteBuffer.wrap(CoderUtils.encodeToByteArray(keyCoder, key,
Coder.Context.NESTED));
- } catch (CoderException e) {
-throw new RuntimeException("Couldn't set key for state");
- }
+ // TODO: use ByteString, eliminate double encoding
+ // https://issues.apache.org/jira/browse/BEAM-7126
+ // We need to have NESTED context here with the ByteStringCoder.
+ // See StateRequestHandlers.
+ final ByteBuffer encodedKey =
+ FlinkKeyUtils.encodeKey(key, keyCoder, Coder.Context.NESTED);
keyedStateBackend.setCurrentKey(encodedKey);
}
};
@@ -400,8 +398,35 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator timer =
cleanupTimers.remove();
+ final ByteBuffer encodedKey = (ByteBuffer) timer.getKey();
+ try {
+// still need to process as timer, see CleanupTimer
+stateBackendLock.lock();
+if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "State cleanup for {} {}",
+ Arrays.toString(encodedKey.array()),
+ timer.getNamespace().getNamespace());
+}
+getKeyedStateBackend().setCurrentKey(encodedKey);
+super.fireTimer(timer);
+ } finally {
+stateBackendLock.unlock();
+ }
+}
+ }
+
@Override
public void fireTimer(InternalTimer timer) {
+if (CleanupTimer.GC_TIMER_ID.equals(timer.getNamespace().getTimerId())) {
[beam] branch master updated (3682d0c -> 8842857)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3682d0c Merge pull request #8327: [BEAM-7091] Fix NPE in DoFnOperator#dispose new e3e57b9 [BEAM-7035] Compatible wire representation for timers in Python SDK new b5778f1 [BEAM-7035] Support deleteTimer by timerId in Flink runner new a002454 [BEAM-7074] FnApiRunner fails to wire multiple timer collections new 8842857 Merge pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK The 21026 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../wrappers/streaming/DoFnOperator.java | 14 +--- .../streaming/ExecutableStageDoFnOperator.java | 8 - sdks/python/apache_beam/coders/coder_impl.py | 22 +++-- .../apache_beam/coders/coders_test_common.py | 14 .../runners/portability/fn_api_runner_test.py | 37 ++ .../apache_beam/runners/worker/bundle_processor.py | 8 +++-- 6 files changed, 85 insertions(+), 18 deletions(-)
[beam] branch master updated: [BEAM-6918] Fixed broken Flink example link
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 63680cb [BEAM-6918] Fixed broken Flink example link new 37d4b40 Merge pull request #8156: [BEAM-6918] Fixed broken Flink example link 63680cb is described below commit 63680cb4718d2a3823de751ad0dc42ee8d849e62 Author: David Yan AuthorDate: Wed Mar 27 15:40:24 2019 -0700 [BEAM-6918] Fixed broken Flink example link --- sdks/python/apache_beam/runners/interactive/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/runners/interactive/README.md b/sdks/python/apache_beam/runners/interactive/README.md index 97c5ad5..76200ba 100644 --- a/sdks/python/apache_beam/runners/interactive/README.md +++ b/sdks/python/apache_beam/runners/interactive/README.md @@ -251,7 +251,7 @@ You can choose to run Interactive Beam on Flink with the following settings. **Note**: Python Flink Runner (combination of PortableRunner and FlinkService) is being actively developed now, so these setups and commands are subject to changes. This guide and -[Interactive Beam Running on Flink.ipynb](exampls/Interactive Beam Running on Flink.ipynb) +[Interactive Beam Running on Flink.ipynb](examples/Interactive%20Beam%20Running%20on%20Flink.ipynb) capture the status of the world when it's last updated. ## TL;DR;
[beam] branch master updated: [BEAM-6876] Cleanup user state in portable Flink Runner
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new baa09ae [BEAM-6876] Cleanup user state in portable Flink Runner
new afcf5ee Merge pull request #8118: [BEAM-6876] Cleanup user state in
portable Flink Runner
baa09ae is described below
commit baa09aedb8eb4d60e3c192f302a25c33fdbc9460
Author: Maximilian Michels
AuthorDate: Fri Mar 22 16:25:14 2019 +0100
[BEAM-6876] Cleanup user state in portable Flink Runner
State had to be explicitly cleaned up in user state using timers which fire
at
the end of a window. This uses the StatefulDoFnRunner to set timers to
clean up
user state at the end of each window.
---
.../org/apache/beam/runners/core/DoFnRunners.java | 2 +-
.../beam/runners/core/StatefulDoFnRunner.java | 15 +-
.../runners/flink/translation/utils/NoopLock.java | 72
.../streaming/ExecutableStageDoFnOperator.java | 185 +
.../streaming/ExecutableStageDoFnOperatorTest.java | 132 ++-
5 files changed, 287 insertions(+), 119 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
index f7646ea..3d929d7 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnRunners.java
@@ -100,7 +100,7 @@ public class DoFnRunners {
DoFn fn,
DoFnRunner doFnRunner,
WindowingStrategy windowingStrategy,
- CleanupTimer cleanupTimer,
+ CleanupTimer cleanupTimer,
StateCleaner stateCleaner) {
return new StatefulDoFnRunner<>(doFnRunner, windowingStrategy,
cleanupTimer, stateCleaner);
}
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
index 6cd580c..14a9502 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java
@@ -52,13 +52,13 @@ public class StatefulDoFnRunner
private final WindowingStrategy windowingStrategy;
private final Counter droppedDueToLateness =
Metrics.counter(StatefulDoFnRunner.class,
DROPPED_DUE_TO_LATENESS_COUNTER);
- private final CleanupTimer cleanupTimer;
+ private final CleanupTimer cleanupTimer;
private final StateCleaner stateCleaner;
public StatefulDoFnRunner(
DoFnRunner doFnRunner,
WindowingStrategy windowingStrategy,
- CleanupTimer cleanupTimer,
+ CleanupTimer cleanupTimer,
StateCleaner stateCleaner) {
this.doFnRunner = doFnRunner;
this.windowingStrategy = windowingStrategy;
@@ -103,7 +103,7 @@ public class StatefulDoFnRunner
window,
cleanupTimer.currentInputWatermarkTime());
} else {
-cleanupTimer.setForWindow(window);
+cleanupTimer.setForWindow(value.getValue(), window);
doFnRunner.processElement(value);
}
}
@@ -151,7 +151,7 @@ public class StatefulDoFnRunner
* time or (b) not need a timer at all because it is a batch runner that
discards state when it is
* done.
*/
- public interface CleanupTimer {
+ public interface CleanupTimer {
/**
* Return the current, local input watermark timestamp for this
computation in the {@link
@@ -160,7 +160,7 @@ public class StatefulDoFnRunner
Instant currentInputWatermarkTime();
/** Set the garbage collect time of the window to timer. */
-void setForWindow(BoundedWindow window);
+void setForWindow(InputT value, BoundedWindow window);
/** Checks whether the given timer is a cleanup timer for the window. */
boolean isForWindow(
@@ -174,7 +174,8 @@ public class StatefulDoFnRunner
}
/** A {@link StatefulDoFnRunner.CleanupTimer} implemented via {@link
TimerInternals}. */
- public static class TimeInternalsCleanupTimer implements
StatefulDoFnRunner.CleanupTimer {
+ public static class TimeInternalsCleanupTimer
+ implements StatefulDoFnRunner.CleanupTimer {
public static final String GC_TIMER_ID = "__StatefulParDoGcTimerId";
@@ -202,7 +203,7 @@ public class StatefulDoFnRunner
}
@Override
-public void setForWindow(BoundedWindow window) {
+public void setForWindow(InputT input, BoundedWindow window) {
Instant gcTime = LateDataUtils.garbageCollectionTime(window,
windowingStrategy);
// make sure this fires after any window.maxTimestamp() timers
gcTime = gcTime.plus(GC_DELAY_MS);
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/t
[beam] branch master updated: [flink] debug logging for state access
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a01f1a1 [flink] debug logging for state access
a01f1a1 is described below
commit a01f1a15e00e042137409222580147f5b9218d75
Author: Thomas Weise
AuthorDate: Wed Mar 13 21:41:07 2019 -0700
[flink] debug logging for state access
---
.../streaming/ExecutableStageDoFnOperator.java | 24 ++
1 file changed, 24 insertions(+)
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
index 8b0685b..210da7d 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
@@ -276,6 +276,14 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator bagState =
stateInternals.state(namespace, StateTags.bag(userStateId,
valueCoder));
return bagState.read();
@@ -290,6 +298,14 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator bagState =
stateInternals.state(namespace, StateTags.bag(userStateId,
valueCoder));
while (values.hasNext()) {
@@ -306,6 +322,14 @@ public class ExecutableStageDoFnOperator
extends DoFnOperator bagState =
stateInternals.state(namespace, StateTags.bag(userStateId,
valueCoder));
bagState.clear();
[beam] branch master updated: [BEAM-6681] increase maxHeapSize for apex validatesrunner test
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1e9aab9 [BEAM-6681] increase maxHeapSize for apex validatesrunner test
new e5f1343 Merge pull request #7861 from adude3141/BEAM-6681
1e9aab9 is described below
commit 1e9aab91784e934a9a9fd2bcde647b847b027e89
Author: Michael Luckey <25622840+adude3...@users.noreply.github.com>
AuthorDate: Sat Feb 16 01:33:01 2019 +0100
[BEAM-6681] increase maxHeapSize for apex validatesrunner test
---
runners/apex/build.gradle | 3 +++
1 file changed, 3 insertions(+)
diff --git a/runners/apex/build.gradle b/runners/apex/build.gradle
index a2d96a9..8fd3ace 100644
--- a/runners/apex/build.gradle
+++ b/runners/apex/build.gradle
@@ -99,6 +99,9 @@ task validatesRunnerBatch(type: Test) {
excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesUnboundedPCollections'
}
+
+ // apex runner is run in embedded mode. Increase default HeapSize
+ maxHeapSize = '4g'
}
task validatesRunner {
[beam] branch master updated (b41c6b8 -> b4c4d70)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b41c6b8 Merge pull request #7834 from lostluck/surface new 278d336 [BEAM-5816] Finish Flink bundles exactly once new be15bf1 [BEAM-5816] Finish bundle only during expected shutdown new b4c4d70 Merge pull request #7719: [BEAM-5816] Finish Flink bundles exactly once The 20151 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../wrappers/streaming/DoFnOperator.java | 34 -- .../streaming/ExecutableStageDoFnOperator.java | 3 -- .../streaming/ExecutableStageDoFnOperatorTest.java | 7 + .../wrappers/streaming/DoFnOperatorTest.java | 21 + 4 files changed, 47 insertions(+), 18 deletions(-)
[beam] branch master updated: [BEAM-6488] Portable Flink runner support for running cross-language transforms (#7709)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c50e3fc [BEAM-6488] Portable Flink runner support for running
cross-language transforms (#7709)
c50e3fc is described below
commit c50e3fc96020125c1afd3afb09bca500ee110987
Author: Heejong Lee
AuthorDate: Fri Feb 8 20:31:44 2019 -0800
[BEAM-6488] Portable Flink runner support for running cross-language
transforms (#7709)
Multi-language support in DefaultJobBundleFactory
---
.../control/DefaultJobBundleFactory.java | 199 -
.../control/DefaultJobBundleFactoryTest.java | 28 ++-
2 files changed, 133 insertions(+), 94 deletions(-)
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
index 0764aa0..5881f4c 100644
---
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
+++
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
@@ -17,12 +17,12 @@
*/
package org.apache.beam.runners.fnexecution.control;
+import com.google.auto.value.AutoValue;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import java.io.IOException;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
-import javax.annotation.concurrent.GuardedBy;
import javax.annotation.concurrent.ThreadSafe;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
@@ -66,25 +66,17 @@ import org.slf4j.LoggerFactory;
* EnvironmentFactory} for environment management. Note that returned {@link
StageBundleFactory
* stage bundle factories} are not thread-safe. Instead, a new stage factory
should be created for
* each client. {@link DefaultJobBundleFactory} initializes the Environment
lazily when the forStage
- * is called for a stage. This factory is not capable of handling mixed types
of environment.
+ * is called for a stage.
*/
@ThreadSafe
public class DefaultJobBundleFactory implements JobBundleFactory {
private static final Logger LOG =
LoggerFactory.getLogger(DefaultJobBundleFactory.class);
- private final IdGenerator stageIdGenerator;
private final LoadingCache
environmentCache;
- // Using environment as the initialization marker.
- private Environment environment;
- private ExecutorService executor;
- private GrpcFnServer controlServer;
- private GrpcFnServer loggingServer;
- private GrpcFnServer retrievalServer;
- private GrpcFnServer provisioningServer;
- private GrpcFnServer dataServer;
- private GrpcFnServer stateServer;
- private MapControlClientPool clientPool;
- private EnvironmentFactory environmentFactory;
+ private final Map
environmentFactoryProviderMap;
+ private final ExecutorService executor;
+ private final MapControlClientPool clientPool;
+ private final IdGenerator stageIdGenerator;
public static DefaultJobBundleFactory create(
JobInfo jobInfo, Map
environmentFactoryProviderMap) {
@@ -94,42 +86,42 @@ public class DefaultJobBundleFactory implements
JobBundleFactory {
DefaultJobBundleFactory(
JobInfo jobInfo, Map
environmentFactoryMap) {
IdGenerator stageIdGenerator = IdGenerators.incrementingLongs();
+this.environmentFactoryProviderMap = environmentFactoryMap;
+this.executor = Executors.newCachedThreadPool();
+this.clientPool = MapControlClientPool.create();
this.stageIdGenerator = stageIdGenerator;
this.environmentCache =
-createEnvironmentCache(
-environment -> {
- synchronized (this) {
-checkAndInitialize(jobInfo, environmentFactoryMap,
environment);
- }
- return environmentFactory.createEnvironment(environment);
-});
+createEnvironmentCache(serverFactory -> createServerInfo(jobInfo,
serverFactory));
}
@VisibleForTesting
DefaultJobBundleFactory(
- EnvironmentFactory environmentFactory,
+ Map environmentFactoryMap,
IdGenerator stageIdGenerator,
GrpcFnServer controlServer,
GrpcFnServer loggingServer,
GrpcFnServer retrievalServer,
GrpcFnServer provisioningServer,
GrpcFnServer dataServer,
- GrpcFnServer stateServer)
- throws Exception {
+ GrpcFnServer stateServer) {
+this.environmentFactoryProviderMap = environmentFactoryMap;
this.executor = Executors.newCachedThreadPool();
+this.clientPool = MapControlClientPool.create();
this.stageIdGenerator = stageIdGenerator;
-this.controlServer = control
[beam] branch master updated: [BEAM-4776] AutoValues for MetricResults, MetricQueryResults (#7621)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 11baa1e [BEAM-4776] AutoValues for MetricResults, MetricQueryResults
(#7621)
11baa1e is described below
commit 11baa1ea69d37935be147e89c8ff1b3ddac7731c
Author: Ryan Williams
AuthorDate: Sun Jan 27 22:41:58 2019 -0500
[BEAM-4776] AutoValues for MetricResults, MetricQueryResults (#7621)
---
.../core/metrics/MetricsContainerStepMap.java | 2 +-
.../apache/beam/runners/direct/DirectMetrics.java | 38 ++
.../runners/direct/portable/DirectMetrics.java | 38 ++
.../metrics/CustomMetricQueryResults.java | 2 +-
.../beam/runners/dataflow/DataflowMetrics.java | 15 ++---
...cQueryResults.java => DefaultMetricResult.java} | 30 +++--
.../beam/sdk/metrics/MetricQueryResults.java | 23 +
.../org/apache/beam/sdk/metrics/MetricResult.java | 5 +++
.../org/apache/beam/sdk/metrics/MetricResults.java | 2 +-
9 files changed, 50 insertions(+), 105 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
index e285493..f2c6f03 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java
@@ -217,7 +217,7 @@ public class MetricsContainerStepMap implements
Serializable {
return new QueryResults(filter);
}
-private class QueryResults implements MetricQueryResults {
+private class QueryResults extends MetricQueryResults {
private final MetricsFilter filter;
private QueryResults(MetricsFilter filter) {
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
index d28b297..e3f34da 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
@@ -19,7 +19,6 @@ package org.apache.beam.runners.direct;
import static java.util.Arrays.asList;
-import com.google.auto.value.AutoValue;
import java.util.ArrayList;
import java.util.Map;
import java.util.Map.Entry;
@@ -39,7 +38,6 @@ import
org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
import org.apache.beam.runners.core.metrics.MetricsMap;
import org.apache.beam.sdk.metrics.DistributionResult;
import org.apache.beam.sdk.metrics.GaugeResult;
-import org.apache.beam.sdk.metrics.MetricName;
import org.apache.beam.sdk.metrics.MetricQueryResults;
import org.apache.beam.sdk.metrics.MetricResult;
import org.apache.beam.sdk.metrics.MetricResults;
@@ -224,38 +222,6 @@ class DirectMetrics extends MetricResults {
private final MetricsMap>
gauges;
- @AutoValue
- abstract static class DirectMetricQueryResults implements MetricQueryResults
{
-public static MetricQueryResults create(
-Iterable> counters,
-Iterable> distributions,
-Iterable> gauges) {
- return new AutoValue_DirectMetrics_DirectMetricQueryResults(counters,
distributions, gauges);
-}
- }
-
- @AutoValue
- abstract static class DirectMetricResult implements MetricResult {
-// need to define these here so they appear in the correct order
-// and the generated constructor is usable and consistent
-@Override
-public abstract MetricName getName();
-
-@Override
-public abstract String getStep();
-
-@Override
-public abstract T getCommitted();
-
-@Override
-public abstract T getAttempted();
-
-public static MetricResult create(
-MetricName name, String scope, T committed, T attempted) {
- return new AutoValue_DirectMetrics_DirectMetricResult<>(name, scope,
committed, attempted);
-}
- }
-
DirectMetrics(ExecutorService executorService) {
this.counters = new MetricsMap<>(unusedKey -> new DirectMetric<>(COUNTER,
executorService));
this.distributions =
@@ -280,7 +246,7 @@ class DirectMetrics extends MetricResults {
maybeExtractResult(filter, gaugeResults, gauge);
}
-return DirectMetricQueryResults.create(
+return MetricQueryResults.create(
counterResults.build(), distributionResults.build(),
gaugeResults.build());
}
@@ -290,7 +256,7 @@ class DirectMetrics extends MetricResults {
Map.Entry> entry) {
if (MetricFiltering.matches(filter, entry.getKey())) {
resultsBuilder.add(
- DirectMetricResult.create(
+ MetricRes
[beam] branch master updated: [BEAM-6473] Materialize all inputs of transforms with side input collections
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8fd1443 [BEAM-6473] Materialize all inputs of transforms with side
input collections
new 94a47ac Merge pull request #7640: [BEAM-6473] Materialize all inputs
of transforms with side input collections
8fd1443 is described below
commit 8fd1443a22096ee71eb6a8b28792127448c26acc
Author: Maximilian Michels
AuthorDate: Fri Jan 25 20:07:21 2019 -0500
[BEAM-6473] Materialize all inputs of transforms with side input collections
In the Python ValidatesRunner test `test_flattened_side_input`, the
PCollection
`side` is materialized to be used as a side input of a transform. `side` is
also
used as input to a flatten operation together with the collection
`side3`. However, `side3` is fused away and can't be reached for the flatten
which is not fused due to `side` being materialized.
The solution is to materialize all inputs to transforms which use side input
collections as input.
---
.../core/construction/graph/GreedyStageFuser.java| 20 +---
.../runners/portability/flink_runner_test.py | 4
.../runners/portability/fn_api_runner_test.py| 13 ++---
3 files changed, 23 insertions(+), 14 deletions(-)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
index eb0edaa..605e995 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyStageFuser.java
@@ -24,6 +24,7 @@ import java.util.LinkedHashSet;
import java.util.Queue;
import java.util.Set;
import java.util.function.Supplier;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
import
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
import
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
@@ -169,9 +170,10 @@ public class GreedyStageFuser {
PCollectionNode candidate,
Environment environment,
Set fusedPCollections) {
-for (PTransformNode node : pipeline.getPerElementConsumers(candidate)) {
- if (!(GreedyPCollectionFusers.canFuse(
- node, environment, candidate, fusedPCollections, pipeline))) {
+for (PTransformNode consumer : pipeline.getPerElementConsumers(candidate))
{
+ if (anyInputsSideInputs(consumer, pipeline)
+ || !(GreedyPCollectionFusers.canFuse(
+ consumer, environment, candidate, fusedPCollections, pipeline)))
{
// Some of the consumers can't be fused into this subgraph, so the
PCollection has to be
// materialized.
// TODO: Potentially, some of the consumers can be fused back into
this stage later
@@ -191,4 +193,16 @@ public class GreedyStageFuser {
MATERIALIZE,
FUSE,
}
+
+ private static boolean anyInputsSideInputs(PTransformNode consumer,
QueryablePipeline pipeline) {
+for (String inputPCollectionId :
consumer.getTransform().getInputsMap().values()) {
+ RunnerApi.PCollection pCollection =
+
pipeline.getComponents().getPcollectionsMap().get(inputPCollectionId);
+ PCollectionNode pCollectionNode =
PipelineNode.pCollection(inputPCollectionId, pCollection);
+ if (!pipeline.getSingletonConsumers(pCollectionNode).isEmpty()) {
+return true;
+ }
+}
+return false;
+ }
}
diff --git a/sdks/python/apache_beam/runners/portability/flink_runner_test.py
b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
index 500ef8e..cc267f2 100644
--- a/sdks/python/apache_beam/runners/portability/flink_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/flink_runner_test.py
@@ -162,10 +162,6 @@ if __name__ == '__main__':
def test_error_traceback_includes_user_code(self):
raise unittest.SkipTest("BEAM-6019")
-def test_flattened_side_input(self):
- # BEAM-6473
- super(FlinkRunnerTest, self).test_flattened_side_input(extended=False)
-
def test_metrics(self):
"""Run a simple DoFn that increments a counter, and verify that its
expected value is written to a temporary file by the FileReporter"""
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index 5515646..1e630be 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/
[beam] branch master updated: Comment typos and misc other nits (#7639)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b14375b Comment typos and misc other nits (#7639)
b14375b is described below
commit b14375b9a103c9c410472ffac3567ff7559e99e0
Author: Ryan Williams
AuthorDate: Sun Jan 27 22:29:03 2019 -0500
Comment typos and misc other nits (#7639)
---
.../fn-execution/src/main/proto/beam_fn_api.proto | 2 +-
.../src/main/proto/beam_job_api.proto | 1 -
.../beam/runners/core/metrics/MetricsPusher.java | 2 +-
.../beam/runners/core/metrics/TestMetricsSink.java | 2 +-
runners/flink/job-server/flink_job_server.gradle | 46 +++---
.../runners/flink/FlinkExecutionEnvironments.java | 12 +++---
.../control/DefaultJobBundleFactory.java | 2 +-
.../fnexecution/control/SdkHarnessClient.java | 2 +-
.../jobsubmission/InMemoryJobService.java | 4 +-
.../org/apache/beam/sdk/metrics/MetricsFilter.java | 4 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 4 +-
sdks/python/apache_beam/metrics/cells.py | 2 +-
sdks/python/apache_beam/metrics/execution.py | 4 +-
.../python/apache_beam/metrics/monitoring_infos.py | 2 +-
.../runners/portability/fn_api_runner.py | 4 +-
sdks/python/build.gradle | 2 +-
16 files changed, 46 insertions(+), 49 deletions(-)
diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index eb86618..819320c 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -29,7 +29,7 @@
syntax = "proto3";
/* TODO: Consider consolidating common components in another package
- * and lanaguage namespaces for re-use with Runner Api.
+ * and language namespaces for re-use with Runner Api.
*/
package org.apache.beam.model.fn_execution.v1;
diff --git a/model/job-management/src/main/proto/beam_job_api.proto
b/model/job-management/src/main/proto/beam_job_api.proto
index 80d6f40..8a94af9 100644
--- a/model/job-management/src/main/proto/beam_job_api.proto
+++ b/model/job-management/src/main/proto/beam_job_api.proto
@@ -134,7 +134,6 @@ message GetJobStateResponse {
// the job ended.
message JobMessagesRequest {
string job_id = 1; // (required)
-
}
message JobMessage {
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
index 53f2cfa..9a695b5 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsPusher.java
@@ -70,7 +70,7 @@ public class MetricsPusher implements Serializable {
.setDaemon(true)
.setNameFormat("MetricsPusher-thread")
.build());
- scheduledFuture = scheduler.scheduleAtFixedRate(() -> run(), 0, period,
TimeUnit.SECONDS);
+ scheduledFuture = scheduler.scheduleAtFixedRate(this::run, 0, period,
TimeUnit.SECONDS);
}
}
diff --git
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/TestMetricsSink.java
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/TestMetricsSink.java
index 52029ea..0d24866 100644
---
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/TestMetricsSink.java
+++
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/TestMetricsSink.java
@@ -23,7 +23,7 @@ import org.apache.beam.sdk.metrics.MetricsSink;
/**
* This sink just stores in a static field the first counter (if it exists)
attempted value. This is
- * usefull for tests.
+ * useful for tests.
*/
public class TestMetricsSink implements MetricsSink {
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
index dd114bc..e38e954 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -122,29 +122,29 @@ def portableValidatesRunnerTask(String name, Boolean
streaming) {
jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
jobServerConfig:
"--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0",
testClasspathConfiguration: configurations.validatesPortableRunner,
- numParallelTests: 1,
- pipelineOpts: pipelineOptions,
- environment:
BeamModulePlugin.PortableValidatesRunnerConfiguration.Environment.EMBEDDED,
- testCategories: {
-includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
-excludeCategories
'
[beam] branch master updated: [BEAM-6165] FlinkMetricContainer cleanup (#7637)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e626750 [BEAM-6165] FlinkMetricContainer cleanup (#7637)
e626750 is described below
commit e6267500c5ad0682f558e36ecba45545499e336f
Author: Ryan Williams
AuthorDate: Sun Jan 27 22:26:28 2019 -0500
[BEAM-6165] FlinkMetricContainer cleanup (#7637)
---
.../beam/runners/core/metrics/MetricUrns.java | 43 +++
.../flink/metrics/FlinkMetricContainer.java| 48 +-
2 files changed, 63 insertions(+), 28 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUrns.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUrns.java
new file mode 100644
index 000..6b40603
--- /dev/null
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricUrns.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.metrics;
+
+import static
org.apache.beam.runners.core.metrics.SimpleMonitoringInfoBuilder.USER_COUNTER_URN_PREFIX;
+
+import org.apache.beam.sdk.metrics.MetricName;
+
+/** Utility for parsing a URN to a {@link MetricName}. */
+public class MetricUrns {
+ /**
+ * Parse a {@link MetricName} from a {@link
+ * org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoUrns.Enum}.
+ *
+ * Should be consistent with {@code parse_namespace_and_name} in
monitoring_infos.py.
+ */
+ public static MetricName parseUrn(String urn) {
+if (urn.startsWith(USER_COUNTER_URN_PREFIX)) {
+ urn = urn.substring(USER_COUNTER_URN_PREFIX.length());
+}
+// If it is not a user counter, just use the first part of the URN, i.e.
'beam'
+String[] pieces = urn.split(":", 2);
+if (pieces.length != 2) {
+ throw new IllegalArgumentException("Invalid metric URN: " + urn);
+}
+return MetricName.named(pieces[0], pieces[1]);
+ }
+}
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
index c235a51..23d1f83 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
@@ -17,13 +17,18 @@
*/
package org.apache.beam.runners.flink.metrics;
+import static org.apache.beam.runners.core.metrics.MetricUrns.parseUrn;
import static
org.apache.beam.runners.core.metrics.MetricsContainerStepMap.asAttemptedOnlyMetricResults;
-import static
org.apache.beam.runners.core.metrics.SimpleMonitoringInfoBuilder.USER_COUNTER_URN_PREFIX;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.CounterData;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.DistributionData;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.ExtremaData;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.IntDistributionData;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Metric;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
import org.apache.beam.runners.core.metrics.MetricsContainerImpl;
import org.apache.beam.runners.core.metrics.MetricsContainerStepMap;
import org.apache.beam.sdk.metrics.Distribution;
@@ -90,36 +95,20 @@ public class FlinkMetricContainer {
}
/**
- * Parse a {@link MetricName} from a {@link
- * org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfoUrns.Enum}
- *
- * Should be consistent with {@code parse_namespace_and_name} in
monitoring_infos.py
- *
- * TODO: not flink-specific; where should it live?
+ * Update this container with metrics from the passed {@link
MonitoringInfo}s, and send updates
+ * along to Flink's internal metrics framework.
*/
-
[beam] branch master updated: [BEAM-6418] Lower memory consumption of Flink integration tests (#7512)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6006038 [BEAM-6418] Lower memory consumption of Flink integration
tests (#7512)
6006038 is described below
commit 6006038e5fa537260d44381b4cd11cc7e6310856
Author: Maximilian Michels
AuthorDate: Sat Jan 19 19:44:53 2019 -0500
[BEAM-6418] Lower memory consumption of Flink integration tests (#7512)
---
.../apache/beam/runners/flink/TestFlinkRunner.java | 4
.../beam/runners/flink/PortableExecutionTest.java | 28 +++---
.../runners/flink/PortableStateExecutionTest.java | 28 +++---
.../runners/flink/PortableTimersExecutionTest.java | 28 +++---
.../flink/streaming/GroupByWithNullValuesTest.java | 4 ++--
5 files changed, 66 insertions(+), 26 deletions(-)
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
index cefb161..2344f97 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/TestFlinkRunner.java
@@ -33,6 +33,10 @@ public class TestFlinkRunner extends
PipelineRunner {
private TestFlinkRunner(FlinkPipelineOptions options) {
options.setShutdownSourcesOnFinalWatermark(true);
+if (options.getParallelism() == -1) {
+ // Limit parallelism to avoid too much memory consumption during local
execution
+ options.setParallelism(1);
+}
this.delegate = FlinkRunner.fromOptions(options);
}
diff --git
a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
index 367fee1..aa3a29c 100644
---
a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
+++
b/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableExecutionTest.java
@@ -20,6 +20,7 @@ package org.apache.beam.runners.flink;
import java.io.Serializable;
import java.util.Collections;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
import org.apache.beam.model.jobmanagement.v1.JobApi.JobState.Enum;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.construction.Environments;
@@ -44,13 +45,15 @@ import org.apache.beam.sdk.values.PCollection;
import
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableList;
import
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ListeningExecutorService;
import
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.MoreExecutors;
-import org.junit.After;
-import org.junit.Before;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameter;
import org.junit.runners.Parameterized.Parameters;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* Tests the execution of a pipeline from specification to execution on the
portable Flink runner.
@@ -60,6 +63,8 @@ import org.junit.runners.Parameterized.Parameters;
@RunWith(Parameterized.class)
public class PortableExecutionTest implements Serializable {
+ private static final Logger LOG =
LoggerFactory.getLogger(PortableExecutionTest.class);
+
@Parameters(name = "streaming: {0}")
public static Object[] data() {
return new Object[] {true, false};
@@ -67,16 +72,23 @@ public class PortableExecutionTest implements Serializable {
@Parameter public boolean isStreaming;
- private transient ListeningExecutorService flinkJobExecutor;
+ private static ListeningExecutorService flinkJobExecutor;
- @Before
- public void setup() {
-flinkJobExecutor =
MoreExecutors.listeningDecorator(Executors.newCachedThreadPool());
+ @BeforeClass
+ public static void setup() {
+// Restrict this to only one thread to avoid multiple Flink clusters up at
the same time
+// which is not suitable for memory-constraint environments, i.e. Jenkins.
+flinkJobExecutor =
MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(1));
}
- @After
- public void tearDown() {
+ @AfterClass
+ public static void tearDown() throws InterruptedException {
flinkJobExecutor.shutdown();
+flinkJobExecutor.awaitTermination(10, TimeUnit.SECONDS);
+if (!flinkJobExecutor.isShutdown()) {
+ LOG.warn("Could not shutdown Flink job executor");
+}
+flinkJobExecutor = null;
}
@Test(timeout = 120_000)
diff --git
a/runners/flink/src/test/java/org/apache/beam/runners/flink/PortableStateExecutionTest.java
b/runners/fli
[beam] 01/01: Merge pull request #7517: [BEAM-6440] Fix leakage of timer de-duplication map
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 40b889e2f633482ccef688ecfee250c2c268a364 Merge: b83b302 85e84e5 Author: Thomas Weise AuthorDate: Tue Jan 15 18:20:57 2019 -0800 Merge pull request #7517: [BEAM-6440] Fix leakage of timer de-duplication map .../wrappers/streaming/DoFnOperator.java | 5 ++- .../wrappers}/streaming/DedupingOperatorTest.java | 4 +- .../wrappers}/streaming/DoFnOperatorTest.java | 5 +-- .../wrappers}/streaming/StreamRecordStripper.java | 2 +- .../streaming/WindowDoFnOperatorTest.java | 48 +++--- 5 files changed, 50 insertions(+), 14 deletions(-)
[beam] branch master updated (b83b302 -> 40b889e)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from b83b302 Merge pull request #7523: Apply spotless across Beam
add 85e84e5 [BEAM-6440] Fix leakage of timer de-duplication map
new 40b889e Merge pull request #7517: [BEAM-6440] Fix leakage of timer
de-duplication map
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../wrappers/streaming/DoFnOperator.java | 5 ++-
.../wrappers}/streaming/DedupingOperatorTest.java | 4 +-
.../wrappers}/streaming/DoFnOperatorTest.java | 5 +--
.../wrappers}/streaming/StreamRecordStripper.java | 2 +-
.../streaming/WindowDoFnOperatorTest.java | 48 +++---
5 files changed, 50 insertions(+), 14 deletions(-)
rename runners/flink/src/test/java/org/apache/beam/runners/flink/{ =>
translation/wrappers}/streaming/DedupingOperatorTest.java (95%)
rename runners/flink/src/test/java/org/apache/beam/runners/flink/{ =>
translation/wrappers}/streaming/DoFnOperatorTest.java (99%)
rename runners/flink/src/test/java/org/apache/beam/runners/flink/{ =>
translation/wrappers}/streaming/StreamRecordStripper.java (96%)
rename runners/flink/src/test/java/org/apache/beam/runners/flink/{ =>
translation/wrappers}/streaming/WindowDoFnOperatorTest.java (83%)
[beam] branch master updated (9780341 -> 3db71dd)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9780341 Merge pull request #7399: [BEAM-6350] Reuse PCollectionView when created in translators add d10a971 [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment add 3db71dd Merge pull request #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment No new revisions were added by this update. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 15 ++- runners/flink/job-server/flink_job_server.gradle | 8 +++- .../functions/FlinkDefaultExecutableStageContext.java | 3 ++- .../environment/EmbeddedEnvironmentFactory.java | 9 +++-- 4 files changed, 30 insertions(+), 5 deletions(-)
[beam] branch master updated (cee2f61 -> 6b78709)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cee2f61 Merge pull request #7487 from apache/aaltay-patch-2 add ebfb0f5 [BEAM-6248] Disable tests for Flink 1.7 build target new 6b78709 Merge pull request #7489: [BEAM-6248] Disable tests for Flink 1.7 build target The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/flink/flink_runner.gradle | 4 1 file changed, 4 insertions(+)
[beam] 01/01: Merge pull request #7489: [BEAM-6248] Disable tests for Flink 1.7 build target
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 6b787093a4cea9a326c59eac03b73082392d6488 Merge: cee2f61 ebfb0f5 Author: Thomas Weise AuthorDate: Fri Jan 11 17:56:36 2019 -0800 Merge pull request #7489: [BEAM-6248] Disable tests for Flink 1.7 build target runners/flink/flink_runner.gradle | 4 1 file changed, 4 insertions(+)
[beam] branch release-2.10.0 updated: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms.
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch release-2.10.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.10.0 by this push:
new de94544 [BEAM-6294] Ensure input and output coders are equal for
reshuffle transforms.
de94544 is described below
commit de94544ac70e70c48b407f4b440e08661db59b94
Author: Robert Bradshaw
AuthorDate: Thu Jan 10 14:22:19 2019 +0100
[BEAM-6294] Ensure input and output coders are equal for reshuffle
transforms.
The type declarations were there, but not getting applied due to a
longstanding TODO. This doesn't resolve that TODO completely, but fixes
a large number of cases, including this one.
---
sdks/python/apache_beam/pipeline.py | 5 +++--
sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 6 ++
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/sdks/python/apache_beam/pipeline.py
b/sdks/python/apache_beam/pipeline.py
index 577e773..20ac5f0 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -544,11 +544,12 @@ class Pipeline(object):
def _infer_result_type(self, transform, inputs, result_pcollection):
# TODO(robertwb): Multi-input, multi-output inference.
-# TODO(robertwb): Ideally we'd do intersection here.
type_options = self._options.view_as(TypeOptions)
if (type_options is not None and type_options.pipeline_type_check
and isinstance(result_pcollection, pvalue.PCollection)
-and not result_pcollection.element_type):
+and (not result_pcollection.element_type
+ # TODO(robertwb): Ideally we'd do intersection here.
+ or result_pcollection.element_type == typehints.Any)):
input_element_type = (
inputs[0].element_type
if len(inputs) == 1
diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
index 3fecbe5..dc248d5 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
@@ -363,6 +363,12 @@ class FnApiRunnerTest(unittest.TestCase):
| beam.Map(lambda k_vs: (k_vs[0], sorted(k_vs[1]
assert_that(res, equal_to([('a', [1, 2]), ('b', [3])]))
+ # Runners may special case the Reshuffle transform urn.
+ def test_reshuffle(self):
+with self.create_pipeline() as p:
+ assert_that(p | beam.Create([1, 2, 3]) | beam.Reshuffle(),
+ equal_to([1, 2, 3]))
+
def test_flatten(self):
with self.create_pipeline() as p:
res = (p | 'a' >> beam.Create(['a']),
[beam] branch master updated: [BEAM-6248] Add Flink v1.7 build target to Flink Runner (#7300)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c739ca5 [BEAM-6248] Add Flink v1.7 build target to Flink Runner
(#7300)
c739ca5 is described below
commit c739ca57004ff0bdb62cf03153051db5aac7fb20
Author: Maximilian Michels
AuthorDate: Thu Jan 10 15:41:38 2019 -0500
[BEAM-6248] Add Flink v1.7 build target to Flink Runner (#7300)
---
runners/flink/1.7/build.gradle | 33 ++
.../flink/1.7/job-server-container/build.gradle| 26 +
runners/flink/1.7/job-server/build.gradle | 30
.../runners/flink/FlinkExecutionEnvironments.java | 5 +++-
.../beam/runners/flink/PortableExecutionTest.java | 1 +
.../runners/flink/PortableStateExecutionTest.java | 1 +
.../runners/flink/PortableTimersExecutionTest.java | 1 +
settings.gradle| 7 +
8 files changed, 103 insertions(+), 1 deletion(-)
diff --git a/runners/flink/1.7/build.gradle b/runners/flink/1.7/build.gradle
new file mode 100644
index 000..5982c1b
--- /dev/null
+++ b/runners/flink/1.7/build.gradle
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def basePath = '..'
+
+/* All properties required for loading the Flink build script */
+project.ext {
+ // Set the version of all Flink-related dependencies here.
+ flink_version = '1.7.1'
+ // Look for the source code in the parent module
+ main_source_dirs = ["$basePath/src/main/java"]
+ test_source_dirs = ["$basePath/src/test/java"]
+ main_resources_dirs = ["$basePath/src/main/resources"]
+ test_resources_dirs = ["$basePath/src/test/resources"]
+}
+
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_runner.gradle"
diff --git a/runners/flink/1.7/job-server-container/build.gradle
b/runners/flink/1.7/job-server-container/build.gradle
new file mode 100644
index 000..afdb68a
--- /dev/null
+++ b/runners/flink/1.7/job-server-container/build.gradle
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+def basePath = '../../job-server-container'
+
+project.ext {
+ resource_path = basePath
+}
+
+// Load the main build script which contains all build logic.
+apply from: "$basePath/flink_job_server_container.gradle"
diff --git a/runners/flink/1.7/job-server/build.gradle
b/runners/flink/1.7/job-server/build.gradle
new file mode 100644
index 000..d5251e0
--- /dev/null
+++ b/runners/flink/1.7/job-server/build.gradle
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF AN
[beam] branch master updated (49d5cf5 -> 336fe0c)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 49d5cf5 Merge pull request #7453: [BEAM-6326] Fix portable stateful processing with side input add ee57028 [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms. add 336fe0c Merge pull request #7457: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/pipeline.py | 5 +++-- sdks/python/apache_beam/runners/portability/fn_api_runner_test.py | 6 ++ 2 files changed, 9 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #7453: [BEAM-6326] Fix portable stateful processing with side input
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 49d5cf56db0137f4acfc9e54d9188e6ae998ca06 Merge: 14426af b9e51f6 Author: Thomas Weise AuthorDate: Thu Jan 10 09:25:41 2019 -0800 Merge pull request #7453: [BEAM-6326] Fix portable stateful processing with side input .../FlinkStreamingPortablePipelineTranslator.java | 41 +++--- .../state/FlinkKeyGroupStateInternals.java | 6 ++-- 2 files changed, 40 insertions(+), 7 deletions(-)
[beam] branch master updated (14426af -> 49d5cf5)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 14426af Merge pull request #7459: [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests add b9e51f6 [BEAM-6326] Fix portable stateful processing with side input new 49d5cf5 Merge pull request #7453: [BEAM-6326] Fix portable stateful processing with side input The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../FlinkStreamingPortablePipelineTranslator.java | 41 +++--- .../state/FlinkKeyGroupStateInternals.java | 6 ++-- 2 files changed, 40 insertions(+), 7 deletions(-)
[beam] 01/01: Merge pull request #7459: [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 14426af8a15aae345bf30050c0d9c318d338c317 Merge: f62f490 aad58d1 Author: Thomas Weise AuthorDate: Thu Jan 10 08:54:55 2019 -0800 Merge pull request #7459: [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 4 ++-- runners/flink/job-server/flink_job_server.gradle | 8 ++-- 2 files changed, 8 insertions(+), 4 deletions(-)
[beam] branch master updated (f62f490 -> 14426af)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f62f490 Merge pull request #7449: [BEAM-6397] add map attribute in SyntheticDataPubSubPublisher add aad58d1 [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests new 14426af Merge pull request #7459: [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 4 ++-- runners/flink/job-server/flink_job_server.gradle | 8 ++-- 2 files changed, 8 insertions(+), 4 deletions(-)
[beam] branch revert-7360-reshuffle-flink created (now cb33143)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch revert-7360-reshuffle-flink in repository https://gitbox.apache.org/repos/asf/beam.git. at cb33143 Revert "[BEAM-6294] Use Flink rebalance for shuffle." This branch includes the following new commits: new cb33143 Revert "[BEAM-6294] Use Flink rebalance for shuffle." The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Revert "[BEAM-6294] Use Flink rebalance for shuffle."
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch revert-7360-reshuffle-flink
in repository https://gitbox.apache.org/repos/asf/beam.git
commit cb33143baf0fe22502ea490518eae048904339dd
Author: Thomas Weise
AuthorDate: Wed Jan 9 00:08:15 2019 -0800
Revert "[BEAM-6294] Use Flink rebalance for shuffle."
---
.../FlinkBatchPortablePipelineTranslator.java | 34 +
.../beam/runners/flink/FlinkJobInvocation.java | 82 ++
.../flink/FlinkPortablePipelineTranslator.java | 19 +
.../FlinkStreamingPortablePipelineTranslator.java | 23 +-
sdks/python/apache_beam/transforms/util.py | 8 ---
5 files changed, 29 insertions(+), 137 deletions(-)
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
index b182832..4a09d86 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java
@@ -23,7 +23,6 @@ import static
org.apache.beam.runners.flink.translation.utils.FlinkPipelineTrans
import static
org.apache.beam.runners.flink.translation.utils.FlinkPipelineTranslatorUtils.getWindowingStrategy;
import static
org.apache.beam.runners.flink.translation.utils.FlinkPipelineTranslatorUtils.instantiateCoder;
-import com.google.auto.service.AutoService;
import com.google.common.collect.BiMap;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Iterables;
@@ -44,7 +43,6 @@ import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
-import org.apache.beam.runners.core.construction.NativeTransforms;
import org.apache.beam.runners.core.construction.PTransformTranslation;
import org.apache.beam.runners.core.construction.RehydratedComponents;
import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
@@ -83,7 +81,6 @@ import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.WindowingStrategy;
import
org.apache.beam.vendor.grpc.v1p13p1.com.google.protobuf.InvalidProtocolBufferException;
-import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
@@ -122,8 +119,7 @@ public class FlinkBatchPortablePipelineTranslator
* Creates a batch translation context. The resulting Flink execution dag
will live in a new
* {@link ExecutionEnvironment}.
*/
- @Override
- public BatchTranslationContext createTranslationContext(
+ public static BatchTranslationContext createTranslationContext(
JobInfo jobInfo,
FlinkPipelineOptions pipelineOptions,
@Nullable String confDir,
@@ -162,8 +158,7 @@ public class FlinkBatchPortablePipelineTranslator
* flink {@link ExecutionEnvironment} that the execution plan will be
applied to.
*/
public static class BatchTranslationContext
- implements FlinkPortablePipelineTranslator.TranslationContext,
- FlinkPortablePipelineTranslator.Executor {
+ implements FlinkPortablePipelineTranslator.TranslationContext {
private final JobInfo jobInfo;
private final FlinkPipelineOptions options;
@@ -190,11 +185,6 @@ public class FlinkBatchPortablePipelineTranslator
return options;
}
-@Override
-public JobExecutionResult execute(String jobName) throws Exception {
- return getExecutionEnvironment().execute(jobName);
-}
-
public ExecutionEnvironment getExecutionEnvironment() {
return executionEnvironment;
}
@@ -239,23 +229,7 @@ public class FlinkBatchPortablePipelineTranslator
}
@Override
- public Set knownUrns() {
-return urnToTransformTranslator.keySet();
- }
-
- /** Predicate to determine whether a URN is a Flink native transform. */
- @AutoService(NativeTransforms.IsNativeTransform.class)
- public static class IsFlinkNativeTransform implements
NativeTransforms.IsNativeTransform {
-@Override
-public boolean test(RunnerApi.PTransform pTransform) {
- return PTransformTranslation.RESHUFFLE_URN.equals(
- PTransformTranslation.urnForTransformOrNull(pTransform));
-}
- }
-
- @Override
- public FlinkPortablePipelineTranslator.Executor translate(
- BatchTranslationContext context, RunnerApi.Pipeline pipeline) {
+ public void translate(BatchTranslationContext context, RunnerApi.Pipeline
pipeline) {
// Use a QueryablePipeline to traverse transforms topologically.
Queryable
[beam] branch master updated: [BEAM-6009] Modify Github template and README to include Flink PVR for batch/streaming (#7370)
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new abe3220 [BEAM-6009] Modify Github template and README to include Flink PVR for batch/streaming (#7370) abe3220 is described below commit abe3220b21d985f7f1cd5e64929f0a8f54e22a22 Author: Maximilian Michels AuthorDate: Sun Jan 6 07:46:21 2019 +0100 [BEAM-6009] Modify Github template and README to include Flink PVR for batch/streaming (#7370) The old build task has been split into a batch and a streaming job. --- .github/PULL_REQUEST_TEMPLATE.md | 2 +- README.md| 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index bf1b1f6..14f977c 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -15,7 +15,7 @@ Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- -Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://coveralls.io/github/apache/beam?branch=master) ### Post-commit tests status (on master branch) + Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- -Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [ were added to refs/heads/master by this push:
new 926361b [BEAM-5386] Move assertion out of finally block to not
swallow original exception
926361b is described below
commit 926361bcb54129a71c7f5abf5817148f8a7d4d41
Author: Thomas Weise
AuthorDate: Sun Dec 30 13:23:33 2018 -0800
[BEAM-5386] Move assertion out of finally block to not swallow original
exception
---
.../translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
index dc93f8f..9edde35 100644
---
a/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
+++
b/runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapperTest.java
@@ -678,8 +678,8 @@ public class UnboundedSourceWrapperTest {
thread.interrupt();
// try to join but also don't mask exceptions with test timeout
thread.join(1000);
-assertThat(thread.isAlive(), is(false));
}
+ assertThat(thread.isAlive(), is(false));
}
}
[beam] branch master updated: Flink 1.5.6 upgrade (#7322)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ffec485 Flink 1.5.6 upgrade (#7322)
ffec485 is described below
commit ffec485fc38da2df69decd5302700250ecac8757
Author: Thomas Weise
AuthorDate: Thu Dec 27 09:18:26 2018 -0800
Flink 1.5.6 upgrade (#7322)
---
runners/flink/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/runners/flink/build.gradle b/runners/flink/build.gradle
index 0db72f3..39addc3 100644
--- a/runners/flink/build.gradle
+++ b/runners/flink/build.gradle
@@ -19,7 +19,7 @@
/* All properties required for loading the Flink build script. */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.5.5'
+ flink_version = '1.5.6'
// Look for the source code in the current module
main_source_dirs = ['./src/main/java']
test_source_dirs = ['./src/test/java']
[beam] branch master updated: [BEAM-6165] Send metrics to Flink in portable Flink runner (#7183)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1e41220 [BEAM-6165] Send metrics to Flink in portable Flink runner
(#7183)
1e41220 is described below
commit 1e41220977d6c45d293b86f2e581daec3513c66e
Author: Ryan Williams
AuthorDate: Fri Dec 21 01:10:30 2018 +0900
[BEAM-6165] Send metrics to Flink in portable Flink runner (#7183)
* add bulk-update method to Distribution
* update metrics in portable flink stages
* add portable flink metrics test
* use "file" config alias for FileReporter
---
.../runners/core/metrics/DistributionCell.java | 5 ++
.../runners/flink/FlinkExecutionEnvironments.java | 6 +-
.../beam/runners/flink/metrics/FileReporter.java | 75 ++
.../flink/metrics/FlinkMetricContainer.java| 66 +++-
.../apache/beam/runners/flink/metrics/Metrics.java | 56 ++
.../functions/FlinkExecutableStageFunction.java| 21 -
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../streaming/ExecutableStageDoFnOperator.java | 18 -
.../dataflow/worker/DeltaDistributionCell.java | 5 ++
.../org/apache/beam/sdk/metrics/Distribution.java | 2 +
.../java/org/apache/beam/sdk/metrics/Metrics.java | 8 ++
.../runners/portability/flink_runner_test.py | 90 +-
12 files changed, 345 insertions(+), 9 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
index 85425a5..c39fee0 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
@@ -57,6 +57,11 @@ public class DistributionCell implements Distribution,
MetricCellhttp://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink.metrics;
+
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.PrintStream;
+import org.apache.flink.metrics.Metric;
+import org.apache.flink.metrics.MetricConfig;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.reporter.AbstractReporter;
+
+/**
+ * Flink {@link org.apache.flink.metrics.reporter.MetricReporter metrics
reporter} for writing
+ * metrics to a file specified via the "metrics.reporter.file.path" config key
(assuming an alias of
+ * "file" for this reporter in the "metrics.reporters" setting).
+ */
+public class FileReporter extends AbstractReporter {
+ @Override
+ public String filterCharacters(String input) {
+return input;
+ }
+
+ private String path;
+ private PrintStream ps;
+
+ @Override
+ public void open(MetricConfig config) {
+synchronized (this) {
+ if (path == null) {
+path = config.getString("path", null);
+log.info("Opening file: {}", path);
+if (path == null) {
+ throw new IllegalStateException("FileReporter metrics config needs
'path' key");
+}
+try {
+ FileOutputStream fos = new FileOutputStream(path);
+ ps = new PrintStream(fos);
+} catch (FileNotFoundException e) {
+ throw new IllegalStateException("FileReporter couldn't open file",
e);
+}
+ }
+}
+ }
+
+ @Override
+ public void notifyOfRemovedMetric(Metric metric, String metricName,
MetricGroup group) {
+final String name = group.getMetricIdentifier(metricName, this);
+super.notifyOfRemovedMetric(metric, metricName, group);
+synchronized (this) {
+ ps.printf("%s: %s%n", name, Metrics.toString(metric));
+}
+ }
+
+ @Override
+ public void close() {
+ps.close();
+log.info("wrote metrics to {}", path);
+ }
+}
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
index f4857d1..cc0e55a 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java
@@ -17,13 +17,17 @@
*/
package org.apache.beam.runners.flink.metrics;
+
[beam] 01/01: Merge pull request #7329: Adds a link to release notes for Beam 2.9.0
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 7c8f7ba3ecde35ce4165606528da7c8a8681aea5 Merge: c039440 6c39b61 Author: Thomas Weise AuthorDate: Thu Dec 20 08:05:59 2018 -0800 Merge pull request #7329: Adds a link to release notes for Beam 2.9.0 website/src/get-started/downloads.md | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (c039440 -> 7c8f7ba)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c039440 Merge pull request #7331: [BEAM-6276] Fix performance regression. add 6c39b61 Adds a link to release notes for Beam 2.9.0. new 7c8f7ba Merge pull request #7329: Adds a link to release notes for Beam 2.9.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/get-started/downloads.md | 2 ++ 1 file changed, 2 insertions(+)
[beam] 01/01: Merge pull request #7314: WEBSITE: update community nav, add in-person page
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5845e2a4b19612864eb1adb56e818af8843d7f9c Merge: aa0fdc5 384de82 Author: Thomas Weise AuthorDate: Tue Dec 18 17:26:19 2018 -0800 Merge pull request #7314: WEBSITE: update community nav, add in-person page website/src/_includes/section-menu/community.html | 1 + website/src/community/in-person.md| 47 +++ 2 files changed, 48 insertions(+)
[beam] branch master updated (aa0fdc5 -> 5845e2a)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aa0fdc5 Merge pull request #7311: Update release guide with new Jenkins job name. add 384de82 WEBSITE: update community nav, add in-person page new 5845e2a Merge pull request #7314: WEBSITE: update community nav, add in-person page The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/_includes/section-menu/community.html | 1 + website/src/community/in-person.md| 47 +++ 2 files changed, 48 insertions(+) create mode 100644 website/src/community/in-person.md
[beam] branch master updated: Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2555945 Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
2555945 is described below
commit 255594576f853f6a0d0f4d32c921a1501461c9da
Author: Thomas Weise
AuthorDate: Mon Dec 17 18:09:56 2018 -0800
Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
---
sdks/python/build.gradle | 11 ---
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index db18b2b..a17b032 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -220,16 +220,13 @@ def portableWordCountTask(name, streaming) {
"--input=/etc/profile",
"--output=/tmp/py-wordcount-direct",
"--runner=PortableRunner",
+ "--experiments=worker_threads=100",
]
-
- if (streaming) {
+ if (streaming)
options += ["--streaming"]
- } else {
-// workaround for local file output in docker container
+ else
+ // workaround for local file output in docker container
options += ["--environment_cache_millis=1"]
-// [BEAM-5167] Workaround for scheduling issue between SDKHarness and
Flink
-options += ["--parallelism=1"]
- }
if (project.hasProperty("jobEndpoint"))
options += ["--job_endpoint=${project.property('jobEndpoint')}"]
exec {
[beam] 01/01: Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
This is an automated email from the ASF dual-hosted git repository.
thw pushed a commit to branch revert-7174-wc_change_parallalism
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 9bdabf9d29b3a827658c24b6e933a9009d2a7acf
Author: Thomas Weise
AuthorDate: Mon Dec 17 18:09:56 2018 -0800
Revert "[BEAM-5978] Changing parallelism for wordcount to 1"
---
sdks/python/build.gradle | 11 ---
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index db18b2b..a17b032 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -220,16 +220,13 @@ def portableWordCountTask(name, streaming) {
"--input=/etc/profile",
"--output=/tmp/py-wordcount-direct",
"--runner=PortableRunner",
+ "--experiments=worker_threads=100",
]
-
- if (streaming) {
+ if (streaming)
options += ["--streaming"]
- } else {
-// workaround for local file output in docker container
+ else
+ // workaround for local file output in docker container
options += ["--environment_cache_millis=1"]
-// [BEAM-5167] Workaround for scheduling issue between SDKHarness and
Flink
-options += ["--parallelism=1"]
- }
if (project.hasProperty("jobEndpoint"))
options += ["--job_endpoint=${project.property('jobEndpoint')}"]
exec {
[beam] branch revert-7174-wc_change_parallalism created (now 9bdabf9)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch revert-7174-wc_change_parallalism in repository https://gitbox.apache.org/repos/asf/beam.git. at 9bdabf9 Revert "[BEAM-5978] Changing parallelism for wordcount to 1" This branch includes the following new commits: new 9bdabf9 Revert "[BEAM-5978] Changing parallelism for wordcount to 1" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (ca3eb14 -> e108cca)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ca3eb14 Merge pull request #7251 from More robust FnApi Runner. add e4f6517 [BEAM-6227] Do not compare recovered state against structural null value new e108cca Merge pull request #7291: [BEAM-6227] Do not compare recovered state against structural null value The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../wrappers/streaming/state/FlinkStateInternals.java| 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-)
[beam] 01/01: Merge pull request #7291: [BEAM-6227] Do not compare recovered state against structural null value
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e108cca89fc2925eb25591f5b38119bf7016f777 Merge: ca3eb14 e4f6517 Author: Thomas Weise AuthorDate: Mon Dec 17 06:44:30 2018 -0800 Merge pull request #7291: [BEAM-6227] Do not compare recovered state against structural null value .../wrappers/streaming/state/FlinkStateInternals.java| 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-)
svn commit: r31552 - /release/beam/KEYS
Author: thw Date: Sat Dec 15 17:44:38 2018 New Revision: 31552 Log: Add key for Thomas Weise Modified: release/beam/KEYS Modified: release/beam/KEYS == --- release/beam/KEYS (original) +++ release/beam/KEYS Sat Dec 15 17:44:38 2018 @@ -699,3 +699,60 @@ Phk187YqxAVreGGQm8FNIX1oN1CZq8C3oLAy+lV/ Ag== =FiUj -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2015-09-16 [SC] + 10409A66C7C2F297C8581C2A12DEE3E4D920A98C +uid [ unknown] Thomas Weise +sig 312DEE3E4D920A98C 2015-09-16 Thomas Weise +sub rsa4096 2015-09-16 [E] +sig 12DEE3E4D920A98C 2015-09-16 Thomas Weise + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBFX5e6ABEAC/QbCpW3nMSJZCO0hTK2hftQTPeDgn82ea2eC4F1D7nzDTsOpd +EpnrTmIHhyWMdOYgAhnHe06ktOv2bu2elcmjROzm0vQ8HL1EQ/8FKL9AsaoHwlxl +EsUe8LPsg1CdbrXgP6QnZTzMF1Yj+ZTMEz7AqrsUuJ/aSHfukQliU6dxpWj2rLn8 +Uolzlpk86w/NceDf+tcc3ZiBVXX+KhCVTfkv6WpFcsFspiIC258t4qcA64ZigJdd +hUybLpgZBmWoDD7UurUTtMsNBBLSZ65Fuj+QvDuNSeXIgQid/B0mmCQGUQNOW1of +OtosRQETlL9KIGTIrqkkhQ9huk4K2kBgOzZRyHpckvA2mdPMJIZqQCexcWwuQHJ4 +IvIE428GhiEi8Xu7n6NAZHXJFcQ8OjPK2aJZVH4USeKHczPxY6qJkqGOrvWsL+1i +vw1x4BwqNb7mppMFiEsO+5WeVTQ1brCY7naW8c9A18fPmMMBT5oxSyJYA9TwWVEW +0jxIjTYXei46EFE9/oryODwJuZZpslO6Miu+oU4uwyG5ksfOUOTMstZd87Rl0k/P +Mrb5JKEpkyZeddRRU9b0UHqXsR+scISwmCXnUQxwOsv5ATqooH/2+bhoxu5a+VzS +xTuT+7Vr4ONqdkIHps1TUM6glP6oMy2KFirraFY61qJy/RlSCvR16H1l7wARAQAB +tB1UaG9tYXMgV2Vpc2UgPHRod0BhcGFjaGUub3JnPokCOAQTAQIAIgUCVfl7oAIb +AwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQEt7j5NkgqYyjexAAumk0QtyW +9oRvINzhPRloWUecfLMo4WgMkdRPjPrtzpOc7wrK8nMeUU+70MtP73H1Xk23cjcg +hkuzzrdJpHSw/VLKPayKGnmqYXtWRJ2rKtBxREDPK9vJJIWVrIIrGP1G4ZZ62Amy +M8IAEnvpRTyu/c2lw6H1X7rHjzP61aWo9zrU5JZLhIDE5iUWezPNbjWJtI985IEG +gxDEeXmtsKLZPQDHnuxhVwsSHBdv3pHvz+llK1Am8LZGr5n1ooNY1E78oAwDhlJ/ +6rEQCxcDWxUtuvSQ6tNtvX1Tket1TwQtPzu91uXYmSDzax1JczN44Zax5ObvL742 +aeTXmZJGFeu8cL9MnDI+/klLceKTN3ERt3vw+QguAXpPn1aQrDH9TPVaTRZxcirk +G/kyAmOP3kmvnqew32X5VaCKrQQfqscBm7zjZqZRM5EkPZQqGVfI8otTh35XTOe9 +qO+/biP4TwVd2NO1z0qZF18y2GxAEGDQSL70jYiGYS7G787NMNwfLewKwoXImV9Q ++UJk/HliBiVnvMNoX72M2evqR0ZGl/B1zi+DB8a1qM+MOoDz3asf2Mmgr5SelllH +s/31qKdtkoar1F7J/lBjV9+ohgZv1M+md5s5R6YiXR3IjIWHyI6SyMpqI28e18j3 +3TKgQoI3MTlGNGPXvdKDaJQMDOusEAuwxcG5Ag0EVfl7oAEQAMMBDrLMmEGdFAIt +BD41W6cdHMQ0FrX3Sz4F9xinDdk/7bbPf2Ky6sh4+9H2tUe77yoMtcoaRbTEIrqu +4HXERWDgPOp18enWM1qQfezZe8lBAp3DM8sG11wHbMJjv3RKlRGbkWCfM0+ZsrVk +zAqIrKuDg3PHMlrKjZbsfBPR1vebNT7KBkHLzKzZQsWHApDdNEaEct/Ys16+OwsY +ihDVin5NC4P4xc1gr6peJa9Ozw+1x7zrbg3H2lYZ+rJgkokKRmWLed74WjohtkT6 +5A9jbSy4ZM0vgzC89Nm9OHoiPeqjrnxTXfqN7omFKlNlnleSLzM13DThb/vcYp+3 +oBhh7lVyXcTD0fPzx8fJN5/Pl/vzLbjukLSpZTvzKON3ryFl6IdVJjSLsts3KLU/ +c5Po1nI/UZSGopPW6MGWLTwk8MxnGJC+Mcs7qxNhWvi45X/3U1xs6yCU9yoaYOH1 +28ee3B6PE7lmXPX+x9F+5AHF422bbfiu/043e+dVobrVQSbtrZPXLc1m+iX/AztH +NKnN976zGah1KrOyG0k3V1MrXUoMC+lllMj0ffgMo3QQbMjdtKD6AD6dCF4j1qOZ +kqSpIS8URnz3O8d+dmvSB5UvgYRU/eIU28KVL/tDcr4LMfHK2UcRNjnk56ABq/4E +8O39PifL/ChYYIVEBxAbKRjpjQg/ABEBAAGJAh8EGAECAAkFAlX5e6ACGwwACgkQ +Et7j5NkgqYyaJg//XNO1Pz87YbTwPAtVjIA+8Y0Js/5cBvFDwYsNL05QIaHZdmfR +109zYV+AAFWg/BCqAgJESqNzrH2JSsMlMLP8koy1ChTlhbJUBuTrXL6Oxfs4GL3N +FGx6BZC2WBZrHaXK8EtwoFvnnDmHBL9BK7L9E7A4MGyxePk/CUr7GLxInAsjq+B5 +2oXfsaJquyR/cZC1TB6Xf3zcSC0+AURJ7ni+k+ROCK6DrpHxlHhi5tWp8Sgg997S +FpTGd10axcR/McqwKDGITpBiTWcwYA5v+M5Vix8BcE00ZhKqBS1Db3RA6jnnToF5 +fnf6verJpR7G4UWyL/tdRCU9bCiHZPXwhJBBHxrlnL0skoJLoX+4MgNcJPaJAHG9 +nyKIzIzB4iQ1NHt9norDJ0sbrpL428frDr4kSRVGuQ8oaCEWLkndtIUcLc91g2l5 +g/fLNO2+34nd1fMnOtsc53ykXQ3e/MWlTqrcNNdyioNJmOzlLJ47UzbXorVzyF4e +PbRkdwxvAq8a0RMZZT4QW1kJ7fy4SRpN/l5bduwivIpEn35irJsAKXQp2c/gLzke +IoB/BxKOM2jyLFd7vlt9mcfqiLTsGpT/y0EegdhZfsnpvJV8Lf3jSRpF10/yO8lm +4lYEmheOUv73NdFk3csl78eVB97eRKVJJ7PTln+QCVc2f4brYmTeP+4381w= +=B6VM +-END PGP PUBLIC KEY BLOCK-
[beam] branch master updated (88f181c -> 2886473)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 88f181c Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime add fc38359 [BEAM-6227] Fix GroupByKey with null values in Flink Runner new 2886473 Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../streaming/state/FlinkStateInternals.java | 58 +++--- .../flink/streaming/GroupByWithNullValuesTest.java | 92 ++ 2 files changed, 138 insertions(+), 12 deletions(-) create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByWithNullValuesTest.java
[beam] 01/01: Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 28864734cf108a49fa5751aa1f867e97c0e069d0 Merge: 88f181c fc38359 Author: Thomas Weise AuthorDate: Fri Dec 14 08:41:38 2018 -0800 Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner .../streaming/state/FlinkStateInternals.java | 58 +++--- .../flink/streaming/GroupByWithNullValuesTest.java | 92 ++ 2 files changed, 138 insertions(+), 12 deletions(-)
[beam] branch master updated (3fafec1 -> 1518361)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3fafec1 [BEAM-5320] [BEAM-6106] Finish Python 3 porting for testing module (#7262) add bd0103c [BEAM-5978] Changing parallalim for wordcount to 1 new 1518361 Merge pull request #7174: [BEAM-5978] Changing parallelism for wordcount to 1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/build.gradle | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-)
[beam] 01/01: Merge pull request #7174: [BEAM-5978] Changing parallelism for wordcount to 1
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 15183612e6a3f6b0eb4dca88a6bf90a92a456805 Merge: 3fafec1 bd0103c Author: Thomas Weise AuthorDate: Wed Dec 12 17:49:28 2018 -0800 Merge pull request #7174: [BEAM-5978] Changing parallelism for wordcount to 1 sdks/python/build.gradle | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-)
[beam] branch master updated (e8cbbf6 -> 8c8b7c3)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e8cbbf6 Merge pull request #7246: [BEAM-5058] remove unused envdir parameter add ebe81b9 [BEAM-2943] Fix typo in filename on FlinkRunner page new 8c8b7c3 Merge pull request #7254: [BEAM-2943] Fix typo in filename on FlinkRunner page The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/documentation/runners/flink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7254: [BEAM-2943] Fix typo in filename on FlinkRunner page
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8c8b7c3d524ee9e654e4d9b1c322419493f1ff0b Merge: e8cbbf6 ebe81b9 Author: Thomas Weise AuthorDate: Tue Dec 11 20:58:42 2018 -0800 Merge pull request #7254: [BEAM-2943] Fix typo in filename on FlinkRunner page website/src/documentation/runners/flink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (3a182d6 -> e8cbbf6)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3a182d6 Merge pull request #7032: [BEAM-6033] normalize httplib2.Http initialization and usage add fddb684 [BEAM-5058] remove unused envdir parameter new e8cbbf6 Merge pull request #7246: [BEAM-5058] remove unused envdir parameter The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/build.gradle | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam] 01/01: Merge pull request #7246: [BEAM-5058] remove unused envdir parameter
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e8cbbf6e28b59bf8ced28dd1d6265fafff8a313e Merge: 3a182d6 fddb684 Author: Thomas Weise AuthorDate: Tue Dec 11 17:49:13 2018 -0800 Merge pull request #7246: [BEAM-5058] remove unused envdir parameter sdks/python/build.gradle | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam] 01/01: Merge pull request #7129: [BEAM-6122] Update committer guidelines
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e5d9cf42e2499e867be580f5ccf63c0189636d81 Merge: bec7dac 702b9de Author: Thomas Weise AuthorDate: Tue Dec 4 08:59:34 2018 -0800 Merge pull request #7129: [BEAM-6122] Update committer guidelines website/src/contribute/committer-guide.md| 37 +--- website/src/contribute/index.md | 25 +++ website/src/contribute/postcommits-guides.md | 2 +- 3 files changed, 49 insertions(+), 15 deletions(-)
[beam] branch master updated (bec7dac -> e5d9cf4)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bec7dac Merge pull request #7187: [BEAM-1628] Allow empty port for flink master url add 9c018ac [BEAM-6122] Update committer guidelines add 1c67861 Address review comments. add 702b9de Move squash paragraph under merging. new e5d9cf4 Merge pull request #7129: [BEAM-6122] Update committer guidelines The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/contribute/committer-guide.md| 37 +--- website/src/contribute/index.md | 25 +++ website/src/contribute/postcommits-guides.md | 2 +- 3 files changed, 49 insertions(+), 15 deletions(-)
[beam] branch master updated (b917a86 -> e2e31c9)
This is an automated email from the ASF dual-hosted git repository.
thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from b917a86 Merge pull request #7180: [BEAM-6146] Run pre-commit
wordcount in batch and streaming mode
add f040a4e [BEAM-5859] Improve operator names for portable pipelines
add e2e31c9 Merge pull request #7150: [BEAM-5859] Improve operator names
for portable pipelines
No new revisions were added by this update.
Summary of changes:
.../construction/ExecutableStageTranslation.java | 21 ++
.../ExecutableStageTranslationTest.java| 83 ++
.../FlinkBatchPortablePipelineTranslator.java | 24 ---
.../FlinkStreamingPortablePipelineTranslator.java | 16 +++--
.../FlinkPipelineTranslatorUtilsTest.java} | 36 +-
5 files changed, 143 insertions(+), 37 deletions(-)
create mode 100644
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExecutableStageTranslationTest.java
copy
runners/{google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/apiary/ApiaryTest.java
=>
flink/src/test/java/org/apache/beam/runners/flink/translation/FlinkPipelineTranslatorUtilsTest.java}
(50%)
[beam] branch master updated (90ef4eb -> b917a86)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 90ef4eb Merge pull request #7090: [BEAM-5984] Enable publishing load test results to Big Query add 4c97d0c [BEAM-6146] Run pre-commit wordcount in batch and streaming mode. add b917a86 Merge pull request #7180: [BEAM-6146] Run pre-commit wordcount in batch and streaming mode No new revisions were added by this update. Summary of changes: sdks/python/build.gradle | 52 +++- 1 file changed, 29 insertions(+), 23 deletions(-)
[beam] 01/01: Merge pull request #7178: [BEAM-6163] Build python boot for mac and support process env on mac
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 6517025e278737f6381cea1452ae5e032f5341c1 Merge: a9dff65 6119242 Author: Thomas Weise AuthorDate: Sat Dec 1 07:24:44 2018 -0800 Merge pull request #7178: [BEAM-6163] Build python boot for mac and support process env on mac sdks/python/build.gradle | 7 ++- sdks/python/container/build.gradle | 4 ++-- 2 files changed, 8 insertions(+), 3 deletions(-)
[beam] branch master updated (a9dff65 -> 6517025)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a9dff65 Merge pull request #7175: [BEAM-5884] Move the nullable attribute onto FieldType. add 6119242 [BEAM-6163] Build python boot for mac and support process env on mac new 6517025 Merge pull request #7178: [BEAM-6163] Build python boot for mac and support process env on mac The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/build.gradle | 7 ++- sdks/python/container/build.gradle | 4 ++-- 2 files changed, 8 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #7031: [BEAM-6058] Adding option for Flink configuration directory and setting config in execution environment
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c2aaf2d613f25df1754d67f61d29d7124ac06b50 Merge: 02c763b f84feac Author: Thomas Weise AuthorDate: Tue Nov 27 16:55:37 2018 -0800 Merge pull request #7031: [BEAM-6058] Adding option for Flink configuration directory and setting config in execution environment runners/flink/job-server/build.gradle | 2 ++ .../FlinkBatchPortablePipelineTranslator.java | 11 ++-- .../runners/flink/FlinkExecutionEnvironments.java | 32 -- .../beam/runners/flink/FlinkJobInvocation.java | 10 +-- .../apache/beam/runners/flink/FlinkJobInvoker.java | 1 + .../beam/runners/flink/FlinkJobServerDriver.java | 15 ++ .../FlinkStreamingPortablePipelineTranslator.java | 8 -- .../flink/FlinkExecutionEnvironmentsTest.java | 4 +-- .../beam/runners/flink/PortableExecutionTest.java | 1 + .../runners/flink/PortableStateExecutionTest.java | 1 + .../runners/flink/PortableTimersExecutionTest.java | 1 + 11 files changed, 61 insertions(+), 25 deletions(-)
[beam] branch master updated (02c763b -> c2aaf2d)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 02c763b [BEAM-5197] Fix UnboundedSourceWrapper#testWatermarkEmission (#7138) add f84feac [BEAM-6058] Adding option for flink configuration directory and setting config in exectution environment new c2aaf2d Merge pull request #7031: [BEAM-6058] Adding option for Flink configuration directory and setting config in execution environment The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/flink/job-server/build.gradle | 2 ++ .../FlinkBatchPortablePipelineTranslator.java | 11 ++-- .../runners/flink/FlinkExecutionEnvironments.java | 32 -- .../beam/runners/flink/FlinkJobInvocation.java | 10 +-- .../apache/beam/runners/flink/FlinkJobInvoker.java | 1 + .../beam/runners/flink/FlinkJobServerDriver.java | 15 ++ .../FlinkStreamingPortablePipelineTranslator.java | 8 -- .../flink/FlinkExecutionEnvironmentsTest.java | 4 +-- .../beam/runners/flink/PortableExecutionTest.java | 1 + .../runners/flink/PortableStateExecutionTest.java | 1 + .../runners/flink/PortableTimersExecutionTest.java | 1 + 11 files changed, 61 insertions(+), 25 deletions(-)
