[beam] branch master updated (d005a5e -> 614425d)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from d005a5e Merge pull request #12614 [BEAM-9980] Configure Python
versions for direct test suite tasks via settings.gradle
add 41b849e Add Filter lesson to Go SDK katas
add 8046408 Reformat code for readability
add 6c57a61 Change license info format in task-info.yaml
add a6f51c0 Update stepik course
add 614425d Merge pull request #12613 [BEAM-9680] Add Filter with filter
package lesson to Go SDK Katas
No new revisions were added by this update.
Summary of changes:
.../go/common_transforms/filter/{pardo => filter}/cmd/main.go| 3 ++-
.../filter/filter}/pkg/task/task.go | 7 ---
.../filter/filter}/task-info.yaml| 8
.../go/common_transforms/filter/filter/task-remote-info.yaml | 2 ++
.../Filter/Filter => go/common_transforms/filter/filter}/task.md | 9 -
.../common_transforms/filter/{pardo => filter}/test/task_test.go | 6 +++---
learning/katas/go/common_transforms/filter/lesson-info.yaml | 1 +
learning/katas/go/course-remote-info.yaml| 2 +-
8 files changed, 21 insertions(+), 17 deletions(-)
copy learning/katas/go/common_transforms/filter/{pardo => filter}/cmd/main.go
(94%)
copy learning/katas/go/{core_transforms/combine/simple_function =>
common_transforms/filter/filter}/pkg/task/task.go (86%)
copy learning/katas/go/{core_transforms/cogroupbykey/cogroupbykey =>
common_transforms/filter/filter}/task-info.yaml (96%)
create mode 100644
learning/katas/go/common_transforms/filter/filter/task-remote-info.yaml
copy learning/katas/{java/Common Transforms/Filter/Filter =>
go/common_transforms/filter/filter}/task.md (63%)
copy learning/katas/go/common_transforms/filter/{pardo =>
filter}/test/task_test.go (90%)
[beam] tag nightly-master updated (ff20e29 -> d005a5e)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag nightly-master was modified! *** from ff20e29 (commit) to d005a5e (commit) from ff20e29 [BEAM-10757] Add enum34 to manual_licenses (#12624) add 34b6ad7 [BEAM-BEAM-10667] Make github actions bulding wheels bucket name stored as a secret add fcd767f Merge pull request #12513: [BEAM-10667] Make gh-actions wheels bucket name stored as a secret add 2b2b8e7 Merge pull request #12597: [BEAM-10685] Added integration test for streaming for SnowflakeIO add 85efc38 [BEAM-10756] Fix empty pull response to not ack and to not throw IndexOutOfBoundsException add ecfc389 Merge pull request #12623 from lukecwik/beam10756 add 3114f0c [BEAM-10753] Add Slack link invitation on README add 423abb7 Update README.md add 38999c6 Merge pull request #12620 from irvifa/docs/add-slack-invitation-link add 7577e00 [BEAM-3301] Adding SDF Go Dataflow translation. (#12629) add 2fce720 clarifying unclear comments add f3ac482 Merge pull request #12631 from ettirapp/filesystem-fixes add 85af44e Allow Nexmark launcher to publish human-readable events to pubsub. add 7d384789 Merge pull request #12606 - Allow Nexmark launcher to publish human-readable events to pubsub. add 317997f [BEAM-10752] Use TestPubsubSignal in PubsubToBigqueryIT (#12625) add e666a51 Include coders embedded in CombinePayload when converting x-lang transforms to proto definition add f2959c3 Merge pull request #12586: [BEAM-8019] Don't drop coders embedded in CombinePayload add 275b7c9 [BEAM-10200] Add optional experiment to enable heap dump through the … (#12537) add ca2e7e7 [BEAM-9979] Support a reset call that happens after the bundle is done allowing resetting of internal state of a transform runner. add 1c6b07c Merge pull request #12636 from lukecwik/beam9979 add a673e34 [BEAM-9919] Refining xlang types and wrappers add 1b6b60e Merge pull request #12632 [BEAM-9919] Refining xlang types and wrappers add 9620d62 [BEAM-9919] xlang transforms wrapped with namespaces add 63f9808 [BEAM-9919] cleaning namespace.go add 5f6cf0b Merge pull request #12635 [BEAM-9919] Isolating xlang transforms by namespace add 49b5192 [BEAM-9547] Implement dataframes top, join, merge. (#12516) add af2f7af [BEAM-9920] Enabling artifact staging for xlang transforms add 0c65ca3 documentation(beam/sdks/go): added apache license and function docstring add 891e775 fix(beam/sdks/go): correcting changed function name add 801af80 Merge pull request #12644 from [BEAM-9920] Enabling artifact staging for xlang transforms add 8995905 [BEAM-9980] Configure Python versions for direct test suite tasks via settings.gradle add d005a5e Merge pull request #12614 [BEAM-9980] Configure Python versions for direct test suite tasks via settings.gradle No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 6 +- .github/workflows/python_tests.yml | 1 + .../job_PostCommit_Python_MongoDBIO_IT.groovy | 3 +- CI.md | 1 + README.md | 1 + scripts/ci/ci_check_are_gcp_variables_set.sh | 2 +- sdks/go/examples/xlang/wordcount/wordcount.go | 115 ++ .../go/examples/xlang/wordcount/xlang_wordcount.go | 111 - sdks/go/pkg/beam/core/graph/edge.go| 50 ++-- sdks/go/pkg/beam/core/graph/xlang.go | 145 +--- sdks/go/pkg/beam/core/runtime/graphx/translate.go | 25 +- sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 54 + sdks/go/pkg/beam/core/runtime/xlangx/namespace.go | 125 ++ sdks/go/pkg/beam/core/runtime/xlangx/resolve.go| 70 ++ sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 46 ++-- .../beam/runners/dataflow/dataflowlib/messages.go | 1 + .../beam/runners/dataflow/dataflowlib/translate.go | 13 ++ .../pkg/beam/runners/universal/runnerlib/stage.go | 3 +- sdks/go/pkg/beam/runners/universal/universal.go| 3 + sdks/go/pkg/beam/xlang.go | 82 --- .../sdk/extensions/gcp/storage/GcsFileSystem.java | 8 +- .../sdk/extensions/sql/PubsubToBigqueryIT.java | 28 +-- .../beam/fn/harness/BeamFnDataReadRunner.java | 1 + .../beam/fn/harness/BeamFnDataWriteRunner.java | 1 + .../beam/fn/harness/BoundedSourceRunner.java | 1 + .../org/apache/beam/fn/harness/CombineRunners.java | 1 + .../org/apache/beam/fn/harness/FlattenRunner.java | 1 + .../apache/beam/fn/harness/FnApiDoFnRunner.java| 1 + .../org/apache/beam/fn/harness/MapFnRunners.java | 1 +
[beam] branch master updated (801af80 -> d005a5e)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 801af80 Merge pull request #12644 from [BEAM-9920] Enabling artifact
staging for xlang transforms
add 8995905 [BEAM-9980] Configure Python versions for direct test suite
tasks via settings.gradle
new d005a5e Merge pull request #12614 [BEAM-9980] Configure Python
versions for direct test suite tasks via settings.gradle
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../job_PostCommit_Python_MongoDBIO_IT.groovy | 3 +-
.../{dataflow/py35 => direct}/build.gradle | 8 +-
sdks/python/test-suites/direct/common.gradle | 96 +
sdks/python/test-suites/direct/py2/build.gradle| 99 +-
sdks/python/test-suites/direct/py35/build.gradle | 31 ---
sdks/python/test-suites/direct/py38/build.gradle | 11 ---
sdks/python/test-suites/gradle.properties | 3 +
settings.gradle| 1 +
8 files changed, 107 insertions(+), 145 deletions(-)
copy sdks/python/test-suites/{dataflow/py35 => direct}/build.gradle (82%)
[beam] 01/01: Merge pull request #12614 [BEAM-9980] Configure Python versions for direct test suite tasks via settings.gradle
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit d005a5e609967f0c643b4466d82b9031aae02a09
Merge: 801af80 8995905
Author: tvalentyn
AuthorDate: Wed Aug 19 19:13:49 2020 -0700
Merge pull request #12614 [BEAM-9980] Configure Python versions for direct
test suite tasks via settings.gradle
.../job_PostCommit_Python_MongoDBIO_IT.groovy | 3 +-
.../test-suites/direct/{py38 => }/build.gradle | 17 +---
sdks/python/test-suites/direct/common.gradle | 96 +
sdks/python/test-suites/direct/py2/build.gradle| 99 +-
sdks/python/test-suites/direct/py35/build.gradle | 31 ---
sdks/python/test-suites/direct/py38/build.gradle | 11 ---
sdks/python/test-suites/gradle.properties | 3 +
settings.gradle| 1 +
8 files changed, 106 insertions(+), 155 deletions(-)
[beam] branch master updated (49b5192 -> 801af80)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 49b5192 [BEAM-9547] Implement dataframes top, join, merge. (#12516) new af2f7af [BEAM-9920] Enabling artifact staging for xlang transforms new 0c65ca3 documentation(beam/sdks/go): added apache license and function docstring new 891e775 fix(beam/sdks/go): correcting changed function name new 801af80 Merge pull request #12644 from [BEAM-9920] Enabling artifact staging for xlang transforms The 28533 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/go/examples/xlang/wordcount/wordcount.go | 4 +- sdks/go/pkg/beam/core/runtime/xlangx/resolve.go| 70 ++ .../pkg/beam/runners/universal/runnerlib/stage.go | 3 +- sdks/go/pkg/beam/runners/universal/universal.go| 3 + 4 files changed, 77 insertions(+), 3 deletions(-) create mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/resolve.go
[beam] branch master updated (5f6cf0b -> 49b5192)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5f6cf0b Merge pull request #12635 [BEAM-9919] Isolating xlang transforms by namespace add 49b5192 [BEAM-9547] Implement dataframes top, join, merge. (#12516) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/dataframe/frames.py| 251 - sdks/python/apache_beam/dataframe/frames_test.py | 28 +++ .../apache_beam/dataframe/pandas_doctests_test.py | 27 ++- 3 files changed, 292 insertions(+), 14 deletions(-)
[beam] branch release-2.24.0 updated: [BEAM-10702, BEAM-10757] Cherrypick #12571: Do not implicitly decompress artifacts (#12619)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a commit to branch release-2.24.0 in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/release-2.24.0 by this push: new 33d2437 [BEAM-10702, BEAM-10757] Cherrypick #12571: Do not implicitly decompress artifacts (#12619) 33d2437 is described below commit 33d24372098a50f822a20416091f71287caf2917 Author: Eugene Kirpichov AuthorDate: Wed Aug 19 16:40:34 2020 -0700 [BEAM-10702, BEAM-10757] Cherrypick #12571: Do not implicitly decompress artifacts (#12619) * [BEAM-10702] Do not implicitly decompress artifacts Also adds a Julia set test on portable local runner, which uses a setup.py and hence exercises the artifact staging codepath. This is a squashed cherrypick of https://github.com/apache/beam/pull/12571 * Add enum34 to manual_licenses --- CHANGES.md | 3 +- .../runners/portability/artifact_service.py| 4 ++- .../portability/fn_api_runner/worker_handlers.py | 7 - .../container/license_scripts/dep_urls_py.yaml | 4 ++- .../license_scripts/manual_licenses/enum34/LICENSE | 32 ++ sdks/python/test-suites/portable/common.gradle | 25 + 6 files changed, 70 insertions(+), 5 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index a4882b3..45e29c9 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -63,16 +63,15 @@ you may use `--experiments=use_legacy_bq_sink`. * Add cross-language support to Java's JdbcIO, now available in the Python module `apache_beam.io.external.jdbc` ([BEAM-10135](https://issues.apache.org/jira/browse/BEAM-10135), [BEAM-10136](https://issues.apache.org/jira/browse/BEAM-10136)). * Add support of AWS SDK v2 for KinesisIO.Read (Java) ([BEAM-9702](https://issues.apache.org/jira/browse/BEAM-9702)). -* Support for X source added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). * Add streaming support to SnowflakeIO in Java SDK ([BEAM-9896](https://issues.apache.org/jira/browse/BEAM-9896 )) * Support reading and writing to Google Healthcare DICOM APIs in Python SDK ([BEAM-10601](https://issues.apache.org/jira/browse/BEAM-10601)) ## New Features / Improvements * Shared library for simplifying management of large shared objects added to Python SDK. Example use case is sharing a large TF model object across threads ([BEAM-10417](https://issues.apache.org/jira/browse/BEAM-10417)). -* X feature added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). * Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle ([BEAM-8543](https://issues.apache.org/jira/browse/BEAM-8543)). * OnTimerContext should not create a new one when processing each element/timer in FnApiDoFnRunner ([BEAM-9839](https://issues.apache.org/jira/browse/BEAM-9839)) +* Fixed BEAM-10702 (Python) - Python portable runner with embedded job endpoint was unable to run pipelines which had dependencies in setup.py or requirements.txt ([BEAM-10702](https://issues.apache.org/jira/browse/BEAM-10702)). ## Breaking Changes diff --git a/sdks/python/apache_beam/runners/portability/artifact_service.py b/sdks/python/apache_beam/runners/portability/artifact_service.py index f2bbf53..1f3ec1c 100644 --- a/sdks/python/apache_beam/runners/portability/artifact_service.py +++ b/sdks/python/apache_beam/runners/portability/artifact_service.py @@ -41,6 +41,7 @@ import grpc from future.moves.urllib.request import urlopen from apache_beam.io import filesystems +from apache_beam.io.filesystems import CompressionTypes from apache_beam.portability import common_urns from apache_beam.portability.api import beam_artifact_api_pb2 from apache_beam.portability.api import beam_artifact_api_pb2_grpc @@ -263,7 +264,8 @@ class BeamFilesystemHandler(object): self._root = root def file_reader(self, path): -return filesystems.FileSystems.open(path) +return filesystems.FileSystems.open( +path, compression_type=CompressionTypes.UNCOMPRESSED) def file_writer(self, name=None): full_path = filesystems.FileSystems.join(self._root, name) diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py index 9d27549..1f1d483 100644 --- a/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py +++ b/sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers.py @@ -45,6 +45,7 @@ from typing import overload import grpc from apache_beam.io import filesystems +from apache_beam.io.filesystems import CompressionTypes from apache_beam.portability import common_urns from apache_beam.portability import python_urns from apache_beam.portability.api import beam_artifact_api_pb2_grpc @@ -464,9 +465,13 @@ class
[beam] branch master updated (1b6b60e -> 5f6cf0b)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1b6b60e Merge pull request #12632 [BEAM-9919] Refining xlang types and wrappers add 9620d62 [BEAM-9919] xlang transforms wrapped with namespaces add 63f9808 [BEAM-9919] cleaning namespace.go add 5f6cf0b Merge pull request #12635 [BEAM-9919] Isolating xlang transforms by namespace No new revisions were added by this update. Summary of changes: sdks/go/examples/xlang/wordcount/wordcount.go | 12 ++- sdks/go/pkg/beam/core/graph/xlang.go | 2 + sdks/go/pkg/beam/core/runtime/graphx/translate.go | 7 +- sdks/go/pkg/beam/core/runtime/xlangx/namespace.go | 125 ++ sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 1 - sdks/go/pkg/beam/xlang.go | 11 +- 6 files changed, 147 insertions(+), 11 deletions(-) create mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/namespace.go
[beam] branch master updated (1c6b07c -> 1b6b60e)
This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1c6b07c Merge pull request #12636 from lukecwik/beam9979
add a673e34 [BEAM-9919] Refining xlang types and wrappers
add 1b6b60e Merge pull request #12632 [BEAM-9919] Refining xlang types
and wrappers
No new revisions were added by this update.
Summary of changes:
.../wordcount/{xlang_wordcount.go => wordcount.go} | 0
sdks/go/pkg/beam/core/graph/edge.go| 50 ---
sdks/go/pkg/beam/core/graph/xlang.go | 143 +
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 20 +--
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 54
sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 45 +++
sdks/go/pkg/beam/xlang.go | 71 +-
7 files changed, 222 insertions(+), 161 deletions(-)
rename sdks/go/examples/xlang/wordcount/{xlang_wordcount.go => wordcount.go}
(100%)
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/xlang.go
[beam] branch master updated: [BEAM-9979] Support a reset call that happens after the bundle is done allowing resetting of internal state of a transform runner.
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ca2e7e7 [BEAM-9979] Support a reset call that happens after the
bundle is done allowing resetting of internal state of a transform runner.
new 1c6b07c Merge pull request #12636 from lukecwik/beam9979
ca2e7e7 is described below
commit ca2e7e79e1f2631da44bbb62cedc745b3727a990
Author: Luke Cwik
AuthorDate: Wed Aug 19 14:41:05 2020 -0700
[BEAM-9979] Support a reset call that happens after the bundle is done
allowing resetting of internal state of a transform runner.
This is preliminary work needed to be able to reset the
BeamFnDataReadRunner to solve the race condition.
---
.../beam/fn/harness/BeamFnDataReadRunner.java | 1 +
.../beam/fn/harness/BeamFnDataWriteRunner.java | 1 +
.../beam/fn/harness/BoundedSourceRunner.java | 1 +
.../org/apache/beam/fn/harness/CombineRunners.java | 1 +
.../org/apache/beam/fn/harness/FlattenRunner.java | 1 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java| 1 +
.../org/apache/beam/fn/harness/MapFnRunners.java | 1 +
.../beam/fn/harness/PTransformRunnerFactory.java | 20 --
.../fn/harness/control/ProcessBundleHandler.java | 27 +++--
.../beam/fn/harness/AssignWindowsRunnerTest.java | 5 ++-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 4 ++
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 1 +
.../beam/fn/harness/BoundedSourceRunnerTest.java | 1 +
.../apache/beam/fn/harness/CombineRunnersTest.java | 15 ---
.../apache/beam/fn/harness/FlattenRunnerTest.java | 10 +++--
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 19 +
.../apache/beam/fn/harness/MapFnRunnersTest.java | 3 ++
.../harness/control/ProcessBundleHandlerTest.java | 46 --
18 files changed, 136 insertions(+), 22 deletions(-)
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
index 34cdcb4..fe52877 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
@@ -102,6 +102,7 @@ public class BeamFnDataReadRunner {
PCollectionConsumerRegistry pCollectionConsumerRegistry,
PTransformFunctionRegistry startFunctionRegistry,
PTransformFunctionRegistry finishFunctionRegistry,
+Consumer addResetFunction,
Consumer tearDownFunctions,
Consumer addProgressRequestCallback,
BundleSplitListener splitListener,
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
index 684f399..0de2d3f 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataWriteRunner.java
@@ -92,6 +92,7 @@ public class BeamFnDataWriteRunner {
PCollectionConsumerRegistry pCollectionConsumerRegistry,
PTransformFunctionRegistry startFunctionRegistry,
PTransformFunctionRegistry finishFunctionRegistry,
+Consumer addResetFunction,
Consumer tearDownFunctions,
Consumer addProgressRequestCallback,
BundleSplitListener splitListener,
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
index 9500d21..ee63a6f 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BoundedSourceRunner.java
@@ -85,6 +85,7 @@ public class BoundedSourceRunner, OutputT>
PCollectionConsumerRegistry pCollectionConsumerRegistry,
PTransformFunctionRegistry startFunctionRegistry,
PTransformFunctionRegistry finishFunctionRegistry,
+Consumer addResetFunction,
Consumer tearDownFunctions,
Consumer addProgressRequestCallback,
BundleSplitListener splitListener,
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
index 9cf80cb..e32ff38 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/CombineRunners.java
@@ -131,6 +131,7 @@ public class CombineRunners {
PCollectionConsumerRegistry pCollectionConsumerRegistry,
PTransformFunctionRegistry
[beam] branch master updated (f2959c3 -> 275b7c9)
This is an automated email from the ASF dual-hosted git repository. goenka pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f2959c3 Merge pull request #12586: [BEAM-8019] Don't drop coders embedded in CombinePayload add 275b7c9 [BEAM-10200] Add optional experiment to enable heap dump through the … (#12537) No new revisions were added by this update. Summary of changes: .../runners/portability/portable_runner.py | 7 -- .../apache_beam/runners/worker/sdk_worker.py | 6 +++-- .../apache_beam/runners/worker/sdk_worker_main.py | 19 +++ .../apache_beam/runners/worker/worker_status.py| 28 +++--- .../runners/worker/worker_status_test.py | 9 +++ 5 files changed, 52 insertions(+), 17 deletions(-)
[beam] branch master updated (317997f -> f2959c3)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 317997f [BEAM-10752] Use TestPubsubSignal in PubsubToBigqueryIT (#12625) add e666a51 Include coders embedded in CombinePayload when converting x-lang transforms to proto definition add f2959c3 Merge pull request #12586: [BEAM-8019] Don't drop coders embedded in CombinePayload No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/core.py | 23 +++ 1 file changed, 15 insertions(+), 8 deletions(-)
[beam] branch master updated (7d384789 -> 317997f)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7d384789 Merge pull request #12606 - Allow Nexmark launcher to publish human-readable events to pubsub. add 317997f [BEAM-10752] Use TestPubsubSignal in PubsubToBigqueryIT (#12625) No new revisions were added by this update. Summary of changes: .../sdk/extensions/sql/PubsubToBigqueryIT.java | 28 -- 1 file changed, 15 insertions(+), 13 deletions(-)
[beam] branch master updated (f3ac482 -> 7d384789)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f3ac482 Merge pull request #12631 from ettirapp/filesystem-fixes add 85af44e Allow Nexmark launcher to publish human-readable events to pubsub. add 7d384789 Merge pull request #12606 - Allow Nexmark launcher to publish human-readable events to pubsub. No new revisions were added by this update. Summary of changes: .../beam/sdk/nexmark/NexmarkConfiguration.java | 17 +++ .../apache/beam/sdk/nexmark/NexmarkLauncher.java | 24 -- .../apache/beam/sdk/nexmark/NexmarkOptions.java| 6 ++ .../org/apache/beam/sdk/nexmark/NexmarkUtils.java | 8 4 files changed, 53 insertions(+), 2 deletions(-)
[beam] branch master updated: clarifying unclear comments
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2fce720 clarifying unclear comments
new f3ac482 Merge pull request #12631 from ettirapp/filesystem-fixes
2fce720 is described below
commit 2fce720b800afd1bce0b7469734de1646c029fd8
Author: Etta Rapp
AuthorDate: Wed Aug 19 16:19:01 2020 +
clarifying unclear comments
---
.../sdk/extensions/gcp/storage/GcsFileSystem.java | 8 ++--
.../org/apache/beam/sdk/io/aws/s3/S3FileSystem.java | 19 ++-
2 files changed, 20 insertions(+), 7 deletions(-)
diff --git
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
index ae4c4f0..c11994f 100644
---
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
+++
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystem.java
@@ -108,8 +108,12 @@ class GcsFileSystem extends FileSystem {
ret.add(nonGlobsMatchResults.next());
}
}
-checkState(!globsMatchResults.hasNext(), "Expect no more elements in
globsMatchResults.");
-checkState(!nonGlobsMatchResults.hasNext(), "Expect no more elements in
nonGlobsMatchResults.");
+checkState(
+!globsMatchResults.hasNext(),
+"Internal error encountered in GcsFilesystem: expected no more
elements in globsMatchResults.");
+checkState(
+!nonGlobsMatchResults.hasNext(),
+"Internal error encountered in GcsFilesystem: expected no more
elements in globsMatchResults.");
return ret.build();
}
diff --git
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
index db0a9e2..af11a61 100644
---
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
+++
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java
@@ -83,6 +83,7 @@ import org.checkerframework.checker.nullness.qual.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+/** {@link FileSystem} implementation for Amazon S3. */
class S3FileSystem extends FileSystem {
private static final Logger LOG =
LoggerFactory.getLogger(S3FileSystem.class);
@@ -159,15 +160,23 @@ class S3FileSystem extends FileSystem {
ImmutableList.Builder matchResults = ImmutableList.builder();
for (Boolean isGlob : isGlobBooleans) {
if (isGlob) {
-checkState(globMatches.hasNext(), "Expect globMatches has next.");
+checkState(
+globMatches.hasNext(),
+"Internal error encountered in S3Filesystem: expected more
elements in globMatches.");
matchResults.add(globMatches.next());
} else {
-checkState(nonGlobMatches.hasNext(), "Expect nonGlobMatches has
next.");
+checkState(
+nonGlobMatches.hasNext(),
+"Internal error encountered in S3Filesystem: expected more
elements in nonGlobMatches.");
matchResults.add(nonGlobMatches.next());
}
}
-checkState(!globMatches.hasNext(), "Expect no more elements in
globMatches.");
-checkState(!nonGlobMatches.hasNext(), "Expect no more elements in
nonGlobMatches.");
+checkState(
+!globMatches.hasNext(),
+"Internal error encountered in S3Filesystem: expected no more elements
in globMatches.");
+checkState(
+!nonGlobMatches.hasNext(),
+"Internal error encountered in S3Filesystem: expected no more elements
in nonGlobMatches.");
return matchResults.build();
}
@@ -377,7 +386,7 @@ class S3FileSystem extends FileSystem {
private static MatchResult.Metadata createBeamMetadata(
S3ResourceId path, String contentEncoding) {
-checkArgument(path.getSize().isPresent(), "path has size");
+checkArgument(path.getSize().isPresent(), "The resource id should have a
size.");
checkNotNull(contentEncoding, "contentEncoding");
boolean isReadSeekEfficient =
!NON_READ_SEEK_EFFICIENT_ENCODINGS.contains(contentEncoding);
[beam] branch master updated (38999c6 -> 7577e00)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 38999c6 Merge pull request #12620 from irvifa/docs/add-slack-invitation-link add 7577e00 [BEAM-3301] Adding SDF Go Dataflow translation. (#12629) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/runners/dataflow/dataflowlib/messages.go | 1 + sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go | 13 + 2 files changed, 14 insertions(+)
[beam] branch master updated (ecfc389 -> 38999c6)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ecfc389 Merge pull request #12623 from lukecwik/beam10756 new 3114f0c [BEAM-10753] Add Slack link invitation on README new 423abb7 Update README.md new 38999c6 Merge pull request #12620 from irvifa/docs/add-slack-invitation-link The 28512 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: README.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated: [BEAM-10756] Fix empty pull response to not ack and to not throw IndexOutOfBoundsException
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 85efc38 [BEAM-10756] Fix empty pull response to not ack and to not
throw IndexOutOfBoundsException
new ecfc389 Merge pull request #12623 from lukecwik/beam10756
85efc38 is described below
commit 85efc3864960fb9f0553fb4671c71e2058169288
Author: Luke Cwik
AuthorDate: Tue Aug 18 16:05:32 2020 -0700
[BEAM-10756] Fix empty pull response to not ack and to not throw
IndexOutOfBoundsException
---
.../java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
index 76d4734..ba52a2e 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsubSignal.java
@@ -253,6 +253,9 @@ public class TestPubsubSignal implements TestRule {
do {
try {
signal = pubsub.pull(DateTime.now().getMillis(),
signalSubscriptionPath, 1, false);
+if (signal.isEmpty()) {
+ continue;
+}
pubsub.acknowledge(
signalSubscriptionPath,
signal.stream().map(IncomingMessage::ackId).collect(toList()));
break;
@@ -267,7 +270,7 @@ public class TestPubsubSignal implements TestRule {
}
} while (DateTime.now().isBefore(endPolling));
-if (signal == null) {
+if (signal == null || signal.isEmpty()) {
throw new AssertionError(
String.format(
"Did not receive signal on %s in %ss",
[beam] branch master updated: Merge pull request #12597: [BEAM-10685] Added integration test for streaming for SnowflakeIO
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2b2b8e7 Merge pull request #12597: [BEAM-10685] Added integration
test for streaming for SnowflakeIO
2b2b8e7 is described below
commit 2b2b8e71b7697b2933ddcef7e2454077684ddee8
Author: Kasia Kucharczyk <2536609+kkuch...@users.noreply.github.com>
AuthorDate: Wed Aug 19 15:48:41 2020 +0200
Merge pull request #12597: [BEAM-10685] Added integration test for
streaming for SnowflakeIO
---
sdks/java/io/snowflake/build.gradle| 1 +
.../io/snowflake/test/StreamingSnowflakeIOIT.java | 196 +
.../beam/sdk/io/snowflake/test/TestUtils.java | 18 ++
3 files changed, 215 insertions(+)
diff --git a/sdks/java/io/snowflake/build.gradle
b/sdks/java/io/snowflake/build.gradle
index ea8204d..f2b48fe 100644
--- a/sdks/java/io/snowflake/build.gradle
+++ b/sdks/java/io/snowflake/build.gradle
@@ -36,6 +36,7 @@ dependencies {
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile project(path: ":sdks:java:io:common", configuration:
"testRuntime")
testCompile project(path: ":sdks:java:testing:test-utils", configuration:
"testRuntime")
+ testCompile 'com.google.cloud:google-cloud-storage:1.102.0'
testCompile library.java.avro
testCompile library.java.junit
testCompile library.java.hamcrest_core
diff --git
a/sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/StreamingSnowflakeIOIT.java
b/sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/StreamingSnowflakeIOIT.java
new file mode 100644
index 000..4584a99
--- /dev/null
+++
b/sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/StreamingSnowflakeIOIT.java
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake.test;
+
+import static
org.apache.beam.sdk.io.snowflake.test.TestUtils.SnowflakeIOITPipelineOptions;
+import static
org.apache.beam.sdk.io.snowflake.test.TestUtils.getTestRowDataMapper;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets.newHashSet;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.common.TestRow;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import
org.apache.beam.sdk.io.snowflake.credentials.SnowflakeCredentialsFactory;
+import org.apache.beam.sdk.io.snowflake.enums.StreamingLogLevel;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * The test for streaming writing to SnowflakeIO. Because of using {@code
TestStream} it has to be
+ * run with DirectRunner. The test requires a Snowflake's snowpipe name with
the copy into
+ * STREAMING_IOIT table statement.
+ *
+ * Example run:
+ *
+ *
+ * ./gradlew --info -p sdks/java/io/snowflake integrationTest
-DintegrationTestPipelineOptions='[
+ * "--serverName=",
+ * "--username=",
+ * "--privateKeyPath=",
+ * "--privateKeyPassphrase=",
+ * "--database=",
+ * "--schema=",
+ * "--stagingBucketName=",
+ * "--storageIntegrationName=",
+ * "--snowPipe=",
+ * "--numberOfRecords=",
+ * "--runner=DirectRunner"]'
+ * --tests
org.apache.beam.sdk.io.snowflake.test.SnowflakeStreamingIOIT.writeStreamThenRead
+ * -DintegrationTestRunner=direct
+ *
+ */
+public class
[beam] branch master updated (ff20e29 -> fcd767f)
This is an automated email from the ASF dual-hosted git repository. kamilwu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ff20e29 [BEAM-10757] Add enum34 to manual_licenses (#12624) add 34b6ad7 [BEAM-BEAM-10667] Make github actions bulding wheels bucket name stored as a secret add fcd767f Merge pull request #12513: [BEAM-10667] Make gh-actions wheels bucket name stored as a secret No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 6 -- .github/workflows/python_tests.yml | 1 + CI.md| 1 + scripts/ci/ci_check_are_gcp_variables_set.sh | 2 +- 4 files changed, 7 insertions(+), 3 deletions(-)
