[beam] branch master updated (f6ea889 -> 47afc59)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f6ea889 Merge pull request #10696 from aaltay/typesandrels add 47afc59 [BEAM-9186] Allow injection of custom equality function. (#10637) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/testing/util.py | 50 sdks/python/apache_beam/testing/util_test.py | 5 +++ 2 files changed, 33 insertions(+), 22 deletions(-)
[beam] branch master updated: Change Dataflow Python containers
This is an automated email from the ASF dual-hosted git repository. altay pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 22df4a5 Change Dataflow Python containers new f6ea889 Merge pull request #10696 from aaltay/typesandrels 22df4a5 is described below commit 22df4a51ba10fb623ab0d9c318ae194782b99767 Author: Ahmet Altay AuthorDate: Mon Jan 27 14:52:15 2020 -0800 Change Dataflow Python containers --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index e9b34d4..c24c5b0 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -40,10 +40,10 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Update this version to the next version whenever there is a change that will # require changes to legacy Dataflow worker execution environment. -BEAM_CONTAINER_VERSION = 'beam-master-20191220' +BEAM_CONTAINER_VERSION = 'beam-master-20200127' # Update this version to the next version whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20191220' +BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20200127' # TODO(BEAM-5939): Remove these shared names once Dataflow worker is updated. PICKLED_MAIN_SESSION_FILE = 'pickled_main_session'
[beam] branch master updated (0a37f19 -> d1b70d6)
This is an automated email from the ASF dual-hosted git repository. goenka pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0a37f19 Merge pull request #10697 from angoenka/fix_lint add d1b70d6 [BEAM-8626] Implement status fn api handler in python sdk (#10598) No new revisions were added by this update. Summary of changes: .../apache_beam/runners/worker/data_plane_test.py | 26 +--- .../apache_beam/runners/worker/sdk_worker.py | 18 ++- .../apache_beam/runners/worker/sdk_worker_main.py | 29 ++-- .../runners/worker/sdk_worker_main_test.py | 4 +- .../apache_beam/runners/worker/worker_status.py| 148 + .../runners/worker/worker_status_test.py | 92 + sdks/python/apache_beam/testing/util.py| 42 ++ 7 files changed, 313 insertions(+), 46 deletions(-) create mode 100644 sdks/python/apache_beam/runners/worker/worker_status.py create mode 100644 sdks/python/apache_beam/runners/worker/worker_status_test.py
[beam] branch master updated: Fixing Lint
This is an automated email from the ASF dual-hosted git repository. goenka pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 1f17361 Fixing Lint new 0a37f19 Merge pull request #10697 from angoenka/fix_lint 1f17361 is described below commit 1f17361c30b605c737e2cdd4c46368b8e758d6f8 Author: Ankur AuthorDate: Mon Jan 27 16:14:34 2020 -0800 Fixing Lint --- sdks/python/apache_beam/io/gcp/datastore/v1new/types.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py b/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py index 497b76a..ca2cb63 100644 --- a/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py +++ b/sdks/python/apache_beam/io/gcp/datastore/v1new/types.py @@ -44,7 +44,8 @@ class Query(object): Args: kind: (str) The kind to query. project: (str) Required. Project associated with query. - namespace: (str, ValueProvider(str)) (Optional) Namespace to restrict results to. + namespace: (str, ValueProvider(str)) (Optional) Namespace to restrict +results to. ancestor: (:class:`~apache_beam.io.gcp.datastore.v1new.types.Key`) (Optional) key of the ancestor to which this query's results are restricted.
[beam] branch pabloem-patch-2 created (now f47b363)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch pabloem-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git. at f47b363 Fixing lint issue breaking master. No new revisions were added by this update.
[beam] branch release-2.19.0 updated (5286261 -> ae1086c)
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a change to branch release-2.19.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from 5286261 Merge pull request #10695 from boyuanzz/cherry-pick add ff8ce83 [Gradle Release Plugin] - pre tag commit: 'v2.19.0-RC1'. new ae1086c [Gradle Release Plugin] - new version commit: 'v2.19.0-SNAPSHOT'. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes:
[beam] annotated tag v2.19.0-RC1 updated (ff8ce83 -> b658ef9)
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a change to annotated tag v2.19.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag v2.19.0-RC1 was modified! *** from ff8ce83 (commit) to b658ef9 (tag) tagging ff8ce834ff24e847c90872a4ea545f987e3e0a2d (commit) replaces java-ulr-removal by Boyuan Zhang on Mon Jan 27 17:04:49 2020 -0800 - Log - [Gradle Release Plugin] - creating tag: 'v2.19.0-RC1'. --- No new revisions were added by this update. Summary of changes:
[beam] 01/01: [Gradle Release Plugin] - new version commit: 'v2.19.0-SNAPSHOT'.
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a commit to branch release-2.19.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit ae1086c5bc5ea697f9efde0df2d0ec92919bc731 Author: Boyuan Zhang AuthorDate: Mon Jan 27 17:04:49 2020 -0800 [Gradle Release Plugin] - new version commit: 'v2.19.0-SNAPSHOT'. --- gradle.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gradle.properties b/gradle.properties index f474e02..1e485e9 100644 --- a/gradle.properties +++ b/gradle.properties @@ -23,7 +23,7 @@ offlineRepositoryRoot=offline-repository signing.gnupg.executable=gpg signing.gnupg.useLegacyGpg=true -version=2.19.0-RC1 +version=2.19.0-SNAPSHOT python_sdk_version=2.19.0 javaVersion=1.8
[beam] branch asf-site updated: Publishing website 2020/01/28 01:04:05 at commit f68180c
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c6bcd44 Publishing website 2020/01/28 01:04:05 at commit f68180c
c6bcd44 is described below
commit c6bcd4432e0840f0bc50526381ab6cefaac4f36f
Author: jenkins
AuthorDate: Tue Jan 28 01:04:06 2020 +
Publishing website 2020/01/28 01:04:05 at commit f68180c
---
website/generated-content/contribute/release-guide/index.html | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 2a19fea..37e2d65 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1532,7 +1532,7 @@ Thanks everyone!
First update required configurations listed in RC_VALIDATE_CONFIGS in
https://github.com/apache/beam/blob/master/release/src/main/scripts/script.config";>script.config
Then run
- cd beam/release/src/main/scripts &&
./run_rc_validation.sh
+ ./beam/release/src/main/scripts/run_rc_validation.sh
@@ -1583,7 +1583,7 @@ Thanks everyone!
Apex Local Runner
./gradlew :runners:apex:runQuickstartJavaApex \
--Prepourl=https://repository.apache.org/content/repositories/orgapachebeam${KEY}
\
+-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY}
\
-Pver=${RELEASE_VERSION}
Flink Local Runner
@@ -1610,7 +1610,7 @@ Thanks everyone!
Pre-request
Create your own BigQuery dataset
-bq mk --project=${YOUR_GCP_PROJECT} ${YOUR_DATASET}
+bq mk --project_id=${YOUR_GCP_PROJECT} ${YOUR_DATASET}
Create yout PubSub topic
@@ -1686,7 +1686,7 @@ pip install dist/apache-beam-2.5.0.tar.gz[gcp]
Please repeat following steps for every following test.
bq rm -rf --project=${YOUR_PROJECT} ${USER}_test
-bq mk --project=${YOUR_PROJECT} ${USER}_test
+bq mk --project_id=${YOUR_PROJECT} ${USER}_test
gsutil rm -rf ${YOUR_GS_STORAGE]
gsutil mb -p ${YOUR_PROJECT} ${YOUR_GS_STORAGE}
gcloud alpha pubsub topics create --project=${YOUR_PROJECT}
${YOUR_PUBSUB_TOPIC}
[beam] branch master updated (fb5e0e6 -> f68180c)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fb5e0e6 Merge pull request #10673 from ibzib/flink-doc add f68180c [BEAM-9201] Release scripts fixes: run_rc_validation.sh, verify_release_build.sh (#10609) No new revisions were added by this update. Summary of changes: release/src/main/scripts/run_rc_validation.sh| 12 ++-- release/src/main/scripts/verify_release_build.sh | 6 -- website/src/contribute/release-guide.md | 8 3 files changed, 14 insertions(+), 12 deletions(-)
svn commit: r37767 - /dev/beam/2.19.0/python/
Author: boyuanz Date: Tue Jan 28 00:42:12 2020 New Revision: 37767 Log: Staging python wheels for 2.19.0 release RC1 Added: dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-manylinux1_i686.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-manylinux1_i686.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-manylinux1_i686.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-manylinux1_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-manylinux1_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-manylinux1_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27mu-manylinux1_i686.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27mu-manylinux1_i686.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27mu-manylinux1_i686.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27mu-manylinux1_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27mu-manylinux1_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27mu-manylinux1_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-manylinux1_i686.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-manylinux1_i686.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-manylinux1_i686.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-manylinux1_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-manylinux1_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp35-cp35m-manylinux1_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-manylinux1_i686.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-manylinux1_i686.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-manylinux1_i686.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-manylinux1_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-manylinux1_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp36-cp36m-manylinux1_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-manylinux1_i686.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-manylinux1_i686.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-manylinux1_i686.whl.sha512 dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-manylinux1_x86_64.whl (with props) dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-manylinux1_x86_64.whl.asc dev/beam/2.19.0/python/apache_beam-2.19.0-cp37-cp37m-manylinux1_x86_64.whl.sha512 Added: dev/beam/2.19.0/python/apache_beam-2.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl == Binary file - no diff
[beam-wheels] branch release-2.19.0 created (now bafcd5d)
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a change to branch release-2.19.0 in repository https://gitbox.apache.org/repos/asf/beam-wheels.git. at bafcd5d Update user guide with helpful links (#14) No new revisions were added by this update.
[beam] branch asf-site updated: Publishing website 2020/01/27 23:13:33 at commit fb5e0e6
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 1116d93 Publishing website 2020/01/27 23:13:33 at commit fb5e0e6 1116d93 is described below commit 1116d935abe6bfa3d8ac3b82b92c9e701a20b258 Author: jenkins AuthorDate: Mon Jan 27 23:13:34 2020 + Publishing website 2020/01/27 23:13:33 at commit fb5e0e6 --- .../documentation/runners/flink/index.html | 31 +++--- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/website/generated-content/documentation/runners/flink/index.html b/website/generated-content/documentation/runners/flink/index.html index 1e3081e..56b3f83 100644 --- a/website/generated-content/documentation/runners/flink/index.html +++ b/website/generated-content/documentation/runners/flink/index.html @@ -322,7 +322,7 @@ period. Artifact Id - 2.17.0 + 2.17.0-2.18.0 1.9.x beam-runners-flink-1.9 @@ -481,27 +481,30 @@ If you have a Flink JobManager running on flinkMaster. Otherwise an embedded Flink cluster will be started for the job. - -As of now you will need a copy of Apache Beam’s source code. You can -download it on the Downloads page. + + +Starting with Beam 2.18.0, pre-built Docker images are available at Docker Hub. -Pre-built Docker images are available at Docker-Hub: +JobService: +https://hub.docker.com/r/apachebeam/flink1.7_job_server";>Flink 1.7, +https://hub.docker.com/r/apachebeam/flink1.8_job_server";>Flink 1.8, +https://hub.docker.com/r/apachebeam/flink1.9_job_server";>Flink 1.9. + +Beam SDK: https://hub.docker.com/r/apachebeam/python2.7_sdk";>Python 2.7, https://hub.docker.com/r/apachebeam/python3.5_sdk";>Python 3.5, https://hub.docker.com/r/apachebeam/python3.6_sdk";>Python 3.6, https://hub.docker.com/r/apachebeam/python3.7_sdk";>Python 3.7. To run a pipeline on an embedded Flink cluster: -+ -1. Start the JobService endpoint: ./gradlew :runners:flink:1.9:job-server:runShadow +1. Start the JobService endpoint: docker run --net=host apachebeam/flink1.9_job_server:latest The JobService is the central instance where you submit your Beam pipeline to. The JobService will create a Flink job for the pipeline and execute the job. -To execute the job on a Flink cluster, the Beam JobService needs to be -provided with the Flink JobManager address. 2. Submit the Python pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService), and environment_type set to LOOPBACK. For example: @@ -520,13 +523,13 @@ provided with the Flink JobManager address. -To run on a separate https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html";>Flink cluster: +To run on a separate https://ci.apache.org/projects/flink/flink-docs-release-1.9/getting-started/tutorials/local_setup.html";>Flink cluster: 1. Start a Flink cluster which exposes the Rest interface on localhost:8081 by default. -2. Start JobService with Flink Rest endpoint: ./gradlew :runners:flink:1.9:job-server:runShadow -PflinkMasterUrl=localhost:8081. +2. Start JobService with Flink Rest endpoint: docker run --net=host apachebeam/flink1.9_job_server:latest --flink-master=localhost:8081. 3. Submit the pipeline as above. @@ -535,7 +538,7 @@ See here for details. Steps 2 and 3 can be automated in Python by using the FlinkRunner, -plus the optional flink_version and flink_master* options, e.g.: +plus the optional flink_version and flink_master options, e.g.: import apache_beam as beam @@ -543,7 +546,7 @@ plus the optional flink_version and options = PipelineOptions([ "--runner=FlinkRunner", -"--flink_version=1.8", +"--flink_version=1.9", "--flink_master=localhost:8081", "--environment_type=LOOPBACK" ]) @@ -551,8 +554,6 @@ plus the optional flink_version and ... -* Note: For Beam versions < 2.17.0, use flink_master_url instead of flink_master. - Additional information and caveats Monitoring your job
[beam] branch master updated (5bea20c -> fb5e0e6)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5bea20c [BEAM-9184] Add ToSet combiner (#10636) new 8716747 [DO NOT MERGE][BEAM-9177] Update Flink runner webpage for 2.18 new d233125 Update Beam version chart new fb5e0e6 Merge pull request #10673 from ibzib/flink-doc The 25290 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/documentation/runners/flink.md | 29 +++-- 1 file changed, 15 insertions(+), 14 deletions(-)
[beam] branch master updated (50557c8 -> 5bea20c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 50557c8 Merge pull request #10691: [website] Add warning on Beam 2.18.0 blog post for Avro 1.9.0 users add 5bea20c [BEAM-9184] Add ToSet combiner (#10636) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/combiners.py| 31 ++ .../apache_beam/transforms/combiners_test.py | 12 + 2 files changed, 43 insertions(+)
[beam] branch asf-site updated: Publishing website 2020/01/27 23:04:00 at commit 50557c8
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 392d7b6 Publishing website 2020/01/27 23:04:00 at commit 50557c8 392d7b6 is described below commit 392d7b6f6cfedf95ec35e6ec1948095b9c644390 Author: jenkins AuthorDate: Mon Jan 27 23:04:01 2020 + Publishing website 2020/01/27 23:04:00 at commit 50557c8 --- website/generated-content/blog/2020/01/13/beam-2.18.0.html | 1 + website/generated-content/feed.xml | 1 + 2 files changed, 2 insertions(+) diff --git a/website/generated-content/blog/2020/01/13/beam-2.18.0.html b/website/generated-content/blog/2020/01/13/beam-2.18.0.html index cdbc5a1..ad6881b 100644 --- a/website/generated-content/blog/2020/01/13/beam-2.18.0.html +++ b/website/generated-content/blog/2020/01/13/beam-2.18.0.html @@ -283,6 +283,7 @@ For more information on changes in 2.18.0, check out the https://issues.apache.org/jira/browse/BEAM-9065";>BEAM-9065 - Spark runner accumulates metrics (incorrectly) between runs https://issues.apache.org/jira/browse/BEAM-9123";>BEAM-9123 - HadoopResourceId returns wrong directory name See a full list of open https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.18.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC";>issues that affect this version. + https://issues.apache.org/jira/browse/BEAM-9144";>BEAM-9144 - If you are using Avro 1.9.x with Beam you should not upgrade to this version. There is an issue with timestamp conversions. A fix will be available in the next release. List of Contributors diff --git a/website/generated-content/feed.xml b/website/generated-content/feed.xml index 681763c..131b3d9 100644 --- a/website/generated-content/feed.xml +++ b/website/generated-content/feed.xml @@ -127,6 +127,7 @@ For more information on changes in 2.18.0, check out theBEAM-9065; - Spark runner accumulates metrics (incorrectly) between runs BEAM-9123; - HadoopResourceId returns wrong directory name See a full list of open issues that affect this version. +BEAM-9144; - If you are using Avro 1.9.x with Beam you should not upgrade to this version. There is an issue with timestamp conversions. A fix will be available in the next release. List of Contributors
[beam] branch master updated (b02a325 -> 50557c8)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b02a325 Merge pull request #10618: [BEAM-8630] Turn on BeamZetaSqlCalcRel add fdf7edb [website] Add warning on Beam 2.18.0 blog post for Avro 1.9.0 users add 50557c8 Merge pull request #10691: [website] Add warning on Beam 2.18.0 blog post for Avro 1.9.0 users No new revisions were added by this update. Summary of changes: website/src/_posts/2020-01-13-beam-2.18.0.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (527d153 -> b02a325)
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 527d153 [BEAM-9183, BEAM-9026] Initialize and cleanup the state of RuntimeValueProvider in tests that modify it. (#10688) add 34e5a1e Turn on BeamZetaSqlCalcRel add b02a325 Merge pull request #10618: [BEAM-8630] Turn on BeamZetaSqlCalcRel No new revisions were added by this update. Summary of changes: .../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 4 +-- .../sdk/extensions/sql/zetasql/SqlOperators.java | 16 .../beam/sdk/extensions/sql/zetasql/TestInput.java | 8 +++--- .../sql/zetasql/ZetaSQLQueryPlanner.java | 11 +++- .../translation/ArrayScanToJoinConverter.java | 4 +-- .../translation/ArrayScanToUncollectConverter.java | 3 +-- .../zetasql/translation/ExpressionConverter.java | 5 ++-- .../sql/zetasql/translation/RelConverter.java | 30 -- .../translation/SingleRowScanConverter.java| 27 --- .../sql/zetasql/ZetaSQLDialectSpecTest.java| 9 +++ 10 files changed, 66 insertions(+), 51 deletions(-)
svn commit: r37762 - in /dev/beam/2.19.0/python: ./ apache-beam-2.19.0.zip apache-beam-2.19.0.zip.asc apache-beam-2.19.0.zip.sha512
Author: boyuanz Date: Mon Jan 27 22:46:47 2020 New Revision: 37762 Log: Staging python 2.19.0 RC1 artifacts Added: dev/beam/2.19.0/python/ dev/beam/2.19.0/python/apache-beam-2.19.0.zip (with props) dev/beam/2.19.0/python/apache-beam-2.19.0.zip.asc dev/beam/2.19.0/python/apache-beam-2.19.0.zip.sha512 Added: dev/beam/2.19.0/python/apache-beam-2.19.0.zip == Binary file - no diff available. Propchange: dev/beam/2.19.0/python/apache-beam-2.19.0.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.19.0/python/apache-beam-2.19.0.zip.asc == --- dev/beam/2.19.0/python/apache-beam-2.19.0.zip.asc (added) +++ dev/beam/2.19.0/python/apache-beam-2.19.0.zip.asc Mon Jan 27 22:46:47 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEE1r/BFeqmmChaiTFlmd84Xah3xZYFAl4vaA0ACgkQmd84Xah3 +xZZk9w/+LWmLmSROUQorES8k58cxPHBOY2oTO5ieLsp1HffE9WqMbQ7BZsuJ3BEE +J56HS9OL4wO+AgG+zHt4j7HEZtVsghLZQ3b/0fvGYrHxJMbwelfcP+kDP13D+USi +KiqMhsHj9MsuHr10tajnGh91VnBqN9s5fh4UcxxYNCzOEenItAUVWuJYUM7kApiM +CI4NtztaIHmIvj0pj8nrku54vA/sVfDImYZL6t+Ww3RKyjS7lt2aaLp5ng/wNcrL +PTirZkXcV/dPiCoIQf1z6vxGYyVstUQDFie2rdkn7561zomez/6e37ZcunavsB0S +1bGPJagUkIHUWG9IB+B1cqimL8WvxivXx9YV1lBu0euoyTkMCqLd0ZYfIkDh2R82 +ZD+kBwtNd+lC7nRCaCjAuVmXzBhvVB+FsPEoczhr+bUslRPIa0LVaJDqbFZ+Fqhd +EfOORIYtwLr7OlBzPjurI6/THboys6MSvNo/jxPRVhmB3awgqEGNNWoB+3lma99a +1wX+ejDxzV5eP7n16UPQdXflIg/6XrfkDh5O/ZY7KstSC/HlMt1vzD1+/3lH5Yjg +6wpCfAK2C74w/QDVzGlUlO0BRvRpiote0BeC/wqnzrDEpCNqRVvTGGUvUVp5pR3h +aDFOMgeC29NC/tTmAwn+5w1f7OKI4wwX1hf1obm/bcR4NAHKtVc= +=yhbX +-END PGP SIGNATURE- Added: dev/beam/2.19.0/python/apache-beam-2.19.0.zip.sha512 == --- dev/beam/2.19.0/python/apache-beam-2.19.0.zip.sha512 (added) +++ dev/beam/2.19.0/python/apache-beam-2.19.0.zip.sha512 Mon Jan 27 22:46:47 2020 @@ -0,0 +1 @@ +23dd8c27ba78f146b1790be3310d4540f8c687def71f159b2ff03a42f46c80e9d6b88dfe545be757ed22e150ae92f47bbe80300c202b62a047d6f0a31f6bb150 apache-beam-2.19.0.zip
[beam] branch aaltay-patch-2 updated (b93380d -> 41504d7)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch aaltay-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git. from b93380d Change Dataflow Python containers at head add 41504d7 Change container names No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
svn commit: r37761 - in /dev/beam/2.19.0: ./ apache-beam-2.19.0-source-release.zip apache-beam-2.19.0-source-release.zip.asc apache-beam-2.19.0-source-release.zip.sha512
Author: boyuanz Date: Mon Jan 27 22:21:12 2020 New Revision: 37761 Log: Staging 2.19.0 Release RC1 Added: dev/beam/2.19.0/ dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip (with props) dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.asc dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.sha512 Added: dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip == Binary file - no diff available. Propchange: dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.asc == --- dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.asc (added) +++ dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.asc Mon Jan 27 22:21:12 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEE1r/BFeqmmChaiTFlmd84Xah3xZYFAl4vYi4ACgkQmd84Xah3 +xZYLihAAswrEVwD+RejjfK6uXosnBni+pG4ZfqO2NgN8wQhbhZ7d1Ua8gii8bpSU +9DbHZS4f1nt5fI3E6Yf87XLUqbh6P8NaR2/plVHi95ciLm2G9aXQhgOFaaiKskMg +5m4BnIFRkzk8ajxW7Pb+Cu9hBb/Wap+CCu/m21FeGOMUaeKmTNOJULyOL6vidAxH +gW3dla/pu8QbTXRqMKEqrX3tIzWmSz3Atb3jaMaQxyBt/6qP0fYK9j7HZmgwMgRR +sOzrHNm9PyUaaZQAzHpBR8vKzMrynz5JXA6PD3V0ssyO90vQY25aRdd4Ovgso3ra +7iMZ8ea9gyjq5n85aQNpGRmhxSSSwntRbn1meubq9bdfRgj99GBoarjn8PyzaY6v +wAogmk/mSXbMr9C9yYlnuBAWvHkpYnNVbDRXmNGRNf1yJrpwrXVh3w1hbD+hlptu +0tMXA2KmGswmx89x679n9DWmRsdw1aF9Z3qrHGTbVUuypQzjAwKBVbo2sEi2cbIi +gGbrzzMz3FCJ25Ul3OKfMx74aar7H3QMkFDPRGDOrmiSQorCqFiugG8AjsVF5Usm +Fs/3pSwH8EnbgHdkz2jfNLuNEosNMNnvtB5FcPwDWkpgbWF0wvWwz0Vv/nj96RPk +r84QRJUM/zyEq/uaipKXze373df862fPP/MVtoi1VIKdLqQU50I= +=oAOk +-END PGP SIGNATURE- Added: dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.sha512 == --- dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.sha512 (added) +++ dev/beam/2.19.0/apache-beam-2.19.0-source-release.zip.sha512 Mon Jan 27 22:21:12 2020 @@ -0,0 +1 @@ +b78cf79ce34b3e7a7128722fcd6f99a7b39541db6f42f099270fd9bedaddf2075677d509eba0424b8987a8d8f2cd89d98528bce2e56c89fb507689203e46373f apache-beam-2.19.0-source-release.zip
[beam] branch release-2.19.0 updated: [BEAM-8939] Cherry-pick: A bash script that cancels stale dataflow jobs
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch release-2.19.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.19.0 by this push:
new 7e2a165 [BEAM-8939] Cherry-pick: A bash script that cancels stale
dataflow jobs
new 5286261 Merge pull request #10695 from boyuanzz/cherry-pick
7e2a165 is described below
commit 7e2a16533cda9151861a26f716a59c6994491e0c
Author: Kamil Wasilewski
AuthorDate: Tue Jan 14 09:33:45 2020 +0100
[BEAM-8939] Cherry-pick: A bash script that cancels stale dataflow jobs
(cherry picked from commit 4d3295c4279a853758e4218b437a92edba63bd89)
---
.../jenkins/job_CancelStaleDataflowJobs.groovy | 1 -
.test-infra/tools/build.gradle | 25 +
.test-infra/tools/stale_dataflow_jobs_cleaner.go | 117 -
.test-infra/tools/stale_dataflow_jobs_cleaner.sh | 23
.../tools/stale_dataflow_jobs_cleaner_test.go | 74 -
5 files changed, 25 insertions(+), 215 deletions(-)
diff --git a/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
b/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
index a03a1d0..e32d14a 100644
--- a/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
+++ b/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
@@ -37,7 +37,6 @@ job("beam_CancelStaleDataflowJobs") {
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':beam-test-tools:check')
tasks(':beam-test-tools:cancelStaleDataflowJobs')
commonJobProperties.setGradleSwitches(delegate)
}
diff --git a/.test-infra/tools/build.gradle b/.test-infra/tools/build.gradle
index aabeca0..53445b6 100644
--- a/.test-infra/tools/build.gradle
+++ b/.test-infra/tools/build.gradle
@@ -16,27 +16,6 @@
* limitations under the License.
*/
-plugins {
- id 'org.apache.beam.module'
-}
-
-applyGoNature()
-
-repositories { mavenCentral() }
-
-clean {
- delete '.gogradle'
-}
-
-golang {
- packagePath = 'github.com/apache/beam/.test-infra/tools'
-}
-
-check.dependsOn goTest
-
-task cancelStaleDataflowJobs(type: com.github.blindpirate.gogradle.Go) {
- dependsOn goVendor
- go('get golang.org/x/oauth2/google')
- go('get google.golang.org/api/dataflow/v1b3')
- go('run stale_dataflow_jobs_cleaner.go')
+task cancelStaleDataflowJobs(type: Exec) {
+ commandLine './stale_dataflow_jobs_cleaner.sh'
}
diff --git a/.test-infra/tools/stale_dataflow_jobs_cleaner.go
b/.test-infra/tools/stale_dataflow_jobs_cleaner.go
deleted file mode 100644
index 6361e27..000
--- a/.test-infra/tools/stale_dataflow_jobs_cleaner.go
+++ /dev/null
@@ -1,117 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package main
-
-import (
- "context"
- "log"
- "strings"
- "time"
-
- "golang.org/x/oauth2/google"
- df "google.golang.org/api/dataflow/v1b3"
-)
-
-const (
- longRunningPrefix = "long-running-"
-)
-
-// client contains methods for listing and cancelling jobs, extracted to allow
easier testing.
-type client interface {
- CurrentTime() time.Time
- ListJobs(projectId string) ([]*df.Job, error)
- CancelJob(job *df.Job) error
-}
-
-// dataflowClient implements the client interface for Google Cloud Dataflow.
-type dataflowClient struct {
- s *df.ProjectsJobsService
-}
-
-// newDataflowClient creates a new Dataflow ProjectsJobsService.
-func newDataflowClient() (*dataflowClient, error) {
- ctx := context.Background()
- cl, err := google.DefaultClient(ctx, df.CloudPlatformScope)
- if err != nil {
- return nil, err
- }
- service, err := df.New(cl)
- if err != nil {
- return nil, err
- }
- return &dataflowClient{s: df.NewProjectsJobsService(service)}, nil
-}
-
-// CurrentTime gets the time Now.
-func (c dataflowClient) CurrentTime() time.Time {
- return time.Now()
-}
-
-// ListJobs lists the active Dataflow jobs for a project.
-func (c dataflowClient) ListJobs(projectId string) ([]*df.Job, error) {
- resp, err :=
c.s.Aggregated(projectId).Filter("ACTIVE").Fields("j
[beam] branch release-2.19.0 updated: [BEAM-9065] Reset MetricsContainerStepMapAccumulator upon initialization of MetricsAccumulator singleton because metrics need to be reset when several pipelines a
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch release-2.19.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.19.0 by this push:
new 3f07189 [BEAM-9065] Reset MetricsContainerStepMapAccumulator upon
initialization of MetricsAccumulator singleton because metrics need to be reset
when several pipelines are run inside the same JVM
new 85417f1 Merge pull request #10670 from
echauchot/release-2.19.0-patched
3f07189 is described below
commit 3f07189a2a01dbc9c39282c1889d17e24f7f3f06
Author: Etienne Chauchot
AuthorDate: Wed Jan 8 11:36:36 2020 +0100
[BEAM-9065] Reset MetricsContainerStepMapAccumulator upon initialization of
MetricsAccumulator singleton because metrics need to be reset when several
pipelines are run inside the same JVM
(cherry picked from commit fd3da1e68f18560954f9c736146c7565e4a1af0f)
---
.../java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java | 2 ++
.../runners/spark/structuredstreaming/metrics/MetricsAccumulator.java | 2 ++
2 files changed, 4 insertions(+)
diff --git
a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
index 2d83c3c..8313d73 100644
---
a/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
+++
b/runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/MetricsAccumulator.java
@@ -74,6 +74,8 @@ public class MetricsAccumulator {
}
}
LOG.info("Instantiated metrics accumulator: " + instance.value());
+} else {
+ instance.reset();
}
}
diff --git
a/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java
b/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java
index f674ef9..e182cf0 100644
---
a/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java
+++
b/runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java
@@ -51,6 +51,8 @@ public class MetricsAccumulator {
}
}
LOG.info("Instantiated metrics accumulator: " + instance.value());
+} else {
+ instance.reset();
}
}
[beam] branch master updated (1753b43 -> 527d153)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1753b43 Merge pull request #10543: [BEAM-8941] Implement simple DSL for load tests add 527d153 [BEAM-9183, BEAM-9026] Initialize and cleanup the state of RuntimeValueProvider in tests that modify it. (#10688) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/internal/gcp/json_value_test.py | 4 +++- sdks/python/apache_beam/io/filebasedsource_test.py | 2 ++ sdks/python/apache_beam/options/pipeline_options_test.py | 10 -- sdks/python/apache_beam/options/value_provider_test.py | 7 ++- 4 files changed, 19 insertions(+), 4 deletions(-)
[beam] branch master updated (31a5a33 -> 1753b43)
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 31a5a33 Merge pull request #10683 from
EDjur/BEAM-7810/valueprovider-namespace
add 801a6bb [BEAM-8941] Implement simple DSL for load tests
add 1753b43 Merge pull request #10543: [BEAM-8941] Implement simple DSL
for load tests
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/LoadTestConfig.groovy | 622 +
.../job_LoadTests_Combine_Java_Smoke.groovy| 89 +++
groovy => job_LoadTests_GBK_Java_Smoke.groovy} | 6 +-
3 files changed, 714 insertions(+), 3 deletions(-)
create mode 100644 .test-infra/jenkins/LoadTestConfig.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_Combine_Java_Smoke.groovy
rename .test-infra/jenkins/{job_LoadTests_Java_Smoke.groovy =>
job_LoadTests_GBK_Java_Smoke.groovy} (97%)
