[beam] 01/01: [BEAM-7218] Backport of #7487
This is an automated email from the ASF dual-hosted git repository. udim pushed a commit to branch pytz-upper-bound in repository https://gitbox.apache.org/repos/asf/beam.git commit ed17beea89c7501e64accbd9466a969189462b09 Author: Udi Meiri AuthorDate: Wed Jun 19 16:35:31 2019 -0700 [BEAM-7218] Backport of #7487 --- sdks/python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/setup.py b/sdks/python/setup.py index ce7591e..d4e5010 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -113,7 +113,7 @@ REQUIRED_PACKAGES = [ # grpcio 1.8.1 and above requires protobuf 3.5.0.post1. 'protobuf>=3.5.0.post1,<4', 'pydot>=1.2.0,<1.3', -'pytz>=2018.3,<=2018.4', +'pytz>=2018.3', 'pyyaml>=3.12,<4.0.0', 'pyvcf>=0.6.8,<0.7.0', 'six>=1.9,<1.12',
[beam] branch pytz-upper-bound created (now ed17bee)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch pytz-upper-bound in repository https://gitbox.apache.org/repos/asf/beam.git. at ed17bee [BEAM-7218] Backport of #7487 This branch includes the following new commits: new ed17bee [BEAM-7218] Backport of #7487 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Create release branch for version 2.14.0.
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch release-2.14.0
in repository https://gitbox.apache.org/repos/asf/beam.git
commit b4704ee530c7f769e1f306c2dec3f0cbe6c5bfd1
Author: Anton Kedin
AuthorDate: Wed Jun 19 16:02:10 2019 -0700
Create release branch for version 2.14.0.
---
runners/google-cloud-dataflow-java/build.gradle | 2 +-
sdks/python/apache_beam/version.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index 35ea595..8e0798a 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -39,7 +39,7 @@ processResources {
filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
'dataflow.legacy_environment_major_version' : '7',
'dataflow.fnapi_environment_major_version' : '7',
-'dataflow.container_version' : 'beam-master-20190415'
+'dataflow.container_version' : 'beam-2.14.0'
]
}
diff --git a/sdks/python/apache_beam/version.py
b/sdks/python/apache_beam/version.py
index 1688685..c9fa3b8 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.15.0.dev'
+__version__ = '2.14.0'
[beam] branch release-2.14.0 created (now b4704ee)
This is an automated email from the ASF dual-hosted git repository. anton pushed a change to branch release-2.14.0 in repository https://gitbox.apache.org/repos/asf/beam.git. at b4704ee Create release branch for version 2.14.0. This branch includes the following new commits: new b4704ee Create release branch for version 2.14.0. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated: [BEAM-7513] Adding RowCount to BigQueryTable.
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e868b80 [BEAM-7513] Adding RowCount to BigQueryTable.
new b2fd4e3 Merge pull request #8892 from riazela/bigquery-rowcount
e868b80 is described below
commit e868b8043287cece938a1864066f33ec33f813e8
Author: Alireza Samadian
AuthorDate: Wed Jun 19 09:36:19 2019 -0700
[BEAM-7513] Adding RowCount to BigQueryTable.
---
.../apache/beam/sdk/extensions/sql/BeamSqlCli.java | 7 +-
.../beam/sdk/extensions/sql/BeamSqlTable.java | 7 +
.../beam/sdk/extensions/sql/SqlTransform.java | 2 +
.../sdk/extensions/sql/impl/BeamCalciteSchema.java | 5 +-
.../sql/impl/BeamCalciteSchemaFactory.java | 12 +-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 49 ++-
.../BeamRowCountStatistics.java} | 37 +++--
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 26 +++-
.../sdk/extensions/sql/impl/JdbcConnection.java| 9 ++
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 6 +-
.../sql/meta/provider/bigquery/BigQueryTable.java | 39 +
.../sdk/extensions/sql/impl/BeamSqlEnvTest.java| 7 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java| 31 ++--
.../extensions/sql/impl/parser/BeamDDLTest.java| 7 +-
...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 7 +-
.../meta/provider/bigquery/BigQueryRowCountIT.java | 161 +
.../meta/provider/bigquery/BigQueryTestTable.java | 45 ++
.../bigquery/BigQueryTestTableProvider.java| 71 +
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 19 +++
20 files changed, 498 insertions(+), 51 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
index 5e44c6c..8bdb1bf 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlCli.java
@@ -25,6 +25,7 @@ import
org.apache.beam.sdk.extensions.sql.impl.rel.BeamEnumerableConverter;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils;
import org.apache.beam.sdk.extensions.sql.meta.store.MetaStore;
import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
/** {@link BeamSqlCli} provides methods to execute Beam SQL with an
interactive client. */
@Experimental
@@ -34,15 +35,17 @@ public class BeamSqlCli {
private MetaStore metaStore;
public BeamSqlCli metaStore(MetaStore metaStore) {
-return metaStore(metaStore, false);
+return metaStore(metaStore, false, PipelineOptionsFactory.create());
}
- public BeamSqlCli metaStore(MetaStore metaStore, boolean autoLoadUdfUdaf) {
+ public BeamSqlCli metaStore(
+ MetaStore metaStore, boolean autoLoadUdfUdaf, PipelineOptions
pipelineOptions) {
this.metaStore = metaStore;
BeamSqlEnv.BeamSqlEnvBuilder builder = BeamSqlEnv.builder(metaStore);
if (autoLoadUdfUdaf) {
builder.autoLoadUserDefinedFunctions();
}
+builder.setPipelineOptions(pipelineOptions);
this.env = builder.build();
return this;
}
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
index 14f1b80..63f7158 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.sdk.extensions.sql;
+import org.apache.beam.sdk.extensions.sql.impl.BeamRowCountStatistics;
+import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
@@ -36,4 +38,9 @@ public interface BeamSqlTable {
/** Get the schema info of the table. */
Schema getSchema();
+
+ /** Estimates the number of rows or returns null if there is no estimation.
*/
+ default BeamRowCountStatistics getRowCount(PipelineOptions options) {
+return BeamRowCountStatistics.UNKNOWN;
+ }
}
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
index e45daca..afa4438 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlTransform.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/
[beam] branch master updated: Moving to 2.15.0-SNAPSHOT on master branch.
This is an automated email from the ASF dual-hosted git repository.
anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6d21877 Moving to 2.15.0-SNAPSHOT on master branch.
6d21877 is described below
commit 6d218777a7bef0679128db2fec5ec257987f0890
Author: Anton Kedin
AuthorDate: Wed Jun 19 15:27:20 2019 -0700
Moving to 2.15.0-SNAPSHOT on master branch.
---
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
gradle.properties | 2 +-
sdks/python/apache_beam/version.py | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 699344f..66f8654 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -314,7 +314,7 @@ class BeamModulePlugin implements Plugin {
// Automatically use the official release version if we are performing a
release
// otherwise append '-SNAPSHOT'
-project.version = '2.14.0'
+project.version = '2.15.0'
if (!isRelease(project)) {
project.version += '-SNAPSHOT'
}
diff --git a/gradle.properties b/gradle.properties
index f0ed4b4..299a616 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -23,4 +23,4 @@ offlineRepositoryRoot=offline-repository
signing.gnupg.executable=gpg
signing.gnupg.useLegacyGpg=true
-version=2.14.0-SNAPSHOT
+version=2.15.0-SNAPSHOT
diff --git a/sdks/python/apache_beam/version.py
b/sdks/python/apache_beam/version.py
index d4053e6..1688685 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.14.0.dev'
+__version__ = '2.15.0.dev'
svn commit: r34584 - /dev/beam/KEYS
Author: kenn Date: Wed Jun 19 22:08:53 2019 New Revision: 34584 Log: Fix metadata for an...@apache.org dev key Modified: dev/beam/KEYS Modified: dev/beam/KEYS == --- dev/beam/KEYS (original) +++ dev/beam/KEYS Wed Jun 19 22:08:53 2019 @@ -1050,7 +1050,9 @@ VJQ2jFnQsSa2a5Kg5L9LxsgsA2q5tLprKq6jg+Tc pub rsa4096 2019-06-19 [SC] 89E2FFCAE7E99CF6E6827CFEF7349F2310FFB193 uid [ultimate] Anton Kedin (CODE SIGNING KEY) +sig 3F7349F2310FFB193 2019-06-19 Anton Kedin (CODE SIGNING KEY) sub rsa4096 2019-06-19 [E] +sig F7349F2310FFB193 2019-06-19 Anton Kedin (CODE SIGNING KEY) -BEGIN PGP PUBLIC KEY BLOCK- mQINBF0KpmkBEADmcavSGmPp8+eFXle+wUHElIz9S8VphY1VlM+JJYkaDy7jmEkX
svn commit: r34583 - /release/beam/KEYS
Author: kenn Date: Wed Jun 19 22:08:17 2019 New Revision: 34583 Log: Add an...@apache.org to Beam release KEYS Modified: release/beam/KEYS Modified: release/beam/KEYS == --- release/beam/KEYS (original) +++ release/beam/KEYS Wed Jun 19 22:08:17 2019 @@ -1047,3 +1047,61 @@ ClRre5N12PPAiMPMIAhlV06wnBGN9+/QT7j1Mj0D VJQ2jFnQsSa2a5Kg5L9LxsgsA2q5tLprKq6jg+TccWHMXYRVgWibZyusUWU= =WFak -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2019-06-19 [SC] + 89E2FFCAE7E99CF6E6827CFEF7349F2310FFB193 +uid [ultimate] Anton Kedin (CODE SIGNING KEY) +sig 3F7349F2310FFB193 2019-06-19 Anton Kedin (CODE SIGNING KEY) +sub rsa4096 2019-06-19 [E] +sig F7349F2310FFB193 2019-06-19 Anton Kedin (CODE SIGNING KEY) +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF0KpmkBEADmcavSGmPp8+eFXle+wUHElIz9S8VphY1VlM+JJYkaDy7jmEkX +OG2WFSFM7xj3SBfbgHoKte+pGAyt83jCptMDB2x9RjfmAUyiB438uLww+BKFKscV +TH6WnKwoV4Erjpq7qjB32GzZWIUtcqrZ33PpwENUdbl9gzH+uo/fhBP5l7asCe0R +jGx09ai/jifQP84o4QprsNu3Prud6lU3ihXkblO35GerOGL6h+L7xKA8GBgw8vjf +hFrMeA0b6bth/cTF0bdEWhLKyp0Sqn4ybZXP4yxrOdsCHOq/qmBq1Jwzm5zW21FR +4xlglCdFn0GdYuDOxO0EK98KV4EAulZB3Ld0LzoblD0DERLo6GUvz4gQ2Xbs7G8a +I8/ODed0dqqBDccrWuXrUsMxvQ6lwdxhjOwW6eWCOSVLsPGGSP3+ZDKwYc7XeR6+ +5N0ts/dR0J/1IT7R2Iw93OXMfOmb4Zb8gUShkFYgHhFNyN3qZ4ZUrBJx/TWQYaX+ +9xV7SbZzMX1oXDUXEQFpTGW811Eicyd+9pZIG3cNTymz7bUe2zStDiThnqRout5l +wWg97hGWHrEgYwa9I+xIkaD9Eo1gA/IUgf7XQN4EMmK3ZVbz8up7STenCk9HidCb +kqAK42CWQCAFrbpAYY43+VWylLTdHIQQgTY08rTrgc3PxwIhwoEt2mHyWQARAQAB +tDFBbnRvbiBLZWRpbiAoQ09ERSBTSUdOSU5HIEtFWSkgPGFudG9uQGFwYWNoZS5v +cmc+iQJOBBMBCgA4FiEEieL/yufpnPbmgnz+9zSfIxD/sZMFAl0KpmkCGwMFCwkI +BwIGFQoJCAsCBBYCAwECHgECF4AACgkQ9zSfIxD/sZOW8RAApetCz0jJ6nw+1m2b +BjgmZVSC/gu4wUsdAYKxTN5baXX4W+aiJk04Gn32ErsiZZwXIrzqWogIbu6mIXj4 +vxqKZuOv48fsNVkwfPiP9p/m0B9l+HxJl+1klZWCqjuxTy3j4Fk7fjPPuFD+iS6c +i4PFOvazo0rtbfetsw9FuPR9uRNP7pbOqOEyZF+kUk9BkffwsUf9LYPKBJktfBNy +1bSmmZ5KqsGUp0BFgR6KVCvyTFgKQDluDcIPTUBsNJxM4Mui9L6lS9v/kOF/YI+y +HsJVUwIeFRsmsKgSVam7VTZFbSX1EhQYqPHUQtbjCZ2JOyYRPiXKzYQ32MAY/RTw +lpFvtjnmqgv0lmAdsRSKeDnju1mlsONzejhrOFOsYc2JyWUc94qzOe7VpeTYKwPX +gxFcr/SX+jgreSIFl7pTcLXyZ2olsiXNoCGzIBgXZDG2ow0CVNoa2jPOkpZkws5t +2rVa2zaEPtwcyjIExZPrhmn7dmpx9zBhmuk+PEPLjE7kXRWIVvfYm1VJmjPNWMv7 +HJp5nx2WDK74ULOFGVpDvJf6hA1F3LO7QWyZw8tOFAMYsOmP1VLNk/TIarptKn3p +immrMsgM60eTDISx/cJRu0B7rzpwZzQxy0vyecDC44jfhylQju6ATXvGq2KZ6PP4 +9kyB9AhRJFJaQYNDo3xsJn+0rIG5Ag0EXQqmaQEQANZRvJiGQABQOhBHIGEcnYS2 +E/0Y7NtPfafesU5cmOmoTZptAx02Ad67loXu8ZYcCde6WUAYiWNBN3Gyul/rlx6L +zFIe4VzzWbbhv5WmQ77yCINtk670enwn6fPYoSLppIG2ykdkhnwXP5t8t1cmKeOM +za/fL4XtPMM+UVVrEi3tVFoT5+pT5flNgzuiZbFhLnGJxHk4oaIcBJ3C7OGjtNy3 +cI/+XOyqA1b0A4gKYa6I07KCFfFO4lOFby9BkF502MGe45evkAue7NkkQNPrf83Z +nRIUJXFYVnEDAXjbKD+izfC3ixaZ48WZ3Fmgs+2owQaEvZ15i3Vj3napu6X8HYUn +u5x1GIH5/Q1CoIuJ98gkjm8ITpoE77tLhL+xiXpYD4sLhV/iuztRb/AMxtGWCW8B +dZH/8bBFVPhGt6iAhcn6EecWGyRfdPDpY2eqLzfUpsmZnF6fsjrsxToB6LbkPmIq +Y4xdMECCJ/dnjvPYGawsWmAPwzgDp2ZLnjAvFzHUbyKG3ylGuxdMCobDFIIUl2gG +Ta2zcbd3YSR6on7RqCcMfk5uHb2I2Rz/PXT4bFb/JmgPRq949r0c/+7UGrxYhasI +Pn/11SYKcyc9409k2DX12lgcWqnQA69f5cs+Q2Afafi+MOEkQLOYkMALISZ4Y5J4 +uCHBsM/3JeM8rbfTWrMxABEBAAGJAjYEGAEKACAWIQSJ4v/K5+mc9uaCfP73NJ8j +EP+xkwUCXQqmaQIbDAAKCRD3NJ8jEP+xk0XDD/9fw9hj2SCC5M0ZLB2E/wnamQ1r +9gHVjwtDrxNdUfiNL1ilP+Y+RBPhcK6bpsfnrTvR20In2mnaCOQkCfaOnDYoB/wB +F0CGPvRMWzFcU35e5Vr068K12lJ/5shIESZ+AqKoXdQHbjux53U0+An9MkAcQild +ro1qMHrvRYJjTgXSTZgb20UBAhNXpmfSSkue2eebOP4XWaKA9fWr7lX/qdutbpgi +gA7iWG5hkIQPd0ifgXOatjAvcm5PVbWaOWTaaQJf6gcSs5TTjCv6v44GByRgMqDZ +9fJbV64uEjwDFuQ6FbsMCcIBYH6SFT4QDDcLL9Cb2vnnV/GWmthDlYjiSJT6FfGQ +YiRkIQQiLYwLofTfbQ3FcuO+0cPpcuG4EkTGQl9WyOo1KYakxeGPqxSGYIm2cevm +DD0lhm9AYjzzh0/jTjM5MMWKed5k53C/hFExpJ05pVz3YiSl/MuQHkw1yZwR0KwG +KeTU21YG4MA2zFwsaiVNRBOEETZjeR7Hzc1sKxaabHfeK0uZbDxff5+LS1yCd3UW +LGF51NvEvVVSZCTrzcPiG6im+zW45Db+IAxXVShxMN1TDhv81mXqUqdHDdCw8dmB +95mHyl72MUaCohP7zxjgnjfu7oU15p320tQk5iQxRucEMV/6gdY4qD83lLoQd4Cw +CUPK/XSFtO9iFPQk/w== +=k7F7 +-END PGP PUBLIC KEY BLOCK-
svn commit: r34582 - /dev/beam/KEYS
Author: kenn Date: Wed Jun 19 22:01:04 2019 New Revision: 34582 Log: Add an...@apache.org to Beam KEYS file Modified: dev/beam/KEYS Modified: dev/beam/KEYS == --- dev/beam/KEYS (original) +++ dev/beam/KEYS Wed Jun 19 22:01:04 2019 @@ -1047,3 +1047,59 @@ ClRre5N12PPAiMPMIAhlV06wnBGN9+/QT7j1Mj0D VJQ2jFnQsSa2a5Kg5L9LxsgsA2q5tLprKq6jg+TccWHMXYRVgWibZyusUWU= =WFak -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2019-06-19 [SC] + 89E2FFCAE7E99CF6E6827CFEF7349F2310FFB193 +uid [ultimate] Anton Kedin (CODE SIGNING KEY) +sub rsa4096 2019-06-19 [E] +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF0KpmkBEADmcavSGmPp8+eFXle+wUHElIz9S8VphY1VlM+JJYkaDy7jmEkX +OG2WFSFM7xj3SBfbgHoKte+pGAyt83jCptMDB2x9RjfmAUyiB438uLww+BKFKscV +TH6WnKwoV4Erjpq7qjB32GzZWIUtcqrZ33PpwENUdbl9gzH+uo/fhBP5l7asCe0R +jGx09ai/jifQP84o4QprsNu3Prud6lU3ihXkblO35GerOGL6h+L7xKA8GBgw8vjf +hFrMeA0b6bth/cTF0bdEWhLKyp0Sqn4ybZXP4yxrOdsCHOq/qmBq1Jwzm5zW21FR +4xlglCdFn0GdYuDOxO0EK98KV4EAulZB3Ld0LzoblD0DERLo6GUvz4gQ2Xbs7G8a +I8/ODed0dqqBDccrWuXrUsMxvQ6lwdxhjOwW6eWCOSVLsPGGSP3+ZDKwYc7XeR6+ +5N0ts/dR0J/1IT7R2Iw93OXMfOmb4Zb8gUShkFYgHhFNyN3qZ4ZUrBJx/TWQYaX+ +9xV7SbZzMX1oXDUXEQFpTGW811Eicyd+9pZIG3cNTymz7bUe2zStDiThnqRout5l +wWg97hGWHrEgYwa9I+xIkaD9Eo1gA/IUgf7XQN4EMmK3ZVbz8up7STenCk9HidCb +kqAK42CWQCAFrbpAYY43+VWylLTdHIQQgTY08rTrgc3PxwIhwoEt2mHyWQARAQAB +tDFBbnRvbiBLZWRpbiAoQ09ERSBTSUdOSU5HIEtFWSkgPGFudG9uQGFwYWNoZS5v +cmc+iQJOBBMBCgA4FiEEieL/yufpnPbmgnz+9zSfIxD/sZMFAl0KpmkCGwMFCwkI +BwIGFQoJCAsCBBYCAwECHgECF4AACgkQ9zSfIxD/sZOW8RAApetCz0jJ6nw+1m2b +BjgmZVSC/gu4wUsdAYKxTN5baXX4W+aiJk04Gn32ErsiZZwXIrzqWogIbu6mIXj4 +vxqKZuOv48fsNVkwfPiP9p/m0B9l+HxJl+1klZWCqjuxTy3j4Fk7fjPPuFD+iS6c +i4PFOvazo0rtbfetsw9FuPR9uRNP7pbOqOEyZF+kUk9BkffwsUf9LYPKBJktfBNy +1bSmmZ5KqsGUp0BFgR6KVCvyTFgKQDluDcIPTUBsNJxM4Mui9L6lS9v/kOF/YI+y +HsJVUwIeFRsmsKgSVam7VTZFbSX1EhQYqPHUQtbjCZ2JOyYRPiXKzYQ32MAY/RTw +lpFvtjnmqgv0lmAdsRSKeDnju1mlsONzejhrOFOsYc2JyWUc94qzOe7VpeTYKwPX +gxFcr/SX+jgreSIFl7pTcLXyZ2olsiXNoCGzIBgXZDG2ow0CVNoa2jPOkpZkws5t +2rVa2zaEPtwcyjIExZPrhmn7dmpx9zBhmuk+PEPLjE7kXRWIVvfYm1VJmjPNWMv7 +HJp5nx2WDK74ULOFGVpDvJf6hA1F3LO7QWyZw8tOFAMYsOmP1VLNk/TIarptKn3p +immrMsgM60eTDISx/cJRu0B7rzpwZzQxy0vyecDC44jfhylQju6ATXvGq2KZ6PP4 +9kyB9AhRJFJaQYNDo3xsJn+0rIG5Ag0EXQqmaQEQANZRvJiGQABQOhBHIGEcnYS2 +E/0Y7NtPfafesU5cmOmoTZptAx02Ad67loXu8ZYcCde6WUAYiWNBN3Gyul/rlx6L +zFIe4VzzWbbhv5WmQ77yCINtk670enwn6fPYoSLppIG2ykdkhnwXP5t8t1cmKeOM +za/fL4XtPMM+UVVrEi3tVFoT5+pT5flNgzuiZbFhLnGJxHk4oaIcBJ3C7OGjtNy3 +cI/+XOyqA1b0A4gKYa6I07KCFfFO4lOFby9BkF502MGe45evkAue7NkkQNPrf83Z +nRIUJXFYVnEDAXjbKD+izfC3ixaZ48WZ3Fmgs+2owQaEvZ15i3Vj3napu6X8HYUn +u5x1GIH5/Q1CoIuJ98gkjm8ITpoE77tLhL+xiXpYD4sLhV/iuztRb/AMxtGWCW8B +dZH/8bBFVPhGt6iAhcn6EecWGyRfdPDpY2eqLzfUpsmZnF6fsjrsxToB6LbkPmIq +Y4xdMECCJ/dnjvPYGawsWmAPwzgDp2ZLnjAvFzHUbyKG3ylGuxdMCobDFIIUl2gG +Ta2zcbd3YSR6on7RqCcMfk5uHb2I2Rz/PXT4bFb/JmgPRq949r0c/+7UGrxYhasI +Pn/11SYKcyc9409k2DX12lgcWqnQA69f5cs+Q2Afafi+MOEkQLOYkMALISZ4Y5J4 +uCHBsM/3JeM8rbfTWrMxABEBAAGJAjYEGAEKACAWIQSJ4v/K5+mc9uaCfP73NJ8j +EP+xkwUCXQqmaQIbDAAKCRD3NJ8jEP+xk0XDD/9fw9hj2SCC5M0ZLB2E/wnamQ1r +9gHVjwtDrxNdUfiNL1ilP+Y+RBPhcK6bpsfnrTvR20In2mnaCOQkCfaOnDYoB/wB +F0CGPvRMWzFcU35e5Vr068K12lJ/5shIESZ+AqKoXdQHbjux53U0+An9MkAcQild +ro1qMHrvRYJjTgXSTZgb20UBAhNXpmfSSkue2eebOP4XWaKA9fWr7lX/qdutbpgi +gA7iWG5hkIQPd0ifgXOatjAvcm5PVbWaOWTaaQJf6gcSs5TTjCv6v44GByRgMqDZ +9fJbV64uEjwDFuQ6FbsMCcIBYH6SFT4QDDcLL9Cb2vnnV/GWmthDlYjiSJT6FfGQ +YiRkIQQiLYwLofTfbQ3FcuO+0cPpcuG4EkTGQl9WyOo1KYakxeGPqxSGYIm2cevm +DD0lhm9AYjzzh0/jTjM5MMWKed5k53C/hFExpJ05pVz3YiSl/MuQHkw1yZwR0KwG +KeTU21YG4MA2zFwsaiVNRBOEETZjeR7Hzc1sKxaabHfeK0uZbDxff5+LS1yCd3UW +LGF51NvEvVVSZCTrzcPiG6im+zW45Db+IAxXVShxMN1TDhv81mXqUqdHDdCw8dmB +95mHyl72MUaCohP7zxjgnjfu7oU15p320tQk5iQxRucEMV/6gdY4qD83lLoQd4Cw +CUPK/XSFtO9iFPQk/w== +=k7F7 +-END PGP PUBLIC KEY BLOCK-
[beam] branch master updated: Add common code infrastructure for element-wise snippets
This is an automated email from the ASF dual-hosted git repository.
altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c32acce Add common code infrastructure for element-wise snippets
new 3124531 Merge pull request #8811 from
davidcavazos/element-wise-snippets
c32acce is described below
commit c32accec7b6d254851d156facf03154198cf2e54
Author: David Cavazos
AuthorDate: Mon Jun 10 16:01:56 2019 -0700
Add common code infrastructure for element-wise snippets
---
.../examples/snippets/transforms/__init__.py | 18 ++
.../snippets/transforms/element_wise/__init__.py | 18 ++
sdks/python/apache_beam/examples/snippets/util.py | 56 ++
.../apache_beam/examples/snippets/util_test.py | 67 ++
4 files changed, 159 insertions(+)
diff --git a/sdks/python/apache_beam/examples/snippets/transforms/__init__.py
b/sdks/python/apache_beam/examples/snippets/transforms/__init__.py
new file mode 100644
index 000..6569e3f
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/transforms/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
diff --git
a/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
new file mode 100644
index 000..6569e3f
--- /dev/null
+++
b/sdks/python/apache_beam/examples/snippets/transforms/element_wise/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
diff --git a/sdks/python/apache_beam/examples/snippets/util.py
b/sdks/python/apache_beam/examples/snippets/util.py
new file mode 100644
index 000..6e6e9e0
--- /dev/null
+++ b/sdks/python/apache_beam/examples/snippets/util.py
@@ -0,0 +1,56 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import argparse
+import shlex
+import subprocess as sp
+
+
+def parse_example(argv=None):
+ """Parse the command line arguments and return it as a string function call.
+
+ Examples:
+python path/to/snippets.py function_name
+python path/to/snippets.py function_name arg1
+python path/to/snippets.py function_name arg1 arg2 ... argN
+ """
+ parser = argparse.ArgumentParser()
+ parser.add_argument('example', help='Name of the example to run.')
+ parser.add_argument('args', nargs=argparse.REMAINDER,
+ help='Arguments for example.')
+ args = parser.parse_args(argv)
+
+ # Return the example as a string representing the Python fu
[beam] branch master updated (221e0a5 -> 441b6b1)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 221e0a5 Merge pull request #8856: [BEAM-6877] Fix trivial_inference for Python 3.x add 441b6b1 [BEAM-6955] Use base version component of Beam Python SDK version when choosing Dataflow container image to use. (#8885) No new revisions were added by this update. Summary of changes: .../runners/dataflow/internal/apiclient.py | 18 +- .../runners/dataflow/internal/apiclient_test.py| 66 ++ 2 files changed, 82 insertions(+), 2 deletions(-)
[beam] branch master updated (4f7d802 -> 221e0a5)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4f7d802 [BEAM-7467] Do not run gearpump pipeline remotely by default add db30622 [BEAM-6877] Fix trivial_inference for Python 3.x add 29807c8 Address review comments add 4dc1c0b Allow more time for IT pipeline to run new 221e0a5 Merge pull request #8856: [BEAM-6877] Fix trivial_inference for Python 3.x The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../examples/streaming_wordcount_it_test.py| 2 +- sdks/python/apache_beam/typehints/opcodes.py | 89 .../apache_beam/typehints/trivial_inference.py | 158 +++-- .../typehints/trivial_inference_test.py| 53 +-- sdks/python/apache_beam/typehints/typehints.py | 10 ++ .../python/apache_beam/typehints/typehints_test.py | 11 ++ 6 files changed, 231 insertions(+), 92 deletions(-)
[beam] 01/01: Merge pull request #8856: [BEAM-6877] Fix trivial_inference for Python 3.x
This is an automated email from the ASF dual-hosted git repository. udim pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 221e0a5273a1c8f590df2c9f68666849a443250c Merge: 4f7d802 4dc1c0b Author: Udi Meiri AuthorDate: Wed Jun 19 11:13:13 2019 -0700 Merge pull request #8856: [BEAM-6877] Fix trivial_inference for Python 3.x .../examples/streaming_wordcount_it_test.py| 2 +- sdks/python/apache_beam/typehints/opcodes.py | 89 .../apache_beam/typehints/trivial_inference.py | 158 +++-- .../typehints/trivial_inference_test.py| 53 +-- sdks/python/apache_beam/typehints/typehints.py | 10 ++ .../python/apache_beam/typehints/typehints_test.py | 11 ++ 6 files changed, 231 insertions(+), 92 deletions(-)
[beam] branch master updated (29868a7 -> 4f7d802)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 29868a7 [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8 add 814713c [BEAM-7467] Do not run gearpump pipeline remotely by default new 4f7d802 [BEAM-7467] Do not run gearpump pipeline remotely by default The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/beam/runners/gearpump/GearpumpPipelineOptions.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: [BEAM-7467] Do not run gearpump pipeline remotely by default
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4f7d802ae7b532a3306ee207db5b85cd366a84ad Merge: 29868a7 814713c Author: Lukasz Cwik AuthorDate: Wed Jun 19 09:55:10 2019 -0700 [BEAM-7467] Do not run gearpump pipeline remotely by default .../java/org/apache/beam/runners/gearpump/GearpumpPipelineOptions.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (25955c6 -> 29868a7)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 25955c6 [BEAM-7579] Use bucket with default key in ITs add 15d7be3 [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8 in hot paths new 29868a7 [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dataflow/worker/StreamingDataflowWorker.java | 27 +- 1 file changed, 16 insertions(+), 11 deletions(-)
[beam] 01/01: [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 29868a7019981b6ac6ef3ef847708343b6ea8094 Merge: 25955c6 15d7be3 Author: Lukasz Cwik AuthorDate: Wed Jun 19 09:49:55 2019 -0700 [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8 .../dataflow/worker/StreamingDataflowWorker.java | 27 +- 1 file changed, 16 insertions(+), 11 deletions(-)
[beam] branch master updated (eb5a8c2 -> 25955c6)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from eb5a8c2 Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy
must accept BinaryFunction
add 5c2b362 [BEAM-7579] Use bucket with default key in ITs
add 4d33e4b Address review comments
new 25955c6 [BEAM-7579] Use bucket with default key in ITs
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
runners/google-cloud-dataflow-java/build.gradle| 14 +++
...rDoWithWindowedSideInputs.java => UsesKms.java} | 10
.../google-cloud-platform-core/build.gradle| 10
.../beam/sdk/extensions/gcp/util/GcsUtilIT.java| 10 +---
sdks/java/io/google-cloud-platform/build.gradle| 13 +++
.../beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java | 3 +++
.../beam/sdk/io/gcp/storage/GcsKmsKeyIT.java | 27 --
7 files changed, 48 insertions(+), 39 deletions(-)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/testing/{UsesSplittableParDoWithWindowedSideInputs.java
=> UsesKms.java} (77%)
[beam] 01/01: [BEAM-7579] Use bucket with default key in ITs
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 25955c6c5c20f542e5a5d11159b7bda5fbed56a8 Merge: eb5a8c2 4d33e4b Author: Lukasz Cwik AuthorDate: Wed Jun 19 09:24:05 2019 -0700 [BEAM-7579] Use bucket with default key in ITs runners/google-cloud-dataflow-java/build.gradle| 14 +++ .../java/org/apache/beam/sdk/testing/UsesKms.java | 26 + .../google-cloud-platform-core/build.gradle| 10 .../beam/sdk/extensions/gcp/util/GcsUtilIT.java| 10 +--- sdks/java/io/google-cloud-platform/build.gradle| 13 +++ .../beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java | 3 +++ .../beam/sdk/io/gcp/storage/GcsKmsKeyIT.java | 27 -- 7 files changed, 69 insertions(+), 34 deletions(-)
[beam] branch master updated (5e5e7a5 -> eb5a8c2)
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5e5e7a5 Merge pull request #8849: [BEAM-7561] Enable HdfsFileSystems.match to match a directory add b0bf009 [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction add 175a8d2 [BEAM-7543] deprecate old #combineBy(Sums.ofLongs(), TypeDescriptors.longs()) new eb5a8c2 Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../euphoria/core/client/operator/CountByKey.java | 2 +- .../euphoria/core/client/operator/ReduceByKey.java | 307 +++-- .../core/client/operator/ReduceWindow.java | 292 +--- .../extensions/euphoria/core/client/util/Sums.java | 58 +++- .../core/translate/ReduceByKeyTranslator.java | 83 +- .../core/client/operator/ReduceByKeyTest.java | 99 +-- .../euphoria/core/client/util/SumsTest.java| 13 +- .../euphoria/core/testkit/ReduceByKeyTest.java | 47 +++- 8 files changed, 799 insertions(+), 102 deletions(-)
[beam] 01/01: Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction
This is an automated email from the ASF dual-hosted git repository. dmvk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit eb5a8c2cbd973972491dedef23aff98e450a9f5a Merge: 5e5e7a5 175a8d2 Author: David Moravek AuthorDate: Wed Jun 19 16:40:32 2019 +0200 Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction .../euphoria/core/client/operator/CountByKey.java | 2 +- .../euphoria/core/client/operator/ReduceByKey.java | 307 +++-- .../core/client/operator/ReduceWindow.java | 292 +--- .../extensions/euphoria/core/client/util/Sums.java | 58 +++- .../core/translate/ReduceByKeyTranslator.java | 83 +- .../core/client/operator/ReduceByKeyTest.java | 99 +-- .../euphoria/core/client/util/SumsTest.java| 13 +- .../euphoria/core/testkit/ReduceByKeyTest.java | 47 +++- 8 files changed, 799 insertions(+), 102 deletions(-)
[beam] 01/01: Merge pull request #8849: [BEAM-7561] Enable HdfsFileSystems.match to match a directory
This is an automated email from the ASF dual-hosted git repository. timrobertson100 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5e5e7a50205ab5a59e40c6baec31ac1d7be3a169 Merge: 08a84e4 edd0e0c Author: Tim Robertson AuthorDate: Wed Jun 19 10:34:25 2019 +0200 Merge pull request #8849: [BEAM-7561] Enable HdfsFileSystems.match to match a directory .../apache/beam/sdk/io/LocalFileSystemTest.java| 21 + .../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 22 ++ .../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 21 + 3 files changed, 52 insertions(+), 12 deletions(-)
[beam] branch master updated (08a84e4 -> 5e5e7a5)
This is an automated email from the ASF dual-hosted git repository. timrobertson100 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 08a84e4 [BEAM-6674] Add schema support to JdbcIO read (#8725) add 408f744 [BEAM-7561] HdfsFileSystems.match can match a directory. add edd0e0c [BEAM-7561] Validate correct directory matching in LocalFileSystemsTest new 5e5e7a5 Merge pull request #8849: [BEAM-7561] Enable HdfsFileSystems.match to match a directory The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/io/LocalFileSystemTest.java| 21 + .../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 22 ++ .../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 21 + 3 files changed, 52 insertions(+), 12 deletions(-)
