[beam] branch master updated (dc0c2bd -> 7967fe7)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dc0c2bd [BEAM-10961] enable strict dependency checking for sdks/java/io/bigquery-io-perf-tests (#14048) add 7967fe7 [BEAM-10961] enable strict dependency checking for sdks/java/io/kinesis (#14058) No new revisions were added by this update. Summary of changes: sdks/java/io/kinesis/build.gradle | 13 ++--- sdks/java/io/kinesis/expansion-service/build.gradle | 4 +++- 2 files changed, 9 insertions(+), 8 deletions(-)
[beam] branch master updated: [BEAM-10961] enable strict dependency checking for sdks/java/io/bigquery-io-perf-tests (#14048)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new dc0c2bd [BEAM-10961] enable strict dependency checking for
sdks/java/io/bigquery-io-perf-tests (#14048)
dc0c2bd is described below
commit dc0c2bd5075991b7b45a26c7c94b2e899f407fcb
Author: shehzaadn-vd <32536022+shehzaadn...@users.noreply.github.com>
AuthorDate: Wed Feb 24 09:17:27 2021 -0800
[BEAM-10961] enable strict dependency checking for
sdks/java/io/bigquery-io-perf-tests (#14048)
* [BEAM-10961] enable strict dependency checking for
sdks/java/io/bigquery-io-perf-tests
* [BEAM-10961] (1) nitpick: space (2) change testImplementation to
testCompile
---
sdks/java/io/bigquery-io-perf-tests/build.gradle | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/sdks/java/io/bigquery-io-perf-tests/build.gradle
b/sdks/java/io/bigquery-io-perf-tests/build.gradle
index 8d5cf89..ee5c123 100644
--- a/sdks/java/io/bigquery-io-perf-tests/build.gradle
+++ b/sdks/java/io/bigquery-io-perf-tests/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(exportJavadoc: false, publish: false)
+applyJavaNature(enableStrictDependencies: true, exportJavadoc: false, publish:
false)
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
@@ -25,7 +25,7 @@ description = "Apache Beam :: SDKs :: Java :: Google BigQuery
IO Performance tes
ext.summary = "Performance tests for Google BigQuery IO sources and sinks"
dependencies {
-compile library.java.google_api_services_bigquery
+testCompile library.java.google_api_services_bigquery
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile project(path: ":sdks:java:testing:test-utils", configuration:
"testRuntime")
testCompile project(path:
":sdks:java:extensions:google-cloud-platform-core", configuration:
"testRuntime")
@@ -34,7 +34,4 @@ dependencies {
testCompile project(path: ":sdks:java:io:google-cloud-platform",
configuration: "testRuntime")
testCompile project(":sdks:java:io:synthetic")
testCompile library.java.junit
-testCompile library.java.hamcrest_core
-testCompile library.java.jaxb_api
-testCompile library.java.jaxb_impl
}
[beam] branch master updated (3a24bde -> e5b16a5)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3a24bde [BEAM-10961] enable strict dependency checking for sdks/java/io/expansion-service (#14051) add e5b16a5 [BEAM-10961] enable strict dependency checking for sdks/java/io/elasticsearch (#14050) No new revisions were added by this update. Summary of changes: sdks/java/io/elasticsearch/build.gradle | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-)
[beam] branch master updated (0f0c5c1 -> 3a24bde)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0f0c5c1 [BEAM-10961] enable strict dependency checking for sdks/java/io/jdbc (#14055) add 3a24bde [BEAM-10961] enable strict dependency checking for sdks/java/io/expansion-service (#14051) No new revisions were added by this update. Summary of changes: sdks/java/io/expansion-service/build.gradle | 5 + 1 file changed, 5 insertions(+)
[beam] branch master updated (654552a -> 0f0c5c1)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 654552a [BEAM-10961] enable strict dependency checking for sdks/java/io/jms (#14056) add 0f0c5c1 [BEAM-10961] enable strict dependency checking for sdks/java/io/jdbc (#14055) No new revisions were added by this update. Summary of changes: sdks/java/io/jdbc/build.gradle | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-)
[beam] branch master updated (be451b0 -> 654552a)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from be451b0 [BEAM-10961] enable strict dependency checking for sdks/java/io/kafka (#14057) add 654552a [BEAM-10961] enable strict dependency checking for sdks/java/io/jms (#14056) No new revisions were added by this update. Summary of changes: sdks/java/io/jms/build.gradle | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[beam] branch master updated: [BEAM-10961] enable strict dependency checking for sdks/java/io/kafka (#14057)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new be451b0 [BEAM-10961] enable strict dependency checking for
sdks/java/io/kafka (#14057)
be451b0 is described below
commit be451b01e39e5729d5de2d974085b4545122a908
Author: shehzaadn-vd <32536022+shehzaadn...@users.noreply.github.com>
AuthorDate: Wed Feb 24 09:13:19 2021 -0800
[BEAM-10961] enable strict dependency checking for sdks/java/io/kafka
(#14057)
* [BEAM-10961] enable strict dependency checking for sdks/java/io/kafka
* [BEAM-10961] (1) fix spacing (2) remove kafka-clients dependency (3) add
reference to BEAM-11761
* Trigger Build
---
sdks/java/io/kafka/build.gradle | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/sdks/java/io/kafka/build.gradle b/sdks/java/io/kafka/build.gradle
index a500ddd..51bb13e 100644
--- a/sdks/java/io/kafka/build.gradle
+++ b/sdks/java/io/kafka/build.gradle
@@ -20,6 +20,7 @@ import java.util.stream.Collectors
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ enableStrictDependencies: true,
automaticModuleName: 'org.apache.beam.sdk.io.kafka',
mavenRepositories: [
[id: 'io.confluent', url: 'https://packages.confluent.io/maven/']
@@ -49,6 +50,8 @@ dependencies {
compile library.java.vendored_guava_26_0_jre
compile project(path: ":sdks:java:core", configuration: "shadow")
compile project(":sdks:java:expansion-service")
+ permitUnusedDeclared project(":sdks:java:expansion-service") // BEAM-11761
+ compile library.java.avro
// Get back to "provided" since 2.14
provided library.java.kafka_clients
compile library.java.slf4j_api
@@ -57,6 +60,13 @@ dependencies {
compile library.java.jackson_databind
compile "org.springframework:spring-expression:4.3.18.RELEASE"
compile ("io.confluent:kafka-avro-serializer:5.3.2") {
+// zookeeper depends on "spotbugs-annotations:3.1.9" which clashes with
current
+// "spotbugs-annotations:3.1.12" used in Beam. Not required.
+exclude group: "org.apache.zookeeper", module: "zookeeper"
+// "kafka-clients" has to be provided since user can use its own version.
+exclude group: "org.apache.kafka", module: "kafka-clients"
+ }
+ compile ("io.confluent:kafka-schema-registry-client:5.3.2") {
// It depends on "spotbugs-annotations:3.1.9" which clashes with current
// "spotbugs-annotations:3.1.12" used in Beam. Not required.
exclude group: "org.apache.zookeeper", module: "zookeeper"
@@ -71,8 +81,6 @@ dependencies {
// For testing Cross-language transforms
testCompile project(":runners:core-construction-java")
testCompile library.java.avro
- testCompile library.java.hamcrest_core
- testCompile library.java.hamcrest_library
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
[beam] branch master updated (a7a7d5f -> a831773)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a7a7d5f Merge pull request #14061 from sonam-vend/sdks-io-mqtt add a831773 [BEAM-10961] enable strict dependency checking for sdks/java/io/hcatalog (#14053) No new revisions were added by this update. Summary of changes: sdks/java/io/hcatalog/build.gradle | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated: [BEAM-10961] enable strict dependency checking for sdks/java/io/mqtt
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 62a746a [BEAM-10961] enable strict dependency checking for
sdks/java/io/mqtt
new a7a7d5f Merge pull request #14061 from sonam-vend/sdks-io-mqtt
62a746a is described below
commit 62a746a7c7d018e8b5793c0bbea1723ba83ea522
Author: Shehzaad Nakhoda
AuthorDate: Wed Feb 24 02:19:10 2021 -0800
[BEAM-10961] enable strict dependency checking for sdks/java/io/mqtt
---
sdks/java/io/mqtt/build.gradle | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/sdks/java/io/mqtt/build.gradle b/sdks/java/io/mqtt/build.gradle
index a384274..ea6ecd9 100644
--- a/sdks/java/io/mqtt/build.gradle
+++ b/sdks/java/io/mqtt/build.gradle
@@ -17,7 +17,7 @@
*/
plugins { id 'org.apache.beam.module' }
-applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.mqtt')
+applyJavaNature(enableStrictDependencies: true, automaticModuleName:
'org.apache.beam.sdk.io.mqtt')
description = "Apache Beam :: SDKs :: Java :: IO :: MQTT"
ext.summary = "IO to read and write to a MQTT broker."
@@ -34,8 +34,6 @@ dependencies {
testCompile library.java.activemq_mqtt
testCompile library.java.activemq_kahadb_store
testCompile library.java.junit
- testCompile library.java.hamcrest_core
- testCompile library.java.hamcrest_library
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration:
"shadow")
}
[beam] branch master updated: [BEAM-10961] enable strict dependency checking for sdks/java/io/splunk
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a955dcc [BEAM-10961] enable strict dependency checking for
sdks/java/io/splunk
new 8d5428a Merge pull request #14064 from sonam-vend/sdks-io-splunk
a955dcc is described below
commit a955dccc912f3187d646039005ce13ee4cb7d076
Author: Shehzaad Nakhoda
AuthorDate: Wed Feb 24 03:38:40 2021 -0800
[BEAM-10961] enable strict dependency checking for sdks/java/io/splunk
---
sdks/java/io/splunk/build.gradle | 12 +---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/sdks/java/io/splunk/build.gradle b/sdks/java/io/splunk/build.gradle
index a8a31b7..5e47d0e 100644
--- a/sdks/java/io/splunk/build.gradle
+++ b/sdks/java/io/splunk/build.gradle
@@ -19,18 +19,24 @@
plugins {
id 'org.apache.beam.module'
}
-applyJavaNature(automaticModuleName: 'org.apache.beam.sdk.io.splunk')
+applyJavaNature(enableStrictDependencies: true, automaticModuleName:
'org.apache.beam.sdk.io.splunk')
description = "Apache Beam :: SDKs :: Java :: IO :: Splunk"
ext.summary = "IO to write events to Splunk Http Event Collector (HEC)"
dependencies {
compile platform(library.java.google_cloud_platform_libraries_bom)
-compile library.java.slf4j_api
+compile project(path: ":sdks:java:core", configuration: "shadow")
compile library.java.google_api_client
+permitUnusedDeclared library.java.google_api_client // BEAM-11761
compile library.java.google_http_client_apache_v2
compile library.java.google_code_gson
-compile project(path: ":sdks:java:core", configuration: "shadow")
+compile library.java.google_http_client
+compile library.java.http_client
+compile library.java.http_core
+compile library.java.joda_time
+compile library.java.slf4j_api
+compile library.java.vendored_guava_26_0_jre
testCompile library.java.junit
testCompile group: 'org.mock-server', name: 'mockserver-junit-rule',
version: '5.10.0'
testCompile group: 'org.mock-server', name: 'mockserver-client-java',
version: '5.10.0'
[beam] branch master updated: [BEAM-10961] enable strict dependency checking for sdks/java/io/synthetic
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ce5a400 [BEAM-10961] enable strict dependency checking for
sdks/java/io/synthetic
new d1439aa Merge pull request #14065 from sonam-vend/sdks-io-synthetic
ce5a400 is described below
commit ce5a40091917a2eed2dc448b0f01051afa06c26f
Author: Shehzaad Nakhoda
AuthorDate: Wed Feb 24 03:47:38 2021 -0800
[BEAM-10961] enable strict dependency checking for sdks/java/io/synthetic
---
sdks/java/io/synthetic/build.gradle | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/sdks/java/io/synthetic/build.gradle
b/sdks/java/io/synthetic/build.gradle
index cb9cbf8..68d916f 100644
--- a/sdks/java/io/synthetic/build.gradle
+++ b/sdks/java/io/synthetic/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ enableStrictDependencies: true,
exportJavadoc: false,
automaticModuleName: 'org.apache.beam.sdk.io.synthetic',
)
@@ -31,11 +32,11 @@ dependencies {
compile library.java.jackson_core
compile library.java.jackson_annotations
compile library.java.jackson_databind
+ compile library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
compile project(path: ":sdks:java:core", configuration: "shadow")
testCompile library.java.vendored_guava_26_0_jre
testCompile library.java.junit
- testCompile library.java.hamcrest_core
- testCompile library.java.hamcrest_library
testRuntimeOnly project(path: ":runners:direct-java", configuration:
"shadow")
}
[beam] branch master updated (bb8ab1b -> e9673d9)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bb8ab1b Merge pull request #14054 from sonam-vend/sdks-io-influxdb new 7a687fc [BEAM-10961] enable strict dependency checking for sdks/java/io/xml new a72e54a [BEAM-10961] fix spacing new e9673d9 Merge pull request #14068 from sonam-vend/sdks-io-xml The 30748 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/xml/build.gradle | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-)
[beam] branch master updated (a775022 -> bb8ab1b)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a775022 Merge pull request #14063 from sonam-vend/sdks-io-solr new 89abbe4 [BEAM-10961] enable strict dependency checking for sdks/java/io/influxdb new 0a27264 [BEAM-10961] nit: spacing new bb8ab1b Merge pull request #14054 from sonam-vend/sdks-io-influxdb The 30745 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/influxdb/build.gradle | 5 - 1 file changed, 4 insertions(+), 1 deletion(-)
[beam] branch master updated: [BEAM-10961] enable strict dependency checking for sdks/java/io/solr
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e4e7d65 [BEAM-10961] enable strict dependency checking for
sdks/java/io/solr
new a775022 Merge pull request #14063 from sonam-vend/sdks-io-solr
e4e7d65 is described below
commit e4e7d65a4fe785ef8a22609752a18ea7beef4a8a
Author: Shehzaad Nakhoda
AuthorDate: Wed Feb 24 03:28:43 2021 -0800
[BEAM-10961] enable strict dependency checking for sdks/java/io/solr
---
sdks/java/io/solr/build.gradle | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sdks/java/io/solr/build.gradle b/sdks/java/io/solr/build.gradle
index 33a9b5c..fb43163 100644
--- a/sdks/java/io/solr/build.gradle
+++ b/sdks/java/io/solr/build.gradle
@@ -18,6 +18,7 @@
plugins { id 'org.apache.beam.module' }
applyJavaNature(
+ enableStrictDependencies: true,
automaticModuleName: 'org.apache.beam.sdk.io.solr',
)
@@ -30,12 +31,11 @@ dependencies {
compile library.java.vendored_guava_26_0_jre
compile project(path: ":sdks:java:core", configuration: "shadow")
compile library.java.commons_compress
+ compile library.java.joda_time
+ compile library.java.slf4j_api
compile "org.apache.solr:solr-solrj:$solrVersion"
- compileOnly "org.apache.httpcomponents:httpclient:4.5.6"
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile project(":sdks:java:io:common")
- testCompile library.java.hamcrest_core
- testCompile library.java.hamcrest_library
testCompile library.java.junit
testCompile library.java.slf4j_api
testCompile "org.apache.solr:solr-test-framework:$solrVersion"
[beam] branch master updated (c137e88 -> f9b3acf)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c137e88 Merge pull request #14001: [BEAM-11824] Set merge status in WindowingStrategyTranslation new 9d2e843 [BEAM-10961] enable strict dependency checking for sdks/java/io/azure/ new 10f74b7 [BEAM-10961] (1) fix azure-storage-common version (2) add reference to BEAM-11761 new f9b3acf Merge pull request #14047 from sonam-vend/sdks-io-azure The 30740 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/azure/build.gradle | 11 +-- 1 file changed, 9 insertions(+), 2 deletions(-)
[beam] branch master updated: [BEAM-10961] Enable strict dependencies on Clickhouse IO.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 00c731f [BEAM-10961] Enable strict dependencies on Clickhouse IO.
new 900e5be Merge pull request #13902 from ibzib/BEAM-10961-clickhouse
00c731f is described below
commit 00c731f460b0041d984904d61f44211c7f5a3e7e
Author: Kyle Weaver
AuthorDate: Fri Feb 5 09:44:50 2021 -0800
[BEAM-10961] Enable strict dependencies on Clickhouse IO.
---
sdks/java/io/clickhouse/build.gradle | 4
1 file changed, 4 insertions(+)
diff --git a/sdks/java/io/clickhouse/build.gradle
b/sdks/java/io/clickhouse/build.gradle
index 42deda7..a447852 100644
--- a/sdks/java/io/clickhouse/build.gradle
+++ b/sdks/java/io/clickhouse/build.gradle
@@ -22,6 +22,7 @@ plugins {
}
applyJavaNature(
automaticModuleName: 'org.apache.beam.sdk.io.clickhouse',
+ enableStrictDependencies: true,
generatedClassPatterns: [
/^org\.apache\.beam\.sdk\.io\.clickhouse\.impl\.parser\..*/,
],
@@ -54,7 +55,10 @@ def clickhouse_jdbc_version = "0.2.4"
dependencies {
javacc "net.java.dev.javacc:javacc:7.0.9"
compile project(path: ":sdks:java:core", configuration: "shadow")
+ compile library.java.guava
compile library.java.joda_time
+ compile library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
compile "ru.yandex.clickhouse:clickhouse-jdbc:$clickhouse_jdbc_version"
testCompile library.java.slf4j_api
testCompile library.java.junit
[beam] branch master updated (92ddb5c -> 4d24f66)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 92ddb5c Merge pull request #14042: [BEAM-11852] Update the title of our use-case on the new Beam Website new db147ba [BEAM-10671] Add environmentOptions to Java. new 84b1d54 fix checkstyle/spotbugs new 4ac92e7 [BEAM-10671] Add more documentation for EnvironmentOptions. new 254e8f6 Revert unintentional change. new 4d24f66 Merge pull request #13908 from ibzib/BEAM-10671-java The 30733 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../runners/core/construction/Environments.java| 125 ++--- .../core/construction/EnvironmentsTest.java| 84 +- .../beam/sdk/options/PortablePipelineOptions.java | 28 + .../sdk/options/PortablePipelineOptionsTest.java | 30 + 4 files changed, 250 insertions(+), 17 deletions(-)
[beam] branch master updated: [BEAM-11834] Enable arrays literals to have null values.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e9bdda9 [BEAM-11834] Enable arrays literals to have null values.
new 3a0a3b4 Merge pull request #14017 from ibzib/BEAM-11834
e9bdda9 is described below
commit e9bdda9e037600dcda597fd6b105966b0ebeb073
Author: Kyle Weaver
AuthorDate: Thu Feb 18 13:09:54 2021 -0800
[BEAM-11834] Enable arrays literals to have null values.
---
.../beam/sdk/extensions/sql/impl/utils/CalciteUtils.java | 2 +-
.../sql/zetasql/ZetaSqlCalciteTranslationUtils.java | 5 -
.../extensions/sql/zetasql/ZetaSqlDialectSpecTest.java | 16
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
index acd6e6c..10ad199 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
@@ -234,7 +234,7 @@ public class CalciteUtils {
return FieldType.row(toSchema(calciteType));
default:
-return toFieldType(calciteType.getSqlTypeName());
+return
toFieldType(calciteType.getSqlTypeName()).withNullable(calciteType.isNullable());
}
}
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlCalciteTranslationUtils.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlCalciteTranslationUtils.java
index 203337c..dd59c26 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlCalciteTranslationUtils.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlCalciteTranslationUtils.java
@@ -290,7 +290,10 @@ public final class ZetaSqlCalciteTranslationUtils {
private static RexNode arrayValueToRexNode(Value value, RexBuilder
rexBuilder) {
return rexBuilder.makeCall(
-toCalciteArrayType(value.getType().asArray().getElementType(), false,
rexBuilder),
+toCalciteArrayType(
+value.getType().asArray().getElementType(),
+value.getElementList().stream().anyMatch(v -> v.isNull()),
+rexBuilder),
SqlStdOperatorTable.ARRAY_VALUE_CONSTRUCTOR,
value.getElementList().stream()
.map(v -> toRexNode(v, rexBuilder))
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
index df5959e..4a07989 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
@@ -2421,6 +2421,22 @@ public class ZetaSqlDialectSpecTest extends
ZetaSqlTestBase {
}
@Test
+ public void testUnnestLiteralWithNullElements() {
+String sql = "SELECT * FROM UNNEST(ARRAY['foo', NULL, 'bar']);";
+ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+
+PCollection stream = BeamSqlRelUtils.toPCollection(pipeline,
beamRelNode);
+Schema schema = Schema.builder().addNullableField("str_field",
FieldType.STRING).build();
+PAssert.that(stream)
+.containsInAnyOrder(
+Row.withSchema(schema).addValues("foo").build(),
+Row.withSchema(schema).addValues((String) null).build(),
+Row.withSchema(schema).addValues("bar").build());
+
pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
public void testUNNESTParameters() {
String sql = "SELECT * FROM UNNEST(@p0);";
ImmutableMap params =
[beam] branch master updated (8d5b7c7 -> 0c08687)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8d5b7c7 Merge pull request #13993 from baeminbo/patch-1 new 5d3ffe7 [BEAM-11747] Disable BeamJavaUdfCalcRule by default. new 3dfbaaf spotless new 0c08687 Merge pull request #14010 from ibzib/disable-calcrule The 30660 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java | 6 ++ .../apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java | 2 ++ 2 files changed, 4 insertions(+), 4 deletions(-)
[beam] branch master updated (5996801 -> fac5f97)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5996801 [BEAM-11589] Migrated release build.gradle file to Kotlin script (#13948) new 7b4962e [BEAM-11811] [Dataflow Java] Don't allow numWorkers > maxNumWorkers. new cbe2b9c Don't create unassigned variable. new fac5f97 Merge pull request #13989 from ibzib/BEAM-11811 The 30626 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dataflow/DataflowPipelineTranslator.java | 7 ++ .../dataflow/DataflowPipelineTranslatorTest.java | 25 ++ 2 files changed, 32 insertions(+)
[beam] branch master updated: [BEAM-10925] Don't publish udf-test-provider to Maven.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e95c5dd [BEAM-10925] Don't publish udf-test-provider to Maven.
new b2c79ad Merge pull request #13967 from ibzib/dont-publish-test
e95c5dd is described below
commit e95c5dd2e8d37440e04ab1362c0021c7215634b7
Author: Kyle Weaver
AuthorDate: Thu Feb 11 14:11:11 2021 -0800
[BEAM-10925] Don't publish udf-test-provider to Maven.
---
sdks/java/extensions/sql/udf-test-provider/build.gradle | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/sdks/java/extensions/sql/udf-test-provider/build.gradle
b/sdks/java/extensions/sql/udf-test-provider/build.gradle
index 6769eba..8d2485b 100644
--- a/sdks/java/extensions/sql/udf-test-provider/build.gradle
+++ b/sdks/java/extensions/sql/udf-test-provider/build.gradle
@@ -20,7 +20,10 @@ plugins {
id 'org.apache.beam.module'
}
-applyJavaNature(automaticModuleName:
'org.apache.beam.sdk.extensions.sql.provider')
+applyJavaNature(
+automaticModuleName: 'org.apache.beam.sdk.extensions.sql.provider',
+publish: false,
+)
description = "Apache Beam :: SDKs :: Java :: Extensions :: SQL :: UDF test
provider"
ext.summary = "Java UDFs for testing. This project must be built separately
from its parent so the UDF provider is not included in the context classloader
for tests."
[beam] branch master updated: [BEAM-11789] Upgrade gradle-dependency-analyze plugin.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 514a27e [BEAM-11789] Upgrade gradle-dependency-analyze plugin.
new 44652b8 Merge pull request #13949 from ibzib/BEAM-11789-kt
514a27e is described below
commit 514a27e49bb618d1286ea5a8b07f7087022fe8c4
Author: Kyle Weaver
AuthorDate: Wed Feb 10 10:07:30 2021 -0800
[BEAM-11789] Upgrade gradle-dependency-analyze plugin.
---
buildSrc/build.gradle.kts | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/buildSrc/build.gradle.kts b/buildSrc/build.gradle.kts
index 9422346..bf46eb5 100644
--- a/buildSrc/build.gradle.kts
+++ b/buildSrc/build.gradle.kts
@@ -55,7 +55,7 @@ dependencies {
runtime("net.ltgt.gradle:gradle-errorprone-plugin:1.2.1")
// Enable errorprone Java static analysis
runtime("org.ajoberstar.grgit:grgit-gradle:4.0.2")
// Enable website git publish to asf-site branch
runtime("com.avast.gradle:gradle-docker-compose-plugin:0.13.2")
// Enable docker compose tasks
- runtime("ca.cutterslade.gradle:gradle-dependency-analyze:1.4.2")
// Enable dep analysis
+ runtime("ca.cutterslade.gradle:gradle-dependency-analyze:1.4.3")
// Enable dep analysis
runtime("gradle.plugin.net.ossindex:ossindex-gradle-plugin:0.4.11")
// Enable dep vulnerability analysis
runtime("org.checkerframework:checkerframework-gradle-plugin:0.5.11")
// Enable enhanced static checking plugin
}
[beam] branch master updated: [BEAM-10961] make mongo-java-driver dependency a named variable
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 827c938 [BEAM-10961] make mongo-java-driver dependency a named
variable
new 705649d Merge pull request #13910 from
sonam-vend/sdks-io-mongodb-driver-version
827c938 is described below
commit 827c9387d7b7efa55717d6dccde672787c72b3de
Author: Shehzaad Nakhoda
AuthorDate: Sat Feb 6 03:34:52 2021 -0800
[BEAM-10961] make mongo-java-driver dependency a named variable
---
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 1 +
sdks/java/io/mongodb/build.gradle | 6 +++---
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 31dbd7c..504a2e8 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -590,6 +590,7 @@ class BeamModulePlugin implements Plugin {
kafka :
"org.apache.kafka:kafka_2.11:$kafka_version",
kafka_clients :
"org.apache.kafka:kafka-clients:$kafka_version",
mockito_core:
"org.mockito:mockito-core:3.7.7",
+mongo_java_driver :
"org.mongodb:mongo-java-driver:3.12.7",
nemo_compiler_frontend_beam :
"org.apache.nemo:nemo-compiler-frontend-beam:$nemo_version",
netty_all :
"io.netty:netty-all:$netty_version",
netty_handler :
"io.netty:netty-handler:$netty_version",
diff --git a/sdks/java/io/mongodb/build.gradle
b/sdks/java/io/mongodb/build.gradle
index 30ede18..9250d5b 100644
--- a/sdks/java/io/mongodb/build.gradle
+++ b/sdks/java/io/mongodb/build.gradle
@@ -25,11 +25,11 @@ description = "Apache Beam :: SDKs :: Java :: IO :: MongoDB"
ext.summary = "IO to read and write on MongoDB."
dependencies {
- compile library.java.vendored_guava_26_0_jre
compile project(path: ":sdks:java:core", configuration: "shadow")
- compile library.java.slf4j_api
compile library.java.joda_time
- compile "org.mongodb:mongo-java-driver:3.12.7"
+ compile library.java.mongo_java_driver
+ compile library.java.slf4j_api
+ compile library.java.vendored_guava_26_0_jre
testCompile library.java.junit
testCompile project(path: ":sdks:java:io:common", configuration:
"testRuntime")
testCompile project(path: ":sdks:java:testing:test-utils", configuration:
"testRuntime")
[beam] branch master updated (5ea504d -> 3261fe3)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5ea504d [BEAM-11715] Partial revert of "Combiner packing in Dataflow" (#13763) (#13884) add 3261fe3 [BEAM-10961] Enable strict dependency checking for sdks/java/io/snowflake (#13800) No new revisions were added by this update. Summary of changes: sdks/java/io/snowflake/build.gradle | 5 +++-- sdks/java/io/snowflake/expansion-service/build.gradle | 4 +++- 2 files changed, 6 insertions(+), 3 deletions(-)
[beam] branch master updated (0cfa80f -> 9bbd5bd)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0cfa80f Remove an unused reference to staleTimerSet and reword the commentary. (#13268) new ef5aeef [BEAM-10925] Enable user-defined Java scalar functions in ZetaSQL. new feee800 address review comments new 9bbd5bd Merge pull request #13891 from ibzib/scalarfn-test The 30515 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../extensions/sql/provider/UdfTestProvider.java | 38 +++- sdks/java/extensions/sql/zetasql/build.gradle | 7 + .../sdk/extensions/sql/zetasql/SqlAnalyzer.java| 54 - .../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 61 - .../zetasql/translation/ExpressionConverter.java | 28 ++- .../extensions/sql/zetasql/ZetaSqlJavaUdfTest.java | 248 + .../sdk/extensions/sql/zetasql/ZetaSqlUdfTest.java | 26 +++ 7 files changed, 448 insertions(+), 14 deletions(-) create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
[beam] branch master updated (4be9e20 -> 654ad2b)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4be9e20 Merge pull request #13890 from Bump Dataflow worker container version. new 291ced1 Create Spark Metrics in directory using Spark History Server format new f044ec1 java spotless apply new c578c84 add --spark-history-dir option in spark job server new cb1b46f add driver distribution logs new 864603a remove start import check new f0c7412 spotless apply new ad359f5 render all metrics & add eventLogEnabled new bb7c05e add exception when eventLogEnabled is true new cac34a7 handle eventLogEnabled new e669894 apply spotless new 8be02d6 remove star import new 5392f66 run java spotless new 1da04b4 fix renderName is static from now on new 9bec263 spark.executor.id loop and switch to boolean new 31f86b8 spotless apply new c98a89e remove uncessary code new 6529c7a scope.Option.apply new 03c2457 remove options from spark job server configuration & fix the spark listener executor added logic new d3b7838 remove options from spark job server configuration & fix the spark listener executor added logic new ba68363 remove options from spark job server configuration & fix the spark listener executor added logic new 6acf592 remove options from spark job server configuration & fix the spark listener executor added logic new a513fbf remove options from spark job server configuration & fix the spark listener executor added logic new f383e17 remove options from spark job server configuration & fix the spark listener executor added logic new e105b9f minor cleanup new 068a33e minor cleanup new b35c282 minor improvments new 4b64b5f remove whitespace changes new 6329b57 remove whitespace changes new 654ad2b Merge pull request #13743 from tszerszen/sparkmetrics The 30503 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/spark/SparkPipelineOptions.java | 12 + .../beam/runners/spark/SparkPipelineRunner.java| 58 -- .../runners/spark/metrics/SparkBeamMetric.java | 24 +++-- .../runners/spark/metrics/SparkBeamMetricTest.java | 2 +- 4 files changed, 87 insertions(+), 9 deletions(-)
[beam] branch release-2.28.0 updated: [BEAM-11732] Revert flink-clients from runtime to compile configuration.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch release-2.28.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.28.0 by this push:
new be8c78e [BEAM-11732] Revert flink-clients from runtime to compile
configuration.
new a008241 Merge pull request #13874 from ibzib/flink-clients
be8c78e is described below
commit be8c78e15852898ae0ba96402b5cc01d3bc57936
Author: Kyle Weaver
AuthorDate: Mon Feb 1 11:44:21 2021 -0800
[BEAM-11732] Revert flink-clients from runtime to compile configuration.
---
runners/flink/flink_runner.gradle | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/runners/flink/flink_runner.gradle
b/runners/flink/flink_runner.gradle
index edd5882..32d7f36 100644
--- a/runners/flink/flink_runner.gradle
+++ b/runners/flink/flink_runner.gradle
@@ -138,7 +138,10 @@ dependencies {
compile library.java.slf4j_api
compile library.java.joda_time
compile library.java.args4j
- runtimeOnly "org.apache.flink:flink-clients_2.11:$flink_version"
+ compile "org.apache.flink:flink-clients_2.11:$flink_version"
+ // Runtime dependencies are not included in Beam's generated pom.xml, so we
must declare flink-clients in compile
+ // configuration (https://issues.apache.org/jira/browse/BEAM-11732).
+ permitUnusedDeclared "org.apache.flink:flink-clients_2.11:$flink_version"
compile "org.apache.flink:flink-core:$flink_version"
compile "org.apache.flink:flink-metrics-core:$flink_version"
compile "org.apache.flink:flink-java:$flink_version"
[beam] branch master updated: Setup InfluxDbIO_IT jenkins job cron
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b0d8d60 Setup InfluxDbIO_IT jenkins job cron
new 41ebec8 Merge pull request #13849 from y1chi/master
b0d8d60 is described below
commit b0d8d602763e27eb529d6f8ed5bae965098d0996
Author: Yichi Zhang
AuthorDate: Fri Jan 29 14:21:37 2021 -0800
Setup InfluxDbIO_IT jenkins job cron
---
.test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy | 1 +
1 file changed, 1 insertion(+)
diff --git a/.test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy
b/.test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy
index 8cd75ed..28b3d34 100644
--- a/.test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy
+++ b/.test-infra/jenkins/job_PerformanceTests_InfluxDBIO_IT.groovy
@@ -22,6 +22,7 @@ String jobName = "beam_PerformanceTests_InfluxDbIO_IT"
job(jobName) {
common.setTopLevelMainJobProperties(delegate)
+ common.setAutoJob(delegate,'H */6 * * *')
common.enablePhraseTriggeringFromPullRequest(
delegate,
'Java InfluxDbIO Performance Test',
[beam] branch master updated: [BEAM-11732] Revert flink-clients from runtime to compile configuration.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2930de1 [BEAM-11732] Revert flink-clients from runtime to compile
configuration.
new 423659c Merge pull request #13868 from ibzib/BEAM-11732
2930de1 is described below
commit 2930de1dbe49a82dafb7de792bd1359594b2e90d
Author: Kyle Weaver
AuthorDate: Mon Feb 1 11:44:21 2021 -0800
[BEAM-11732] Revert flink-clients from runtime to compile configuration.
---
runners/flink/flink_runner.gradle | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/runners/flink/flink_runner.gradle
b/runners/flink/flink_runner.gradle
index edd5882..32d7f36 100644
--- a/runners/flink/flink_runner.gradle
+++ b/runners/flink/flink_runner.gradle
@@ -138,7 +138,10 @@ dependencies {
compile library.java.slf4j_api
compile library.java.joda_time
compile library.java.args4j
- runtimeOnly "org.apache.flink:flink-clients_2.11:$flink_version"
+ compile "org.apache.flink:flink-clients_2.11:$flink_version"
+ // Runtime dependencies are not included in Beam's generated pom.xml, so we
must declare flink-clients in compile
+ // configuration (https://issues.apache.org/jira/browse/BEAM-11732).
+ permitUnusedDeclared "org.apache.flink:flink-clients_2.11:$flink_version"
compile "org.apache.flink:flink-core:$flink_version"
compile "org.apache.flink:flink-metrics-core:$flink_version"
compile "org.apache.flink:flink-java:$flink_version"
[beam] branch master updated: [BEAM-10379] Remove BIT_XOR from ZetaSQL supported functions list.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4ea3602 [BEAM-10379] Remove BIT_XOR from ZetaSQL supported functions
list.
new 41bd965 Merge pull request #13866 from ibzib/xor-rollback
4ea3602 is described below
commit 4ea3602dc9183327174ced07a0ad779888449918
Author: Kyle Weaver
AuthorDate: Mon Feb 1 11:12:16 2021 -0800
[BEAM-10379] Remove BIT_XOR from ZetaSQL supported functions list.
It does not pass internal compliance tests because nulls are not propagated
correctly.
---
.../sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java | 3 ++-
.../apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java | 2 ++
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
index 59f01b1..8680c38 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SupportedZetaSqlBuiltinFunctions.java
@@ -415,7 +415,8 @@ class SupportedZetaSqlBuiltinFunctions {
// JIRA link: https://issues.apache.org/jira/browse/BEAM-10379
// FunctionSignatureId.FN_BIT_AND_INT64, // bit_and
FunctionSignatureId.FN_BIT_OR_INT64, // bit_or
- FunctionSignatureId.FN_BIT_XOR_INT64, // bit_xor
+ // TODO(BEAM-10379) Re-enable when nulls are handled properly.
+ // FunctionSignatureId.FN_BIT_XOR_INT64, // bit_xor
// FunctionSignatureId.FN_LOGICAL_AND, // logical_and
// FunctionSignatureId.FN_LOGICAL_OR, // logical_or
// Approximate aggregate functions.
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
index dcf21bb..9f5bd28 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlDialectSpecTest.java
@@ -4069,6 +4069,8 @@ public class ZetaSqlDialectSpecTest extends
ZetaSqlTestBase {
}
@Test
+ @Ignore(
+ "Null values are not handled properly, so BIT_XOR is temporarily removed
from SupportedZetaSqlBuiltinFunctions.
https://issues.apache.org/jira/browse/BEAM-10379;)
public void testZetaSQLBitXor() {
String sql = "SELECT BIT_XOR(x) AS bit_xor FROM UNNEST([5678, 1234]) AS x";
ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
[beam] branch master updated: [BEAM-10925] Add rule to replace Calc with BeamCalcRel for ZetaSQL UDFs.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 5127734 [BEAM-10925] Add rule to replace Calc with BeamCalcRel for
ZetaSQL UDFs.
new a497ff2 Merge pull request #13841 from ibzib/calc-rule
5127734 is described below
commit 5127734d44e83d82776f49219bb662656e388b5b
Author: Kyle Weaver
AuthorDate: Fri Jan 29 13:12:23 2021 -0800
[BEAM-10925] Add rule to replace Calc with BeamCalcRel for ZetaSQL UDFs.
---
...taSqlCalcRule.java => BeamJavaUdfCalcRule.java} | 15
.../sql/zetasql/BeamZetaSqlCalcRule.java | 2 +-
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java| 3 ++
.../sql/zetasql/ZetaSQLQueryPlanner.java | 41 +-
4 files changed, 52 insertions(+), 9 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamJavaUdfCalcRule.java
similarity index 82%
copy from
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
copy to
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamJavaUdfCalcRule.java
index 2e7ea0f..23d0f76 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamJavaUdfCalcRule.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.zetasql;
+import org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamLogicalConvention;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.Convention;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptRule;
@@ -26,18 +27,18 @@ import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.convert.Con
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Calc;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalCalc;
-/** A {@code ConverterRule} to replace {@link Calc} with {@link
BeamZetaSqlCalcRel}. */
-public class BeamZetaSqlCalcRule extends ConverterRule {
- public static final BeamZetaSqlCalcRule INSTANCE = new BeamZetaSqlCalcRule();
+/** {@link ConverterRule} to replace {@link Calc} with {@link BeamCalcRel}. */
+public class BeamJavaUdfCalcRule extends ConverterRule {
+ public static final BeamJavaUdfCalcRule INSTANCE = new BeamJavaUdfCalcRule();
- private BeamZetaSqlCalcRule() {
+ private BeamJavaUdfCalcRule() {
super(
-LogicalCalc.class, Convention.NONE, BeamLogicalConvention.INSTANCE,
"BeamZetaSqlCalcRule");
+LogicalCalc.class, Convention.NONE, BeamLogicalConvention.INSTANCE,
"BeamJavaUdfCalcRule");
}
@Override
public boolean matches(RelOptRuleCall x) {
-return true;
+return ZetaSQLQueryPlanner.hasUdfInProjects(x);
}
@Override
@@ -45,7 +46,7 @@ public class BeamZetaSqlCalcRule extends ConverterRule {
final Calc calc = (Calc) rel;
final RelNode input = calc.getInput();
-return new BeamZetaSqlCalcRel(
+return new BeamCalcRel(
calc.getCluster(),
calc.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
RelOptRule.convert(input,
input.getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
index 2e7ea0f..2f6c60d 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRule.java
@@ -37,7 +37,7 @@ public class BeamZetaSqlCalcRule extends ConverterRule {
@Override
public boolean matches(RelOptRuleCall x) {
-return true;
+return !ZetaSQLQueryPlanner.hasUdfInProjects(x);
}
@Override
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
index f4db1f1..4889183 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
@@ -78,6 +78,9 @@ p
[beam] branch master updated: [BEAM-10925] Add optional jar path field to ZetaSqlScalarFunctionImpl.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 477fc28 [BEAM-10925] Add optional jar path field to
ZetaSqlScalarFunctionImpl.
new e47f0e5 Merge pull request #13844 from ibzib/jar-path
477fc28 is described below
commit 477fc281184cea0d2f3100507b54b9d4b3ab1bee
Author: Kyle Weaver
AuthorDate: Fri Jan 29 13:37:00 2021 -0800
[BEAM-10925] Add optional jar path field to ZetaSqlScalarFunctionImpl.
This is required to pass the jar path to BeamCalcRel (see #13835).
---
.../sql/zetasql/translation/SqlOperators.java| 20 ++--
.../translation/ZetaSqlScalarFunctionImpl.java | 13 +++--
2 files changed, 21 insertions(+), 12 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java
index fcc41a4..c4d31f2 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SqlOperators.java
@@ -134,7 +134,8 @@ public class SqlOperators {
"LIKE",
BeamBuiltinMethods.LIKE_METHOD,
SqlSyntax.BINARY,
- SqlAnalyzer.ZETASQL_FUNCTION_GROUP_NAME);
+ SqlAnalyzer.ZETASQL_FUNCTION_GROUP_NAME,
+ "");
public static final SqlOperator VALIDATE_TIMESTAMP =
createUdfOperator(
@@ -181,7 +182,9 @@ public class SqlOperators {
/**
* Create a dummy SqlFunction of type OTHER_FUNCTION from given function
name and return type.
- * These functions will be unparsed in BeamZetaSqlCalcRel and then executed
by ZetaSQL evaluator.
+ * These functions will be unparsed in either {@link
+ * org.apache.beam.sdk.extensions.sql.zetasql.BeamZetaSqlCalcRel} (for
built-in functions) or
+ * {@link org.apache.beam.sdk.extensions.sql.impl.rel.BeamCalcRel} (for
user-defined functions).
*/
public static SqlFunction createZetaSqlFunction(String name, SqlTypeName
returnType) {
return new SqlFunction(
@@ -220,16 +223,21 @@ public class SqlOperators {
null,
null,
paramTypes,
-ZetaSqlScalarFunctionImpl.create(methodClass, methodName, funGroup));
+ZetaSqlScalarFunctionImpl.create(methodClass, methodName, funGroup,
""));
+ }
+
+ static SqlUserDefinedFunction createUdfOperator(
+ String name, Method method, String funGroup, String jarPath) {
+return createUdfOperator(name, method, SqlSyntax.FUNCTION, funGroup,
jarPath);
}
static SqlUserDefinedFunction createUdfOperator(String name, Method method,
String funGroup) {
-return createUdfOperator(name, method, SqlSyntax.FUNCTION, funGroup);
+return createUdfOperator(name, method, SqlSyntax.FUNCTION, funGroup, "");
}
private static SqlUserDefinedFunction createUdfOperator(
- String name, Method method, final SqlSyntax syntax, String funGroup) {
-Function function = ZetaSqlScalarFunctionImpl.create(method, funGroup);
+ String name, Method method, final SqlSyntax syntax, String funGroup,
String jarPath) {
+Function function = ZetaSqlScalarFunctionImpl.create(method, funGroup,
jarPath);
final RelDataTypeFactory typeFactory = createTypeFactory();
List argTypes = new ArrayList<>();
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java
index 5255eec..2508ac3 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java
@@ -32,8 +32,8 @@ public class ZetaSqlScalarFunctionImpl extends
ScalarFunctionImpl {
public final String functionGroup;
private ZetaSqlScalarFunctionImpl(
- Method method, CallImplementor implementor, String functionGroup) {
-super(method, implementor);
+ Method method, CallImplementor implementor, String functionGroup, String
jarPath) {
+super(method, implementor, jarPath);
this.functionGroup = functionGroup;
}
@@ -49,8 +49,9 @@ public class ZetaSqlScalarFunctionImpl extends
ScalarFunctionImpl {
* divergent translation paths.
* @return created {@link ScalarFunction} or null
*/
- public static Function create(Class c
[beam] branch master updated (96280f7 -> bb88778)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 96280f7 Merge pull request #13835 from ibzib/calcfn-jars new b3c931a [BEAM-10925] Add Java UDFs to UserFunctionDefinitions holder. new 2e0f0a4 Add javadoc comment to JavaScalarFunction::getJarPath. new bb88778 Merge pull request #13842 from ibzib/java-udf-wrapper The 30427 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../translation/UserFunctionDefinitions.java| 21 - 1 file changed, 20 insertions(+), 1 deletion(-)
[beam] branch master updated (4d50329 -> 96280f7)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4d50329 Merge pull request #13836: [BEAM-11709] Disable Gradle module metadata generation for Beam BOM new ffb0091 [BEAM-10925] Load user-provided jars in CalcFn. new 989c317 [BEAM-10925] Fix code style new 6f4c664 Wrap classloader creation in doPrivileged block. new 96280f7 Merge pull request #13835 from ibzib/calcfn-jars The 30424 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/extensions/sql/impl/JavaUdfLoader.java | 50 +++--- .../extensions/sql/impl/ScalarFunctionImpl.java| 16 ++- .../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 42 +- 3 files changed, 99 insertions(+), 9 deletions(-)
[beam] branch master updated (96b4a58 -> 42dfc39)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 96b4a58 Merge pull request #13821: [BEAM-11658] Match .snappy files into the given (de)compressor new 997680a Implemented BIT_XOR for Zetasql dialect new 90ef75a replaced integer with Bigint type new 1f2830c added int32 field in testcase new 5062684 INT32 to INT64 new 4edb5a2 fixed spotless checks new e6b3c62 SQL_Java11 PreCommit issue fixed new adc0f2c removed * dependencies new 4f428a5 removed * dependencies from zetasql test file new e7dc079 removed * dependency from SQloperators.java new 2a7c635 Fixed check style violations new d5b520b fixed checkstyle violations from sqloperators table new 7e2e7af fixed violations from beambuiltinagg file new 7add8cf bit_Xor to support bytes as well new b195374 deleted extra introduced class new 25303ab added meaningful name for unit test new 42dfc39 Merge pull request #13569 from sonam-vend/feature/bit_xor The 30417 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../impl/transform/BeamBuiltinAggregations.java| 40 ++ .../zetasql/SupportedZetaSqlBuiltinFunctions.java | 2 +- .../translation/SqlOperatorMappingTable.java | 1 + .../sql/zetasql/translation/SqlOperators.java | 7 .../sql/zetasql/ZetaSqlDialectSpecTest.java| 13 +++ 5 files changed, 62 insertions(+), 1 deletion(-)
[beam] branch master updated: [BEAM-11689] Add public.nexus.pentaho.org to offline repositories.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 870cc22 [BEAM-11689] Add public.nexus.pentaho.org to offline
repositories.
new aab90f2 Merge pull request #13823 from ibzib/pentaho-offline
870cc22 is described below
commit 870cc2253544bc7307bb11c3f3957cfdc6759318
Author: Kyle Weaver
AuthorDate: Wed Jan 27 11:11:48 2021 -0800
[BEAM-11689] Add public.nexus.pentaho.org to offline repositories.
---
buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy | 1 +
1 file changed, 1 insertion(+)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
index 258ab08..ad4099f 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
@@ -81,6 +81,7 @@ class Repositories {
jcenter()
maven { url "https://plugins.gradle.org/m2/; }
maven { url "https://repo.spring.io/plugins-release; }
+maven { url
"https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release; }
maven { url "https://packages.confluent.io/maven/; }
maven { url project.offlineRepositoryRoot }
}
[beam] branch master updated (4edb26f -> 280107a)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4edb26f Merge pull request #13818 from iemejia/BEAM-11697-update-flink-versions new 1b6c8e3 Enabled strict dependency on flink runner new 0144bdb Added new line new e7dacc2 removed checker-qual new 75bc373 Removed flink-clients dep new 02e3666 set enableStrictDependencies:true new 5d83602 made flink-clients runtimeOnly new 248e990 Remove stray colon. new 280107a Merge pull request #13658 from sonam-vend/flink-runner The 30367 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/flink/1.10/job-server/build.gradle | 4 runners/flink/1.11/job-server/build.gradle | 4 runners/flink/1.12/job-server/build.gradle | 4 runners/flink/1.8/job-server/build.gradle | 4 runners/flink/1.9/job-server/build.gradle | 4 runners/flink/flink_runner.gradle | 12 ++-- 6 files changed, 30 insertions(+), 2 deletions(-)
[beam] branch master updated: [BEAM-11697] Upgrade Flink runner to Flink versions 1.12.1 and 1.11.3
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new cb4c2de [BEAM-11697] Upgrade Flink runner to Flink versions 1.12.1
and 1.11.3
new 4edb26f Merge pull request #13818 from
iemejia/BEAM-11697-update-flink-versions
cb4c2de is described below
commit cb4c2de4fe5dc289db75853f0acbb481c7532f72
Author: Ismaël MejÃa
AuthorDate: Wed Jan 27 10:33:05 2021 +0100
[BEAM-11697] Upgrade Flink runner to Flink versions 1.12.1 and 1.11.3
---
runners/flink/1.11/build.gradle | 2 +-
runners/flink/1.12/build.gradle | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/runners/flink/1.11/build.gradle b/runners/flink/1.11/build.gradle
index b1b2e11..229d569 100644
--- a/runners/flink/1.11/build.gradle
+++ b/runners/flink/1.11/build.gradle
@@ -20,7 +20,7 @@ def basePath = '..'
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.11.1'
+ flink_version = '1.11.3'
// Version specific code overrides.
main_source_overrides = ["${basePath}/1.8/src/main/java",
"${basePath}/1.9/src/main/java", "${basePath}/1.10/src/main/java",
'./src/main/java']
test_source_overrides = ["${basePath}/1.8/src/test/java",
"${basePath}/1.9/src/test/java", "${basePath}/1.10/src/test/java",
'./src/test/java']
diff --git a/runners/flink/1.12/build.gradle b/runners/flink/1.12/build.gradle
index 5bfa7e3..d716862 100644
--- a/runners/flink/1.12/build.gradle
+++ b/runners/flink/1.12/build.gradle
@@ -20,7 +20,7 @@ def basePath = '..'
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.12.0'
+ flink_version = '1.12.1'
// Version specific code overrides.
main_source_overrides = ["${basePath}/1.8/src/main/java",
"${basePath}/1.9/src/main/java", "${basePath}/1.10/src/main/java",
"${basePath}/1.11/src/main/java", './src/main/java']
test_source_overrides = ["${basePath}/1.8/src/test/java",
"${basePath}/1.9/src/test/java", "${basePath}/1.10/src/test/java",
"${basePath}/1.11/src/test/java", './src/test/java']
[beam] branch master updated: [BEAM-11689] Use public.nexus.pentaho.org for pentaho dependencies instead of Spring.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new dca92f7 [BEAM-11689] Use public.nexus.pentaho.org for pentaho
dependencies instead of Spring.
new 3d7c3a7 Merge pull request #13813 from ibzib/BEAM-11689
dca92f7 is described below
commit dca92f7f07e1deb4c42280d8d0164c0928cfadc6
Author: Kyle Weaver
AuthorDate: Tue Jan 26 14:24:53 2021 -0800
[BEAM-11689] Use public.nexus.pentaho.org for pentaho dependencies instead
of Spring.
The Spring-hosted repository has been made password-protected.
---
buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
index 13a0c10..258ab08 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
@@ -40,9 +40,9 @@ class Repositories {
mavenLocal()
jcenter()
- // Spring only for resolving pentaho dependency.
+ // For pentaho dependencies.
maven {
-url "https://repo.spring.io/plugins-release/;
+url
"https://public.nexus.pentaho.org/repository/proxy-public-3rd-party-release;
content { includeGroup "org.pentaho" }
}
[beam] branch master updated: [BEAM-9541] Create a Gradle task to push all docker images, and use it instead of bash when building the release candidate.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4c81134 [BEAM-9541] Create a Gradle task to push all docker images,
and use it instead of bash when building the release candidate.
new f4f4611 Merge pull request #13811 from ibzib/flink-docker-push
4c81134 is described below
commit 4c81134dd9206fd6cb9060a9b973fc111b8fbd64
Author: Kyle Weaver
AuthorDate: Tue Jan 26 13:05:26 2021 -0800
[BEAM-9541] Create a Gradle task to push all docker images, and use it
instead of bash when building the release candidate.
Bash arrays make no sense and I've seen the ones in this file cause at
least two bugs already.
Delegating to Gradle also lets as many of the tasks run in parallel as the
OS allows.
Also use LOCAL_CLONE_DIR instead of LOCAL_PYTHON_STAGING_DIR for clarity,
and rm it afterwards.
---
build.gradle | 9 +
.../src/main/scripts/build_release_candidate.sh| 38 --
sdks/java/container/build.gradle | 5 +++
sdks/python/container/build.gradle | 6
4 files changed, 26 insertions(+), 32 deletions(-)
diff --git a/build.gradle b/build.gradle
index 12f355d..d1c5857 100644
--- a/build.gradle
+++ b/build.gradle
@@ -307,6 +307,15 @@ task typescriptPreCommit() {
dependsOn ":sdks:python:test-suites:tox:py38:jest"
}
+task pushAllDockerImages() {
+ dependsOn ":runners:spark:job-server:container:dockerPush"
+ dependsOn ":sdks:java:container:pushAll"
+ dependsOn ":sdks:python:container:pushAll"
+ for (version in project.ext.allFlinkVersions) {
+dependsOn ":runners:flink:${version}:job-server-container:dockerPush"
+ }
+}
+
// Configure the release plugin to do only local work; the release manager
determines what, if
// anything, to push. On failure, the release manager can reset the branch
without pushing.
release {
diff --git a/release/src/main/scripts/build_release_candidate.sh
b/release/src/main/scripts/build_release_candidate.sh
index 1c618ca..8c57261 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -57,13 +57,6 @@ PYTHON_ARTIFACTS_DIR=python
BEAM_ROOT_DIR=beam
WEBSITE_ROOT_DIR=beam-site
-DOCKER_IMAGE_DEFAULT_REPO_ROOT=apache
-DOCKER_IMAGE_DEFAULT_REPO_PREFIX=beam_
-
-JAVA_VER=("java8" "java11")
-PYTHON_VER=("python3.6" "python3.7" "python3.8")
-FLINK_VER=("1.8" "1.9" "1.10")
-
echo "Setting Up Environment Variables==="
echo "Which release version are you working on: "
read RELEASE
@@ -257,39 +250,20 @@ read confirmation
if [[ $confirmation = "y" ]]; then
echo "Staging SDK docker images on docker hub="
cd ~
- if [[ -d ${LOCAL_PYTHON_STAGING_DIR} ]]; then
-rm -rf ${LOCAL_PYTHON_STAGING_DIR}
+ if [[ -d ${LOCAL_CLONE_DIR} ]]; then
+rm -rf ${LOCAL_CLONE_DIR}
fi
- mkdir -p ${LOCAL_PYTHON_STAGING_DIR}
- cd ${LOCAL_PYTHON_STAGING_DIR}
+ mkdir -p ${LOCAL_CLONE_DIR}
+ cd ${LOCAL_CLONE_DIR}
echo '---Cloning Beam Release Branch-'
git clone ${GIT_REPO_URL}
cd ${BEAM_ROOT_DIR}
git checkout ${RELEASE_BRANCH}
- echo '---Generating and Pushing Python
images-'
- ./gradlew :sdks:python:container:buildAll -Pdocker-pull-licenses
-Pdocker-tag=${RELEASE}_rc${RC_NUM}
- for ver in "${PYTHON_VER[@]}"; do
-docker push
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}_rc${RC_NUM}
&
- done
+ ./gradlew :pushAllDockerImages -Pdocker-pull-licenses
-Pdocker-tag=${RELEASE}_rc${RC_NUM}
- echo '---Generating and Pushing Java images-'
- echo "Building containers for the following Java versions:" "${JAVA_VER[@]}"
- for ver in "${JAVA_VER[@]}"; do
-./gradlew :sdks:java:container:${ver}:dockerPush -Pdocker-pull-licenses
-Pdocker-tag=${RELEASE}_rc${RC_NUM}
- done
-
- echo '-Generating and Pushing Flink job server
images-'
- echo "Building containers for the following Flink versions:"
"${FLINK_VER[@]}"
- for ver in "${FLINK_VER[@]}"; do
-./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
- done
-
- echo '-Generating and Pushing Spark job server
image-'
- ./gradlew ":runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
-
- rm -rf ~/${PYTHON_ARTIFACTS_DI
[beam] branch master updated: [BEAM-9541] Update Python SDK's Flink version list.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new f2c9b6e [BEAM-9541] Update Python SDK's Flink version list. new 7c4a21a8 Merge pull request #13810 from ibzib/flink-python-version f2c9b6e is described below commit f2c9b6e1aa5d38385f4c168107c85d4fe7f0f259 Author: Kyle Weaver AuthorDate: Tue Dec 29 14:24:13 2020 -0800 [BEAM-9541] Update Python SDK's Flink version list. --- sdks/python/apache_beam/options/pipeline_options.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/options/pipeline_options.py b/sdks/python/apache_beam/options/pipeline_options.py index 96f445a..bda9df1 100644 --- a/sdks/python/apache_beam/options/pipeline_options.py +++ b/sdks/python/apache_beam/options/pipeline_options.py @@ -1214,7 +1214,8 @@ class JobServerOptions(PipelineOptions): class FlinkRunnerOptions(PipelineOptions): - PUBLISHED_FLINK_VERSIONS = ['1.7', '1.8', '1.9', '1.10'] + # These should stay in sync with gradle.properties. + PUBLISHED_FLINK_VERSIONS = ['1.8', '1.9', '1.10', '1.11', '1.12'] @classmethod def _add_argparse_args(cls, parser):
[beam] branch master updated: [BEAM-9541] Push all docker images from RC instead of hard-coding them.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 34ce707 [BEAM-9541] Push all docker images from RC instead of
hard-coding them.
new 867a298 Merge pull request #13798 from ibzib/container-pull
34ce707 is described below
commit 34ce707370f2b0ff539e0ac24c84485aeb07207e
Author: Kyle Weaver
AuthorDate: Fri Jan 22 17:21:02 2021 -0800
[BEAM-9541] Push all docker images from RC instead of hard-coding them.
This greatly simplifies the script. It also spares us from having to
hard-code yet another list of supported Flink versions, which I'm trying to
avoid. There is the minor downside that listing images requires them to be
present locally from build_release_candidate.sh, so we can't clean them up in
between builds. However, I don't think that was ever really necessary except to
save disk space.
---
.../src/main/scripts/build_release_candidate.sh| 10 ---
release/src/main/scripts/publish_docker_images.sh | 87 +++---
.../site/content/en/contribute/release-guide.md| 4 +-
3 files changed, 13 insertions(+), 88 deletions(-)
diff --git a/release/src/main/scripts/build_release_candidate.sh
b/release/src/main/scripts/build_release_candidate.sh
index f846e82..1c618ca 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -290,16 +290,6 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
rm -rf ~/${PYTHON_ARTIFACTS_DIR}
-
- echo '---Clean up images at local-'
- for ver in "${PYTHON_VER[@]}"; do
- docker rmi -f
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}_rc${RC_NUM}
- done
- docker rmi -f
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}java_sdk:${RELEASE}_rc${RC_NUM}
- for ver in "${FLINK_VER[@]}"; do
-docker rmi -f
"${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}flink${ver}_job_server:${RELEASE}_rc${RC_NUM}"
- done
- docker rmi -f
"${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}spark_job_server:${RELEASE}_rc${RC_NUM}"
fi
echo "[Current Step]: Update beam-site"
diff --git a/release/src/main/scripts/publish_docker_images.sh
b/release/src/main/scripts/publish_docker_images.sh
index c40e7a2..6bf013b 100755
--- a/release/src/main/scripts/publish_docker_images.sh
+++ b/release/src/main/scripts/publish_docker_images.sh
@@ -27,10 +27,6 @@ set -e
DOCKER_IMAGE_DEFAULT_REPO_ROOT=apache
DOCKER_IMAGE_DEFAULT_REPO_PREFIX=beam_
-JAVA_VER=("java8" "java11")
-PYTHON_VER=("python3.6" "python3.7" "python3.8")
-FLINK_VER=("1.8" "1.9" "1.10")
-
echo "Publish SDK docker images to Docker Hub."
echo "Setting Up Environment Variables==="
@@ -47,84 +43,21 @@ echo "Do you want to proceed? [y|N]"
read confirmation
if [[ $confirmation = "y" ]]; then
- echo '---Tagging and Pushing Python images-'
- for ver in "${PYTHON_VER[@]}"; do
-# Pull verified RC from dockerhub.
-docker pull
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}_${RC_VERSION}
-
-# Tag with ${RELEASE} and push to dockerhub.
-docker tag
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}_${RC_VERSION}
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}
-docker push
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}
-
-# Tag with latest and push to dockerhub.
-docker tag
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}_${RC_VERSION}
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:latest
-docker push
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:latest
-
-# Cleanup images from local
-docker rmi -f
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}_${RC_VERSION}
-docker rmi -f
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:${RELEASE}
-docker rmi -f
${DOCKER_IMAGE_DEFAULT_REPO_ROOT}/${DOCKER_IMAGE_DEFAULT_REPO_PREFIX}${ver}_sdk:latest
- done
-
- echo '---Tagging and Pushing Java images-'
- for ver in "${JAVA_VER[@]}"; do
-# Pull verified RC from dockerhub.
-docker pull
${DOCKER_IMAGE_DEFAULT
[beam] branch master updated (67989ca -> 4f43726)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 67989ca Merge pull request #13806: BEAM-4986: Bump Apache Parquet to
1.11.1
new 71b7d8b [BEAM-10925] Load SQL UDFs from jar.
new 25b8903 Move JavaUdfLoader from zetasql -> sql.
new ee4d14a Make JavaUdfDefinitions a subclass of JavaUdfLoader.
new 1fbf45c Don't set context classloader.
new 4f43726 Merge pull request #13629 from ibzib/udf-loader
The 30341 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
sdks/java/extensions/sql/build.gradle | 14 ++
.../sdk/extensions/sql/impl/JavaUdfLoader.java | 193 +
.../sdk/extensions/sql/impl/JavaUdfLoaderTest.java | 100 +++
.../{perf-tests => udf-test-provider}/build.gradle | 22 +--
.../extensions/sql/provider/UdfTestProvider.java} | 31 ++--
.../sdk/extensions/sql/provider}/package-info.java | 4 +-
settings.gradle| 1 +
7 files changed, 339 insertions(+), 26 deletions(-)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JavaUdfLoader.java
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/JavaUdfLoaderTest.java
copy sdks/java/extensions/sql/{perf-tests => udf-test-provider}/build.gradle
(68%)
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/util/WindowTracing.java
=>
extensions/sql/udf-test-provider/src/main/java/org/apache/beam/sdk/extensions/sql/provider/UdfTestProvider.java}
(58%)
copy
{examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options
=>
sdks/java/extensions/sql/udf-test-provider/src/main/java/org/apache/beam/sdk/extensions/sql/provider}/package-info.java
(88%)
[beam] branch master updated (d7bee01 -> 809a0b7)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d7bee01 strict dependency for some sdks/java/io modules - trivial changes (#13794) new 9bc7820 [BEAM-9541] Jenkins tests use latest Flink version. new 704bae9 Write latest Flink version in CommonTestProperties. new 809a0b7 Merge pull request #13675 from ibzib/flink-jenkins The 30308 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/jenkins/CommonTestProperties.groovy | 6 +- ...job_PostCommit_CrossLanguageValidatesRunner_Flink.groovy | 3 ++- .../jenkins/job_PostCommit_Java_Nexmark_Flink.groovy| 13 +++-- ...stCommit_Java_PortableValidatesRunner_Flink_Batch.groovy | 3 ++- ...mmit_Java_PortableValidatesRunner_Flink_Streaming.groovy | 5 +++-- .../job_PostCommit_Java_ValidatesRunner_Flink.groovy| 3 ++- .../job_PostCommit_Java_ValidatesRunner_Flink_Java11.groovy | 7 --- .test-infra/jenkins/job_Publish_Docker_Snapshots.groovy | 3 ++- 8 files changed, 27 insertions(+), 16 deletions(-)
[beam] branch master updated (9e0920c -> d7bee01)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9e0920c [BEAM-11542] Add support for projecting GroupBy objects (#13637) add d7bee01 strict dependency for some sdks/java/io modules - trivial changes (#13794) No new revisions were added by this update. Summary of changes: sdks/java/io/amqp/build.gradle| 4 +--- sdks/java/io/cassandra/build.gradle | 1 + sdks/java/io/common/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/build.gradle | 1 + sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle | 1 + sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle | 1 + sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/build.gradle | 1 + .../io/elasticsearch-tests/elasticsearch-tests-common/build.gradle| 1 + sdks/java/io/hadoop-common/build.gradle | 4 +--- sdks/java/io/hadoop-file-system/build.gradle | 4 +--- sdks/java/io/hbase/build.gradle | 2 +- sdks/java/io/kudu/build.gradle| 3 +-- sdks/java/io/mongodb/build.gradle | 4 +--- sdks/java/io/rabbitmq/build.gradle| 4 +--- sdks/java/io/redis/build.gradle | 4 +--- 15 files changed, 15 insertions(+), 22 deletions(-)
[beam] branch master updated (da2f431 -> 52dcc36)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from da2f431 [BEAM-11570] Disable classloader check for [auto] mode. (#13722) add 52dcc36 [BEAM-10961] Enabled strict dependency on Direct Java runner (#13656) No new revisions were added by this update. Summary of changes: runners/direct-java/build.gradle | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-)
[beam] branch master updated (0460daa -> da2f431)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0460daa [BEAM-10961] Enabled strict dependency on Jet Runner (#13663) add da2f431 [BEAM-11570] Disable classloader check for [auto] mode. (#13722) No new revisions were added by this update. Summary of changes: .../beam/runners/flink/FlinkExecutionEnvironments.java| 15 ++- 1 file changed, 14 insertions(+), 1 deletion(-)
[beam] branch master updated (cf402a1 -> 0460daa)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cf402a1 [BEAM-10961] Enabled strict dependency on SDKS-Testing (#13689) add 0460daa [BEAM-10961] Enabled strict dependency on Jet Runner (#13663) No new revisions were added by this update. Summary of changes: runners/jet/build.gradle | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
[beam] branch master updated (5d58250 -> cf402a1)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5d58250 [BEAM-10961] enabled strict dependency on sdks-core and sdks-expansion-service (#13685) add cf402a1 [BEAM-10961] Enabled strict dependency on SDKS-Testing (#13689) No new revisions were added by this update. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 4 sdks/java/testing/expansion-service/build.gradle | 7 +++ sdks/java/testing/jpms-tests/build.gradle| 2 +- sdks/java/testing/kafka-service/build.gradle | 2 +- sdks/java/testing/load-tests/build.gradle| 8 ++-- sdks/java/testing/nexmark/build.gradle | 9 - sdks/java/testing/test-utils/build.gradle| 12 7 files changed, 27 insertions(+), 17 deletions(-)
[beam] branch master updated (4141550 -> 5d58250)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4141550 [BEAM-10961] Enabled strict dependencies on example module (#13649) add 5d58250 [BEAM-10961] enabled strict dependency on sdks-core and sdks-expansion-service (#13685) No new revisions were added by this update. Summary of changes: sdks/java/container/license_scripts/dep_urls_java.yaml | 4 ++-- sdks/java/core/build.gradle| 5 - sdks/java/expansion-service/build.gradle | 3 ++- 3 files changed, 8 insertions(+), 4 deletions(-)
[beam] branch master updated (262273c -> 4141550)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 262273c [BEAM-10961] Enabled strict Dependency on Portability (#13666) add 4141550 [BEAM-10961] Enabled strict dependencies on example module (#13649) No new revisions were added by this update. Summary of changes: examples/java/build.gradle | 13 - examples/kotlin/build.gradle | 14 ++ 2 files changed, 18 insertions(+), 9 deletions(-)
[beam] branch master updated (8b759d1 -> 262273c)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8b759d1 Merge pull request #13779 from [BEAM-11677] Expose commit_offset_in_finalize and timestamp_policy to ReadFromKafka add 262273c [BEAM-10961] Enabled strict Dependency on Portability (#13666) No new revisions were added by this update. Summary of changes: runners/portability/java/build.gradle | 11 +-- 1 file changed, 9 insertions(+), 2 deletions(-)
[beam] branch master updated (61ea255 -> a9e31ba)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 61ea255 [BEAM-11092][Go SDK] Ack Harness monitoring requests (#13788) add a9e31ba [BEAM-10961] Enabled strict dependencies on samza (#13667) No new revisions were added by this update. Summary of changes: runners/samza/build.gradle| 21 +++-- runners/samza/job-server/build.gradle | 2 ++ 2 files changed, 13 insertions(+), 10 deletions(-)
[beam] branch master updated (d1c8c24 -> 7e87324)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d1c8c24 [BEAM-10961] Enabled strict dependencies on java-job-service (#13662) add 7e87324 [BEAM-10961] Enabled Strict dependency on Fn-execution Runner (#13661) No new revisions were added by this update. Summary of changes: runners/java-fn-execution/build.gradle | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-)
[beam] branch master updated (0bfdf6c -> d1c8c24)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0bfdf6c Merge pull request #13657 from sonam-vend/runner-extension-java add d1c8c24 [BEAM-10961] Enabled strict dependencies on java-job-service (#13662) No new revisions were added by this update. Summary of changes: runners/java-job-service/build.gradle | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-)
[beam] branch master updated (eed16a5 -> 0bfdf6c)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from eed16a5 Merge pull request #13650 from sonam-vend/runner-core-construction new c5e6cc5 Enabled strict dependency on Extensions Java runner new a9c6cb6 set enableStrictDependencies: true new 0bfdf6c Merge pull request #13657 from sonam-vend/runner-extension-java The 30278 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/extensions-java/metrics/build.gradle | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[beam] branch master updated (b131cc88 -> eed16a5)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b131cc88 [BEAM-10961] Enabled strict dependencies on sdks-fn-execution and sdks-harness (#13687) new 6628a27 Enabled strict dependency on core-construction runnber module new 51a495f set enableStrictDependencies: true new eed16a5 Merge pull request #13650 from sonam-vend/runner-core-construction The 30275 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/core-construction-java/build.gradle | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (c27b8d6 -> b131cc88)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c27b8d6 [BEAM-10961] Enabled strict dependency on Twister2 (#13669) add b131cc88 [BEAM-10961] Enabled strict dependencies on sdks-fn-execution and sdks-harness (#13687) No new revisions were added by this update. Summary of changes: sdks/java/fn-execution/build.gradle | 3 +-- sdks/java/harness/build.gradle | 5 + 2 files changed, 2 insertions(+), 6 deletions(-)
[beam] branch master updated (8d6fd6a -> c27b8d6)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8d6fd6a [BEAM-10961] Enabled strict dependency on Core Java runner (#13654) add c27b8d6 [BEAM-10961] Enabled strict dependency on Twister2 (#13669) No new revisions were added by this update. Summary of changes: runners/twister2/build.gradle | 17 - 1 file changed, 16 insertions(+), 1 deletion(-)
[beam] branch master updated (35a49d1 -> 8d6fd6a)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 35a49d1 Kafka To Pub/Sub Template Blogpost (#13365) add 8d6fd6a [BEAM-10961] Enabled strict dependency on Core Java runner (#13654) No new revisions were added by this update. Summary of changes: runners/core-java/build.gradle | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam] branch master updated (9f4ed60 -> cd1f664)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9f4ed60 [BEAM-10961] Enabled strict dependency on Spark (#13668) new 4f32816 Enabled strict dependency on Local Java new b538b59 set enableStrictDependencies: true new cd1f664 Merge pull request #13665 from sonam-vend/local-java-runner The 30261 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/local-java/build.gradle | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[beam] branch master updated (cdc7226 -> 9f4ed60)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cdc7226 Merge pull request #13774 from pcoet/BEAM-10095 add 9f4ed60 [BEAM-10961] Enabled strict dependency on Spark (#13668) No new revisions were added by this update. Summary of changes: runners/spark/build.gradle| 7 ++- runners/spark/job-server/build.gradle | 4 +++- 2 files changed, 9 insertions(+), 2 deletions(-)
[beam] branch master updated: BEAM-10095: Add Runner and SDK links to Beam overview page
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 5a360a8 BEAM-10095: Add Runner and SDK links to Beam overview page new cdc7226 Merge pull request #13774 from pcoet/BEAM-10095 5a360a8 is described below commit 5a360a86d74f40af4d92dc6297190d5847acdaa8 Author: David Huntsperger AuthorDate: Tue Jan 19 19:00:35 2021 + BEAM-10095: Add Runner and SDK links to Beam overview page --- .../site/content/en/get-started/beam-overview.md | 24 -- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/website/www/site/content/en/get-started/beam-overview.md b/website/www/site/content/en/get-started/beam-overview.md index 73c05eb..40aacb0 100644 --- a/website/www/site/content/en/get-started/beam-overview.md +++ b/website/www/site/content/en/get-started/beam-overview.md @@ -23,7 +23,7 @@ limitations under the License. Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam's supported **distributed processing back-ends**, which include [Apache Flink](https://flink.apache.org), [Apache Spark](http://spark.apache.org), and [Google Cloud Dataflow](https://cloud.google.com/dataflow). -Beam is particularly useful for [Embarrassingly Parallel](https://en.wikipedia.org/wiki/Embarassingly_parallel) data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, [...] +Beam is particularly useful for [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarassingly_parallel) data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, [...] ## Apache Beam SDKs @@ -31,9 +31,9 @@ The Beam SDKs provide a unified programming model that can represent and transfo Beam currently supports the following language-specific SDKs: -- Java  -- Python  -- Go +- [Apache Beam Java SDK](/documentation/sdks/java)  +- [Apache Beam Python SDK](/documentation/sdks/python)  +- [Apache Beam Go SDK](/documentation/sdks/go) A Scala interface is also available as [Scio](https://github.com/spotify/scio). @@ -41,14 +41,16 @@ A Scala The Beam Pipeline Runners translate the data processing pipeline you define with your Beam program into the API compatible with the distributed processing back-end of your choice. When you run your Beam program, you'll need to specify an [appropriate runner](/documentation/runners/capability-matrix) for the back-end where you want to execute your pipeline. -Beam currently supports Runners that work with the following distributed processing back-ends: +Beam currently supports the following runners: -- Apache Flink  -- Apache Samza -- Apache Spark  -- Google Cloud Dataflow  -- Hazelcast Jet  -- Twister2  +- [Direct Runner](/documentation/runners/direct) +- [Apache Flink Runner](/documentation/runners/flink)  +- [Apache Nemo Runner](/documentation/runners/nemo) +- [Apache Samza Runner](/documentation/runners/samza) +- [Apache Spark Runner](/documentation/runners/spark)  +- [Google Cloud Dataflow Runner](/documentation/runners/dataflow)  +- [Hazelcast Jet Runner](/documentation/runners/jet)  +- [Twister2 Runner](/documentation/runners/twister2)  **Note:** You can always execute your pipeline locally for testing and debugging purposes.
[beam] branch master updated (17c7162 -> f371a26)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 17c7162 Merge pull request #13753 from TobKed/gradle-command-action-as-submodule new 8ea198a Enabled strict dependency new ba7702a removed * depdendency new 8d7eb82 enabled strict dependency check to false new 7f8c721 Fixed checker-qual issue new 7d172ad removed * dep new f39bc2d Fixed requested changes new 54902d8 autovalue_version variable added new a68b9e7 resolved merge conflicts new 52f831a Alphabetize dependency lists. new f371a26 Merge pull request #13690 from sonam-vend/feature/strict-dependency The 30230 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 12 1 file changed, 8 insertions(+), 4 deletions(-)
[beam] branch master updated: Add gradle-command-action as submodule
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1dd1513 Add gradle-command-action as submodule
new 17c7162 Merge pull request #13753 from
TobKed/gradle-command-action-as-submodule
1dd1513 is described below
commit 1dd1513b483f0fa49cfcc062736bda73c2f2e835
Author: Tobiasz Kędzierski
AuthorDate: Thu Jan 14 15:00:20 2021 +0100
Add gradle-command-action as submodule
---
.github/actions/gradle-command-action | 1 +
.github/workflows/java_tests.yml | 19 ++-
.gitmodules | 3 +++
3 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/.github/actions/gradle-command-action
b/.github/actions/gradle-command-action
new file mode 16
index 000..90ccf05
--- /dev/null
+++ b/.github/actions/gradle-command-action
@@ -0,0 +1 @@
+Subproject commit 90ccf054e6b9905f30f98c938bce4c6acd323b6b
diff --git a/.github/workflows/java_tests.yml b/.github/workflows/java_tests.yml
index 62c51fb..0b8cf2f 100644
--- a/.github/workflows/java_tests.yml
+++ b/.github/workflows/java_tests.yml
@@ -67,9 +67,12 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v2
+with:
+ persist-credentials: false
+ submodules: recursive
# :sdks:java:core:test
- name: Run :sdks:java:core:test
-uses: eskatos/gradle-command-action@v1
+uses: ./.github/actions/gradle-command-action
with:
arguments: :sdks:java:core:test
- name: Upload test logs for :sdks:java:core:test
@@ -80,7 +83,7 @@ jobs:
path: sdks/java/core/build/reports/tests/test
# :sdks:java:harness:test
- name: Run :sdks:java:harness:test
-uses: eskatos/gradle-command-action@v1
+uses: ./.github/actions/gradle-command-action
with:
arguments: :sdks:java:harness:test
if: always()
@@ -92,7 +95,7 @@ jobs:
path: sdks/java/harness/build/reports/tests/test
# :runners:core-java:test
- name: Run :runners:core-java:test
-uses: eskatos/gradle-command-action@v1
+uses: ./.github/actions/gradle-command-action
with:
arguments: :runners:core-java:test
if: always()
@@ -113,8 +116,11 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v2
+with:
+ persist-credentials: false
+ submodules: recursive
- name: Run WordCount Unix
-uses: eskatos/gradle-command-action@v1
+uses: ./.github/actions/gradle-command-action
with:
arguments: -p examples/ integrationTest --tests
org.apache.beam.examples.WordCountIT
-DintegrationTestRunner=direct
@@ -143,6 +149,9 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v2
+with:
+ persist-credentials: false
+ submodules: recursive
- name: Authenticate on GCP
uses: google-github-actions/setup-gcloud@master
with:
@@ -151,7 +160,7 @@ jobs:
project_id: ${{ secrets.GCP_PROJECT_ID }}
export_default_credentials: true
- name: Run WordCount
-uses: eskatos/gradle-command-action@v1
+uses: ./.github/actions/gradle-command-action
with:
arguments: -p examples/ integrationTest --tests
org.apache.beam.examples.WordCountIT
-DintegrationTestPipelineOptions=["--runner=DataflowRunner","--project=${{
secrets.GCP_PROJECT_ID }}","--tempRoot=gs://${{ secrets.GCP_TESTING_BUCKET
}}/tmp/"]
diff --git a/.gitmodules b/.gitmodules
index 3a6406c..fa6e30a 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -7,3 +7,6 @@
[submodule ".github/actions/github-push-action"]
path = .github/actions/github-push-action
url = https://github.com/ad-m/github-push-action
+[submodule ".github/actions/gradle-command-action"]
+ path = .github/actions/gradle-command-action
+ url = https://github.com/eskatos/gradle-command-action
[beam] branch master updated (7f93e42 -> 75d3864)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7f93e42 Merge pull request #13711 from emilymye/fixjavaartifact new 51183ac Add version and sdk_version as inputs to ProcessResources task. new 2c63124 Also add docker image to sdk.properties new a1d97b8 Fix incorrect input property new 0a7edfc Update sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties new 75d3864 Merge pull request #13691 from runningcode/no/sdk-properties The 30192 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/core/build.gradle | 4 sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties | 2 ++ 2 files changed, 6 insertions(+)
[beam] branch master updated (15d282b -> 7f93e42)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 15d282b Merge pull request #13699: Update Dataflow worker harness container image version. new 1396a70 don't use artifact to output java 3rd party licenses new ca207f6 remove java 3pl directory from copy new 7f93e42 Merge pull request #13711 from emilymye/fixjavaartifact The 30187 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/container/build.gradle | 4 +--- sdks/java/container/common.gradle | 10 +- 2 files changed, 6 insertions(+), 8 deletions(-)
[beam] branch master updated: [BEAM-11579] Specify flink_job_server_jar instead of using default.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3fa9846 [BEAM-11579] Specify flink_job_server_jar instead of using
default.
new 4e83ac5 Merge pull request #13682 from ibzib/BEAM-11579
3fa9846 is described below
commit 3fa98460d0540997da9027465f599c74ee414449
Author: Kyle Weaver
AuthorDate: Wed Jan 6 19:58:13 2021 -0800
[BEAM-11579] Specify flink_job_server_jar instead of using default.
The default reads from pipeline_options.py, which hasn't been updated from
Flink 1.10 to 1.12 yet.
---
sdks/python/test-suites/portable/common.gradle | 9 -
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/sdks/python/test-suites/portable/common.gradle
b/sdks/python/test-suites/portable/common.gradle
index 2793768..cb1b5ff 100644
--- a/sdks/python/test-suites/portable/common.gradle
+++ b/sdks/python/test-suites/portable/common.gradle
@@ -181,10 +181,17 @@ project.task("postCommitPy${pythonVersionSuffix}IT") {
"apache_beam.io.gcp.tests.xlang_spannerio_it_test",
]
def testOpts = ["--tests=${tests.join(',')}"]
+def pipelineOpts = [
+"--runner=FlinkRunner",
+"--project=apache-beam-testing",
+"--environment_type=LOOPBACK",
+"--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it",
+
"--flink_job_server_jar=${project(":runners:flink:${latestFlinkVersion}:job-server").shadowJar.archivePath}",
+]
def cmdArgs = mapToArgString([
"test_opts": testOpts,
"suite": "postCommitIT-flink-py${pythonVersionSuffix}",
-"pipeline_opts": "--runner=FlinkRunner
--project=apache-beam-testing --environment_type=LOOPBACK
--temp_location=gs://temp-storage-for-end-to-end-tests/temp-it",
+"pipeline_opts": pipelineOpts.join(" "),
])
def kafkaJar =
project(":sdks:java:testing:kafka-service:").buildTestKafkaServiceJar.archivePath
exec {
[beam] branch master updated: [BEAM-11419] Update Flink supported versions in the website
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 2239b3a [BEAM-11419] Update Flink supported versions in the website new c5badae Merge pull request #13620 from iemejia/website-flink-versions 2239b3a is described below commit 2239b3a1fb1fbc1e36d7454bcae7bb7323031ebe Author: Ismaël MejÃa AuthorDate: Mon Dec 28 15:19:28 2020 +0100 [BEAM-11419] Update Flink supported versions in the website --- .../site/content/en/documentation/runners/flink.md | 42 +- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/website/www/site/content/en/documentation/runners/flink.md b/website/www/site/content/en/documentation/runners/flink.md index d173f70..e1f814c 100644 --- a/website/www/site/content/en/documentation/runners/flink.md +++ b/website/www/site/content/en/documentation/runners/flink.md @@ -326,7 +326,45 @@ To find out which version of Flink is compatible with Beam please see the table Artifact Id - 2.21.0 + 2.27.0 + 1.12.x * + beam-runners-flink-1.12 + + + 1.11.x * + beam-runners-flink-1.11 + + + 1.10.x + beam-runners-flink-1.10 + + + 1.9.x + beam-runners-flink-1.9 + + + 1.8.x + beam-runners-flink-1.8 + + + 2.25.0 - 2.26.0 + 1.11.x * + beam-runners-flink-1.11 + + + 1.10.x + beam-runners-flink-1.10 + + + 1.9.x + beam-runners-flink-1.9 + + + 1.8.x + beam-runners-flink-1.8 + + + 2.21.0 - 2.24.0 1.10.x beam-runners-flink-1.10 @@ -421,6 +459,8 @@ To find out which version of Flink is compatible with Beam please see the table +* This version does not have a published docker image for the Flink Job Service. + For retrieving the right Flink version, see the [Flink downloads page](https://flink.apache.org/downloads.html). For more information, the [Flink Documentation](https://ci.apache.org/projects/flink/flink-docs-stable/) can be helpful.
[beam] branch master updated: [BEAM-11504] Clean up direct runner parallelism section.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 36bf02c [BEAM-11504] Clean up direct runner parallelism section.
new d72b5d9 Merge pull request #13589 from ibzib/BEAM-11504
36bf02c is described below
commit 36bf02cfb2b37a894f6555c0038f06ba6a779e3a
Author: Kyle Weaver
AuthorDate: Mon Dec 21 13:12:10 2020 -0800
[BEAM-11504] Clean up direct runner parallelism section.
- Remove unparseable language tags.
- Add corresponding Java instructions.
- Remove obsolete (version < 2.19) Python instructions.
- Defer to programming guide for general instructions for setting pipeline
options.
---
.../content/en/documentation/runners/direct.md | 112 +
1 file changed, 25 insertions(+), 87 deletions(-)
diff --git a/website/www/site/content/en/documentation/runners/direct.md
b/website/www/site/content/en/documentation/runners/direct.md
index 0168dcc..1249aa9 100644
--- a/website/www/site/content/en/documentation/runners/direct.md
+++ b/website/www/site/content/en/documentation/runners/direct.md
@@ -57,6 +57,8 @@ Here are some resources with information about how to test
your pipelines.
## Pipeline options for the Direct Runner
+For general instructions on how to set pipeline options, see the [programming
guide](/documentation/programming-guide/#configuring-pipeline-options).
+
When executing your pipeline from the command-line, set `runner` to `direct`
or `DirectRunner`. The default values for the other pipeline options are
generally sufficient.
See the reference documentation for the
@@ -74,105 +76,41 @@ Local execution is limited by the memory available in your
local environment. It
If your pipeline uses an unbounded data source or sink, you must set the
`streaming` option to `true`.
-{:.language-py}
-### Execution Mode
+### Parallel execution
-{:.language-py}
+{{< paragraph class="language-py" >}}
Python
[FnApiRunner](https://beam.apache.org/contribute/runner-guide/#the-fn-api)
supports multi-threading and multi-processing mode.
+{{< /paragraph >}}
+
+ Setting parallelism
-{:.language-py}
-Setting parallelism
+{{< paragraph class="language-java" >}}
+The number of worker threads is defined by the `targetParallelism` pipeline
option.
+By default, `targetParallelism` is the greater of the number of available
processors and 3.
+{{< /paragraph >}}
-{:.language-py}
-Number of threads or subprocesses is defined by setting the
`direct_num_workers` option.
+{{< paragraph class="language-py" >}}
+Number of threads or subprocesses is defined by setting the
`direct_num_workers` pipeline option.
From 2.22.0, `direct_num_workers = 0` is supported. When `direct_num_workers`
is set to 0, it will set the number of threads/subprocess to the number of
cores of the machine where the pipeline is running.
+{{< /paragraph >}}
-{:.language-py}
-* There are several ways to set this option.
-```py
-python wordcount.py --input xx --output xx --direct_num_workers 2
-```
-
-{:.language-py}
-* Setting with `PipelineOptions`.
-```py
-from apache_beam.options.pipeline_options import PipelineOptions
-pipeline_options = PipelineOptions(['--direct_num_workers', '2'])
-```
-
-{:.language-py}
-* Adding to existing `PipelineOptions`.
-```py
-from apache_beam.options.pipeline_options import DirectOptions
-pipeline_options = PipelineOptions(xxx)
-pipeline_options.view_as(DirectOptions).direct_num_workers = 2
-```
-
-{:.language-py}
+{{< paragraph class="language-py" >}}
Setting running mode
+{{< /paragraph >}}
-{:.language-py}
-From 2.19, a new option was added to set running mode. We can use
`direct_running_mode` option to set the running mode.
+{{< paragraph class="language-py" >}}
+In Beam 2.19.0 and newer, you can use the `direct_running_mode` pipeline
option to set the running mode.
`direct_running_mode` can be one of [`'in_memory'`, `'multi_threading'`,
`'multi_processing'`].
+{{< /paragraph >}}
-{:.language-py}
+{{< paragraph class="language-py" >}}
in_memory: Runner and workers' communication happens in memory (not
through gRPC). This is a default mode.
+{{< /paragraph >}}
-{:.language-py}
+{{< paragraph class="language-py" >}}
multi_threading: Runner and workers communicate through gRPC and each
worker runs in a thread.
+{{< /paragraph >}}
-{:.language-py}
+{{< paragraph class="language-py" >}}
multi_processing: Runner and workers communicate through gRPC and each
worker runs in a subprocess.
-
-{:.language-py}
-Same as other options, `direct_running_mode` can be passed through CLI or set
with `PipelineOptions`.
-
-{:.language-py}
-For the ver
[beam] branch master updated (d42c67c -> 2ad2854)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d42c67c [BEAM-11580] Add disk_size_gb flag for Go SDK. (#13693) new 854ed5b [BEAM-10983] Add getting started from Spark page new c484e68 Add "From Apache Spark" entry to side menu new 17aed03 Add links to transform catalog new eaff2e9 Merge branch 'master' of github.com:apache/beam into coming-from-spark new 26a2a04 Add more links new 2f39799 Add RDD/DataFrame clarification new a751df8 Merge branch 'master' of github.com:apache/beam into coming-from-spark new 6c338ce Add link to from-spark page new a3406b3 Added extra explanations new 67af84a Merge branch 'master' of github.com:apache/beam into coming-from-spark new 9557470 Fixed typos + removed italics new 2ad2854 Merge pull request #12963 from davidcavazos/coming-from-spark The 30141 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../site/content/en/get-started/beam-overview.md | 3 + .../www/site/content/en/get-started/from-spark.md | 268 + .../partials/section-menu/en/get-started.html | 7 +- 3 files changed, 275 insertions(+), 3 deletions(-) create mode 100644 website/www/site/content/en/get-started/from-spark.md
[beam] branch master updated: Fix sdk.properties caching, remove timestamp.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c72fac9 Fix sdk.properties caching, remove timestamp.
new d3422f3 Merge pull request #13652 from runningcode/no/remove-timestamp
c72fac9 is described below
commit c72fac9673e4fb8de63c981715bc748735b90de6
Author: Nelson Osacky
AuthorDate: Mon Jan 4 21:00:48 2021 +0100
Fix sdk.properties caching, remove timestamp.
This PR does two things.
1. It removes the timestamp from the sdk.properties file.
2. This doesn't ignore the `sdk.properties` anymore for the purposes of the
runtime classpath. This means that the tests will rerun when the versions
change.
This PR assumes that [there is no need for the timestamp in the
sdk.properties](https://github.com/apache/beam/pull/13597#issuecomment-750423282).
If there is such a need, the other option to resolve this issue is to wait
until Gradle 6.8 is released which will have the ability to ignore specific
properties [such as the
timestamp](https://docs.gradle.org/6.8-rc-3/release-notes.html#more-cache-hits-for-tasks-with-runtime-classpaths)
in the sdk.properties file.
See the discussion on [the PR](https://github.com/apache/beam/pull/13597)
for more information.
---
build.gradle | 9 -
sdks/java/core/build.gradle | 1 -
.../core/src/main/resources/org/apache/beam/sdk/sdk.properties | 3 ---
3 files changed, 13 deletions(-)
diff --git a/build.gradle b/build.gradle
index 15cfc24..12f355d 100644
--- a/build.gradle
+++ b/build.gradle
@@ -381,12 +381,3 @@ if (project.hasProperty('compileAndRunTestsWithJava11')) {
}
}
}
-
-// Ignore sdk.properties so it doesn't spoil the build cache unnecessarily
(see
https://docs.gradle.org/6.6/userguide/more_about_tasks.html#sec:configure_input_normalization)
-allprojects {
- normalization {
-runtimeClasspath {
- ignore('**/sdk.properties')
-}
- }
-}
\ No newline at end of file
diff --git a/sdks/java/core/build.gradle b/sdks/java/core/build.gradle
index e8b07f6..99264d0 100644
--- a/sdks/java/core/build.gradle
+++ b/sdks/java/core/build.gradle
@@ -54,7 +54,6 @@ processResources {
'pom.sdk_version': sdk_version,
'pom.docker_image_default_repo_root': docker_image_default_repo_root,
'pom.docker_image_default_repo_prefix': docker_image_default_repo_prefix,
-'timestamp': new Date().format("-MM-dd HH:mm")
]
}
diff --git
a/sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties
b/sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties
index 3320a4c..af09037 100644
--- a/sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties
+++ b/sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties
@@ -18,6 +18,3 @@
version=@pom.version@
sdk_version=@pom.sdk_version@
-
-build.date=@timestamp@
-
[beam] branch master updated (b2ad4dd -> 60834ae)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b2ad4dd Merge pull request #13670 from [BEAM-11569] Clone ad-m/github-push-action into Beam repository to comply with Apache security requirements new 9ebdc8a replace Assert.asserThat with MacherAssert.assertThat across entire codebase new 3915c3f [BEAM-11572] Add lint rule to avoid Assert.assertThat usage new 9df5db9 Fix Assertion in Nexmark (non-test) class. new 60834ae Merge pull request #13673 from anantdamle/beam_11572_fix_assertThat The 30118 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../examples/complete/game/LeaderBoardTest.java| 2 +- .../cookbook/CombinePerKeyExamplesTest.java| 13 +-- .../beam/learning/katas/util/ContainsKvs.java | 2 +- .../core/construction/CoderTranslationTest.java| 2 +- .../runners/core/construction/CommonCoderTest.java | 2 +- .../CreatePCollectionViewTranslationTest.java | 2 +- .../DeduplicatedFlattenFactoryTest.java| 2 +- .../construction/DefaultArtifactResolverTest.java | 2 +- .../EmptyFlattenAsCreateFactoryTest.java | 2 +- .../core/construction/EnvironmentsTest.java| 2 +- .../ExecutableStageTranslationTest.java| 2 +- .../construction/ForwardingPTransformTest.java | 2 +- .../construction/GroupByKeyTranslationTest.java| 2 +- .../runners/core/construction/ModelCodersTest.java | 2 +- .../core/construction/MorePipelineTest.java| 2 +- .../construction/PCollectionTranslationTest.java | 2 +- .../core/construction/PTransformMatchersTest.java | 2 +- .../construction/PTransformReplacementsTest.java | 2 +- .../construction/PTransformTranslationTest.java| 2 +- .../core/construction/ParDoTranslationTest.java| 2 +- .../PipelineOptionsTranslationTest.java| 2 +- .../core/construction/PipelineTranslationTest.java | 2 +- .../core/construction/ReadTranslationTest.java | 2 +- .../construction/RehydratedComponentsTest.java | 2 +- .../core/construction/ReplacementOutputsTest.java | 2 +- .../construction/ReshuffleTranslationTest.java | 2 +- .../core/construction/SchemaTranslationTest.java | 2 +- .../core/construction/SdkComponentsTest.java | 2 +- .../SingleInputOutputOverrideFactoryTest.java | 2 +- .../construction/TestStreamTranslationTest.java| 2 +- .../core/construction/TransformInputsTest.java | 2 +- .../core/construction/TriggerTranslationTest.java | 2 +- .../core/construction/UnconsumedReadsTest.java | 2 +- .../core/construction/ValidateRunnerXlangTest.java | 2 +- .../WindowingStrategyTranslationTest.java | 2 +- .../construction/WriteFilesTranslationTest.java| 2 +- .../construction/graph/ExecutableStageTest.java| 2 +- .../core/construction/graph/FusedPipelineTest.java | 2 +- .../graph/GreedyPipelineFuserTest.java | 2 +- .../construction/graph/GreedyStageFuserTest.java | 2 +- .../graph/ImmutableExecutableStageTest.java| 2 +- .../core/construction/graph/NetworksTest.java | 2 +- .../construction/graph/OutputDeduplicatorTest.java | 2 +- .../construction/graph/ProtoOverridesTest.java | 2 +- .../construction/graph/QueryablePipelineTest.java | 2 +- .../runners/core/InMemoryStateInternalsTest.java | 2 +- .../runners/core/InMemoryTimerInternalsTest.java | 2 +- .../core/LateDataDroppingDoFnRunnerTest.java | 2 +- .../beam/runners/core/LateDataUtilsTest.java | 2 +- ...BoundedSplittableProcessElementInvokerTest.java | 2 +- .../beam/runners/core/ReduceFnRunnerTest.java | 2 +- .../beam/runners/core/SideInputHandlerTest.java| 2 +- .../beam/runners/core/SimpleDoFnRunnerTest.java| 2 +- .../SimplePushbackSideInputDoFnRunnerTest.java | 2 +- .../runners/core/SplittableParDoProcessFnTest.java | 2 +- .../beam/runners/core/StateNamespacesTest.java | 2 +- .../beam/runners/core/TimerInternalsTest.java | 2 +- .../beam/runners/core/WindowMatchersTest.java | 2 +- .../beam/runners/core/metrics/CounterCellTest.java | 2 +- .../beam/runners/core/metrics/DirtyStateTest.java | 2 +- .../runners/core/metrics/DistributionCellTest.java | 2 +- .../core/metrics/ExecutionStateSamplerTest.java| 2 +- .../core/metrics/ExecutionStateTrackerTest.java| 2 +- .../beam/runners/core/metrics/GaugeCellTest.java | 2 +- .../core/metrics/MetricsContainerImplTest.java | 2 +- .../core/metrics/MetricsContainerStepMapTest.java | 2 +- .../beam/runners/core/metrics/MetricsMapTest.java | 2 +- .../runners/core/metrics/MetricsPus
[beam] branch master updated (3f8a4ff -> 9b85af0)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3f8a4ff [BEAM-8829] only drop event_timestamp when it exists (#13638) new f120eb7 [BEAM-9541] Add flink_versions to gradle.properties. new 7eacb40 [BEAM-9541] All Gradle tasks use latest Flink version. new 08706df Disable Flink classloader leak check when using local execution mode. new e12a0d8 [BEAM-11570] Comment with link to context. new 9b85af0 Merge pull request #13633 from ibzib/latest-flink The 30111 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 10 ++--- examples/java/build.gradle | 2 +- examples/kotlin/build.gradle | 2 +- gradle.properties | 3 +++ runners/flink/flink_runner.gradle | 19 +++- .../runners/flink/FlinkExecutionEnvironments.java | 12 +++ sdks/go/test/build.gradle | 8 +++ sdks/java/maven-archetypes/examples/build.gradle | 2 +- sdks/java/testing/jpms-tests/build.gradle | 2 +- sdks/python/test-suites/portable/common.gradle | 25 +++--- 10 files changed, 51 insertions(+), 34 deletions(-)
[beam] branch master updated: Oxford comma
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d84699e Oxford comma new ba305fa Merge pull request #13631 from elharo/patch-3 d84699e is described below commit d84699ecee3dfa2303e60ba2dd178441431ab833 Author: Elliotte Rusty Harold AuthorDate: Tue Dec 29 07:59:21 2020 -0500 Oxford comma --- website/www/site/content/en/get-started/beam-overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/www/site/content/en/get-started/beam-overview.md b/website/www/site/content/en/get-started/beam-overview.md index baf8610..5599fb6 100644 --- a/website/www/site/content/en/get-started/beam-overview.md +++ b/website/www/site/content/en/get-started/beam-overview.md @@ -58,7 +58,7 @@ Get started using Beam for your data processing tasks. 1. [Try Apache Beam](/get-started/try-apache-beam) in an online interactive environment. -1. Follow the Quickstart for the [Java SDK](/get-started/quickstart-java), the [Python SDK](/get-started/quickstart-py) or the [Go SDK](/get-started/quickstart-go). +1. Follow the Quickstart for the [Java SDK](/get-started/quickstart-java), the [Python SDK](/get-started/quickstart-py), or the [Go SDK](/get-started/quickstart-go). 1. See the [WordCount Examples Walkthrough](/get-started/wordcount-example) for examples that introduce various features of the SDKs.
[beam] branch master updated: [BEAM-10925] Change UserFunctionDefinitions into an AutoValue and Builder class.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8b050a3 [BEAM-10925] Change UserFunctionDefinitions into an AutoValue
and Builder class.
new 356e2ae Merge pull request #13625 from ibzib/ufd-builder
8b050a3 is described below
commit 8b050a33cfd49b7cc33a5b2c507cbc403d12fd1b
Author: Kyle Weaver
AuthorDate: Mon Dec 28 11:29:53 2020 -0800
[BEAM-10925] Change UserFunctionDefinitions into an AutoValue and Builder
class.
We are planning on adding several new types of UDF, and adding them all in
the constructor will get verbose.
---
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 5 +++-
.../sql/zetasql/translation/ConversionContext.java | 2 +-
.../zetasql/translation/ExpressionConverter.java | 4 +++-
.../translation/UserFunctionDefinitions.java | 28 +++---
4 files changed, 27 insertions(+), 12 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
index 6259d12..ea9e332 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
@@ -131,7 +131,10 @@ class ZetaSQLPlannerImpl {
}
UserFunctionDefinitions userFunctionDefinitions =
-new UserFunctionDefinitions(udfBuilder.build(), udtvfBuilder.build());
+UserFunctionDefinitions.newBuilder()
+.setSqlScalarFunctions(udfBuilder.build())
+.setSqlTableValuedFunctions(udtvfBuilder.build())
+.build();
ExpressionConverter expressionConverter =
new ExpressionConverter(cluster, params, userFunctionDefinitions);
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
index 9523b76..3f87c54 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
@@ -79,7 +79,7 @@ public class ConversionContext {
}
Map, ResolvedNode> getUserDefinedTableValuedFunctions() {
-return
getExpressionConverter().userFunctionDefinitions.sqlTableValuedFunctions;
+return
getExpressionConverter().userFunctionDefinitions.sqlTableValuedFunctions();
}
Map getFunctionArgumentRefMapping() {
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
index a4d0f03..c732c43 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
@@ -667,7 +667,9 @@ public class ExpressionConverter {
}
} else if (USER_DEFINED_FUNCTIONS.equals(funGroup)) {
ResolvedCreateFunctionStmt createFunctionStmt =
-
userFunctionDefinitions.sqlScalarFunctions.get(functionCall.getFunction().getNamePath());
+ userFunctionDefinitions
+ .sqlScalarFunctions()
+ .get(functionCall.getFunction().getNamePath());
ResolvedExpr functionExpression =
createFunctionStmt.getFunctionExpression();
ImmutableMap.Builder innerFunctionArguments =
ImmutableMap.builder();
for (int i = 0; i < functionCall.getArgumentList().size(); i++) {
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
index a4544e7..792c31c 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
@@ -17,27 +17,37 @@
*/
package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+import com.google.auto.value.AutoValue;
import com.google.zetasql.resolvedast.Resolv
[beam] branch master updated (6171a36 -> c2e07fb)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6171a36 [BEAM-11456] Fix flaky table provider tests (Kafka and Pubsub) (#13542) add c2e07fb [BEAM-10925] Convert ScalarFn to Method. (#13593) No new revisions were added by this update. Summary of changes: sdks/java/extensions/sql/build.gradle | 1 + .../sdk/extensions/sql/impl/ScalarFnReflector.java | 73 +++ .../extensions/sql/impl/ScalarFnReflectorTest.java | 141 + 3 files changed, 215 insertions(+) create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFnReflector.java create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFnReflectorTest.java
[beam] branch master updated: [BEAM-9637] Add all runners to Python --runner help text.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c679de6 [BEAM-9637] Add all runners to Python --runner help text.
new 0350f2e Merge pull request #13584 from ibzib/BEAM-9637
c679de6 is described below
commit c679de69d26de7472ab50fd4704cde24281205d7
Author: Kyle Weaver
AuthorDate: Fri Dec 18 12:41:23 2020 -0800
[BEAM-9637] Add all runners to Python --runner help text.
---
.../python/apache_beam/options/pipeline_options.py | 19 -
sdks/python/apache_beam/runners/runner.py | 24 +++---
2 files changed, 25 insertions(+), 18 deletions(-)
diff --git a/sdks/python/apache_beam/options/pipeline_options.py
b/sdks/python/apache_beam/options/pipeline_options.py
index c169586..0effcf6 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -414,13 +414,30 @@ class StandardOptions(PipelineOptions):
DEFAULT_RUNNER = 'DirectRunner'
+ ALL_KNOWN_RUNNERS = (
+ 'apache_beam.runners.dataflow.dataflow_runner.DataflowRunner',
+ 'apache_beam.runners.direct.direct_runner.BundleBasedDirectRunner',
+ 'apache_beam.runners.direct.direct_runner.DirectRunner',
+ 'apache_beam.runners.direct.direct_runner.SwitchingDirectRunner',
+ 'apache_beam.runners.interactive.interactive_runner.InteractiveRunner',
+ 'apache_beam.runners.portability.flink_runner.FlinkRunner',
+ 'apache_beam.runners.portability.portable_runner.PortableRunner',
+ 'apache_beam.runners.portability.spark_runner.SparkRunner',
+ 'apache_beam.runners.test.TestDirectRunner',
+ 'apache_beam.runners.test.TestDataflowRunner',
+ )
+
+ KNOWN_RUNNER_NAMES = [path.split('.')[-1] for path in ALL_KNOWN_RUNNERS]
+
@classmethod
def _add_argparse_args(cls, parser):
parser.add_argument(
'--runner',
help=(
'Pipeline runner used to execute the workflow. Valid values are '
-'DirectRunner, DataflowRunner.'))
+'one of %s, or the fully qualified name of a PipelineRunner '
+'subclass. If unspecified, defaults to %s.' %
+(', '.join(cls.KNOWN_RUNNER_NAMES), cls.DEFAULT_RUNNER)))
# Whether to enable streaming mode.
parser.add_argument(
'--streaming',
diff --git a/sdks/python/apache_beam/runners/runner.py
b/sdks/python/apache_beam/runners/runner.py
index ba80b27..02ed845 100644
--- a/sdks/python/apache_beam/runners/runner.py
+++ b/sdks/python/apache_beam/runners/runner.py
@@ -31,6 +31,8 @@ from builtins import object
from typing import TYPE_CHECKING
from typing import Optional
+from apache_beam.options.pipeline_options import StandardOptions
+
if TYPE_CHECKING:
from apache_beam import pvalue
from apache_beam import PTransform
@@ -41,22 +43,10 @@ if TYPE_CHECKING:
__all__ = ['PipelineRunner', 'PipelineState', 'PipelineResult']
-_ALL_KNOWN_RUNNERS = (
-'apache_beam.runners.dataflow.dataflow_runner.DataflowRunner',
-'apache_beam.runners.direct.direct_runner.BundleBasedDirectRunner',
-'apache_beam.runners.direct.direct_runner.DirectRunner',
-'apache_beam.runners.direct.direct_runner.SwitchingDirectRunner',
-'apache_beam.runners.interactive.interactive_runner.InteractiveRunner',
-'apache_beam.runners.portability.flink_runner.FlinkRunner',
-'apache_beam.runners.portability.portable_runner.PortableRunner',
-'apache_beam.runners.portability.spark_runner.SparkRunner',
-'apache_beam.runners.test.TestDirectRunner',
-'apache_beam.runners.test.TestDataflowRunner',
-)
-
-_KNOWN_RUNNER_NAMES = [path.split('.')[-1] for path in _ALL_KNOWN_RUNNERS]
-
-_RUNNER_MAP = {path.split('.')[-1].lower(): path for path in
_ALL_KNOWN_RUNNERS}
+_RUNNER_MAP = {
+path.split('.')[-1].lower(): path
+for path in StandardOptions.ALL_KNOWN_RUNNERS
+}
# Allow this alias, but don't make public.
_RUNNER_MAP['pythonrpcdirectrunner'] = (
@@ -110,7 +100,7 @@ def create_runner(runner_name):
raise ValueError(
'Unexpected pipeline runner: %s. Valid values are %s '
'or the fully qualified name of a PipelineRunner subclass.' %
-(runner_name, ', '.join(_KNOWN_RUNNER_NAMES)))
+(runner_name, ', '.join(StandardOptions.KNOWN_RUNNER_NAMES)))
class PipelineRunner(object):
[beam] branch master updated: [BEAM-10925] Add UdfProvider interface.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b1fcfe9 [BEAM-10925] Add UdfProvider interface.
new e4ee5f5 Merge pull request #13587 from ibzib/udf-provider
b1fcfe9 is described below
commit b1fcfe9f69ac59880586a909544c7619ffcb9e86
Author: Kyle Weaver
AuthorDate: Mon Dec 21 11:27:29 2020 -0800
[BEAM-10925] Add UdfProvider interface.
---
.../beam/sdk/extensions/sql/udf/UdfProvider.java | 37 ++
1 file changed, 37 insertions(+)
diff --git
a/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/UdfProvider.java
b/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/UdfProvider.java
new file mode 100644
index 000..fe8b5b5
--- /dev/null
+++
b/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/UdfProvider.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.udf;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Provider for user-defined functions written in Java. Implementations should
be annotated with
+ * {@link com.google.auto.service.AutoService}.
+ */
+public interface UdfProvider {
+ /** Maps function names to scalar function implementations. */
+ default Map userDefinedScalarFunctions() {
+return Collections.emptyMap();
+ }
+
+ /** Maps function names to aggregate function implementations. */
+ default Map> userDefinedAggregateFunctions() {
+return Collections.emptyMap();
+ }
+}
[beam] branch master updated (bfdb3e5 -> 164a7b6)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bfdb3e5 Merge pull request #13472: [BEAM-11391] Improve Nexmark Kafka support to track add 164a7b6 [BEAM-9426] Add JVM properties to JavaJarJobServer. (#13581) No new revisions were added by this update. Summary of changes: .../python/apache_beam/options/pipeline_options.py | 10 +++ .../apache_beam/runners/portability/job_server.py | 10 +-- .../runners/portability/job_server_test.py | 84 ++ 3 files changed, 100 insertions(+), 4 deletions(-) create mode 100644 sdks/python/apache_beam/runners/portability/job_server_test.py
[beam] branch master updated (199fea8 -> 1466db9)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 199fea8 Merge pull request #13579 from ibzib/accumulators
new c4b1359 [BEAM-10925] Create ZetaSQL-specific subclass of
ScalarFunctionImpl that knows which function group it belongs to.
new 542e8ce Make createUd(a)fOperator methods package-private.
new 1466db9 Merge pull request #13309 from ibzib/fngroup
The 30030 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../extensions/sql/impl/ScalarFunctionImpl.java| 47 +++--
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java| 6 ++
.../zetasql/translation/ExpressionConverter.java | 7 +-
.../sql/zetasql/translation/SqlOperators.java | 76 +++---
.../translation/ZetaSqlScalarFunctionImpl.java}| 56 +---
5 files changed, 100 insertions(+), 92 deletions(-)
copy
sdks/java/extensions/sql/{src/main/java/org/apache/beam/sdk/extensions/sql/impl/UdfImpl.java
=>
zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java}
(57%)
[beam] branch master updated: [BEAM-10925] SQL AggregateFn: Separate mutable accumulator from immutables.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 4b1c4aa [BEAM-10925] SQL AggregateFn: Separate mutable accumulator from immutables. new 199fea8 Merge pull request #13579 from ibzib/accumulators 4b1c4aa is described below commit 4b1c4aadb16db36d5caf6a60186b981a6fffd503 Author: Kyle Weaver AuthorDate: Thu Dec 17 14:49:26 2020 -0800 [BEAM-10925] SQL AggregateFn: Separate mutable accumulator from immutables. --- .../org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java b/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java index 28ebf3b..1c1cece 100644 --- a/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java +++ b/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java @@ -54,17 +54,18 @@ public interface AggregateFn< * Returns an accumulator representing the accumulation of all the input values accumulated in the * merging accumulators. * - * @param accumulators only the first accumulator may be modified and returned for efficiency; the - * other accumulators should not be mutated, because they may be shared with other code and - * mutating them could lead to incorrect results or data corruption. + * @param mutableAccumulator This accumulator may be modified and returned for efficiency. + * @param immutableAccumulators These other accumulators should not be mutated, because they may + * be shared with other code and mutating them could lead to incorrect results or data + * corruption. */ - AccumT mergeAccumulators(Iterable accumulators); + AccumT mergeAccumulators(AccumT mutableAccumulator, Iterable immutableAccumulators); /** * Returns the output value that is the result of combining all the input values represented by * the given accumulator. * - * @param accumulator can be modified for efficiency + * @param mutableAccumulator can be modified for efficiency */ - OutputT extractOutput(AccumT accumulator); + OutputT extractOutput(AccumT mutableAccumulator); }
[beam] branch master updated: [BEAM-10925] Add SQL AggregateFn interface.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3de4f88 [BEAM-10925] Add SQL AggregateFn interface.
new a7457c0 Merge pull request #13306 from ibzib/aggregate-udf
3de4f88 is described below
commit 3de4f88aa8839af36e355060c8de62f9a6caf0e7
Author: Kyle Weaver
AuthorDate: Wed Nov 11 13:05:20 2020 -0800
[BEAM-10925] Add SQL AggregateFn interface.
---
.../beam/sdk/extensions/sql/udf/AggregateFn.java | 70 ++
1 file changed, 70 insertions(+)
diff --git
a/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java
b/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java
new file mode 100644
index 000..28ebf3b
--- /dev/null
+++
b/sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.udf;
+
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * An aggregate function that can be executed as part of a SQL query.
+ *
+ * AggregateFn contains a subset of the functionality of {@code
+ * org.apache.beam.sdk.transforms.Combine.CombineFn}.
+ *
+ * AggregateFn is experimental. Compatibility is not
guaranteed across Beam
+ * versions.
+ *
+ * @param type of input values
+ * @param type of mutable accumulator values
+ * @param type of output values
+ */
+public interface AggregateFn<
+InputT extends @Nullable Object,
+AccumT extends @Nullable Object,
+OutputT extends @Nullable Object> {
+
+ /**
+ * Returns a new, mutable accumulator value, representing the accumulation
of zero input values.
+ */
+ AccumT createAccumulator();
+
+ /**
+ * Adds the given input value to the given accumulator, returning the new
accumulator value.
+ *
+ * @param mutableAccumulator may be modified and returned for efficiency
+ * @param input should not be mutated
+ */
+ AccumT addInput(AccumT mutableAccumulator, InputT input);
+
+ /**
+ * Returns an accumulator representing the accumulation of all the input
values accumulated in the
+ * merging accumulators.
+ *
+ * @param accumulators only the first accumulator may be modified and
returned for efficiency; the
+ * other accumulators should not be mutated, because they may be shared
with other code and
+ * mutating them could lead to incorrect results or data corruption.
+ */
+ AccumT mergeAccumulators(Iterable accumulators);
+
+ /**
+ * Returns the output value that is the result of combining all the input
values represented by
+ * the given accumulator.
+ *
+ * @param accumulator can be modified for efficiency
+ */
+ OutputT extractOutput(AccumT accumulator);
+}
[beam] branch master updated (1ccdee8 -> 8de3288)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1ccdee8 [BEAM-11432] put latest tag when publishing SDK head
containers to gcr (#13517)
add 8de3288 [BEAM-10925] Add interface for SQL Java scalar UDF. (#13305)
No new revisions were added by this update.
Summary of changes:
.../sql/{perf-tests => udf}/build.gradle | 16 -
.../beam/sdk/extensions/sql/udf/ScalarFn.java} | 42 --
.../beam/sdk/extensions/sql/udf}/package-info.java | 9 ++---
settings.gradle| 1 +
4 files changed, 37 insertions(+), 31 deletions(-)
copy sdks/java/extensions/sql/{perf-tests => udf}/build.gradle (73%)
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/options/Validation.java
=>
extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/ScalarFn.java}
(52%)
copy {runners/direct-java/src/main/java/org/apache/beam/runners/direct =>
sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf}/package-info.java
(69%)
[beam] branch master updated (5f50f17 -> adf5515)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5f50f17 Merge pull request #13459: [BEAM-11384] Change StreamingDataflowWorker from encoding key add adf5515 Better logging for Python Docker environment (#13508) No new revisions were added by this update. Summary of changes: .../runners/portability/fn_api_runner/worker_handlers.py | 7 +-- sdks/python/apache_beam/transforms/environments.py| 11 --- 2 files changed, 13 insertions(+), 5 deletions(-)
[beam] branch master updated: [BEAM-11413] Fix default job server for Spark tests.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 0498a17 [BEAM-11413] Fix default job server for Spark tests. new 7b39702 Merge pull request #13498 from ibzib/BEAM-11413 0498a17 is described below commit 0498a17f723efda0f4e6884fbbf471b64fabd8b5 Author: Kyle Weaver AuthorDate: Mon Dec 7 13:52:19 2020 -0800 [BEAM-11413] Fix default job server for Spark tests. --- sdks/python/apache_beam/runners/portability/spark_runner_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/runners/portability/spark_runner_test.py b/sdks/python/apache_beam/runners/portability/spark_runner_test.py index 7898618..1b873cf 100644 --- a/sdks/python/apache_beam/runners/portability/spark_runner_test.py +++ b/sdks/python/apache_beam/runners/portability/spark_runner_test.py @@ -87,7 +87,7 @@ class SparkRunnerTest(portable_runner_test.PortableRunnerTest): self.set_spark_job_server_jar( known_args.spark_job_server_jar or job_server.JavaJarJobServer.path_to_beam_jar( -':runners:job-server:shadowJar')) +':runners:spark:job-server:shadowJar')) self.environment_type = known_args.environment_type self.environment_options = known_args.environment_options
[beam] branch master updated (f628ced -> 6a91643)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f628ced Merge pull request #13487 from ibzib/BEAM-11394 new ff1b192 [BEAM-11412] Skip SparkRunnerTest::test_sdf_with_dofn_as_watermark_estimator. new 305dcfd Fix requirements. new 6a91643 Merge pull request #13499 from ibzib/BEAM-11412 The 29873 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/portability/spark_runner_test.py | 4 1 file changed, 4 insertions(+)
[beam] branch master updated (553553d -> f628ced)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 553553d Avoiding queries that check whether a BQ table is empty for streaming inserts (#13490) new dd6cbc8 [BEAM-11394] Make job server Java launcher configurable via pipeline options. new 681ea6f Rename option to disambiguate job servers and expansion servers. new f628ced Merge pull request #13487 from ibzib/BEAM-11394 The 29870 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/options/pipeline_options.py | 6 ++ sdks/python/apache_beam/runners/portability/job_server.py | 3 ++- 2 files changed, 8 insertions(+), 1 deletion(-)
[beam] branch master updated (0a4bdb1 -> 5e132c7)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0a4bdb1 Merge pull request #13444: [BEAM-11363] Create `:sdks:java:extensions:ml:integrationTest` target, run in Java PostCommit new f551c81 Fix popen() typo new 38982ae Add missing equal signs to example in comment new 5e132c7 Merge pull request #13465 from hacosta/nits The 29841 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/portability/flink_runner_test.py| 2 +- .../apache_beam/runners/portability/fn_api_runner/worker_handlers.py| 2 +- sdks/python/apache_beam/runners/portability/spark_runner_test.py| 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (0d0d2d5 -> 1bdc1cd)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0d0d2d5 Merge pull request #13411: [BEAM-11331] Change region flag to awsRegion in io.aws2 add 1bdc1cd [BEAM-11376] Ignoring DicomIOReadIT (#13450) No new revisions were added by this update. Summary of changes: .../test/java/org/apache/beam/sdk/io/gcp/healthcare/DicomIOReadIT.java | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated: [BEAM-10278] Move pipeline jar tests from job server projects to python/test-suites/portable.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f8e8daf [BEAM-10278] Move pipeline jar tests from job server projects
to python/test-suites/portable.
new b002530 Merge pull request #13417 from ibzib/BEAM-10278
f8e8daf is described below
commit f8e8dafdf78f29e551baaf64dc612689a31610d5
Author: Kyle Weaver
AuthorDate: Tue Nov 24 11:45:02 2020 -0800
[BEAM-10278] Move pipeline jar tests from job server projects to
python/test-suites/portable.
---
.../job_PostCommit_PortableJar_Flink.groovy| 3 +-
.../job_PostCommit_PortableJar_Spark.groovy| 2 +-
runners/flink/job-server/flink_job_server.gradle | 66 -
.../test_flink_uber_jar.sh | 0
runners/portability/test_pipeline_jar.sh | 3 +-
runners/spark/job-server/build.gradle | 32 --
sdks/python/test-suites/portable/common.gradle | 69 +-
7 files changed, 73 insertions(+), 102 deletions(-)
diff --git a/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
b/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
index 7572669..856b7bf 100644
--- a/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
+++ b/.test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
@@ -31,7 +31,8 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_PortableJar_Flink',
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runners:flink:1.10:job-server:testPipelineJar')
+
tasks(':sdks:python:test-suites:portable:py36:testPipelineJarFlinkRunner')
+
tasks(':sdks:python:test-suites:portable:py37:testPipelineJarFlinkRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/.test-infra/jenkins/job_PostCommit_PortableJar_Spark.groovy
b/.test-infra/jenkins/job_PostCommit_PortableJar_Spark.groovy
index 8e4f1c5..93e58af 100644
--- a/.test-infra/jenkins/job_PostCommit_PortableJar_Spark.groovy
+++ b/.test-infra/jenkins/job_PostCommit_PortableJar_Spark.groovy
@@ -31,7 +31,7 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_PortableJar_Spark',
steps {
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
- tasks(':runners:spark:job-server:testPipelineJar')
+
tasks(':sdks:python:test-suites:portable:py37:testPipelineJarSparkRunner')
commonJobProperties.setGradleSwitches(delegate)
}
}
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
index 99cd12d..130dd97 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -239,30 +239,6 @@ createCrossLanguageValidatesRunnerTask(
]
)
-def addTestJavaJarCreator(String pyVersion) {
- def pyBuildPath = pyVersion.startsWith("2") ? "2" :
pyVersion.replaceAll("\\.", "")
- project.tasks.create(name: "testJavaJarCreatorPy${pyBuildPath}") {
-dependsOn shadowJar
-dependsOn ":sdks:python:container:py${pyBuildPath}:docker"
-doLast{
- exec {
-executable "sh"
-def options = [
-"--runner FlinkRunner",
-"--job_server_jar ${shadowJar.archivePath}",
-"--env_dir
${project.rootProject.buildDir}/gradleenv/${project.path.hashCode()}",
-"--python_root_dir ${project.rootDir}/sdks/python",
-"--python_version ${pyVersion}",
-"--python_container_image
${project.docker_image_default_repo_root}/"
-+ "${project.docker_image_default_repo_prefix}"
-+ "python${pyVersion}_sdk:${project.sdk_version}",
-]
-args "-c", "../../../portability/test_pipeline_jar.sh ${options.join('
')}"
- }
-}
- }
-}
-
// miniCluster jar starts an embedded Flink cluster intended for use in
testing.
task miniCluster(type: Jar, dependsOn: shadowJar) {
archiveBaseName = "${project.archivesBaseName}-mini-cluster"
@@ -281,45 +257,3 @@ task miniCluster(type: Jar, dependsOn: shadowJar) {
}
zip64 true // jar needs to contain more than 65535 files
}
-
-def addTestFlinkUberJarPy(String pyVersion, boolean saveMainSession) {
- def pyBuildPath = pyVersion.startsWith("2") ? "2" :
pyVersion.replaceAll("\\.", "")
- project.tasks.create(name:
"testFlinkUberJarPy${pyBuildPath}${saveMainSession ? 'saveMainSession' : ''}") {
-dependsOn miniCluster
-dependsOn shadowJar
-dependsOn "
[beam] branch master updated: [BEAM-10974] Skip GroupByKeyTest.testLargeKeys10MB.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a0f1a54 [BEAM-10974] Skip GroupByKeyTest.testLargeKeys10MB.
new 1a34854 Merge pull request #13218 from ibzib/BEAM-10974
a0f1a54 is described below
commit a0f1a5483e37f70dd0ce73520e8ef0b95c63af55
Author: Kyle Weaver
AuthorDate: Wed Oct 28 20:51:29 2020 -0700
[BEAM-10974] Skip GroupByKeyTest.testLargeKeys10MB.
---
runners/flink/job-server/flink_job_server.gradle | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
index c43865d..87e6e1e 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -150,7 +150,8 @@ def portableValidatesRunnerTask(String name, Boolean
streaming, Boolean checkpoi
} else {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
excludeCategories
'org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders'
-excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above100MB'
+// Larger keys are possible, but they require more memory.
+excludeCategories 'org.apache.beam.sdk.testing.LargeKeys$Above10MB'
excludeCategories 'org.apache.beam.sdk.testing.UsesCommittedMetrics'
excludeCategories
'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
excludeCategories 'org.apache.beam.sdk.testing.UsesCustomWindowMerging'
[beam] branch master updated (019ebb5 -> 909a519)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 019ebb5 Merge pull request #13304 from ibzib/reflect-helpers add 909a519 [BEAM-10379] Add bit_and back into Calcite SQL. (#13062) No new revisions were added by this update. Summary of changes: .../impl/transform/BeamBuiltinAggregations.java| 143 --- .../extensions/sql/BeamSqlDslAggregationTest.java | 263 + 2 files changed, 158 insertions(+), 248 deletions(-)
[beam] branch master updated: [BEAM-10925] Move general-usage helper methods from DoFnSignatures to ReflectHelpers.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b9f0538 [BEAM-10925] Move general-usage helper methods from
DoFnSignatures to ReflectHelpers.
new 019ebb5 Merge pull request #13304 from ibzib/reflect-helpers
b9f0538 is described below
commit b9f0538290375ab17ec3814e327c7c2a9dcc9a50
Author: Kyle Weaver
AuthorDate: Fri Oct 30 18:25:33 2020 -0700
[BEAM-10925] Move general-usage helper methods from DoFnSignatures to
ReflectHelpers.
---
.../sdk/transforms/reflect/DoFnSignatures.java | 77 --
.../beam/sdk/util/common/ReflectHelpers.java | 60 +
2 files changed, 72 insertions(+), 65 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
index 550b547..e730e8a 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
@@ -21,7 +21,6 @@ import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Prec
import com.google.auto.value.AutoValue;
import java.lang.annotation.Annotation;
-import java.lang.reflect.AnnotatedElement;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
@@ -33,7 +32,6 @@ import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashMap;
-import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -542,7 +540,7 @@ public class DoFnSignatures {
findAnnotatedMethod(errors, DoFn.NewWatermarkEstimator.class, fnClass,
false);
Collection onTimerMethods =
-declaredMethodsWithAnnotation(DoFn.OnTimer.class, fnClass, DoFn.class);
+ReflectHelpers.declaredMethodsWithAnnotation(DoFn.OnTimer.class,
fnClass, DoFn.class);
HashMap onTimerMethodMap =
Maps.newHashMapWithExpectedSize(onTimerMethods.size());
for (Method onTimerMethod : onTimerMethods) {
@@ -569,7 +567,7 @@ public class DoFnSignatures {
// Check for TimerFamily
Collection onTimerFamilyMethods =
-declaredMethodsWithAnnotation(DoFn.OnTimerFamily.class, fnClass,
DoFn.class);
+ReflectHelpers.declaredMethodsWithAnnotation(DoFn.OnTimerFamily.class,
fnClass, DoFn.class);
HashMap onTimerFamilyMethodMap =
Maps.newHashMapWithExpectedSize(onTimerFamilyMethods.size());
@@ -1867,7 +1865,8 @@ public class DoFnSignatures {
private static ImmutableMap
analyzeTimerFamilyDeclarations(
ErrorReporter errors, Class fnClazz) {
Map declarations = new HashMap<>();
-for (Field field : declaredFieldsWithAnnotation(DoFn.TimerFamily.class,
fnClazz, DoFn.class)) {
+for (Field field :
+ReflectHelpers.declaredFieldsWithAnnotation(DoFn.TimerFamily.class,
fnClazz, DoFn.class)) {
// TimerSpec fields may generally be private, but will be accessed via
the signature
field.setAccessible(true);
String id =
@@ -1882,7 +1881,8 @@ public class DoFnSignatures {
private static ImmutableMap
analyzeTimerDeclarations(
ErrorReporter errors, Class fnClazz) {
Map declarations = new HashMap<>();
-for (Field field : declaredFieldsWithAnnotation(DoFn.TimerId.class,
fnClazz, DoFn.class)) {
+for (Field field :
+ReflectHelpers.declaredFieldsWithAnnotation(DoFn.TimerId.class,
fnClazz, DoFn.class)) {
// TimerSpec fields may generally be private, but will be accessed via
the signature
field.setAccessible(true);
// Add fixed prefix to avoid key collision with TimerFamily.
@@ -2200,66 +2200,11 @@ public class DoFnSignatures {
return DoFnSignature.GetSizeMethod.create(m, windowT,
methodContext.getExtraParameters());
}
- private static Collection declaredMethodsWithAnnotation(
- Class anno, Class startClass, Class
stopClass) {
-return declaredMembersWithAnnotation(anno, startClass, stopClass,
GET_METHODS);
- }
-
- private static Collection declaredFieldsWithAnnotation(
- Class anno, Class startClass, Class
stopClass) {
-return declaredMembersWithAnnotation(anno, startClass, stopClass,
GET_FIELDS);
- }
-
- private interface MemberGetter {
-MemberT[] getMembers(Class clazz);
- }
-
- private static final MemberGetter GET_METHODS =
Class::getDeclaredMethods;
-
- private static final MemberGetter GET_FIELDS =
Class::getDeclaredFields;
-
- private static
- Collection declaredMembersWithAnnotation(
- Class anno,
- Class startClass,
- Class stopClass,
- MemberGetter getter) {
-Collection m
[beam] branch master updated: [BEAM-10925] Create wrapper for user function definitions.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 314efa7 [BEAM-10925] Create wrapper for user function definitions.
new ca9e4d2 Merge pull request #13307 from ibzib/udf-wrapper
314efa7 is described below
commit 314efa755e0baf4162de406dc96d2c9c307457bd
Author: Kyle Weaver
AuthorDate: Wed Nov 11 13:30:51 2020 -0800
[BEAM-10925] Create wrapper for user function definitions.
Also, resolve native SQL scalar functions by path (list) instead of full
name (string) to match UDTVF.
---
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 22 +--
.../sql/zetasql/translation/ConversionContext.java | 24 ++--
.../zetasql/translation/ExpressionConverter.java | 10 ++---
.../translation/UserFunctionDefinitions.java | 43 ++
4 files changed, 61 insertions(+), 38 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
index 8431bf0..6259d12 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
@@ -35,8 +35,8 @@ import
org.apache.beam.sdk.extensions.sql.impl.QueryPlanner.QueryParameters;
import
org.apache.beam.sdk.extensions.sql.zetasql.translation.ConversionContext;
import
org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter;
import
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter;
+import
org.apache.beam.sdk.extensions.sql.zetasql.translation.UserFunctionDefinitions;
import
org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
-import
org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
@@ -51,6 +51,7 @@ import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Framework
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Program;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** ZetaSQLPlannerImpl. */
@SuppressWarnings({
@@ -100,7 +101,8 @@ class ZetaSQLPlannerImpl {
SimpleCatalog catalog =
analyzer.createPopulatedCatalog(defaultSchemaPlus.getName(), options,
tables);
-ImmutableMap.Builder udfBuilder =
ImmutableMap.builder();
+ImmutableMap.Builder, ResolvedCreateFunctionStmt> udfBuilder =
+ImmutableMap.builder();
ImmutableMap.Builder, ResolvedNode> udtvfBuilder =
ImmutableMap.builder();
ResolvedStatement statement;
@@ -109,13 +111,7 @@ class ZetaSQLPlannerImpl {
statement = analyzer.analyzeNextStatement(parseResumeLocation, options,
catalog);
if (statement.nodeKind() == RESOLVED_CREATE_FUNCTION_STMT) {
ResolvedCreateFunctionStmt createFunctionStmt =
(ResolvedCreateFunctionStmt) statement;
-// ResolvedCreateFunctionStmt does not include the full function name,
so build it here.
-String functionFullName =
-String.format(
-"%s:%s",
-SqlAnalyzer.USER_DEFINED_FUNCTIONS,
-String.join(".", createFunctionStmt.getNamePath()));
-udfBuilder.put(functionFullName, createFunctionStmt);
+udfBuilder.put(createFunctionStmt.getNamePath(), createFunctionStmt);
} else if (statement.nodeKind() == RESOLVED_CREATE_TABLE_FUNCTION_STMT) {
ResolvedCreateTableFunctionStmt createTableFunctionStmt =
(ResolvedCreateTableFunctionStmt) statement;
@@ -134,10 +130,12 @@ class ZetaSQLPlannerImpl {
"Statement list must end in a SELECT statement, not " +
statement.nodeKindString());
}
+UserFunctionDefinitions userFunctionDefinitions =
+new UserFunctionDefinitions(udfBuilder.build(), udtvfBuilder.build());
+
ExpressionConverter expressionConverter =
-new ExpressionConverter(cluster, params, udfBuilder.build());
-ConversionContext context =
-ConversionContext.of(config, expressionConverter, cluster, trait,
udtvfBuilder.build());
+new ExpressionConverter(cluster, params, userFunctionDefiniti
[beam] branch master updated: [BEAM-10925] Create wrapper for user function definitions.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 314efa7 [BEAM-10925] Create wrapper for user function definitions.
new ca9e4d2 Merge pull request #13307 from ibzib/udf-wrapper
314efa7 is described below
commit 314efa755e0baf4162de406dc96d2c9c307457bd
Author: Kyle Weaver
AuthorDate: Wed Nov 11 13:30:51 2020 -0800
[BEAM-10925] Create wrapper for user function definitions.
Also, resolve native SQL scalar functions by path (list) instead of full
name (string) to match UDTVF.
---
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 22 +--
.../sql/zetasql/translation/ConversionContext.java | 24 ++--
.../zetasql/translation/ExpressionConverter.java | 10 ++---
.../translation/UserFunctionDefinitions.java | 43 ++
4 files changed, 61 insertions(+), 38 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
index 8431bf0..6259d12 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
@@ -35,8 +35,8 @@ import
org.apache.beam.sdk.extensions.sql.impl.QueryPlanner.QueryParameters;
import
org.apache.beam.sdk.extensions.sql.zetasql.translation.ConversionContext;
import
org.apache.beam.sdk.extensions.sql.zetasql.translation.ExpressionConverter;
import
org.apache.beam.sdk.extensions.sql.zetasql.translation.QueryStatementConverter;
+import
org.apache.beam.sdk.extensions.sql.zetasql.translation.UserFunctionDefinitions;
import
org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableList;
-import
org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.adapter.java.JavaTypeFactory;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptCluster;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RelOptPlanner;
@@ -51,6 +51,7 @@ import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Framework
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Program;
import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Util;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
/** ZetaSQLPlannerImpl. */
@SuppressWarnings({
@@ -100,7 +101,8 @@ class ZetaSQLPlannerImpl {
SimpleCatalog catalog =
analyzer.createPopulatedCatalog(defaultSchemaPlus.getName(), options,
tables);
-ImmutableMap.Builder udfBuilder =
ImmutableMap.builder();
+ImmutableMap.Builder, ResolvedCreateFunctionStmt> udfBuilder =
+ImmutableMap.builder();
ImmutableMap.Builder, ResolvedNode> udtvfBuilder =
ImmutableMap.builder();
ResolvedStatement statement;
@@ -109,13 +111,7 @@ class ZetaSQLPlannerImpl {
statement = analyzer.analyzeNextStatement(parseResumeLocation, options,
catalog);
if (statement.nodeKind() == RESOLVED_CREATE_FUNCTION_STMT) {
ResolvedCreateFunctionStmt createFunctionStmt =
(ResolvedCreateFunctionStmt) statement;
-// ResolvedCreateFunctionStmt does not include the full function name,
so build it here.
-String functionFullName =
-String.format(
-"%s:%s",
-SqlAnalyzer.USER_DEFINED_FUNCTIONS,
-String.join(".", createFunctionStmt.getNamePath()));
-udfBuilder.put(functionFullName, createFunctionStmt);
+udfBuilder.put(createFunctionStmt.getNamePath(), createFunctionStmt);
} else if (statement.nodeKind() == RESOLVED_CREATE_TABLE_FUNCTION_STMT) {
ResolvedCreateTableFunctionStmt createTableFunctionStmt =
(ResolvedCreateTableFunctionStmt) statement;
@@ -134,10 +130,12 @@ class ZetaSQLPlannerImpl {
"Statement list must end in a SELECT statement, not " +
statement.nodeKindString());
}
+UserFunctionDefinitions userFunctionDefinitions =
+new UserFunctionDefinitions(udfBuilder.build(), udtvfBuilder.build());
+
ExpressionConverter expressionConverter =
-new ExpressionConverter(cluster, params, udfBuilder.build());
-ConversionContext context =
-ConversionContext.of(config, expressionConverter, cluster, trait,
udtvfBuilder.build());
+new ExpressionConverter(cluster, params, userFunctionDefiniti
[beam] branch master updated (412db21 -> 405204c)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 412db21 Merge pull request #13281 from TheNeuralBit/beam-11159-fix new bd53149 [BEAM-10188] Include links in the release checklist to verify release is published to Github. new ee8b114 [BEAM-10188] Make publishing its own step for extra clarity. new 405204c Merge pull request #13276 from ibzib/release-tag The 29566 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/content/en/contribute/release-guide.md | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-)
[beam] branch master updated: listSubscription should remove TestPubsub's own sub
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c56c33b listSubscription should remove TestPubsub's own sub
new 412db21 Merge pull request #13281 from TheNeuralBit/beam-11159-fix
c56c33b is described below
commit c56c33b2f5fc042c57776e9b07c7bce7650be8e4
Author: Brian Hulette
AuthorDate: Fri Nov 6 16:02:50 2020 -0800
listSubscription should remove TestPubsub's own sub
---
.../main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
index 8df0004..2c32c55 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
@@ -47,6 +47,7 @@ import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Streams;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.hamcrest.Matcher;
import org.joda.time.DateTime;
@@ -227,9 +228,12 @@ public class TestPubsub implements TestRule {
return subscriptionPath;
}
- private Iterable listSubscriptions(TopicPath topicPath) {
+ private List listSubscriptions(TopicPath topicPath) {
Preconditions.checkNotNull(topicAdmin);
-return topicAdmin.listTopicSubscriptions(topicPath.getPath()).iterateAll();
+// Exclude subscriptionPath, the subscription that we created
+return
Streams.stream(topicAdmin.listTopicSubscriptions(topicPath.getPath()).iterateAll())
+.filter((path) -> !path.equals(subscriptionPath.getPath()))
+.collect(Collectors.toList());
}
/** Publish messages to {@link #topicPath()}. */
[beam] branch master updated: Enforce $JAVA_HOME setting in RC build script.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new aaae474 Enforce $JAVA_HOME setting in RC build script.
new 996c8d2 Merge pull request #13143 from ibzib/release-jdk
aaae474 is described below
commit aaae4746987bf6b587286082ca87a86bacbdca40
Author: Kyle Weaver
AuthorDate: Mon Oct 19 15:38:36 2020 -0700
Enforce $JAVA_HOME setting in RC build script.
---
release/src/main/scripts/build_release_candidate.sh | 11 +++
1 file changed, 11 insertions(+)
diff --git a/release/src/main/scripts/build_release_candidate.sh
b/release/src/main/scripts/build_release_candidate.sh
index 7a16104..074b7ff 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -25,6 +25,17 @@
set -e
+if [[ "$JAVA_HOME" ]]; then
+ version=$("$JAVA_HOME/bin/java" -version 2>&1 | awk -F '"' '/version/ {print
$2}')
+ if [[ ! `echo $version | sed "s/1\.8\..*/1.8/"` == "1.8" ]]; then
+echo "Java version $version detected. Set \$JAVA_HOME to point to a JDK 8
installation."
+exit 1
+ fi
+else
+ echo "\$JAVA_HOME must be set."
+ exit 1
+fi
+
LOCAL_CLONE_DIR=build_release_candidate
LOCAL_JAVA_STAGING_DIR=java_staging_dir
LOCAL_PYTHON_STAGING_DIR=python_staging_dir
[beam] branch master updated: Update the test invocation cmd for poratble runner.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 15bd9ae Update the test invocation cmd for poratble runner. new 2a16dd1 Merge pull request #13019 from boyuanzz/cmd_fix 15bd9ae is described below commit 15bd9ae45874c5e32795e39f07679027dd1bd21e Author: Boyuan Zhang AuthorDate: Tue Oct 6 11:42:31 2020 -0700 Update the test invocation cmd for poratble runner. --- sdks/python/apache_beam/runners/portability/flink_runner_test.py | 6 ++ sdks/python/apache_beam/runners/portability/spark_runner_test.py | 6 ++ 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/sdks/python/apache_beam/runners/portability/flink_runner_test.py b/sdks/python/apache_beam/runners/portability/flink_runner_test.py index e83adac..584b31c 100644 --- a/sdks/python/apache_beam/runners/portability/flink_runner_test.py +++ b/sdks/python/apache_beam/runners/portability/flink_runner_test.py @@ -56,10 +56,8 @@ from apache_beam.transforms.sql import SqlTransform # Run as # -# pytest flink_runner_test.py \ -# [--test_pipeline_options "--flink_job_server_jar=/path/to/job_server.jar \ -# --environment_type=DOCKER"] \ -# [FlinkRunnerTest.test_method, ...] +# pytest flink_runner_test.py[::TestClass::test_case] \ +# --test-pipeline-options "--environment_type=LOOPBACK" _LOGGER = logging.getLogger(__name__) diff --git a/sdks/python/apache_beam/runners/portability/spark_runner_test.py b/sdks/python/apache_beam/runners/portability/spark_runner_test.py index a522aca..53f0fbb 100644 --- a/sdks/python/apache_beam/runners/portability/spark_runner_test.py +++ b/sdks/python/apache_beam/runners/portability/spark_runner_test.py @@ -35,10 +35,8 @@ from apache_beam.runners.portability import portable_runner_test # Run as # -# pytest spark_runner_test.py \ -# [--test_pipeline_options "--spark_job_server_jar=/path/to/job_server.jar \ -# --environment_type=DOCKER"] \ -# [SparkRunnerTest.test_method, ...] +# pytest spark_runner_test.py[::TestClass::test_case] \ +# --test-pipeline-options "--environment_type=LOOPBACK" _LOGGER = logging.getLogger(__name__)
