[beam] branch master updated (7eeffe0 -> 6b68536)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 7eeffe0 [BEAM-11936] Enable errorprone unused checks (#16262)
add 6b68536 Add Nexmark Query 14 (#16337)
No new revisions were added by this update.
Summary of changes:
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 3 +++
.../apache/beam/sdk/nexmark/NexmarkQueryName.java | 1 +
.../nexmark/queries/{Query9.java => Query14.java} | 22 ++
3 files changed, 14 insertions(+), 12 deletions(-)
copy
sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/{Query9.java
=> Query14.java} (63%)
[beam] branch master updated (2b96f43 -> bd9e4d3)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2b96f43 [BEAM-12453]: Add interface to access I/O topic information for a Samza Beam job and PipelineJsonRenderer to create the JSON Beam DAG (#14945) add bd9e4d3 [BEAM-10212] Guard state caching with experiment (#15319) No new revisions were added by this update. Summary of changes: .../fnexecution/control/RemoteExecutionTest.java | 25 ++ .../fn/harness/control/ProcessBundleHandler.java | 22 +++ 2 files changed, 34 insertions(+), 13 deletions(-)
[beam] branch master updated: [BEAM-10212] Integrate caching client (#15214)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9c9903d [BEAM-10212] Integrate caching client (#15214)
9c9903d is described below
commit 9c9903d50b59a6ca956b9d43809dc26c490cb849
Author: anthonyqzhu <43458232+anthonyq...@users.noreply.github.com>
AuthorDate: Fri Aug 6 13:03:28 2021 -0400
[BEAM-10212] Integrate caching client (#15214)
* [BEAM-10212] Add state cache to ProcessBundleHandler
---
.../fnexecution/control/RemoteExecutionTest.java | 407 +
.../fn/harness/control/ProcessBundleHandler.java | 52 ++-
2 files changed, 451 insertions(+), 8 deletions(-)
diff --git
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index 5acb87d..f238601 100644
---
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@ -42,6 +42,7 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.UUID;
+import java.util.concurrent.CompletionStage;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
@@ -55,6 +56,7 @@ import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
import
org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
import
org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleSplitResponse;
@@ -68,6 +70,7 @@ import
org.apache.beam.runners.core.construction.graph.FusedPipeline;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
import
org.apache.beam.runners.core.construction.graph.PipelineNode.PTransformNode;
import org.apache.beam.runners.core.construction.graph.ProtoOverrides;
+import org.apache.beam.runners.core.construction.graph.SideInputReference;
import org.apache.beam.runners.core.construction.graph.SplittableParDoExpander;
import org.apache.beam.runners.core.metrics.DistributionData;
import org.apache.beam.runners.core.metrics.ExecutionStateSampler;
@@ -532,6 +535,160 @@ public class RemoteExecutionTest implements Serializable {
}
}
+ @Test
+ public void testExecutionWithSideInputCaching() throws Exception {
+Pipeline p = Pipeline.create();
+addExperiment(p.getOptions().as(ExperimentalOptions.class), "beam_fn_api");
+// TODO(BEAM-10097): Remove experiment once all portable runners support
this view type
+addExperiment(p.getOptions().as(ExperimentalOptions.class),
"use_runner_v2");
+PCollection input =
+p.apply("impulse", Impulse.create())
+.apply(
+"create",
+ParDo.of(
+new DoFn() {
+ @ProcessElement
+ public void process(ProcessContext ctxt) {
+ctxt.output("zero");
+ctxt.output("one");
+ctxt.output("two");
+ }
+}))
+.setCoder(StringUtf8Coder.of());
+PCollectionView> view = input.apply("createSideInput",
View.asIterable());
+
+input
+.apply(
+"readSideInput",
+ParDo.of(
+new DoFn>() {
+ @ProcessElement
+ public void processElement(ProcessContext context) {
+for (String value : context.sideInput(view)) {
+ context.output(KV.of(context.element(), value));
+}
+ }
+})
+.withSideInputs(view))
+.setCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
+// Force the output to be materialized
+.apply("gbk", GroupByKey.create());
+
+RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(p);
+FusedPipeline fused = GreedyPipelineFuser.fuse(pipelineProto);
+Optional optionalStage =
+Iterables.tryFind(
+fused.getFusedStages(), (ExecutableStage stage) ->
!stage.getSideInputs().isEmpty());
+checkState(optionalStage.isPresent(), "Expected a stage with side
inputs.");
+ExecutableStage stage = opt
[beam] branch master updated (f88ed4e -> 63257ed)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f88ed4e [BEAM-12643] Resolved the concurrent writes issue for parallel tests (#15199) add 63257ed [BEAM-10212] Add caching state client wrapper (#15170) No new revisions were added by this update. Summary of changes: .../fn/harness/state/CachingBeamFnStateClient.java | 178 +++ .../state/CachingBeamFnStateClientTest.java| 335 + .../fn/harness/state/FakeBeamFnStateClient.java| 4 + 3 files changed, 517 insertions(+) create mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/CachingBeamFnStateClient.java create mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/CachingBeamFnStateClientTest.java
[beam] branch master updated: [BEAM-12334] Re-use java 11 flag in build.gradle (#14892)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3615801 [BEAM-12334] Re-use java 11 flag in build.gradle (#14892)
3615801 is described below
commit 36158013318fcfd7f18a432ccbc9b18b945a430c
Author: kileys
AuthorDate: Tue Jun 15 16:47:09 2021 -0700
[BEAM-12334] Re-use java 11 flag in build.gradle (#14892)
---
.test-infra/jenkins/NexmarkBuilder.groovy | 14 ++-
.../job_PostCommit_Java_Nexmark_Dataflow_V2.groovy | 4 ++--
...stCommit_Java_Nexmark_Dataflow_V2_Java11.groovy | 4 ++--
sdks/java/testing/nexmark/build.gradle | 27 --
4 files changed, 17 insertions(+), 32 deletions(-)
diff --git a/.test-infra/jenkins/NexmarkBuilder.groovy
b/.test-infra/jenkins/NexmarkBuilder.groovy
index 1841a3e..d50cc6f 100644
--- a/.test-infra/jenkins/NexmarkBuilder.groovy
+++ b/.test-infra/jenkins/NexmarkBuilder.groovy
@@ -63,6 +63,16 @@ class NexmarkBuilder {
suite(context, "NEXMARK IN ZETASQL STREAMING MODE USING ${runner} RUNNER",
runner, sdk, options, jobSpecificSwitches, javaRuntimeVersion)
}
+ static void nonQueryLanguageJobs(context, Runner runner, SDK sdk,
Map jobSpecificOptions, TriggeringContext triggeringContext,
List jobSpecificSwitches, String javaRuntimeVersion) {
+Map options = getFullOptions(jobSpecificOptions, runner,
triggeringContext)
+
+options.put('streaming', false)
+suite(context, "NEXMARK IN BATCH MODE USING ${runner} RUNNER", runner,
sdk, options, jobSpecificSwitches, javaRuntimeVersion)
+
+options.put('streaming', true)
+suite(context, "NEXMARK IN STREAMING MODE USING ${runner} RUNNER", runner,
sdk, options, jobSpecificSwitches, javaRuntimeVersion)
+ }
+
static void batchOnlyJob(context, Runner runner, SDK sdk, Map jobSpecificOptions, TriggeringContext triggeringContext) {
Map options = getFullOptions(jobSpecificOptions, runner,
triggeringContext)
@@ -122,8 +132,10 @@ class NexmarkBuilder {
// Run with Java 11
gradle {
rootBuildScriptDir(commonJobProperties.checkoutDir)
-tasks(':sdks:java:testing:nexmark:runJava11')
+tasks(':sdks:java:testing:nexmark:run')
commonJobProperties.setGradleSwitches(delegate)
+switches("-PcompileAndRunTestsWithJava11")
+switches("-Pjava11Home=${commonJobProperties.JAVA_11_HOME}")
switches("-Pnexmark.runner=${runner.getDependencyBySDK(sdk)}")
switches("-Pnexmark.args=\"${parseOptions(options)}\"")
if (jobSpecificSwitches != null) {
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2.groovy
b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2.groovy
index d39ab69..9d7a124 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2.groovy
@@ -53,7 +53,7 @@
NoPhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Nexmark_
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 240)
- Nexmark.standardJob(delegate, Runner.DATAFLOW, SDK.JAVA,
JOB_SPECIFIC_OPTIONS, TriggeringContext.POST_COMMIT, JOB_SPECIFIC_SWITCHES,
Nexmark.DEFAULT_JAVA_RUNTIME_VERSION)
+ Nexmark.nonQueryLanguageJobs(delegate, Runner.DATAFLOW, SDK.JAVA,
JOB_SPECIFIC_OPTIONS, TriggeringContext.POST_COMMIT, JOB_SPECIFIC_SWITCHES,
Nexmark.DEFAULT_JAVA_RUNTIME_VERSION)
}
PhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Nexmark_DataflowV2',
@@ -63,5 +63,5 @@
PhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Nexmark_Da
commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 240)
- Nexmark.standardJob(delegate, Runner.DATAFLOW, SDK.JAVA,
JOB_SPECIFIC_OPTIONS, TriggeringContext.PR, JOB_SPECIFIC_SWITCHES,
Nexmark.DEFAULT_JAVA_RUNTIME_VERSION)
+ Nexmark.nonQueryLanguageJobs(delegate, Runner.DATAFLOW, SDK.JAVA,
JOB_SPECIFIC_OPTIONS, TriggeringContext.PR, JOB_SPECIFIC_SWITCHES,
Nexmark.DEFAULT_JAVA_RUNTIME_VERSION)
}
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2_Java11.groovy
b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2_Java11.groovy
index 1c5c380..8fc69a7 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2_Java11.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2_Java11.groovy
@@ -53,7 +53,7 @@
NoPhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Nexmark_
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 240)
- Nexmark.standardJob(delegate, Runner.DATAFLOW, SDK.J
[beam] branch master updated: Graph java 8 and 11 metrics separately (#14860)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a3c744e Graph java 8 and 11 metrics separately (#14860)
a3c744e is described below
commit a3c744e7d0bf9302ba8e8d02bc2ad097708800e3
Author: kileys
AuthorDate: Fri May 21 17:27:05 2021 -0700
Graph java 8 and 11 metrics separately (#14860)
---
.../Nexmark_Dataflow_RunnerV2.json | 30 +++---
1 file changed, 26 insertions(+), 4 deletions(-)
diff --git
a/.test-infra/metrics/grafana/dashboards/perftests_metrics/Nexmark_Dataflow_RunnerV2.json
b/.test-infra/metrics/grafana/dashboards/perftests_metrics/Nexmark_Dataflow_RunnerV2.json
index c45b28a..0a42f66 100644
---
a/.test-infra/metrics/grafana/dashboards/perftests_metrics/Nexmark_Dataflow_RunnerV2.json
+++
b/.test-infra/metrics/grafana/dashboards/perftests_metrics/Nexmark_Dataflow_RunnerV2.json
@@ -111,7 +111,7 @@
],
"orderByTime": "ASC",
"policy": "default",
- "query": "SELECT \"runtimeMs\" FROM
\"forever\".\"nexmark_${ID}_${processingType}\" WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'v2' AND $timeFilter",
+ "query": "SELECT \"runtimeMs\" FROM
\"forever\".\"nexmark_${ID}_${processingType}\" WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'V2' AND $timeFilter",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
@@ -229,7 +229,7 @@
],
"orderByTime": "ASC",
"policy": "default",
- "query": "SELECT \"runtimeMs\" FROM
\"forever\".\"nexmark_${ID}_sql_${processingType}\" WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'v2' AND $timeFilter",
+ "query": "SELECT \"runtimeMs\" FROM
\"forever\".\"nexmark_${ID}_sql_${processingType}\" WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'V2' AND $timeFilter",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
@@ -338,12 +338,12 @@
"steppedLine": false,
"targets": [
{
- "alias": "$m",
+ "alias": "[[m]]_java8",
"groupBy": [],
"measurement": "",
"orderByTime": "ASC",
"policy": "default",
- "query": "SELECT \"runtimeMs\" FROM
\"forever\"./nexmark_${ID}_\\w*${processingType}/ WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'v2' AND $timeFilter GROUP BY
\"runner\"",
+ "query": "SELECT \"runtimeMs\" FROM
\"forever\"./nexmark_${ID}_\\w*${processingType}/ WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'V2' AND \"javaVersion\" = '8' AND
$timeFilter GROUP BY \"runner\"",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
@@ -358,6 +358,28 @@
]
],
"tags": []
+},
+{
+ "alias": "[[m]]_java11",
+ "groupBy": [],
+ "measurement": "",
+ "orderByTime": "ASC",
+ "policy": "default",
+ "query": "SELECT \"runtimeMs\" FROM
\"forever\"./nexmark_${ID}_\\w*${processingType}/ WHERE \"runner\" =
'DataflowRunner' AND \"runnerVersion\" = 'V2' AND \"javaVersion\" = '11' AND
$timeFilter GROUP BY \"runner\"",
+ "rawQuery": true,
+ "refId": "B",
+ "resultFormat": "time_series",
+ "select": [
+[
+ {
+"params": [
+ "value"
+],
+"type": "field"
+ }
+]
+ ],
+ "tags": []
}
],
"thresholds": [],
[beam] branch master updated (4b449f2 -> f25b42e)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4b449f2 [BEAM-9547] Add support for drop_duplicates and duplicated (#14786) add f25b42e Don't compile with 8 if flag is set (#14838) No new revisions were added by this update. Summary of changes: sdks/java/testing/nexmark/build.gradle | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (c7c97b0 -> 22516a7)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from c7c97b0 [BEAM-12361] Ensure that Reshuffle.AssignToShard numBuckets
is respected (#14720)
add 22516a7 Run Jenkins jobs for Nexmark Dataflow V2 Java 8 and 11
(#14774)
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/NexmarkBuilder.groovy | 62 ---
.../job_PostCommit_Java_Nexmark_Dataflow_V2.groovy | 67
...stCommit_Java_Nexmark_Dataflow_V2_Java11.groovy | 67
.../dashboards/perftests_metrics/Nexmark.json | 14 +++-
...Nexmark.json => Nexmark_Dataflow_RunnerV2.json} | 50
sdks/java/testing/nexmark/build.gradle | 89 +-
.../java/org/apache/beam/sdk/nexmark/Main.java | 3 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java| 7 ++
.../testutils/publishing/InfluxDBPublisher.java| 45 +++
9 files changed, 320 insertions(+), 84 deletions(-)
create mode 100644
.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2.groovy
create mode 100644
.test-infra/jenkins/job_PostCommit_Java_Nexmark_Dataflow_V2_Java11.groovy
copy .test-infra/metrics/grafana/dashboards/perftests_metrics/{Nexmark.json =>
Nexmark_Dataflow_RunnerV2.json} (91%)
[beam] branch master updated: [BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient triggering failed precondition. (#14668)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 400b1d7 [BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient
triggering failed precondition. (#14668)
400b1d7 is described below
commit 400b1d73a74b88c3cb7d7f4ac9a20580a7fcd5fd
Author: scwhittle
AuthorDate: Thu Apr 29 09:30:08 2021 -0700
[BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient triggering
failed precondition. (#14668)
---
.../beam/fn/harness/data/QueueingBeamFnDataClient.java | 14 --
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
index 21d0b14..11dea66 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
@@ -83,12 +83,22 @@ public class QueueingBeamFnDataClient implements
BeamFnDataClient {
// empty in which case it returns null.
@Nullable
ConsumerAndData take() throws InterruptedException {
+ // We first poll without blocking to optimize for the case there is data.
+ // If there is no data we end up blocking on take() and thus the extra
+ // poll doesn't matter.
@Nullable ConsumerAndData result = queue.poll();
if (result == null) {
if (closed.get()) {
- return null;
+ // Poll again to ensure that there is nothing in the queue. Once we
observe closed as true
+ // we are guaranteed no additional elements other than the POISON
will be added. However
+ // we can't rely on the previous poll result as it could race with
additional offers and
+ // close.
+ result = queue.poll();
+} else {
+ // We are not closed so we perform a blocking take. We are
guaranteed that additional
+ // elements will be offered or the POISON will be added by close to
unblock this thread.
+ result = queue.take();
}
-result = queue.take();
}
if (result == POISON) {
return null;
[beam] branch master updated (dc636be -> 47cfbcb)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dc636be [BEAM-12209] Fix thread safety of DirectStreamObserver by using atomic for message counting (#14617) add 47cfbcb [BEAM-12127] More optimizations for PCollectionConsumerRegistry: cache monitoring container, avoid randomInt for sampling if resevoir is empty (#14601) No new revisions were added by this update. Summary of changes: .../harness/data/PCollectionConsumerRegistry.java | 29 ++ 1 file changed, 19 insertions(+), 10 deletions(-)
[beam] branch master updated (9209c75 -> dc636be)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9209c75 Merge pull request #14628: [BEAM-12217] MongoDbIO: Read.withFilter() and Read.withProjection() are removed since they are deprecated since Beam 2.12.0 add dc636be [BEAM-12209] Fix thread safety of DirectStreamObserver by using atomic for message counting (#14617) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java | 8 1 file changed, 4 insertions(+), 4 deletions(-)
[beam] branch master updated (8122b33 -> d502185)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8122b33 [BEAM-366] Populate display data in portable job representation (#14470) add d502185 [BEAM-12118] Modify QueuingBeamFnDataClient to avoid completion latency due to polling. (#14480) No new revisions were added by this update. Summary of changes: .../worker/fn/data/BeamFnDataGrpcService.java | 6 + .../sdk/fn/data/BeamFnDataInboundObserver.java | 5 + .../data/CompletableFutureInboundDataClient.java | 6 + .../apache/beam/sdk/fn/data/InboundDataClient.java | 3 + .../fn/harness/control/ProcessBundleHandler.java | 5 +- .../beam/fn/harness/data/BeamFnDataGrpcClient.java | 3 +- .../fn/harness/data/BeamFnTimerGrpcClient.java | 5 + .../fn/harness/data/QueueingBeamFnDataClient.java | 181 - .../fn/harness/data/FakeBeamFnTimerClient.java | 6 + .../harness/data/QueueingBeamFnDataClientTest.java | 121 +- 10 files changed, 296 insertions(+), 45 deletions(-)
[beam] branch master updated (476c2b1 -> 43fba5f)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 476c2b1 [BEAM-12142] Reduce ThreadLocal interaction with MetricsEnvironment by leaving unchanging (#14498) add 43fba5f [BEAM-12117] When reusing QueueingBeamFnDataClient, reset it to avoid its set of inbound clients from growing without bound. (#14458) No new revisions were added by this update. Summary of changes: .../fn/harness/control/ProcessBundleHandler.java | 1 + .../fn/harness/data/QueueingBeamFnDataClient.java | 68 ++ 2 files changed, 44 insertions(+), 25 deletions(-)
[beam] branch master updated (d3cfb71 -> 476c2b1)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d3cfb71 Optimize ExecutionStateTracker reset to not iterate over all trackers (#14495) add 476c2b1 [BEAM-12142] Reduce ThreadLocal interaction with MetricsEnvironment by leaving unchanging (#14498) No new revisions were added by this update. Summary of changes: .../beam/sdk/metrics/MetricsEnvironment.java | 33 .../beam/sdk/metrics/MetricsEnvironmentTest.java | 36 -- 2 files changed, 54 insertions(+), 15 deletions(-)
[beam] branch master updated (9a9d10a -> d3cfb71)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9a9d10a Avoid multiple lookups in ConcurrentHashMap by utilizing compute* methods instead of chained get/put. (#14494) add d3cfb71 Optimize ExecutionStateTracker reset to not iterate over all trackers (#14495) No new revisions were added by this update. Summary of changes: .../apache/beam/runners/core/metrics/ExecutionStateTracker.java | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (1b86266 -> 9a9d10a)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1b86266 [BEAM-12104] Pin Dataflow wordcount to ubuntu 18.04 (#14451) add 9a9d10a Avoid multiple lookups in ConcurrentHashMap by utilizing compute* methods instead of chained get/put. (#14494) No new revisions were added by this update. Summary of changes: .../core/metrics/MetricsContainerStepMap.java | 27 +++--- 1 file changed, 14 insertions(+), 13 deletions(-)
[beam] branch master updated (b5a8b54 -> c1035ab)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b5a8b54 Turn on mpyp checks for filesystem (#14425) add c1035ab [BEAM-12112] Disable streaming mode for PORTABILITY_BATCH (#14452) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java | 6 ++ 1 file changed, 6 insertions(+)
[beam] branch master updated (5ffb3ee -> abbe14f)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5ffb3ee [BEAM-9615] Misc final schema cleanups. (#14285) add abbe14f [BEAM-12083] Nexmark Query 13. (#14404) No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/transforms/Reshuffle.java | 52 +- .../beam/sdk/nexmark/NexmarkConfiguration.java | 13 +++ .../apache/beam/sdk/nexmark/NexmarkLauncher.java | 4 + .../apache/beam/sdk/nexmark/NexmarkQueryName.java | 1 + .../apache/beam/sdk/nexmark/queries/Query13.java | 107 + 5 files changed, 151 insertions(+), 26 deletions(-) create mode 100644 sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query13.java
[beam] branch master updated: [BEAM-11887] Change SortingFlinkCombineRunner usage to only for Sessions (#14120)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2da7448 [BEAM-11887] Change SortingFlinkCombineRunner usage to only
for Sessions (#14120)
2da7448 is described below
commit 2da7448e1a4be2870eeb0e994937beb7e4d4ad1c
Author: Yichi Zhang
AuthorDate: Thu Mar 11 17:46:00 2021 -0800
[BEAM-11887] Change SortingFlinkCombineRunner usage to only for Sessions
(#14120)
---
.../functions/FlinkMergingNonShuffleReduceFunction.java | 4 ++--
.../flink/translation/functions/FlinkPartialReduceFunction.java | 9 -
.../runners/flink/translation/functions/FlinkReduceFunction.java | 9 -
3 files changed, 10 insertions(+), 12 deletions(-)
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
index b34649f..b1b95c6 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkMergingNonShuffleReduceFunction.java
@@ -23,7 +23,7 @@ import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
@@ -83,7 +83,7 @@ public class FlinkMergingNonShuffleReduceFunction<
new FlinkSideInputReader(sideInputs, getRuntimeContext());
AbstractFlinkCombineRunner reduceRunner;
-if
(windowingStrategy.getWindowFn().windowCoder().equals(IntervalWindow.getCoder()))
{
+if (windowingStrategy.getWindowFn() instanceof Sessions) {
reduceRunner = new SortingFlinkCombineRunner<>();
} else {
reduceRunner = new HashingFlinkCombineRunner<>();
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
index f98b9df..7b0f5d5 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkPartialReduceFunction.java
@@ -23,7 +23,7 @@ import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
@@ -98,11 +98,10 @@ public class FlinkPartialReduceFunction();
} else {
- if (windowingStrategy.needsMerge()
- &&
!windowingStrategy.getWindowFn().windowCoder().equals(IntervalWindow.getCoder()))
{
-reduceRunner = new HashingFlinkCombineRunner<>();
- } else {
+ if (windowingStrategy.needsMerge() && windowingStrategy.getWindowFn()
instanceof Sessions) {
reduceRunner = new SortingFlinkCombineRunner<>();
+ } else {
+reduceRunner = new HashingFlinkCombineRunner<>();
}
}
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
index 80ce7ef..1399869 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkReduceFunction.java
@@ -23,7 +23,7 @@ import org.apache.beam.sdk.io.FileSystems;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.transforms.CombineFnBase;
import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.util.WindowedValue;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollectionView;
@@ -98,11 +98,10 @@ public c
[beam] branch master updated (b675eb5 -> f0d1aef)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b675eb5 Merge pull request #13660 from sonam-vend/runner-google-cloud-dataflow add f0d1aef [BEAM-11962] Disable failing test (#14202) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/examples/cookbook/MapClassIntegrationIT.java | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (4e7d645 -> b55dd49)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 4e7d645 [BEAM-11659] Allow Kafka sql table provider to have a nested
mode and raw binary payloads (#14016)
add b55dd49 [BEAM-11659] Implement Pub/Sub Lite Table Provider (#13920)
No new revisions were added by this update.
Summary of changes:
.../src/main/resources/beam/suppressions.xml | 1 +
.../java/org/apache/beam/sdk/schemas/Schema.java | 35 +++
.../beam/sdk/schemas/io/DeadLetteredTransform.java | 87
.../org/apache/beam/sdk/schemas/SchemaTest.java| 19 ++
.../sdk/schemas/io/DeadLetteredTransformTest.java | 102 +
.../PubsubLiteSubscriptionTable.java} | 63 +++---
.../pubsublite/PubsubLiteTableProvider.java| 219 +++
.../PubsubLiteTopicTable.java} | 60 +++---
.../sql/meta/provider/pubsublite/RowHandler.java | 171 +++
.../meta/provider/pubsublite}/package-info.java| 2 +-
.../pubsublite/PubsubLiteTableProviderTest.java| 239 +
.../meta/provider/pubsublite/RowHandlerTest.java | 237
12 files changed, 1169 insertions(+), 66 deletions(-)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/DeadLetteredTransform.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/DeadLetteredTransformTest.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/{seqgen/GenerateSequenceTable.java
=> pubsublite/PubsubLiteSubscriptionTable.java} (52%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/PubsubLiteTableProvider.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/{seqgen/GenerateSequenceTable.java
=> pubsublite/PubsubLiteTopicTable.java} (52%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/RowHandler.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/schemas/io =>
extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite}/package-info.java
(93%)
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/PubsubLiteTableProviderTest.java
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsublite/RowHandlerTest.java
[beam] branch master updated (21feb59 -> 4e7d645)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 21feb59 Merge pull request #14173: [BEAM-11941] Upgrade Flink runner to Flink version 1.12.2 add 4e7d645 [BEAM-11659] Allow Kafka sql table provider to have a nested mode and raw binary payloads (#14016) No new revisions were added by this update. Summary of changes: .../sql/meta/provider/kafka/BeamKafkaCSVTable.java | 37 ++- .../sql/meta/provider/kafka/BeamKafkaTable.java| 15 +- .../meta/provider/kafka/KafkaTableProvider.java| 46 ++-- .../provider/kafka/NestedPayloadKafkaTable.java| 181 + .../kafka/PayloadSerializerKafkaTable.java | 25 +- .../sql/meta/provider/kafka/Schemas.java | 107 .../provider/kafka/BeamKafkaTableAvroTest.java | 3 +- .../meta/provider/kafka/BeamKafkaTableCSVTest.java | 2 +- .../provider/kafka/BeamKafkaTableJsonTest.java | 3 +- .../provider/kafka/BeamKafkaTableProtoTest.java| 9 +- .../meta/provider/kafka/BeamKafkaTableTest.java| 42 ++- .../provider/kafka/BeamKafkaTableThriftTest.java | 9 +- .../meta/provider/kafka/KafkaTableProviderIT.java | 61 + .../provider/kafka/KafkaTableProviderTest.java | 69 - .../sql/meta/provider/kafka/KafkaTestTable.java| 9 +- .../kafka/NestedPayloadKafkaTableTest.java | 290 + .../org/apache/beam/sdk/io/kafka/KafkaRecord.java | 4 +- 17 files changed, 843 insertions(+), 69 deletions(-) create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/NestedPayloadKafkaTable.java create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/Schemas.java create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/NestedPayloadKafkaTableTest.java
[beam] branch master updated (137a0b6 -> 40c4386)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 137a0b6 Merge pull request #14072: Merge Fn API and runner v2 configurations for DataflowRunner add 40c4386 [BEAM-9378] Add ignored tests which fail in various ways when querying nested structures (#14077) No new revisions were added by this update. Summary of changes: .../extensions/sql/BeamSqlDslNestedRowsTest.java | 83 ++ .../sql/zetasql/ZetaSqlDialectSpecTest.java| 18 + 2 files changed, 101 insertions(+)
[beam] branch master updated (0566418 -> 09e65cb)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0566418 Removed conscrypt from a shaded dataflow-worker jar (#13846) add 09e65cb [BEAM-11747] Reject the mixed Java UDF and ZetaSQL builtin operator cases (#13912) No new revisions were added by this update. Summary of changes: .../sql/zetasql/BeamJavaUdfCalcRule.java | 2 +- .../sql/zetasql/BeamZetaSqlCalcRule.java | 2 +- .../sql/zetasql/ZetaSQLQueryPlanner.java | 48 +++--- .../extensions/sql/zetasql/ZetaSqlJavaUdfTest.java | 14 +++ 4 files changed, 50 insertions(+), 16 deletions(-)
[beam] 01/01: Revert "[BEAM-11659] Extract common payload serialization behavior for Row transforms. (#13825)"
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch revert-13825-payload_utils
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 0012fc6f2b8d3650ff50a1f7dcc101f5a4c488d1
Author: Rui Wang
AuthorDate: Wed Feb 10 15:10:08 2021 -0800
Revert "[BEAM-11659] Extract common payload serialization behavior for Row
transforms. (#13825)"
This reverts commit e6eba9edb2632bfd132337873adc1f5c9f5bc703.
---
.../src/main/resources/beam/suppressions.xml | 1 -
.../org/apache/beam/sdk/schemas/io/GenericDlq.java | 17 ++-
.../beam/sdk/schemas/io/GenericDlqProvider.java| 6 +-
.../org/apache/beam/sdk/schemas/io/Providers.java | 54
.../io/payloads/AvroPayloadSerializerProvider.java | 42 --
.../io/payloads/JsonPayloadSerializerProvider.java | 52 ---
.../sdk/schemas/io/payloads/PayloadSerializer.java | 51 ---
.../io/payloads/PayloadSerializerProvider.java | 37 -
.../schemas/io/payloads/PayloadSerializers.java| 47 ---
.../beam/sdk/schemas/io/payloads/package-info.java | 24
.../org/apache/beam/sdk/util/RowJsonUtils.java | 2 +-
.../io/AvroPayloadSerializerProviderTest.java | 64 -
.../io/JsonPayloadSerializerProviderTest.java | 60
.../meta/provider/kafka/BeamKafkaAvroTable.java| 89
.../meta/provider/kafka/BeamKafkaJsonTable.java| 107 ++
.../meta/provider/kafka/BeamKafkaProtoTable.java | 119
.../meta/provider/kafka/BeamKafkaThriftTable.java | 138 ++
.../meta/provider/kafka/KafkaTableProvider.java| 86 +---
.../kafka/PayloadSerializerKafkaTable.java | 69 -
.../payloads/ProtoPayloadSerializerProvider.java | 85
.../payloads/ThriftPayloadSerializerProvider.java | 108 ---
.../sql/meta/provider/payloads/package-info.java | 24
.../provider/kafka/BeamKafkaTableAvroTest.java | 12 +-
.../provider/kafka/BeamKafkaTableJsonTest.java | 12 +-
.../provider/kafka/BeamKafkaTableProtoTest.java| 36 ++---
.../provider/kafka/BeamKafkaTableThriftTest.java | 35 ++---
.../provider/kafka/KafkaTableProviderTest.java | 12 +-
.../ProtoPayloadSerializerProviderTest.java| 111 ---
.../ThriftPayloadSerializerProviderTest.java | 154 -
.../beam/sdk/io/gcp/pubsub/PubsubMessageToRow.java | 98 ++---
.../sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java | 49 +--
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 60 ++--
.../sdk/io/gcp/pubsub/PubsubMessageToRowTest.java | 40 --
33 files changed, 732 insertions(+), 1169 deletions(-)
diff --git a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
index 89b7744..ee03dc3 100644
--- a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
@@ -98,7 +98,6 @@
-
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlq.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlq.java
index 03c9912..f059c11 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlq.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlq.java
@@ -19,8 +19,10 @@ package org.apache.beam.sdk.schemas.io;
import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.ServiceLoader;
import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.annotations.Experimental.Kind;
import org.apache.beam.sdk.annotations.Internal;
@@ -35,8 +37,19 @@ import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Splitter;
public final class GenericDlq {
private GenericDlq() {}
- private static final Map PROVIDERS =
- Providers.loadProviders(GenericDlqProvider.class);
+ private static final Map PROVIDERS =
loadDlqProviders();
+
+ private static Map loadDlqProviders() {
+Map providers = new HashMap<>();
+for (GenericDlqProvider provider :
ServiceLoader.load(GenericDlqProvider.class)) {
+ checkArgument(
+ !providers.containsKey(provider.identifier()),
+ "Duplicate providers exist with identifier `%s`.",
+ provider.identifier());
+ providers.put(provider.identifier(), provider);
+}
+return providers;
+ }
@SuppressWarnings("dereference.of.nullable")
public static PTransform, PDone> getDlqTransform(String
fullConfig) {
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlqProvider.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/i
[beam] branch revert-13825-payload_utils created (now 0012fc6)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch revert-13825-payload_utils in repository https://gitbox.apache.org/repos/asf/beam.git. at 0012fc6 Revert "[BEAM-11659] Extract common payload serialization behavior for Row transforms. (#13825)" This branch includes the following new commits: new 0012fc6 Revert "[BEAM-11659] Extract common payload serialization behavior for Row transforms. (#13825)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated: [BEAM-10925] Test Java UDF on columns
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fc8f4e6 [BEAM-10925] Test Java UDF on columns
new 16f757c Merge pull request #13947 from amaliujia/rw-more-testing
fc8f4e6 is described below
commit fc8f4e630763bac6227e52a567cfe63052b4422f
Author: amaliujia
AuthorDate: Tue Feb 9 22:39:16 2021 -0800
[BEAM-10925] Test Java UDF on columns
---
.../extensions/sql/zetasql/ZetaSqlJavaUdfTest.java | 21 +
1 file changed, 21 insertions(+)
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
index 34b1d23..9d118be 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
@@ -95,6 +95,27 @@ public class ZetaSqlJavaUdfTest extends ZetaSqlTestBase {
}
@Test
+ public void testJavaUdfColumnReference() {
+String sql =
+String.format(
+"CREATE FUNCTION increment(i INT64) RETURNS INT64 LANGUAGE java "
++ "OPTIONS (path='%s'); "
++ "SELECT increment(Key) FROM KeyValue;",
+jarPath);
+ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+BeamRelNode beamRelNode = zetaSQLQueryPlanner.convertToBeamRel(sql);
+PCollection stream = BeamSqlRelUtils.toPCollection(pipeline,
beamRelNode);
+
+Schema singleField = Schema.builder().addInt64Field("field1").build();
+
+PAssert.that(stream)
+.containsInAnyOrder(
+Row.withSchema(singleField).addValues(15L).build(),
+Row.withSchema(singleField).addValues(16L).build());
+
pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
+ }
+
+ @Test
public void testNestedJavaUdf() {
String sql =
String.format(
[beam] branch master updated (c6db9ca -> e6eba9e)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from c6db9ca [BEAM-11324] Impose consistent ordering on partitionings used
in PartitioningSession (#13854)
add e6eba9e [BEAM-11659] Extract common payload serialization behavior
for Row transforms. (#13825)
No new revisions were added by this update.
Summary of changes:
.../src/main/resources/beam/suppressions.xml | 1 +
.../org/apache/beam/sdk/schemas/io/GenericDlq.java | 17 +--
.../beam/sdk/schemas/io/GenericDlqProvider.java| 6 +-
...nvalidLocationException.java => Providers.java} | 33 +++--
.../AvroPayloadSerializerProvider.java}| 25 ++--
.../io/payloads/JsonPayloadSerializerProvider.java | 52 +++
.../payloads/PayloadSerializer.java} | 31 -
.../payloads/PayloadSerializerProvider.java} | 19 ++-
.../PayloadSerializers.java} | 42 +++---
.../schemas/io/{ => payloads}/package-info.java| 2 +-
.../org/apache/beam/sdk/util/RowJsonUtils.java | 2 +-
.../io/AvroPayloadSerializerProviderTest.java | 64 +
.../io/JsonPayloadSerializerProviderTest.java | 60
.../meta/provider/kafka/BeamKafkaAvroTable.java| 89
.../meta/provider/kafka/BeamKafkaJsonTable.java| 107 --
.../meta/provider/kafka/BeamKafkaProtoTable.java | 119
.../meta/provider/kafka/BeamKafkaThriftTable.java | 138 --
.../meta/provider/kafka/KafkaTableProvider.java| 86 +++-
.../kafka/PayloadSerializerKafkaTable.java | 69 +
.../payloads/ProtoPayloadSerializerProvider.java | 85
.../payloads/ThriftPayloadSerializerProvider.java | 108 +++
.../sql/meta/provider/payloads}/package-info.java | 2 +-
.../provider/kafka/BeamKafkaTableAvroTest.java | 12 +-
.../provider/kafka/BeamKafkaTableJsonTest.java | 12 +-
.../provider/kafka/BeamKafkaTableProtoTest.java| 36 +++--
.../provider/kafka/BeamKafkaTableThriftTest.java | 35 +++--
.../provider/kafka/KafkaTableProviderTest.java | 12 +-
.../ProtoPayloadSerializerProviderTest.java| 111 +++
.../ThriftPayloadSerializerProviderTest.java | 154 +
.../beam/sdk/io/gcp/pubsub/PubsubMessageToRow.java | 98 +++--
.../sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java | 49 ++-
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 60 ++--
.../sdk/io/gcp/pubsub/PubsubMessageToRowTest.java | 40 ++
33 files changed, 988 insertions(+), 788 deletions(-)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{InvalidLocationException.java
=> Providers.java} (52%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{InvalidLocationException.java
=> payloads/AvroPayloadSerializerProvider.java} (61%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/payloads/JsonPayloadSerializerProvider.java
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{Factory.java =>
io/payloads/PayloadSerializer.java} (59%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{Factory.java =>
io/payloads/PayloadSerializerProvider.java} (66%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{Failure.java
=> payloads/PayloadSerializers.java} (53%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{ =>
payloads}/package-info.java (95%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/AvroPayloadSerializerProviderTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/JsonPayloadSerializerProviderTest.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaAvroTable.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaJsonTable.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaProtoTable.java
delete mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaThriftTable.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/PayloadSerializerKafkaTable.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/payloads/ProtoPayloadSerializerProvider.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/payloads/ThriftPayloadSerializerProvider.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/schemas/io =>
extensions/sql/src/main/java/org/apache/b
[beam] branch master updated (73731ec -> 623a9c0)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 73731ec Merge pull request #13783 from [BEAM-10120] Add dynamic timer support to portable Flink add 623a9c0 [BEAM-11076] Reuse TriggerProto translation in StreamingGroupAlsoByWindowViaWindowSetFn (#13831) No new revisions were added by this update. Summary of changes: .../dataflow/worker/StreamingGroupAlsoByWindowViaWindowSetFn.java | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
[beam] branch master updated (f4f4611 -> 9863b57)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f4f4611 Merge pull request #13811 from ibzib/flink-docker-push add 9863b57 [BEAM-11691] Skip JavaUdfLoaderTest instead of failing when jar path system properties aren't set. (#13814) No new revisions were added by this update. Summary of changes: .../beam/sdk/extensions/sql/impl/JavaUdfLoaderTest.java | 16 +++- 1 file changed, 3 insertions(+), 13 deletions(-)
[beam] branch master updated (7e87324 -> a724602)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 7e87324 [BEAM-10961] Enabled Strict dependency on Fn-execution Runner
(#13661)
add a724602 [BEAM-11659]: Add DlqProvider, a way of handling failures in
a generic way. (#13773)
No new revisions were added by this update.
Summary of changes:
...{InvalidLocationException.java => Failure.java} | 30 +---
.../org/apache/beam/sdk/schemas/io/GenericDlq.java | 66 +
...chemaException.java => GenericDlqProvider.java} | 21 +++---
.../apache/beam/sdk/schemas/io/GenericDlqTest.java | 61
.../beam/sdk/schemas/io/StoringDlqProvider.java| 72 ++
.../sdk/io/gcp/bigquery/BigQueryDlqProvider.java | 85 ++
.../beam/sdk/io/gcp/pubsub/PubsubDlqProvider.java | 66 +
.../beam/sdk/io/gcp/pubsublite/DlqProvider.java| 77
8 files changed, 457 insertions(+), 21 deletions(-)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{InvalidLocationException.java
=> Failure.java} (57%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlq.java
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{InvalidSchemaException.java
=> GenericDlqProvider.java} (68%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/GenericDlqTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/StoringDlqProvider.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryDlqProvider.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubDlqProvider.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/DlqProvider.java
[beam] branch master updated (c57d71f -> 6675586)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c57d71f Make DataCatalogTableProvider.createDataCatalogClient public (#13752) add 6675586 [BEAM-11637] Use accumulators properly in BitAnd. (#13745) No new revisions were added by this update. Summary of changes: .../impl/transform/BeamBuiltinAggregations.java| 56 +++--- 1 file changed, 38 insertions(+), 18 deletions(-)
[beam] branch master updated (23d404d -> 6432274)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 23d404d [BEAM-11569] Enabling wheels to be built using Beam-based submodules workflow (#13736) add 6432274 [BEAM-11624] Add hash functions in beam sql (#13733) No new revisions were added by this update. Summary of changes: .../sql/zetasql/SupportedZetaSqlBuiltinFunctions.java | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-)
[beam] branch master updated: BEAM-11536. Test "beam:window_fn:serialized_java:v1" in WindowStrategyTranslation
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0c84a51 BEAM-11536. Test "beam:window_fn:serialized_java:v1" in
WindowStrategyTranslation
new 5e17b69 Merge pull request #13630 from amaliujia/BEAM-11536
0c84a51 is described below
commit 0c84a5140e8b3248911a8620740a31befd4ccb55
Author: amaliujia
AuthorDate: Mon Dec 28 23:13:12 2020 -0800
BEAM-11536. Test "beam:window_fn:serialized_java:v1" in
WindowStrategyTranslation
---
.../WindowingStrategyTranslationTest.java | 159 -
1 file changed, 158 insertions(+), 1 deletion(-)
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
index f68ed2e..d5d1a11 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslationTest.java
@@ -21,19 +21,37 @@ import static org.hamcrest.Matchers.equalTo;
import static org.junit.Assert.assertThat;
import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CustomCoder;
+import org.apache.beam.sdk.coders.VarIntCoder;
import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
import org.apache.beam.sdk.transforms.windowing.Sessions;
import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
import org.apache.beam.sdk.transforms.windowing.Trigger;
import org.apache.beam.sdk.transforms.windowing.Window.ClosingBehavior;
import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
+import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.WindowingStrategy;
import org.apache.beam.sdk.values.WindowingStrategy.AccumulationMode;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.checkerframework.checker.nullness.qual.Nullable;
import org.joda.time.Duration;
+import org.joda.time.Instant;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
@@ -98,7 +116,8 @@ public class WindowingStrategyTranslationTest {
.withMode(AccumulationMode.RETRACTING_FIRED_PANES)
.withTrigger(REPRESENTATIVE_TRIGGER)
.withAllowedLateness(Duration.millis(100))
-.withTimestampCombiner(TimestampCombiner.LATEST)));
+.withTimestampCombiner(TimestampCombiner.LATEST)),
+toProtoAndBackSpec(WindowingStrategy.of(new CustomWindowFn(;
}
@Parameter(0)
@@ -143,4 +162,142 @@ public class WindowingStrategyTranslationTest {
proto.getAssignsToOneWindow(),
equalTo(windowingStrategy.getWindowFn().assignsToOneWindow()));
}
+
+ private static class CustomWindow extends IntervalWindow {
+private boolean isBig;
+
+CustomWindow(Instant start, Instant end, boolean isBig) {
+ super(start, end);
+ this.isBig = isBig;
+}
+
+@Override
+public boolean equals(@Nullable Object o) {
+ if (this == o) {
+return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+return false;
+ }
+ CustomWindow that = (CustomWindow) o;
+ return super.equals(o) && this.isBig == that.isBig;
+}
+
+@Override
+public int hashCode() {
+ return Objects.hash(super.hashCode(), isBig);
+}
+ }
+
+ private static class CustomWindowCoder extends CustomCoder {
+
+private static final CustomWindowCoder INSTANCE = new CustomWindowCoder();
+private static final Coder INTERVAL_WINDOW_CODER =
IntervalWindow.getCoder();
+private static final VarIntCoder VAR_INT_CODER = VarIntCoder.of();
+
+public static CustomWindowCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(CustomWindow window, OutputStream outStream) throws
IOException {
+ INTERVAL_WINDOW_CODER.encode(window, outStream);
+ VAR_INT_CODER.encode(win
[beam] branch master updated: [BEAM-10925] Default to empty maps in UserFunctionDefinitions.Builder.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 894231f [BEAM-10925] Default to empty maps in
UserFunctionDefinitions.Builder.
new 7ef8bd1 Merge pull request #13628 from ibzib/ufd-builder
894231f is described below
commit 894231fc87bd4cc1a203a4753723ff9a609d2b36
Author: Kyle Weaver
AuthorDate: Mon Dec 28 11:51:22 2020 -0800
[BEAM-10925] Default to empty maps in UserFunctionDefinitions.Builder.
---
.../extensions/sql/zetasql/translation/UserFunctionDefinitions.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
index 792c31c..395ad5e 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/UserFunctionDefinitions.java
@@ -48,6 +48,8 @@ public abstract class UserFunctionDefinitions {
}
public static Builder newBuilder() {
-return new AutoValue_UserFunctionDefinitions.Builder();
+return new AutoValue_UserFunctionDefinitions.Builder()
+.setSqlScalarFunctions(ImmutableMap.of())
+.setSqlTableValuedFunctions(ImmutableMap.of());
}
}
[beam] branch master updated: [BEAM-11407] Add IT test to Bigtable for BeamSQL
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3cacd4a [BEAM-11407] Add IT test to Bigtable for BeamSQL
new 1a5505b Merge pull request #13512 from piotr-szuberski/bigtable-it
3cacd4a is described below
commit 3cacd4a1acaa5e37b326970aa47a89ee025ad1b4
Author: Piotr Szuberski
AuthorDate: Mon Dec 14 09:21:51 2020 +0100
[BEAM-11407] Add IT test to Bigtable for BeamSQL
---
.../provider/bigtable/BigtableClientWrapper.java | 115 ++
.../BigtableTableCreationFailuresTest.java | 2 +-
.../provider/bigtable/BigtableTableFlatTest.java | 110 +-
.../meta/provider/bigtable/BigtableTableIT.java| 200 +
.../meta/provider/bigtable/BigtableTableTest.java | 94
.../provider/bigtable/BigtableTableTestUtils.java | 237 +
.../bigtable/BigtableTableWithRowsTest.java| 120 +--
.../io/gcp/testing/BigtableEmulatorWrapper.java| 75 ---
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 153 -
.../beam/sdk/io/gcp/testing/BigtableUtils.java | 49 +
.../bigtable/BeamRowToBigtableMutationTest.java| 14 +-
.../gcp/bigtable/BigtableRowToBeamRowFlatTest.java | 14 +-
.../io/gcp/bigtable/BigtableRowToBeamRowTest.java | 17 +-
.../{TestUtils.java => BigtableTestUtils.java} | 50 -
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 13 +-
15 files changed, 786 insertions(+), 477 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableClientWrapper.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableClientWrapper.java
new file mode 100644
index 000..6a8b343
--- /dev/null
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableClientWrapper.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.bigtable;
+
+import static org.apache.beam.sdk.io.gcp.bigtable.RowUtils.byteString;
+import static org.apache.beam.sdk.io.gcp.bigtable.RowUtils.byteStringUtf8;
+
+import com.google.auth.Credentials;
+import com.google.bigtable.admin.v2.ColumnFamily;
+import com.google.bigtable.admin.v2.DeleteTableRequest;
+import com.google.bigtable.admin.v2.Table;
+import com.google.bigtable.v2.MutateRowRequest;
+import com.google.bigtable.v2.Mutation;
+import com.google.cloud.bigtable.config.BigtableOptions;
+import com.google.cloud.bigtable.config.CredentialOptions;
+import com.google.cloud.bigtable.grpc.BigtableDataClient;
+import com.google.cloud.bigtable.grpc.BigtableSession;
+import com.google.cloud.bigtable.grpc.BigtableTableAdminClient;
+import java.io.IOException;
+import java.io.Serializable;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+class BigtableClientWrapper implements Serializable {
+ private final BigtableTableAdminClient tableAdminClient;
+ private final BigtableDataClient dataClient;
+ private final BigtableSession session;
+ private final BigtableOptions bigtableOptions;
+
+ BigtableClientWrapper(
+ String project,
+ String instanceId,
+ @Nullable Integer emulatorPort,
+ @Nullable Credentials gcpCredentials)
+ throws IOException {
+BigtableOptions.Builder optionsBuilder =
+BigtableOptions.builder()
+.setProjectId(project)
+.setInstanceId(instanceId)
+.setUserAgent("apache-beam-test");
+if (emulatorPort != null) {
+ optionsBuilder.enableEmulator("localhost", emulatorPort);
+}
+if (gcpCredentials != null) {
+
optionsBuilder.setCredentialOptions(CredentialOptions.credential(gcpCredentials));
+}
+bigtableOptions = optionsBuilder.build();
+
+session = new BigtableSession(bigtableOptions);
+tableAdminClient = session.getTableAdminClient();
+dataClient = session.getDataClient();
[beam] branch master updated: [BEAM-11374] Add key regex filter to Bigtable for BeamSQL
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new aa1498c [BEAM-11374] Add key regex filter to Bigtable for BeamSQL
new d4e9171 Merge pull request #13495 from
piotr-szuberski/bigtable-filters
aa1498c is described below
commit aa1498c0d70da32ae730990b9d64d52eed5da084
Author: Piotr Szuberski
AuthorDate: Mon Dec 7 17:55:38 2020 +0100
[BEAM-11374] Add key regex filter to Bigtable for BeamSQL
---
.../sql/meta/provider/bigtable/BigtableFilter.java | 136 +
.../sql/meta/provider/bigtable/BigtableTable.java | 47 +--
.../meta/provider/bigtable/BigtableFilterTest.java | 114 +
.../provider/bigtable/BigtableTableFlatTest.java | 40 +-
.../apache/beam/sdk/io/gcp/bigtable/RowUtils.java | 10 ++
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 5 +-
.../dsls/sql/extensions/create-external-table.md | 4 +
7 files changed, 336 insertions(+), 20 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilter.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilter.java
new file mode 100644
index 000..c3bdda7
--- /dev/null
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilter.java
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.meta.provider.bigtable;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.beam.sdk.io.gcp.bigtable.RowUtils.KEY;
+import static org.apache.beam.sdk.io.gcp.bigtable.RowUtils.byteStringUtf8;
+import static
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlKind.LIKE;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.bigtable.v2.RowFilter;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.beam.sdk.extensions.sql.meta.BeamSqlTableFilter;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexCall;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexLiteral;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexNode;
+
+/**
+ * BigtableFilter for queries with WHERE clause.
+ *
+ * Currently only queries with a single LIKE statement by key field with https://github.com/google/re2/wiki/Syntax>RE2 Syntax regex type
are supported, e.g.
+ * `SELECT * FROM table WHERE key LIKE '^key\d'`
+ */
+class BigtableFilter implements BeamSqlTableFilter {
+ private final List supported;
+ private final List unsupported;
+ private final Schema schema;
+
+ BigtableFilter(List predicateCNF, Schema schema) {
+supported =
predicateCNF.stream().filter(BigtableFilter::isSupported).collect(toList());
+unsupported =
+predicateCNF.stream().filter(predicate ->
!isSupported(predicate)).collect(toList());
+this.schema = schema;
+ }
+
+ @Override
+ public List getNotSupported() {
+return unsupported;
+ }
+
+ @Override
+ public int numSupported() {
+return BeamSqlTableFilter.expressionsInFilter(supported);
+ }
+
+ public List getSupported() {
+return supported;
+ }
+
+ @Override
+ public String toString() {
+String supStr =
supported.stream().map(RexNode::toString).collect(Collectors.joining());
+String unsupStr =
unsupported.stream().map(RexNode::toString).collect(Collectors.joining());
+return String.format("[supported{%s}, unsupported{%s}]", supStr, unsupStr);
+ }
+
+ RowFilter getFilters() {
+checkArgument(
+supported.size() == 1,
+String.format("Only one LIKE operation is allowed. Got %s operations",
supported.size()));
+return translateRexNodeToRowFilter(
[beam] branch master updated (30f9a60 -> 9862f54)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 30f9a60 [BEAM-11383] Add runtime normalization for sdk.properties.
(#13464)
new 780ce89 [BEAM-11373] Add write to Bigtable Table
new 9862f54 Merge pull request #13452 from
piotr-szuberski/bigtable-table-write
The 29863 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.md | 2 +-
.../sql/meta/provider/bigtable/BigtableTable.java | 12 +-
.../provider/bigtable/BigtableTableFlatTest.java | 72 ++--
.../meta/provider/bigtable/BigtableTableTest.java | 49
.../bigtable/BigtableTableWithRowsTest.java| 9 +-
.../io/gcp/bigtable/BeamRowToBigtableMutation.java | 123 +
.../sdk/io/gcp/bigtable/BigtableRowToBeamRow.java | 8 +-
.../io/gcp/bigtable/BigtableRowToBeamRowFlat.java | 7 +-
.../io/gcp/bigtable/BigtableRowToBeamRowFn.java| 2 +-
.../beam/sdk/io/gcp/bigtable/CellValueParser.java | 32 ++
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 60 ++
...est.java => BeamRowToBigtableMutationTest.java} | 46 ++--
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 83 ++
.../apache/beam/sdk/io/gcp/bigtable/TestUtils.java | 26 +
.../dsls/sql/extensions/create-external-table.md | 8 +-
15 files changed, 477 insertions(+), 62 deletions(-)
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BeamRowToBigtableMutation.java
copy
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/{BigtableRowToBeamRowFlatTest.java
=> BeamRowToBigtableMutationTest.java} (57%)
[beam] branch master updated: [BEAM-11173] Add Bigtable table with read operation
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c1dfa1c [BEAM-11173] Add Bigtable table with read operation
new 18f7a32 Merge pull request #13319 from
piotr-szuberski/bigtable-provider
c1dfa1c is described below
commit c1dfa1caae8ae4a53d00aad0de64b63186fb6859
Author: Piotr Szuberski
AuthorDate: Thu Nov 12 17:37:44 2020 +0100
[BEAM-11173] Add Bigtable table with read operation
---
CHANGES.md | 1 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 1 +
sdks/java/extensions/sql/build.gradle | 1 +
.../sql/meta/provider/bigtable/BigtableTable.java | 201 +
.../provider/bigtable/BigtableTableProvider.java | 85 +
.../sql/meta/provider/bigtable/package-info.java | 20 ++
.../BigtableTableCreationFailuresTest.java | 145 +++
.../provider/bigtable/BigtableTableFlatTest.java | 113
.../meta/provider/bigtable/BigtableTableTest.java | 95 ++
.../bigtable/BigtableTableWithRowsTest.java| 157
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 15 ++
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 22 +++
.../sdk/io/gcp/bigtable/BigtableRowToBeamRow.java | 176 ++
.../io/gcp/bigtable/BigtableRowToBeamRowFlat.java | 107 +++
.../io/gcp/bigtable/BigtableRowToBeamRowFn.java| 46 +
.../beam/sdk/io/gcp/bigtable/CellValueParser.java | 74
.../apache/beam/sdk/io/gcp/bigtable/RowUtils.java | 26 +++
.../io/gcp/testing/BigtableEmulatorWrapper.java| 75
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 94 ++
.../gcp/bigtable/BigtableRowToBeamRowFlatTest.java | 65 +++
.../io/gcp/bigtable/BigtableRowToBeamRowTest.java | 93 ++
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 194
.../apache/beam/sdk/io/gcp/bigtable/TestUtils.java | 83 +
.../dsls/sql/extensions/create-external-table.md | 126 +
24 files changed, 2015 insertions(+)
diff --git a/CHANGES.md b/CHANGES.md
index 244682d..4d74aac 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -94,6 +94,7 @@
* Added support for json payload format in Beam SQL Kafka Table
([BEAM-10893](https://issues.apache.org/jira/browse/BEAM-10893))
* Added support for protobuf payload format in Beam SQL Kafka Table
([BEAM-10892](https://issues.apache.org/jira/browse/BEAM-10892))
* Added support for avro payload format in Beam SQL Pubsub Table
([BEAM-5504](https://issues.apache.org/jira/browse/BEAM-5504))
+* Added Cloud Bigtable Table with Read operation to Beam SQL
([BEAM-11173](https://issues.apache.org/jira/browse/BEAM-11173))
* Added option to disable unnecessary copying between operators in Flink
Runner (Java) ([BEAM-11146](https://issues.apache.org/jira/browse/BEAM-11146))
* Added CombineFn.setup and CombineFn.teardown to Python SDK. These methods
let you initialize the CombineFn's state before any of the other methods of the
CombineFn is executed and clean that state up later on. If you are using
Dataflow, you need to enable Dataflow Runner V2 by passing
`--experiments=use_runner_v2` before using this feature.
([BEAM-3736](https://issues.apache.org/jira/browse/BEAM-3736))
* X feature added (Java/Python)
([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 500a917..01af0aa 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -514,6 +514,7 @@ class BeamModulePlugin implements Plugin {
google_cloud_bigquery :
"com.google.cloud:google-cloud-bigquery", //
google_cloud_platform_libraries_bom sets version
google_cloud_bigquery_storage :
"com.google.cloud:google-cloud-bigquerystorage", //
google_cloud_platform_libraries_bom sets version
google_cloud_bigtable_client_core :
"com.google.cloud.bigtable:bigtable-client-core:1.16.0",
+google_cloud_bigtable_emulator :
"com.google.cloud:google-cloud-bigtable-emulator:0.125.2",
google_cloud_core :
"com.google.cloud:google-cloud-core", // google_cloud_platform_libraries_bom
sets version
google_cloud_core_grpc :
"com.google.cloud:google-cloud-core-grpc", //
google_cloud_platform_libraries_bom sets version
google_cloud_datacatalog_v1beta1:
"com.google.cloud:google-cloud-datacatalog", //
g
[beam] branch master updated (f67cb9a -> 331b36e)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f67cb9a [BEAM-5504] Change Pubsub avro table jira task number in CHANGES.md (#13248) add 331b36e [BEAM-5570] Update javacc dependency (#13094) No new revisions were added by this update. Summary of changes: sdks/java/io/clickhouse/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated: Increase Dataflow V2 validates runner timeout.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 492d64b Increase Dataflow V2 validates runner timeout.
new 7828af8 Merge pull request #13223 from tysonjh/bump-timeout-java-r2-vr
492d64b is described below
commit 492d64b3eb8c66727384e73fb1a5868ca976bf14
Author: Tyson Hamilton
AuthorDate: Thu Oct 29 17:09:17 2020 +
Increase Dataflow V2 validates runner timeout.
The suite is failing consistently with a timeout. One successful run
previously completed with 1 minute to spare (269min). Bumping by an
hour.
---
.../jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_V2.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_V2.groovy
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_V2.groovy
index 631f790..00c6048 100644
--- a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_V2.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_V2.groovy
@@ -27,7 +27,7 @@
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_VR_Dataflow_V2',
description('Runs Java ValidatesRunner suite on the Dataflow runner V2.')
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 330)
// Publish all test results to Jenkins
publishers {
[beam] branch master updated (dab6f90 -> 126554d)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dab6f90 [BEAM-10921] Fix flaky unit tests on Windows (#12866) add 126554d [BEAM-9543] Add blog post for MATCH_RECOGNIZE (#12735) No new revisions were added by this update. Summary of changes: .../site/content/en/blog/pattern-match-beam-sql.md | 167 + 1 file changed, 167 insertions(+) create mode 100644 website/www/site/content/en/blog/pattern-match-beam-sql.md
[beam] branch master updated (d52f545 -> 43b5c5c)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from d52f545 Merge pull request #12803 from ibzib/BEAM-10762
add 43b5c5c [Beam-9543] support MATCH_RECOGNIZE with NFA (#12532)
No new revisions were added by this update.
Summary of changes:
.../beam/sdk/extensions/sql/impl/cep/CEPKind.java | 5 +
.../sdk/extensions/sql/impl/cep/CEPLiteral.java| 197 -
.../sdk/extensions/sql/impl/cep/CEPMeasure.java| 4 +-
.../sdk/extensions/sql/impl/cep/CEPOperator.java | 5 +
.../sdk/extensions/sql/impl/cep/CEPPattern.java| 83 +--
.../sql/impl/cep/{CEPUtil.java => CEPUtils.java} | 56 +-
.../sdk/extensions/sql/impl/cep/Quantifier.java| 22 +-
.../beam/sdk/extensions/sql/impl/nfa/NFA.java | 815 +
.../sdk/extensions/sql/impl/nfa}/package-info.java | 4 +-
.../sdk/extensions/sql/impl/rel/BeamMatchRel.java | 397 +-
.../extensions/sql/impl/rel/BeamMatchRelTest.java | 40 +-
11 files changed, 1258 insertions(+), 370 deletions(-)
rename
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/{CEPUtil.java
=> CEPUtils.java} (81%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/nfa/NFA.java
copy
{runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/resources
=>
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/nfa}/package-info.java
(85%)
[beam] branch master updated: Fix BEAM-10661: Java quickstart using Gradle doesn't work
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9351996 Fix BEAM-10661: Java quickstart using Gradle doesn't work
new ba89e76 Merge pull request #12778 from SergiyKolesnikov/BEAM-10661
9351996 is described below
commit 93519964a029d9d0f72ef7dc100ae51abf918a3f
Author: Sergiy Kolesnikov
AuthorDate: Sun Sep 6 18:41:40 2020 +0200
Fix BEAM-10661: Java quickstart using Gradle doesn't work
getPropety() will return an empty string if "exec.args" property is not
set. This will prevent the NullPointerException.
---
website/www/site/content/en/get-started/quickstart-java.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/www/site/content/en/get-started/quickstart-java.md
b/website/www/site/content/en/get-started/quickstart-java.md
index 364e32f..4e68bf0 100644
--- a/website/www/site/content/en/get-started/quickstart-java.md
+++ b/website/www/site/content/en/get-started/quickstart-java.md
@@ -132,7 +132,7 @@ task execute (type:JavaExec) {
main = System.getProperty("mainClass")
classpath = sourceSets.main.runtimeClasspath
systemProperties System.getProperties()
-args System.getProperty("exec.args").split()
+args System.getProperty("exec.args", "").split()
}
{{< /highlight >}}
1. Rebuild your project by running:
[beam] branch master updated (b9bb2a4 -> 32b67ec)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b9bb2a4 [BEAM-9980] Configure Python versions for Dataflow test suite tasks via settings.gradle (#12239) add 32b67ec [BEAM-9891] Generate query execution summary table after finishing jobs (#12601) No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 67 +++-- .../apache/beam/sdk/tpcds/SqlTransformRunner.java | 74 +++--- .../apache/beam/sdk/tpcds/SummaryGenerator.java| 153 + .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 26 +++- .../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 92 + 5 files changed, 378 insertions(+), 34 deletions(-) create mode 100644 sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SummaryGenerator.java create mode 100644 sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/TpcdsRunResult.java
[beam] branch master updated (af1ce64 -> e44c3c2)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from af1ce64 Merge pull request #12507 from robinyqiu/refactor
add e44c3c2 [BEAM-9891] Added ZetaSQL planner support and uploaded 100G
data (#12502)
No new revisions were added by this update.
Summary of changes:
sdks/java/testing/tpcds/build.gradle | 1 +
.../{BeamTpcds.java => BeamSqlEnvRunner.java} | 64 +-
.../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 114 +---
.../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 58 ++
.../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 52 ++
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 138 +
.../beam/sdk/tpcds/TpcdsParametersReader.java | 3 +-
.../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 672 +
.../tpcds/src/main/resources/queries/query96.sql | 9 +-
.../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 124
10 files changed, 1116 insertions(+), 119 deletions(-)
copy
sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/{BeamTpcds.java
=> BeamSqlEnvRunner.java} (71%)
create mode 100644
sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
create mode 100644
sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
create mode 100644
sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
create mode 100644
sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/TpcdsSchemas.java
create mode 100644
sdks/java/testing/tpcds/src/test/java/org/apache/beam/sdk/tpcds/TpcdsSchemasTest.java
[beam] branch master updated: [BEAM-10633] UdfImpl should be able to return java.util.List.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ab1c53f [BEAM-10633] UdfImpl should be able to return java.util.List.
new aa6e64c Merge pull request #12461 from
amaliujia/rw-support_array_as_return_type
ab1c53f is described below
commit ab1c53fa878f10b72f73e463064f651ea01cee42
Author: amaliujia
AuthorDate: Mon Aug 3 20:17:34 2020 -0700
[BEAM-10633] UdfImpl should be able to return java.util.List.
---
.../extensions/sql/impl/ScalarFunctionImpl.java| 2 +-
.../extensions/sql/impl/utils/CalciteUtils.java| 23 +
.../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 39 ++
3 files changed, 56 insertions(+), 8 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
index 3ef4d9f..b4a9d7e 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFunctionImpl.java
@@ -124,7 +124,7 @@ public class ScalarFunctionImpl extends
UdfImplReflectiveFunctionBase
@Override
public RelDataType getReturnType(RelDataTypeFactory typeFactory) {
-return CalciteUtils.sqlTypeWithAutoCast(typeFactory,
method.getReturnType());
+return CalciteUtils.sqlTypeWithAutoCast(typeFactory,
method.getGenericReturnType());
}
@Override
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
index acb4ee1..d7016cb 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.utils;
+import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
import java.util.Date;
import java.util.Map;
@@ -283,18 +284,26 @@ public class CalciteUtils {
/**
* SQL-Java type mapping, with specified Beam rules:
- * 1. redirect {@link AbstractInstant} to {@link Date} so Calcite can
recognize it.
+ * 1. redirect {@link AbstractInstant} to {@link Date} so Calcite can
recognize it.
+ * 2. For a list, the component type is needed to create a Sql array type.
*
- * @param rawType
- * @return
+ * @param type
+ * @return Calcite RelDataType
*/
- public static RelDataType sqlTypeWithAutoCast(RelDataTypeFactory
typeFactory, Type rawType) {
+ public static RelDataType sqlTypeWithAutoCast(RelDataTypeFactory
typeFactory, Type type) {
// For Joda time types, return SQL type for java.util.Date.
-if (rawType instanceof Class &&
AbstractInstant.class.isAssignableFrom((Class) rawType)) {
+if (type instanceof Class &&
AbstractInstant.class.isAssignableFrom((Class) type)) {
return typeFactory.createJavaType(Date.class);
-} else if (rawType instanceof Class &&
ByteString.class.isAssignableFrom((Class) rawType)) {
+} else if (type instanceof Class &&
ByteString.class.isAssignableFrom((Class) type)) {
return typeFactory.createJavaType(byte[].class);
+} else if (type instanceof ParameterizedType
+&& java.util.List.class.isAssignableFrom(
+(Class) ((ParameterizedType) type).getRawType())) {
+ ParameterizedType parameterizedType = (ParameterizedType) type;
+ Class genericType = (Class)
parameterizedType.getActualTypeArguments()[0];
+ RelDataType collectionElementType =
typeFactory.createJavaType(genericType);
+ return typeFactory.createArrayType(collectionElementType,
UNLIMITED_ARRAY_SIZE);
}
-return typeFactory.createJavaType((Class) rawType);
+return typeFactory.createJavaType((Class) type);
}
}
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
index 75e8a08..c2afc5d 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java
@@ -23,6 +23,7 @@ import static
org.junit.internal.matchers.ThrowableMessageMatcher.hasMessage;
import com.google.auto.service.AutoService;
import java.sql.Timestamp;
+import java.util.Arrays;
import java.util.Map;
im
[beam] branch master updated: Fix add field method in SQL walkthrough
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7620e6c Fix add field method in SQL walkthrough
new 51c95d8 Merge pull request #12463 from iht/fix_sql_doc
7620e6c is described below
commit 7620e6cecc945c1ca2d89784bab45058a8151a5f
Author: Israel Herraiz
AuthorDate: Tue Aug 4 18:17:36 2020 +0200
Fix add field method in SQL walkthrough
---
website/www/site/content/en/documentation/dsls/sql/walkthrough.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
b/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
index a02e14c..59ecb82 100644
--- a/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
+++ b/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
@@ -129,7 +129,7 @@ to either a single `PCollection` or a `PCollectionTuple`
which holds multiple
.builder()
.addInt32Field("appId")
.addInt32Field("reviewerId")
- .withFloatField("rating")
+ .addFloatField("rating")
.addDateTimeField("rowtime")
.build();
[beam] branch master updated (05e5c72 -> c2369bd)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 05e5c72 Fix dictionary changes size error in pickler.py (#12458) add c2369bd [BEAM-9891] TPC-DS module initialization, tables and queries stored (#12436) No new revisions were added by this update. Summary of changes: .../extensions/sql/impl/schema/BeamTableUtils.java | 6 + sdks/java/testing/tpcds/build.gradle | 52 + .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 129 .../org/apache/beam/sdk/tpcds/QueryReader.java | 59 ++ .../beam/sdk/tpcds/TableSchemaJSONLoader.java | 112 ++ .../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 42 .../beam/sdk/tpcds/TpcdsOptionsRegistrar.java | 33 +++ .../beam/sdk/tpcds/TpcdsParametersReader.java | 108 ++ .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 40 .../tpcds/src/main/resources/queries/query1.sql| 38 .../tpcds/src/main/resources/queries/query10.sql | 72 +++ .../tpcds/src/main/resources/queries/query11.sql | 94 + .../tpcds/src/main/resources/queries/query12.sql | 47 + .../tpcds/src/main/resources/queries/query13.sql | 64 ++ .../tpcds/src/main/resources/queries/query14.sql | 223 .../tpcds/src/main/resources/queries/query15.sql | 33 +++ .../tpcds/src/main/resources/queries/query16.sql | 44 .../tpcds/src/main/resources/queries/query17.sql | 58 + .../tpcds/src/main/resources/queries/query18.sql | 47 + .../tpcds/src/main/resources/queries/query19.sql | 38 .../tpcds/src/main/resources/queries/query2.sql| 73 +++ .../tpcds/src/main/resources/queries/query20.sql | 43 .../tpcds/src/main/resources/queries/query21.sql | 43 .../tpcds/src/main/resources/queries/query22.sql | 33 +++ .../tpcds/src/main/resources/queries/query23.sql | 120 +++ .../tpcds/src/main/resources/queries/query24.sql | 119 +++ .../tpcds/src/main/resources/queries/query25.sql | 61 ++ .../tpcds/src/main/resources/queries/query26.sql | 34 +++ .../tpcds/src/main/resources/queries/query27.sql | 36 .../tpcds/src/main/resources/queries/query28.sql | 66 ++ .../tpcds/src/main/resources/queries/query29.sql | 60 ++ .../tpcds/src/main/resources/queries/query3.sql| 34 +++ .../tpcds/src/main/resources/queries/query30.sql | 44 .../tpcds/src/main/resources/queries/query31.sql | 65 ++ .../tpcds/src/main/resources/queries/query32.sql | 41 .../tpcds/src/main/resources/queries/query33.sql | 88 .../tpcds/src/main/resources/queries/query34.sql | 44 .../tpcds/src/main/resources/queries/query35.sql | 71 +++ .../tpcds/src/main/resources/queries/query36.sql | 43 .../tpcds/src/main/resources/queries/query37.sql | 30 +++ .../tpcds/src/main/resources/queries/query38.sql | 36 .../tpcds/src/main/resources/queries/query39.sql | 66 ++ .../tpcds/src/main/resources/queries/query4.sql| 129 .../tpcds/src/main/resources/queries/query40.sql | 41 .../tpcds/src/main/resources/queries/query41.sql | 65 ++ .../tpcds/src/main/resources/queries/query42.sql | 35 .../tpcds/src/main/resources/queries/query43.sql | 32 +++ .../tpcds/src/main/resources/queries/query44.sql | 48 + .../tpcds/src/main/resources/queries/query45.sql | 33 +++ .../tpcds/src/main/resources/queries/query46.sql | 48 + .../tpcds/src/main/resources/queries/query47.sql | 64 ++ .../tpcds/src/main/resources/queries/query48.sql | 79 +++ .../tpcds/src/main/resources/queries/query49.sql | 142 + .../tpcds/src/main/resources/queries/query5.sql| 141 + .../tpcds/src/main/resources/queries/query50.sql | 72 +++ .../tpcds/src/main/resources/queries/query51.sql | 58 + .../tpcds/src/main/resources/queries/query52.sql | 35 .../tpcds/src/main/resources/queries/query53.sql | 41 .../tpcds/src/main/resources/queries/query54.sql | 69 ++ .../tpcds/src/main/resources/queries/query55.sql | 27 +++ .../tpcds/src/main/resources/queries/query56.sql | 82 .../tpcds/src/main/resources/queries/query57.sql | 61 ++ .../tpcds/src/main/resources/queries/query58.sql | 78 +++ .../tpcds/src/main/resources/queries/query59.sql | 57 + .../tpcds/src/main/resources/queries/query6.sql| 39 .../tpcds/src/main/resources/queries/query60.sql | 91 .../tpcds/src/main/resources/queries/query61.sql | 57 + .../tpcds/src/main/resources/queries/query62.sql | 48 + .../tpcds/src/main/resources/queries/query63.sql | 42 .../tpcds/src/main/resources/queries/query64.sql | 134 .../tpcds/src/main/resources/queries
[beam] branch master updated (54fe9f6 -> 5833667)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 54fe9f6 Fix link for S3FileSystem (#12450)
new 4e56953 [BEAM-9543] built the basis for Match_Recog
new 72232fc [BEAM-9543] built the basis for Match_Recog
new 064ada7 [BEAM-9543] implemented `partition by`
new 9cd1a82 [BEAM-9543] implemented `order by`
new c07b8a8 [BEAM-9543] fixed `order by` coder issue
new cdb7e9f [BEAM-9543] fixed `order by` coder issue
new cc63e55 [BEAM-9543] applied regex pattern match
new b2b189d [BEAM-9543] applied regex pattern match
new 08abbab [BEAM-9543] fixed sortKey serialization problem
new 03a33c6 [BEAM-9543] fixed sortKey serialization problem
new ec7c929 [BEAM-9543] fixed serialization problem
new f52d96f [BEAM-9543] recognized simple pattern
new 8d6ffcc [BEAM-9543] recognized simple pattern
new a7d111f [BEAM-9543] fixed code style
new f529b87 [BEAM-9543] supported regex quantifier
new 0bf24db [BEAM-9543] added javadoc
new 422cbe2 [BEAM-9543] removed CEPTypeName.java
new adc2354 [BEAM-9543] added Measures implementation (unfinished)
new ebc41a2 [BEAM-9543] added Measures implementation
new 8793574 [BEAM-9543] added Measures implementation
new 040d1f4 [BEAM-9543] fixed minor issues
new 799491b [BEAM-9543] fixed minor style issues
new 5833667 Merge pull request #12232 from
Mark-Zeng/BEAM-9543-patternRecog
The 28207 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../beam/sdk/extensions/sql/impl/cep/CEPCall.java | 83
.../CEPFieldRef.java} | 35 +-
.../beam/sdk/extensions/sql/impl/cep/CEPKind.java} | 22 +-
.../sdk/extensions/sql/impl/cep/CEPLiteral.java| 195
.../sdk/extensions/sql/impl/cep/CEPMeasure.java| 57 +++
.../sdk/extensions/sql/impl/cep/CEPOperation.java | 48 ++
.../sdk/extensions/sql/impl/cep/CEPOperator.java | 65 +++
.../sdk/extensions/sql/impl/cep/CEPPattern.java| 172 +++
.../beam/sdk/extensions/sql/impl/cep/CEPUtil.java | 249 ++
.../beam/sdk/extensions/sql/impl/cep/OrderKey.java | 74 +++
.../extensions/sql/impl/cep/PatternCondition.java} | 19 +-
.../sdk/extensions/sql/impl/cep/Quantifier.java} | 33 +-
.../sdk/extensions/sql/impl/cep}/package-info.java | 9 +-
.../extensions/sql/impl/planner/BeamRuleSets.java | 4 +-
.../sdk/extensions/sql/impl/rel/BeamMatchRel.java | 529 +
.../rule/{BeamSortRule.java => BeamMatchRule.java} | 41 +-
.../extensions/sql/impl/rel/BeamMatchRelTest.java | 264 ++
17 files changed, 1844 insertions(+), 55 deletions(-)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPCall.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{utils/SerializableRexInputRef.java
=> cep/CEPFieldRef.java} (53%)
copy
sdks/java/{io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/CheckpointGenerator.java
=>
extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPKind.java}
(70%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPLiteral.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPMeasure.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPOperation.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPOperator.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPPattern.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/CEPUtil.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/OrderKey.java
copy
sdks/java/{core/src/main/java/org/apache/beam/sdk/schemas/utils/RowSelector.java
=>
extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/PatternCondition.java}
(66%)
copy
sdks/java/{io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaTimestampType.java
=>
extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep/Quantifier.java}
(51%)
copy {runners/direct-java/src/main/java/org/apache/beam/runners/direct =>
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/cep}/package-info.java
(71%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/a
[beam] branch master updated: remove redundent precommits.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b6ac20d remove redundent precommits.
new 4885e20 Merge pull request #12417 from amaliujia/rw-remove-precommits
b6ac20d is described below
commit b6ac20d9ad579a55a9e9bd2dfc54ffcae3137c26
Author: amaliujia
AuthorDate: Wed Jul 29 12:00:31 2020 -0700
remove redundent precommits.
---
.../jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy | 36 ---
.../job_PreCommit_BeamSQL_ZetaSQL_Java11.groovy| 41 --
2 files changed, 77 deletions(-)
diff --git a/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
b/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
deleted file mode 100644
index 2aa89c2..000
--- a/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import PrecommitJobBuilder
-
-PrecommitJobBuilder builder = new PrecommitJobBuilder(
-scope: this,
-nameBase: 'JavaBeamZetaSQL',
-gradleTask: ':javaPreCommitBeamZetaSQL',
-gradleSwitches: [
- '-PdisableSpotlessCheck=true'
-], // spotless checked in separate pre-commit
-triggerPathPatterns: [
- '^sdks/java/extensions/sql/.*$',
-]
-)
-builder.build {
- publishers {
-archiveJunit('**/build/test-results/**/*.xml')
- }
-}
diff --git a/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL_Java11.groovy
b/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL_Java11.groovy
deleted file mode 100644
index 7efe77d..000
--- a/.test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL_Java11.groovy
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import PrecommitJobBuilder
-import CommonJobProperties as properties
-
-PrecommitJobBuilder builder = new PrecommitJobBuilder(
-scope: this,
-nameBase: 'JavaBeamZetaSQLJava11',
-gradleTask: ':javaPreCommitBeamZetaSQL',
-gradleSwitches: [
- '-PdisableSpotlessCheck=true',
- '-PcompileAndRunTestsWithJava11',
- '-PskipCheckerFramework',
- // Gradle itself is running under JDK8 so plugin configures wrong for
JDK11
- "-Pjava11Home=${CommonJobProperties.JAVA_11_HOME}"
-], // spotless checked in separate pre-commit
-triggerPathPatterns: [
- '^sdks/java/extensions/sql/.*$',
-]
-)
-builder.build {
- publishers {
-archiveJunit('**/build/test-results/**/*.xml')
- }
-}
[beam] branch master updated (88acc52 -> 890bedc)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 88acc52 [BEAM-8543] Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (#11924) add 890bedc [BEAM-10595] Remove rules that not work well with Java UDF (#12400) No new revisions were added by this update. Summary of changes: .../beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java | 10 ++ 1 file changed, 10 insertions(+)
[beam] branch master updated: Implement Numbering functions (#12375)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8d460db Implement Numbering functions (#12375)
8d460db is described below
commit 8d460db620d2ff1257b0e092218294df15b409a1
Author: jhnmora000 <48849676+jhnmora...@users.noreply.github.com>
AuthorDate: Sun Jul 26 14:50:00 2020 -0500
Implement Numbering functions (#12375)
RANK, DENSE_RANK, PERCENT_RANK, ROW_NUMBER
---
.../sdk/extensions/sql/impl/rel/BeamWindowRel.java | 23 ++-
.../transform/BeamBuiltinAnalyticFunctions.java| 183 -
.../transform/agg/AggregationCombineFnAdapter.java | 8 +-
.../extensions/sql/BeamAnalyticFunctionsTest.java | 115 +
4 files changed, 318 insertions(+), 11 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
index 09ca5b1..5d5da8d 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
@@ -28,6 +28,7 @@ import org.apache.beam.sdk.coders.IterableCoder;
import org.apache.beam.sdk.coders.ListCoder;
import org.apache.beam.sdk.extensions.sql.impl.planner.BeamCostModel;
import org.apache.beam.sdk.extensions.sql.impl.planner.NodeStats;
+import
org.apache.beam.sdk.extensions.sql.impl.transform.BeamBuiltinAnalyticFunctions;
import
org.apache.beam.sdk.extensions.sql.impl.transform.agg.AggregationCombineFnAdapter;
import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils;
import org.apache.beam.sdk.schemas.Schema;
@@ -291,10 +292,26 @@ public class BeamWindowRel extends Window implements
BeamRelNode {
aggRange = getRange(indexRange, sortedRowsAsList.get(idx));
}
Object accumulator = fieldAgg.combineFn.createAccumulator();
- final int aggFieldIndex = fieldAgg.inputFields.get(0);
+ // if not inputs are needed, put a mock Field index
+ final int aggFieldIndex =
+ fieldAgg.inputFields.isEmpty() ? -1 :
fieldAgg.inputFields.get(0);
+ long count = 0;
for (Row aggRow : aggRange) {
-accumulator =
-fieldAgg.combineFn.addInput(accumulator,
aggRow.getBaseValue(aggFieldIndex));
+if (fieldAgg.combineFn instanceof
BeamBuiltinAnalyticFunctions.PositionAwareCombineFn) {
+ BeamBuiltinAnalyticFunctions.PositionAwareCombineFn fn =
+ (BeamBuiltinAnalyticFunctions.PositionAwareCombineFn)
fieldAgg.combineFn;
+ accumulator =
+ fn.addInput(
+ accumulator,
+ getOrderByValue(aggRow),
+ count,
+ (long) idx,
+ (long) sortedRowsAsList.size());
+} else {
+ accumulator =
+ fieldAgg.combineFn.addInput(accumulator,
aggRow.getBaseValue(aggFieldIndex));
+}
+count++;
}
Object result = fieldAgg.combineFn.extractOutput(accumulator);
Row processingRow = sortedRowsAsList.get(idx);
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
index 14fe20d..457e0db 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
@@ -17,11 +17,13 @@
*/
package org.apache.beam.sdk.extensions.sql.impl.transform;
+import java.math.BigDecimal;
import java.util.Map;
import java.util.Optional;
import java.util.function.Function;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.values.KV;
import
org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
/** Built-in Analytic Functions for the aggregation analytics functionality. */
@@ -29,10 +31,16 @@ public class BeamBuiltinAnalyticFunctions {
public static final Map>>
BUILTIN_ANALYTIC_FACTORIES =
ImmutableMap.>>builder()
+ // Aggregate Analytic Functions
.putAll(BeamBuiltinAggregations.BUILTIN_AGGREGATOR_FACTORIES)
+ // Navigation Functions
.put("FIRST_VALUE", typeName -> navigationFirstValue())
[beam] branch master updated: [BEAM-10551] Implement Navigation Functions FIRST_VALUE and LAST_VALUE (#12313)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8ea176d [BEAM-10551] Implement Navigation Functions FIRST_VALUE and
LAST_VALUE (#12313)
8ea176d is described below
commit 8ea176d23d8d13d2f9becb107178c36ef5269062
Author: jhnmora000 <48849676+jhnmora...@users.noreply.github.com>
AuthorDate: Wed Jul 22 00:48:02 2020 -0500
[BEAM-10551] Implement Navigation Functions FIRST_VALUE and LAST_VALUE
(#12313)
* Implement navigation functions
- FIRST_VALUE
- LAST_VALUE
Since, navigation functions are executed sequentially
the mergeAccumulators implementation was replaced with
an UnsupportedOperationException
---
.../sdk/extensions/sql/impl/rel/BeamWindowRel.java | 3 +-
.../transform/BeamBuiltinAnalyticFunctions.java| 65 +++-
.../extensions/sql/BeamAnalyticFunctionsTest.java | 113 +
3 files changed, 179 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
index 9d29dd4..09ca5b1 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
@@ -293,7 +293,8 @@ public class BeamWindowRel extends Window implements
BeamRelNode {
Object accumulator = fieldAgg.combineFn.createAccumulator();
final int aggFieldIndex = fieldAgg.inputFields.get(0);
for (Row aggRow : aggRange) {
-fieldAgg.combineFn.addInput(accumulator,
aggRow.getBaseValue(aggFieldIndex));
+accumulator =
+fieldAgg.combineFn.addInput(accumulator,
aggRow.getBaseValue(aggFieldIndex));
}
Object result = fieldAgg.combineFn.extractOutput(accumulator);
Row processingRow = sortedRowsAsList.get(idx);
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
index 086b0f2..14fe20d 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.extensions.sql.impl.transform;
import java.util.Map;
+import java.util.Optional;
import java.util.function.Function;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.Combine;
@@ -29,7 +30,8 @@ public class BeamBuiltinAnalyticFunctions {
BUILTIN_ANALYTIC_FACTORIES =
ImmutableMap.>>builder()
.putAll(BeamBuiltinAggregations.BUILTIN_AGGREGATOR_FACTORIES)
- // Pending Navigation functions
+ .put("FIRST_VALUE", typeName -> navigationFirstValue())
+ .put("LAST_VALUE", typeName -> navigationLastValue())
// Pending Numbering functions
.build();
@@ -42,4 +44,65 @@ public class BeamBuiltinAnalyticFunctions {
throw new UnsupportedOperationException(
String.format("Analytics Function [%s] is not supported",
functionName));
}
+
+ public static Combine.CombineFn navigationFirstValue() {
+return new FirstValueCombineFn();
+ }
+
+ public static Combine.CombineFn navigationLastValue() {
+return new LastValueCombineFn();
+ }
+
+ private static class FirstValueCombineFn extends Combine.CombineFn, T> {
+private FirstValueCombineFn() {}
+
+@Override
+public Optional createAccumulator() {
+ return Optional.empty();
+}
+
+@Override
+public Optional addInput(Optional accumulator, T input) {
+ Optional r = accumulator;
+ if (!accumulator.isPresent()) {
+r = Optional.of(input);
+ }
+ return r;
+}
+
+@Override
+public Optional mergeAccumulators(Iterable> accumulators) {
+ throw new UnsupportedOperationException();
+}
+
+@Override
+public T extractOutput(Optional accumulator) {
+ return accumulator.isPresent() ? accumulator.get() : null;
+}
+ }
+
+ private static class LastValueCombineFn extends Combine.CombineFn, T> {
+private LastValueCombineFn() {}
+
+@Override
+public Optional createAccumulator() {
+ return Optional.empty();
+}
+
+@Override
+public Optional addInput(Optional accumulator, T input)
[beam] branch master updated (613edd3 -> b80dc54)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 613edd3 Make tracking/cleanup of cache and in-environment states per
pipeline (#12249)
add b80dc54 [BEAM-9198] BeamSQL aggregation analytics functionality
(#11975)
No new revisions were added by this update.
Summary of changes:
.../apache/beam/sdk/schemas/transforms/Group.java | 2 +-
.../extensions/sql/impl/planner/BeamRuleSets.java | 5 +
.../sdk/extensions/sql/impl/rel/BeamSortRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamWindowRel.java | 417 +
.../sdk/extensions/sql/impl/rule/BeamCalcRule.java | 23 +-
.../{BeamValuesRule.java => BeamWindowRule.java} | 32 +-
.../transform/BeamBuiltinAnalyticFunctions.java| 45 +++
.../transform/agg/AggregationCombineFnAdapter.java | 24 ++
.../extensions/sql/BeamAnalyticFunctionsTest.java | 363 ++
9 files changed, 895 insertions(+), 18 deletions(-)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamWindowRel.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/{BeamValuesRule.java
=> BeamWindowRule.java} (64%)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAnalyticFunctions.java
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamAnalyticFunctionsTest.java
[beam] branch master updated (21959c9 -> 5b66382)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 21959c9 [BEAM-10283] Add new overloads of withKeyRanges and
withRowFilter met… (#12089)
add 5b66382 [BEAM-9953[SQL][ZetaSQL] Support Pure SQL user-defined
table-valued function. (#12169)
No new revisions were added by this update.
Summary of changes:
...qlUserDefinedSQLNativeTableValuedFunction.java} | 48 +++---
.../sql/impl/rel/BeamTableFunctionScanRel.java | 22 +--
.../sql/impl/utils/TVFStreamingUtils.java | 5 --
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java| 34 +++---
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 12 +++-
.../sql/zetasql/translation/ConversionContext.java | 51 ++-
.../zetasql/translation/ExpressionConverter.java | 45 -
.../zetasql/translation/FilterScanConverter.java | 3 +-
.../sql/zetasql/translation/TVFScanConverter.java | 76 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 52 +++
10 files changed, 267 insertions(+), 81 deletions(-)
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/{planner/BeamJavaTypeFactory.java
=> ZetaSqlUserDefinedSQLNativeTableValuedFunction.java} (55%)
[beam] branch master updated: Update walkthrough.md
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fae9067 Update walkthrough.md
new cf6bb3d Merge pull request #12052 from dhodun/patch-1
fae9067 is described below
commit fae906762d7d5c3f7741bc526075ac59b3a0b9da
Author: dhodun
AuthorDate: Mon Jun 22 14:39:11 2020 -0400
Update walkthrough.md
Fairly sure .setRowSchema() is required in this case.
---
website/www/site/content/en/documentation/dsls/sql/walkthrough.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
b/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
index 3733324..a02e14c 100644
--- a/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
+++ b/website/www/site/content/en/documentation/dsls/sql/walkthrough.md
@@ -97,7 +97,7 @@ A `PCollection` can be obtained multiple ways, for
example:
// Output the Row representing the current POJO
c.output(appRow);
}
- }));
+ })).setRowSchema(appSchema);
{{< /highlight >}}
- **As an output of another `SqlTransform`**. Details in the next section.
[beam] branch master updated (51bd3a4 -> 354b97a)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 51bd3a4 [BEAM-10298] beam-linkage-check.sh should not swallow errors (#12051) add 354b97a [BEAM-9890] Try fixing BIT_AND(NULL) issue (#12110) No new revisions were added by this update. Summary of changes: .../impl/transform/BeamBuiltinAggregations.java| 133 +++ .../extensions/sql/BeamSqlDslAggregationTest.java | 180 + .../sql/zetasql/SqlStdOperatorMappingTable.java| 7 +- .../sql/zetasql/ZetaSQLDialectSpecTest.java| 1 + 4 files changed, 258 insertions(+), 63 deletions(-)
[beam] 01/01: Revert "[BEAM-9890] Support BIT_AND aggregation function in Beam SQL"
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch revert-12079-master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit efe93afbc10084740b5066b93befd8481dc8dad0
Author: Rui Wang
AuthorDate: Mon Jun 29 13:52:19 2020 -0700
Revert "[BEAM-9890] Support BIT_AND aggregation function in Beam SQL"
---
.../impl/transform/BeamBuiltinAggregations.java| 35 --
.../extensions/sql/BeamSqlDslAggregationTest.java | 32
.../sql/zetasql/SqlStdOperatorMappingTable.java| 3 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 17 ---
4 files changed, 1 insertion(+), 86 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
index ab3786b..347fdc12 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
@@ -58,7 +58,6 @@ public class BeamBuiltinAggregations {
.put("$SUM0", BeamBuiltinAggregations::createSum)
.put("AVG", BeamBuiltinAggregations::createAvg)
.put("BIT_OR", BeamBuiltinAggregations::createBitOr)
- .put("BIT_AND", BeamBuiltinAggregations::createBitAnd)
.put("VAR_POP", t -> VarianceFn.newPopulation(t.getTypeName()))
.put("VAR_SAMP", t -> VarianceFn.newSample(t.getTypeName()))
.put("COVAR_POP", t ->
CovarianceFn.newPopulation(t.getTypeName()))
@@ -186,14 +185,6 @@ public class BeamBuiltinAggregations {
String.format("[%s] is not supported in BIT_OR", fieldType));
}
- static CombineFn createBitAnd(Schema.FieldType fieldType) {
-if (fieldType.getTypeName() == TypeName.INT64) {
- return new BitAnd();
-}
-throw new UnsupportedOperationException(
-String.format("[%s] is not supported in BIT_AND", fieldType));
- }
-
static class CustMax> extends
Combine.BinaryCombineFn {
@Override
public T apply(T left, T right) {
@@ -392,30 +383,4 @@ public class BeamBuiltinAggregations {
return accum;
}
}
-
- static class BitAnd extends CombineFn {
-@Override
-public Long createAccumulator() {
- return -1L;
-}
-
-@Override
-public Long addInput(Long accum, T input) {
- return accum & input.longValue();
-}
-
-@Override
-public Long mergeAccumulators(Iterable accums) {
- Long merged = createAccumulator();
- for (long accum : accums) {
-merged = merged & accum;
- }
- return merged;
-}
-
-@Override
-public Long extractOutput(Long accum) {
- return accum;
-}
- }
}
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
index 9c365b2..40b3b63 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
@@ -314,39 +314,7 @@ public class BeamSqlDslAggregationTest extends
BeamSqlDslBase {
PCollection inputRows =
pipeline.apply("longVals",
Create.of(rowsInTableA).withRowSchema(schemaInTableA));
PCollection result = inputRows.apply("sql", SqlTransform.query(sql));
-
-PAssert.that(result).containsInAnyOrder(rowResult);
-
-pipeline.run().waitUntilFinish();
- }
-
- @Test
- public void testBitAndFunction() throws Exception {
-pipeline.enableAbandonedNodeEnforcement(false);
-
-Schema schemaInTableA =
-
Schema.builder().addInt64Field("f_long").addInt32Field("f_int2").build();
-
-Schema resultType = Schema.builder().addInt64Field("finalAnswer").build();
-
-List rowsInTableA =
-TestUtils.RowsBuilder.of(schemaInTableA)
-.addRows(
-0xF001L, 0,
-0x00A1L, 0)
-.getRows();
-
-String sql = "SELECT bit_and(f_long) as bitand " + "FROM PCOLLECTION GROUP
BY f_int2";
-
-Row rowResult = Row.withSchema(resultType).addValues(1L).build();
-
-PCollection inputRows =
-pipeline.apply("longVals",
Create.of(rowsInTableA).withRowSchema(schemaInTableA));
-PCollection result = inputRows.apply("sql", SqlTransform.query(sql));
-
PAssert.that(result).containsInAnyOrder(r
[beam] branch revert-12079-master created (now efe93af)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch revert-12079-master in repository https://gitbox.apache.org/repos/asf/beam.git. at efe93af Revert "[BEAM-9890] Support BIT_AND aggregation function in Beam SQL" This branch includes the following new commits: new efe93af Revert "[BEAM-9890] Support BIT_AND aggregation function in Beam SQL" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated: [BEAM-9890] Support BIT_AND aggregation function in Beam SQL and added tests
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d41dd50 [BEAM-9890] Support BIT_AND aggregation function in Beam SQL
and added tests
new 18ce248 Merge pull request #12079 from Imfuyuwei/master
d41dd50 is described below
commit d41dd5018f90bcac90079f68624710c811f05e50
Author: Yuwei Fu
AuthorDate: Wed Jun 24 06:18:45 2020 +
[BEAM-9890] Support BIT_AND aggregation function in Beam SQL and added tests
---
.../impl/transform/BeamBuiltinAggregations.java| 35 ++
.../extensions/sql/BeamSqlDslAggregationTest.java | 32
.../sql/zetasql/SqlStdOperatorMappingTable.java| 3 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 17 +++
4 files changed, 86 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
index 347fdc12..ab3786b 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
@@ -58,6 +58,7 @@ public class BeamBuiltinAggregations {
.put("$SUM0", BeamBuiltinAggregations::createSum)
.put("AVG", BeamBuiltinAggregations::createAvg)
.put("BIT_OR", BeamBuiltinAggregations::createBitOr)
+ .put("BIT_AND", BeamBuiltinAggregations::createBitAnd)
.put("VAR_POP", t -> VarianceFn.newPopulation(t.getTypeName()))
.put("VAR_SAMP", t -> VarianceFn.newSample(t.getTypeName()))
.put("COVAR_POP", t ->
CovarianceFn.newPopulation(t.getTypeName()))
@@ -185,6 +186,14 @@ public class BeamBuiltinAggregations {
String.format("[%s] is not supported in BIT_OR", fieldType));
}
+ static CombineFn createBitAnd(Schema.FieldType fieldType) {
+if (fieldType.getTypeName() == TypeName.INT64) {
+ return new BitAnd();
+}
+throw new UnsupportedOperationException(
+String.format("[%s] is not supported in BIT_AND", fieldType));
+ }
+
static class CustMax> extends
Combine.BinaryCombineFn {
@Override
public T apply(T left, T right) {
@@ -383,4 +392,30 @@ public class BeamBuiltinAggregations {
return accum;
}
}
+
+ static class BitAnd extends CombineFn {
+@Override
+public Long createAccumulator() {
+ return -1L;
+}
+
+@Override
+public Long addInput(Long accum, T input) {
+ return accum & input.longValue();
+}
+
+@Override
+public Long mergeAccumulators(Iterable accums) {
+ Long merged = createAccumulator();
+ for (long accum : accums) {
+merged = merged & accum;
+ }
+ return merged;
+}
+
+@Override
+public Long extractOutput(Long accum) {
+ return accum;
+}
+ }
}
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
index 40b3b63..9c365b2 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
@@ -314,7 +314,39 @@ public class BeamSqlDslAggregationTest extends
BeamSqlDslBase {
PCollection inputRows =
pipeline.apply("longVals",
Create.of(rowsInTableA).withRowSchema(schemaInTableA));
PCollection result = inputRows.apply("sql", SqlTransform.query(sql));
+
+PAssert.that(result).containsInAnyOrder(rowResult);
+
+pipeline.run().waitUntilFinish();
+ }
+
+ @Test
+ public void testBitAndFunction() throws Exception {
+pipeline.enableAbandonedNodeEnforcement(false);
+
+Schema schemaInTableA =
+
Schema.builder().addInt64Field("f_long").addInt32Field("f_int2").build();
+
+Schema resultType = Schema.builder().addInt64Field("finalAnswer").build();
+
+List rowsInTableA =
+TestUtils.RowsBuilder.of(schemaInTableA)
+.addRows(
+0xF001L, 0,
+0x00A1L, 0)
+.getRows();
+
+String sql = "SELECT bit_and(f_long) as bitand " + "FROM PCOLLECTION GROUP
BY f_int2";
+
+Row rowResult = Row.withSchema(resultType).addValues(1L).build();
+
+PCollection inputRows =
+
[beam] branch master updated (42e45e2 -> 63cc13a)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 42e45e2 Merge pull request #11988 from robinyqiu/cleanup add 63cc13a Support HOP and SESSION as TVF (#11868) No new revisions were added by this update. Summary of changes: .../extensions/sql/impl/TVFSlidingWindowFn.java| 75 +++ .../sql/impl/rel/BeamTableFunctionScanRel.java | 198 +++- .../sql/impl/utils/TVFStreamingUtils.java | 9 + .../sdk/extensions/sql/zetasql/SqlAnalyzer.java| 36 +- .../beam/sdk/extensions/sql/zetasql/TestInput.java | 52 +- .../zetasql/translation/ExpressionConverter.java | 79 ++- .../extensions/sql/zetasql/StreamingSQLTest.java | 550 + .../sql/zetasql/ZetaSQLDialectSpecTest.java| 293 +-- .../extensions/sql/zetasql/ZetaSQLTestBase.java| 122 + 9 files changed, 1076 insertions(+), 338 deletions(-) create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TVFSlidingWindowFn.java create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/StreamingSQLTest.java create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLTestBase.java
[beam] branch master updated: [BEAM-10230] @Ignore: BYTES works with LIKE.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6b03195 [BEAM-10230] @Ignore: BYTES works with LIKE.
new 0ef36fe Merge pull request #11971 from amaliujia/rw-BEAM-10230
6b03195 is described below
commit 6b03195b794151604a49b15cd99bb3f5fa79040c
Author: amaliujia
AuthorDate: Wed Jun 10 11:11:39 2020 -0700
[BEAM-10230] @Ignore: BYTES works with LIKE.
---
.../beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java | 1 +
.../apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java | 1 -
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
index 34f130c..d8a0c72 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
@@ -89,6 +89,7 @@ public class SqlStdOperatorMappingTable {
FunctionSignatureId.FN_CHAR_LENGTH_STRING,
FunctionSignatureId.FN_ENDS_WITH_STRING,
FunctionSignatureId.FN_STRING_LIKE,
+ FunctionSignatureId.FN_BYTE_LIKE,
FunctionSignatureId.FN_COALESCE,
FunctionSignatureId.FN_IF,
FunctionSignatureId.FN_IFNULL,
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
index a74482c..639d180 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -833,7 +833,6 @@ public class ZetaSQLDialectSpecTest {
}
@Test
- @Ignore("Does not support BYTES for like")
public void testLikeBytes() {
String sql = "SELECT @p0 LIKE @p1 AS ColA";
ImmutableMap params =
[beam] branch master updated (c3a2dd8 -> 9c44108)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c3a2dd8 Merge pull request #11940 [BEAM-6215] Additional tests for FlatMap label. add 9c44108 fixup! roll back changes (#11958) No new revisions were added by this update. Summary of changes: .../sql/impl/rel/BeamSetOperatorRelBase.java | 43 ++ 1 file changed, 20 insertions(+), 23 deletions(-)
[beam] branch master updated: [BEAM-10215] @Ignore: Concat now works with varargs
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 93a167b [BEAM-10215] @Ignore: Concat now works with varargs
new c844646 Merge pull request #11952 from amaliujia/rw-BEAM-10215
93a167b is described below
commit 93a167b9eea654f3db80cbff57dfd04028b87aa6
Author: amaliujia
AuthorDate: Mon Jun 8 13:53:45 2020 -0700
[BEAM-10215] @Ignore: Concat now works with varargs
---
.../apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java | 3 ---
1 file changed, 3 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
index adae655..6481702 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -3841,9 +3841,6 @@ public class ZetaSQLDialectSpecTest {
}
@Test
- @Ignore(
- "Calcite codegen does not support UDF with ... args."
- + " See:https://jira.apache.org/jira/browse/CALCITE-2889;)
public void testConcatWithSixParameters() {
String sql = "SELECT concat('abc', 'def', ' ', 'xyz', 'kkk', 'ttt')";
ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
[beam] branch master updated (c0a96edc -> e28593b)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c0a96edc Merge pull request #11626 Cleanup ToString transforms. add e28593b [BEAM-10213] @Ignore: fix the test for testCastToDateWithCase. (#11948) No new revisions were added by this update. Summary of changes: .../extensions/sql/zetasql/ZetaSQLDialectSpecTest.java | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-)
[beam] branch master updated: [BEAM-9191] Add Jira Link to empty @Ignore.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7a8d21f [BEAM-9191] Add Jira Link to empty @Ignore.
new 0a6e2f4 Merge pull request #11953 from amaliujia/rw-BEAM-9191
7a8d21f is described below
commit 7a8d21fc7850f8fbba52e8071a887e208f5cd87c
Author: amaliujia
AuthorDate: Mon Jun 8 14:05:29 2020 -0700
[BEAM-9191] Add Jira Link to empty @Ignore.
---
.../beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
index adae655..bc00018 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -4252,7 +4252,7 @@ public class ZetaSQLDialectSpecTest {
}
@Test
- @Ignore("")
+ @Ignore("https://jira.apache.org/jira/browse/BEAM-9191;)
public void testCastBytesToString1() {
String sql = "SELECT CAST(@p0 AS STRING)";
ImmutableMap params =
@@ -4281,7 +4281,7 @@ public class ZetaSQLDialectSpecTest {
}
@Test
- @Ignore("")
+ @Ignore("https://jira.apache.org/jira/browse/BEAM-9191;)
public void testCastBytesToStringFromTable() {
String sql = "SELECT CAST(bytes_col AS STRING) FROM table_all_types";
ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
[beam] branch master updated (96836a7 -> f59adee)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 96836a7 [BEAM-2939] Fix FnApiDoFnRunner to ensure that we output within the correct window when processing a splittable dofn (#11922) add f59adee [BEAM-10204] @Ignore: re-enable LIKE operator related unit tests. (#11933) No new revisions were added by this update. Summary of changes: .../sql/zetasql/ZetaSQLDialectSpecTest.java| 23 -- 1 file changed, 23 deletions(-)
[beam] branch master updated: [BEAM-10205] @Ignore:BYTES can work with UNION ALL.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1a82c1b [BEAM-10205] @Ignore:BYTES can work with UNION ALL.
new a8315b8 Merge pull request #11934 from amaliujia/rw-BEAM-10205
1a82c1b is described below
commit 1a82c1b7902d5e6a7789d7a0966a5188fabbc4a3
Author: amaliujia
AuthorDate: Fri Jun 5 10:27:27 2020 -0700
[BEAM-10205] @Ignore:BYTES can work with UNION ALL.
---
.../apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
index 8abbb08..31a312d 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
@@ -4590,7 +4590,6 @@ public class ZetaSQLDialectSpecTest {
}
@Test
- @Ignore("Bytes cannot be in UNION ALL")
public void testSelectDistinct2() {
String sql =
"SELECT DISTINCT val.BYTES\n"
[beam] branch master updated (9aaf9ab -> 8046377)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 9aaf9ab Merge pull request #11914 from ibzib/BEAM-10187
add 8046377 [Beam 9879] support STRING_AGG function in Beam ZetaSQL
(#11895)
No new revisions were added by this update.
Summary of changes:
.../beam/sdk/extensions/sql/SqlTransform.java | 2 +-
.../beam/sdk/extensions/sql/impl/UdafImpl.java | 2 +-
.../sdk/extensions/sql/impl/udaf/StringAgg.java| 76 ++
.../extensions/sql/impl/udaf}/package-info.java| 4 +-
.../sdk/extensions/sql/zetasql/SqlOperators.java | 25 +++
.../sql/zetasql/SqlStdOperatorMappingTable.java| 3 +-
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 1 +
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 16 +
8 files changed, 124 insertions(+), 5 deletions(-)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udaf/StringAgg.java
copy sdks/java/{fn-execution/src/main/java/org/apache/beam/sdk/fn/stream =>
extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udaf}/package-info.java
(91%)
[beam] branch master updated: [BEAM-9825] | Implement Intersect, Union, Except transforms (#11610)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 01c11e7 [BEAM-9825] | Implement Intersect,Union,Except transforms
(#11610)
01c11e7 is described below
commit 01c11e7211937bde3c238fe3639f9dfe7774d093
Author: darshanj
AuthorDate: Tue Jun 2 11:54:54 2020 +0800
[BEAM-9825] | Implement Intersect,Union,Except transforms (#11610)
* [BEAM-9825] | Implement Intersect,Union,Except transforms
---
.../java/org/apache/beam/sdk/transforms/Sets.java | 680 +
.../org/apache/beam/sdk/transforms/SetsTest.java | 324 ++
.../sql/impl/rel/BeamSetOperatorRelBase.java | 43 +-
3 files changed, 1027 insertions(+), 20 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sets.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sets.java
new file mode 100644
index 000..16a7281
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sets.java
@@ -0,0 +1,680 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms;
+
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.join.CoGbkResult;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.TupleTag;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables;
+
+/**
+ * The {@code PTransform}s that allow to compute different set functions
across {@link
+ * PCollection}s.
+ *
+ * They come in two variants. 1. Between two {@link PCollection} 2. Between
two or more {@link
+ * PCollection} in a {@link PCollectionList}.
+ *
+ * Following {@code PTransform}s follows SET DISTINCT semantics:
intersectDistinct,
+ * expectDistinct, unionDistinct
+ *
+ * Following {@code PTransform}s follows SET ALL semantics: intersectAll,
expectAll, unionAll
+ *
+ * For example, the following demonstrates intersectDistinct between two
collections {@link
+ * PCollection}s.
+ *
+ * {@code
+ * Pipeline p = ...;
+ *
+ * PCollection left = p.apply(Create.of("1", "2", "3", "3", "4", "5"));
+ * PCollection right = p.apply(Create.of("1", "3", "4", "4", "6"));
+ *
+ * PCollection results =
+ * left.apply(SetFns.intersectDistinct(right)); // results will be
PCollection containing: "1","3","4"
+ *
+ * }
+ *
+ * For example, the following demonstrates intersectDistinct between three
collections {@link
+ * PCollection}s in a {@link PCollectionList}.
+ *
+ * {@code
+ * Pipeline p = ...;
+ *
+ * PCollection first = p.apply(Create.of("1", "2", "3", "3", "4",
"5"));
+ * PCollection second = p.apply(Create.of("1", "3", "4", "4", "6"));
+ * PCollection third = p.apply(Create.of("3", "4", "4"));
+ *
+ * // Following example will perform (first intersect second) intersect third.
+ * PCollection results =
+ * PCollectionList.of(first).and(second).and(third)
+ * .apply(SetFns.intersectDistinct()); // results will be
PCollection containing: "3","4"
+ *
+ * }
+ */
+public class Sets {
+
+ /**
+ * Returns a new {@code PTransform} transform that follows SET DISTINCT
[beam] branch master updated (c4f03a8 -> 0588260)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c4f03a8 [BEAM-9810] Add a Tox (precommit) suite for Python 3.8 #11707 add 0588260 [BEAM-9363] Support TUMBLE aggregation (#11807) No new revisions were added by this update. Summary of changes: .../sql/impl/rel/BeamTableFunctionScanRel.java | 37 +- .../sql/zetasql/ZetaSQLDialectSpecTest.java| 25 +++ 2 files changed, 54 insertions(+), 8 deletions(-)
[beam] branch master updated: [BEAM-10074] | implement hashing functions
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fa028e6 [BEAM-10074] | implement hashing functions
new 706a06c Merge pull request #11817 from darshanj/BEAM-10074
fa028e6 is described below
commit fa028e68e47c2a47858783a5a5f7adc15569c654
Author: darshan jani
AuthorDate: Tue May 26 22:36:35 2020 +0800
[BEAM-10074] | implement hashing functions
---
.../sql/impl/udf/BuiltinHashFunctions.java | 139 +
.../beam/sdk/extensions/sql/BeamSqlDslBase.java| 6 +
.../udf/BeamSalUhfSpecialTypeAndValueTest.java | 69 ++
.../sql/impl/udf/BeamSqlUdfExpressionTest.java | 41 ++
4 files changed, 255 insertions(+)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinHashFunctions.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinHashFunctions.java
new file mode 100644
index 000..c3fc82b
--- /dev/null
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinHashFunctions.java
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.impl.udf;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.schemas.Schema;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.linq4j.function.Strict;
+
+/** Hash Functions. */
+@AutoService(BeamBuiltinFunctionProvider.class)
+public class BuiltinHashFunctions extends BeamBuiltinFunctionProvider {
+
+ /**
+ * MD5(X)
+ *
+ * Calculates the MD5 digest and returns the value as a 16 element {@code
byte[]}.
+ */
+ @UDF(
+ funcName = "MD5",
+ parameterArray = {Schema.TypeName.STRING},
+ returnType = Schema.TypeName.BYTES)
+ @Strict
+ public byte[] md5String(String str) {
+return org.apache.commons.codec.digest.DigestUtils.md5(str);
+ }
+
+ /**
+ * MD5(X)
+ *
+ * Calculates the MD5 digest and returns the value as a 16 element {@code
byte[]}.
+ */
+ @UDF(
+ funcName = "MD5",
+ parameterArray = {Schema.TypeName.BYTES},
+ returnType = Schema.TypeName.BYTES)
+ @Strict
+ public byte[] md5Bytes(byte[] bytes) {
+return org.apache.commons.codec.digest.DigestUtils.md5(bytes);
+ }
+
+ /**
+ * SHA1(X)
+ *
+ * Calculates the SHA-1 digest and returns the value as a {@code byte[]}.
+ */
+ @UDF(
+ funcName = "SHA1",
+ parameterArray = {Schema.TypeName.STRING},
+ returnType = Schema.TypeName.BYTES)
+ @Strict
+ public byte[] sha1String(String str) {
+return org.apache.commons.codec.digest.DigestUtils.sha1(str);
+ }
+
+ /**
+ * SHA1(X)
+ *
+ * Calculates the SHA-1 digest and returns the value as a {@code byte[]}.
+ */
+ @UDF(
+ funcName = "SHA1",
+ parameterArray = {Schema.TypeName.BYTES},
+ returnType = Schema.TypeName.BYTES)
+ @Strict
+ public byte[] sha1Bytes(byte[] bytes) {
+return org.apache.commons.codec.digest.DigestUtils.sha1(bytes);
+ }
+
+ /**
+ * SHA256(X)
+ *
+ * Calculates the SHA-1 digest and returns the value as a {@code byte[]}.
+ */
+ @UDF(
+ funcName = "SHA256",
+ parameterArray = {Schema.TypeName.STRING},
+ returnType = Schema.TypeName.BYTES)
+ @Strict
+ public byte[] sha256String(String str) {
+return org.apache.commons.codec.digest.DigestUtils.sha256(str);
+ }
+
+ /**
+ * SHA256(X)
+ *
+ * Calculates the SHA-1 digest and returns the value as a {@code byte[]}.
+ */
+ @UDF(
+ funcName = "SHA256",
+ parameterArray = {Schema.TypeName.BYTES},
+ returnType = Schema.TypeName.BYTES)
+ @Strict
+ public byte[] sha256Bytes(byte[] bytes) {
+return org.apache.commons.codec.digest.DigestUtils.sha256(bytes);
+ }
+
+ /**
+ * SHA512(X)
+ *
+ * Calculates the SHA-1 digest and returns the value as a {@code byte[]}.
+ */
+ @UDF(
+ funcName = "SHA512",
+ parame
[beam] branch master updated (cf03756 -> e5345fb)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cf03756 [BEAM-9899] Fix some issues around storing schema `id` on user types (#11701) add 9be5a29 Updated Videos and Podcasts page add e5345fb Merge pull request #11724 from apache/aaltay-patch-1 No new revisions were added by this update. Summary of changes: .../documentation/resources/videos-and-podcasts.md | 20 +++- 1 file changed, 19 insertions(+), 1 deletion(-)
[beam] branch master updated (7ad4c4c -> 27656d7)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7ad4c4c Merge pull request #11740: [BEAM-8019] Prevent Dataflow from starting multiple containers for the same image add 27656d7 [BEAM-9984] Support BIT_OR aggregation function in Beam SQL (#11737) No new revisions were added by this update. Summary of changes: .../impl/transform/BeamBuiltinAggregations.java| 44 -- .../extensions/sql/BeamSqlDslAggregationTest.java | 27 + .../sql/zetasql/SqlStdOperatorMappingTable.java| 4 +- .../sql/zetasql/ZetaSQLDialectSpecTest.java| 18 - 4 files changed, 86 insertions(+), 7 deletions(-)
[beam] branch master updated (a70fde0 -> bd4523b)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from a70fde0 [BEAM-9339] Ensure that Dataflow's pipeline proto also
contains the capabilities (#11748)
add bd4523b [BEAM-9363] TUMBLE as TVF (#10946)
No new revisions were added by this update.
Summary of changes:
.../src/main/resources/beam/suppressions.xml | 1 +
.../extensions/sql/impl/planner/BeamRuleSets.java | 3 +-
.../sql/impl/rel/BeamTableFunctionScanRel.java | 162 +
.../sql/impl/rule/BeamTableFunctionScanRule.java | 71 +
.../{package-info.java => TVFStreamingUtils.java} | 8 +-
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java| 47 +-
.../sql/zetasql/SqlWindowTableFunction.java| 116 +++
.../zetasql/translation/ExpressionConverter.java | 64
.../translation/QueryStatementConverter.java | 2 +
.../sql/zetasql/translation/TVFScanConverter.java | 86 +++
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 40 +
11 files changed, 594 insertions(+), 6 deletions(-)
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamTableFunctionScanRel.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamTableFunctionScanRule.java
copy
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/{package-info.java
=> TVFStreamingUtils.java} (80%)
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlWindowTableFunction.java
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TVFScanConverter.java
[beam] branch master updated (4e47dea -> 5de4080)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4e47dea Merge pull request #11713 Run Inventory jobs every 12 hours. add 844bf45 Update release notes link for 2.20.0 add 5de4080 Merge pull request #11714 from apache/aaltay-patch-2 No new revisions were added by this update. Summary of changes: website/www/site/content/en/get-started/downloads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated: [BEAM-9929] Support UNNEST(array_column) in ZetaSQL.
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b130aa1 [BEAM-9929] Support UNNEST(array_column) in ZetaSQL.
new 46f42d8 Merge pull request #11636 from
amaliujia/rw-support_unnest_column
b130aa1 is described below
commit b130aa15cb6d090603793c5429cb6a9c651c0b61
Author: amaliujia
AuthorDate: Thu May 7 17:18:34 2020 -0700
[BEAM-9929] Support UNNEST(array_column) in ZetaSQL.
---
.../translation/ArrayScanColumnRefToUncollect.java | 90 ++
...a => ArrayScanLiteralToUncollectConverter.java} | 4 +-
.../translation/QueryStatementConverter.java | 3 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 20 +
4 files changed, 114 insertions(+), 3 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanColumnRefToUncollect.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanColumnRefToUncollect.java
new file mode 100644
index 000..0a02a4a
--- /dev/null
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanColumnRefToUncollect.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.zetasql.translation;
+
+import com.google.zetasql.resolvedast.ResolvedNode;
+import com.google.zetasql.resolvedast.ResolvedNodes;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.RelNode;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.CorrelationId;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.JoinRelType;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.core.Uncollect;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalCorrelate;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.logical.LogicalProject;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rex.RexInputRef;
+import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.ImmutableBitSet;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+/** Converts array scan that represents a reference to array column literal to
uncollect. */
+public class ArrayScanColumnRefToUncollect extends
RelConverter {
+ ArrayScanColumnRefToUncollect(ConversionContext context) {
+super(context);
+ }
+
+ @Override
+ public boolean canConvert(ResolvedNodes.ResolvedArrayScan zetaNode) {
+return zetaNode.getInputScan() != null
+&& zetaNode.getArrayExpr() instanceof ResolvedNodes.ResolvedColumnRef
+&& zetaNode.getJoinExpr() == null;
+ }
+
+ @Override
+ public List getInputs(ResolvedNodes.ResolvedArrayScan
zetaNode) {
+return ImmutableList.of(zetaNode.getInputScan());
+ }
+
+ @Override
+ public RelNode convert(ResolvedNodes.ResolvedArrayScan zetaNode,
List inputs) {
+assert inputs.size() == 1;
+RelNode input = inputs.get(0);
+RexInputRef columnRef =
+(RexInputRef)
+getExpressionConverter()
+.convertRexNodeFromResolvedExpr(
+zetaNode.getArrayExpr(),
+zetaNode.getInputScan().getColumnList(),
+input.getRowType().getFieldList());
+
+String fieldName =
+String.format(
+"%s%s",
+zetaNode.getElementColumn().getTableName(),
zetaNode.getElementColumn().getName());
+CorrelationId correlationId = new CorrelationId(0);
+RelNode projectNode =
+LogicalProject.create(
+createOneRow(getCluster()),
+Collections.singletonList(
+getCluster()
+.getRexBuilder()
+.makeFieldAccess(
+
getCluster().getRexBuild
[beam] branch master updated: [BEAM-9418] Support ANY_VALUE aggregation functions
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 27876f0 [BEAM-9418] Support ANY_VALUE aggregation functions
new 6453e85 Merge pull request #11333 from jhnmora000/master
27876f0 is described below
commit 27876f035e3afdd478ef5e7f21d8524d17b59e13
Author: John Mora
AuthorDate: Tue Apr 28 22:11:51 2020 -0500
[BEAM-9418] Support ANY_VALUE aggregation functions
The implementation is based on the function Sample#anyCombineFn(int)
of the Java SDK core.Also, the support for ZetaSQL was enabled.
---
.../org/apache/beam/sdk/transforms/Sample.java | 38 +
.../impl/transform/BeamBuiltinAggregations.java| 2 +
.../extensions/sql/BeamSqlDslAggregationTest.java | 49 ++
.../sql/zetasql/SqlStdOperatorMappingTable.java| 3 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java| 33 +++
5 files changed, 124 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
index 2594dee..4b93596 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sample.java
@@ -59,6 +59,14 @@ public class Sample {
}
/**
+ * Returns a {@link CombineFn} that computes a single and potentially
non-uniform sample value of
+ * its inputs.
+ */
+ public static CombineFn anyValueCombineFn() {
+return new AnyValueCombineFn<>();
+ }
+
+ /**
* {@code Sample#any(long)} takes a {@code PCollection} and a limit, and
produces a new {@code
* PCollection} containing up to limit elements of the input {@code
PCollection}.
*
@@ -246,6 +254,36 @@ public class Sample {
}
}
+ /** A {@link CombineFn} that combines into a single element. */
+ private static class AnyValueCombineFn extends CombineFn, T> {
+private SampleAnyCombineFn internal;
+
+private AnyValueCombineFn() {
+ internal = new SampleAnyCombineFn<>(1);
+}
+
+@Override
+public List createAccumulator() {
+ return internal.createAccumulator();
+}
+
+@Override
+public List addInput(List accumulator, T input) {
+ return internal.addInput(accumulator, input);
+}
+
+@Override
+public List mergeAccumulators(Iterable> accumulators) {
+ return internal.mergeAccumulators(accumulators);
+}
+
+@Override
+public T extractOutput(List accumulator) {
+ Iterator it = internal.extractOutput(accumulator).iterator();
+ return it.hasNext() ? it.next() : null;
+}
+ }
+
/**
* {@code CombineFn} that computes a fixed-size sample of a collection of
values.
*
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
index ad99c28..106e609 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations.java
@@ -38,6 +38,7 @@ import org.apache.beam.sdk.transforms.Combine.CombineFn;
import org.apache.beam.sdk.transforms.Count;
import org.apache.beam.sdk.transforms.Max;
import org.apache.beam.sdk.transforms.Min;
+import org.apache.beam.sdk.transforms.Sample;
import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.values.KV;
import
org.apache.beam.vendor.calcite.v1_20_0.com.google.common.collect.ImmutableMap;
@@ -48,6 +49,7 @@ public class BeamBuiltinAggregations {
public static final Map>>
BUILTIN_AGGREGATOR_FACTORIES =
ImmutableMap.>>builder()
+ .put("ANY_VALUE", typeName -> Sample.anyValueCombineFn())
.put("COUNT", typeName -> Count.combineFn())
.put("MAX", BeamBuiltinAggregations::createMax)
.put("MIN", BeamBuiltinAggregations::createMin)
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
index d350062..80964f5 100644
---
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlDslAggregationTest.java
@@ -27,8 +27,10 @@ import static
org.junit.internal.matchers.ThrowableMessageMatcher.hasMessage;
import java.m
[beam] branch master updated (e173fa8 -> 7f7105c)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e173fa8 Merge pull request #11573 from davidcavazos/fix-notebook-urls add 7f7105c Fix typo in go installation link. (#11445) No new revisions were added by this update. Summary of changes: website/src/contribute/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (da4a48d -> 0a5cb43)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from da4a48d [BEAM-9729, BEAM-8486] Runner-side bundle registration cleanup. (#11358) add 0a5cb43 Add new release 2.20.0 to beam website (#11285) No new revisions were added by this update. Summary of changes: website/_config.yml | 2 +- website/src/.htaccess| 2 +- website/src/get-started/downloads.md | 13 ++--- 3 files changed, 12 insertions(+), 5 deletions(-)
[beam] branch master updated (0a5cb43 -> 1d7f640)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0a5cb43 Add new release 2.20.0 to beam website (#11285) add f451643 add 2.20.0 blog post add 4ebcc59 add a known issue add 3162ed2 fixup! update 2.20.0 date add 1d7f640 add 2.20.0 blog post (#11298) No new revisions were added by this update. Summary of changes: website/src/_data/authors.yml| 4 ++ website/src/_posts/2020-04-15-beam-2.20.0.md | 76 2 files changed, 80 insertions(+) create mode 100644 website/src/_posts/2020-04-15-beam-2.20.0.md
[beam] annotated tag v2.20.0 updated (9f0cb64 -> 0509ad3)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to annotated tag v2.20.0 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag v2.20.0 was modified! *** from 9f0cb64 (commit) to 0509ad3 (tag) tagging e0985a6a1a4c4398ba3fd96ae4fef3efb6eaee36 (tag) length 187 bytes by amaliujia on Wed Apr 15 14:04:36 2020 -0700 - Log - 2.20.0 --- No new revisions were added by this update. Summary of changes:
[beam] branch master updated (ec0e157 -> 9336272)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ec0e157 Merge pull request #11389 from Refactor the BCJ and capture controls to be more testable add 9336272 [BEAM-i9751] upgrade zetasql to 2020.04.1 (#11410) No new revisions were added by this update. Summary of changes: sdks/java/extensions/sql/zetasql/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
svn commit: r38874 - /dev/beam/2.20.0/python/
Author: amaliujia Date: Thu Apr 9 17:48:18 2020 New Revision: 38874 Log: python wheels 2.20.0 rc2 Added: dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl.sha512 Added: dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl == Binary file - no diff available
[beam] annotated tag v2.20.0-RC2 updated (9f0cb64 -> e0985a6)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to annotated tag v2.20.0-RC2 in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag v2.20.0-RC2 was modified! *** from 9f0cb64 (commit) to e0985a6 (tag) tagging 9f0cb649d39ee6236ea27f111acb4b66591a80ec (commit) replaces java-ulr-removal by amaliujia on Wed Apr 8 14:38:48 2020 -0700 - Log - [Gradle Release Plugin] - creating tag: 'v2.20.0-RC2'. --- No new revisions were added by this update. Summary of changes:
[beam-wheels] branch release-2.20.0 created (now e233575)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam-wheels.git. at e233575 fix travis config (#16) No new revisions were added by this update.
svn commit: r38846 - in /dev/beam/2.20.0/python: ./ apache-beam-2.20.0.zip apache-beam-2.20.0.zip.asc apache-beam-2.20.0.zip.sha512
Author: amaliujia Date: Tue Apr 7 19:00:37 2020 New Revision: 38846 Log: 2.20.0 python rc2 Added: dev/beam/2.20.0/python/ dev/beam/2.20.0/python/apache-beam-2.20.0.zip (with props) dev/beam/2.20.0/python/apache-beam-2.20.0.zip.asc dev/beam/2.20.0/python/apache-beam-2.20.0.zip.sha512 Added: dev/beam/2.20.0/python/apache-beam-2.20.0.zip == Binary file - no diff available. Propchange: dev/beam/2.20.0/python/apache-beam-2.20.0.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.20.0/python/apache-beam-2.20.0.zip.asc == --- dev/beam/2.20.0/python/apache-beam-2.20.0.zip.asc (added) +++ dev/beam/2.20.0/python/apache-beam-2.20.0.zip.asc Tue Apr 7 19:00:37 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEDXvholLbzuifZJG736ZIYrcD9cgFAl6MzZIACgkQ36ZIYrcD +9cgNHQ/+N/ddmwXa40Kb2tH+yoqXqyziJOqM+tirzo9h2/93ZAUXdqf4lddNYL9v +zs8we4RMLXZD1J2MAZB7BKXnSt36E9znN2rPwg9Mw7vnjfAJl4oWkoqqxzorff7c +8vV1fPyWgB4ZGa0gdwHWd9OuZwUmkWz7epxmed6sMfJVFgVaSLOFCQUmmBybKKg/ +hgTjZ4mutjh0jVMVXBu+f3K6FfEY/1oc5MFsWC4iNVnhkLjM6zWyxfL4ZY3GCMas +UQDeU7iDchhU36MAfshTssdNX5LAOz078hSIh0kEouzMY3nAuKnfjrUtKwqGuCKS +dK9mhFEy4s9ofNW9F0hUE912XDgzMEouAiBQsEG407/i8C22cuxTZVaoBgnLrouM +jId3ZSEbUK5th+UaKaBef4ghvtR/PDD8HYAQJewMbv2v1sAr4Mu3lS469Xtmdops ++BnCe47Vfhxs3Rcrk0In5NrT1Ely977YFRUjdg7xefK4QA0NYhZguPWLkeDnKGeX +FGgEv51uekibh2qw4KcgcOzVKut9sHH0JuHfIDebfmIe4NIYktSDfEo+vdsdu5sc +Ft22KBEmkXJ/jCHw1D71f2sOg2T+l2/VocXIbYKJcjYXXLBaMe0ABcOQ72UDPoqQ +6k2M5Ks/QBx+qsMWZ+ufQVYjSEyln4wqKMTSqTGu/T+5BvOvmDY= +=FtRP +-END PGP SIGNATURE- Added: dev/beam/2.20.0/python/apache-beam-2.20.0.zip.sha512 == --- dev/beam/2.20.0/python/apache-beam-2.20.0.zip.sha512 (added) +++ dev/beam/2.20.0/python/apache-beam-2.20.0.zip.sha512 Tue Apr 7 19:00:37 2020 @@ -0,0 +1 @@ +410b1311df77b0bff39542ed8476a21010c0bff7ea3f42db149028ba2483f7db7ea75a5859c26e211924ac60f9a1b6e780df78593743bca085ea1c76f6fc9eae apache-beam-2.20.0.zip
svn commit: r38845 - in /dev/beam/2.20.0: ./ apache-beam-2.20.0-source-release.zip apache-beam-2.20.0-source-release.zip.asc apache-beam-2.20.0-source-release.zip.sha512
Author: amaliujia Date: Tue Apr 7 18:58:54 2020 New Revision: 38845 Log: 2.20.0 java rc2 Added: dev/beam/2.20.0/ dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip (with props) dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.asc dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.sha512 Added: dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip == Binary file - no diff available. Propchange: dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.asc == --- dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.asc (added) +++ dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.asc Tue Apr 7 18:58:54 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEDXvholLbzuifZJG736ZIYrcD9cgFAl6MzVkACgkQ36ZIYrcD +9chD7Q/+Oywyl6ogRWQfI5JK+Ju2GtAQ4PmBjn0xpwxR2/LrHtTM1kZxG6zdqGFA +dy9mjCuvriwfAfdYKkIg8M8JUFmrGEtpgEWx6xD6SC4Wi9v8L3AYEsOb3nr/qCBs +aSJ3Lyihpfrw7Yx14jxDUxKn85zx4r+oajphOnvtATK+kIGqAVrI4eM5Lj6JS0Tw +/DO1h/TA6SW/VWCJdaEH+jwQM+R4vHitx2YliP91QHK5yfI2du4TbexUVuorJ4o2 +WAj32c7onwcYBHATdS7fUi9FdjKUESoAZsFoqZJJGQhDN9HlutK5w01z8kQuJjXM +duC3zAgT5PH1v85r+/iQ/UNZXLg+nu+TVwr9/2ERgaTcpV3hb/L8vjF19WkBFLFd +90Zq1JY0tybiLoUgD2ZgaRNELJ9HipIKiGadelYBp+jW5KrUzRSHGc04TFebDO7e +h8Cvc5wi3YcSHt4rpaKW2wIRc6wa3nZjjwl/31cx1pKiRS158n00/jAd9jK/t7iy +sXeFUUZL/gi15HmLpZozfJUwVxcSCeJnKrktU8GmWh+W9yw3xN8GDK2TEsU59gp0 +blW/oDP0GmouB6HTF0J7dZqvtB80j/OnsmMCFrIqXbfYqmDApoGNQ5wGZh/Tm8nW +mY6PD3HGv/54Z1+6XoFcPQmITvK1XMLe0h596LFis6tr/Sz9qUw= +=1kCU +-END PGP SIGNATURE- Added: dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.sha512 == --- dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.sha512 (added) +++ dev/beam/2.20.0/apache-beam-2.20.0-source-release.zip.sha512 Tue Apr 7 18:58:54 2020 @@ -0,0 +1 @@ +08775e19715c7f9e51df990993dc971f124463f2008f4180785ddfd939583d1ff5f134aa29c389e2ffc0fbf3ca8c37219f397b6fc7682ce9312213be14721a94 apache-beam-2.20.0-source-release.zip
svn commit: r38843 - /dev/beam/2.20.0/
Author: amaliujia Date: Tue Apr 7 17:59:14 2020 New Revision: 38843 Log: remove 2.20.0 rc1 Removed: dev/beam/2.20.0/
[beam] branch release-2.20.0 updated (ef4b21b -> cd67dbf)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from ef4b21b [BEAM-9557] Fix timer window boundary checking (#11252) new 6bba79a Install typing package only for Python < 3.5.3 (#10821) new cd67dbf Merge pull request #11226: [BEAM-9557] Fix timer window boundary checking The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/runners/core/SimpleDoFnRunner.java | 4 +- .../org/apache/beam/sdk/transforms/ParDoTest.java | 91 +- sdks/python/setup.py | 4 +- 3 files changed, 94 insertions(+), 5 deletions(-)
[beam] 02/02: Merge pull request #11226: [BEAM-9557] Fix timer window boundary checking
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch release-2.20.0
in repository https://gitbox.apache.org/repos/asf/beam.git
commit cd67dbf1c0209824a06f764f3a1d4c591441c5c8
Author: reuvenlax
AuthorDate: Tue Apr 7 09:50:14 2020 -0700
Merge pull request #11226: [BEAM-9557] Fix timer window boundary checking
---
.../apache/beam/runners/core/SimpleDoFnRunner.java | 4 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 91 +-
2 files changed, 92 insertions(+), 3 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index fa5c695..9cc1b8d 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -1190,13 +1190,13 @@ public class SimpleDoFnRunner
implements DoFnRunner expectedSubstrings = Arrays.asList("event-time
timer", "expiration");
+expectedSubstrings.forEach(
+str ->
+Preconditions.checkState(
+message.contains(str),
+"Pipeline didn't fail with the expected strings:
%s",
+expectedSubstrings));
+ }
+}
+
+@OnTimer(timerId)
+public void onTimer() {}
+ };
+
+ pipeline.apply(Create.of(KV.of("hello", 37))).apply(ParDo.of(fn));
+ pipeline.run();
+}
+
+@Test
+@Category({
+ ValidatesRunner.class,
+ UsesTimersInParDo.class,
+ DataflowPortabilityApiUnsupported.class
+})
+public void testOutOfBoundsEventTimeTimerHold() throws Exception {
+ final String timerId = "foo";
+
+ DoFn, Integer> fn =
+ new DoFn, Integer>() {
+
+@TimerId(timerId)
+private final TimerSpec spec =
TimerSpecs.timer(TimeDomain.EVENT_TIME);
+
+@ProcessElement
+public void processElement(
+ProcessContext context, BoundedWindow window,
@TimerId(timerId) Timer timer) {
+ try {
+timer
+.withOutputTimestamp(window.maxTimestamp().plus(1L))
+.set(window.maxTimestamp());
+fail("Should have failed due to out-of-bounds timer.");
+ } catch (RuntimeException e) {
+String message = e.getMessage();
+List expectedSubstrings =
+Arrays.asList("event-time timer", "output timestamp");
+expectedSubstrings.forEach(
+str ->
+Preconditions.checkState(
+message.contains(str),
+"Pipeline didn't fail with the expected strings:
%s",
+expectedSubstrings));
+ }
+}
+
+@OnTimer(timerId)
+public void onTimer() {}
+ };
+
+ pipeline.apply(Create.of(KV.of("hello", 37))).apply(ParDo.of(fn));
+ pipeline.run();
+}
+
+@Test
+@Category({
+ ValidatesRunner.class,
+ UsesTimersInParDo.class,
+ DataflowPortabilityApiUnsupported.class
+})
+public void testOutOfBoundsProcessingTimeTimerHold() throws Exception {
+ final String timerId = "foo";
+
+ DoFn, Integer> fn =
+ new DoFn, Integer>() {
+
+@TimerId(timerId)
+private final TimerSpec spec =
TimerSpecs.timer(TimeDomain.PROCESSING_TIME);
+
+@ProcessElement
+public void processElement(
+ProcessContext context, BoundedWindow window,
@TimerId(timerId) Timer timer) {
+ try {
+timer
+.withOutputTimestamp(window.maxTimestamp().plus(1L))
+.offset(Duration.standardSeconds(1))
+.setRelative();
fail("Should have failed due to processing time with absolute
timer.");
} catch (RuntimeException e) {
String message = e.getMessage();
-List expectedSubstrings = Arrays.asList("event time
timer", "expiration");
+List expectedSubstrings =
+Arrays.asList("processing-time timer", "output timestamp");
expectedSubstrings.forEach(
str ->
Preconditions.checkState(
[beam] 01/02: Install typing package only for Python < 3.5.3 (#10821)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a commit to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 6bba79abbca019bfcbdae714b02806817ebc06a3 Author: Curtis "Fjord" Hawthorne AuthorDate: Tue Mar 10 16:06:26 2020 -0700 Install typing package only for Python < 3.5.3 (#10821) --- sdks/python/setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/setup.py b/sdks/python/setup.py index bd6450e..80f4cb0 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -168,8 +168,8 @@ REQUIRED_PACKAGES = [ # [BEAM-5628] Beam VCF IO is not supported in Python 3. 'pyvcf>=0.6.8,<0.7.0; python_version < "3.0"', # fixes and additions have been made since typing 3.5 -'typing>=3.7.0,<3.8.0; python_version < "3.8.0"', -'typing-extensions>=3.7.0,<3.8.0; python_version < "3.8.0"', +'typing>=3.7.0,<3.8.0; python_version < "3.5.3"', +'typing-extensions>=3.7.0,<3.8.0', ] # [BEAM-8181] pyarrow cannot be installed on 32-bit Windows platforms.
svn commit: r38788 - /dev/beam/2.20.0/python/
Author: amaliujia Date: Fri Apr 3 03:40:50 2020 New Revision: 38788 Log: sign wheels Added: dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27mu-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp35-cp35m-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp36-cp36m-manylinux1_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_i686.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_i686.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_i686.whl.sha512 dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl (with props) dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl.asc dev/beam/2.20.0/python/apache_beam-2.20.0-cp37-cp37m-manylinux1_x86_64.whl.sha512 Added: dev/beam/2.20.0/python/apache_beam-2.20.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl == Binary file - no diff available. Propchange: dev
[beam-wheels] branch release-2.20.0 created (now e233575)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam-wheels.git. at e233575 fix travis config (#16) No new revisions were added by this update.
[beam-wheels] branch release-2.20.0 created (now 0c52ab9)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam-wheels.git. at 0c52ab9 update travis cofig to remove some warnning messages and fix no job matrix problem. (#15) No new revisions were added by this update.
[beam-wheels] branch master updated: fix travis config (#16)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam-wheels.git
The following commit(s) were added to refs/heads/master by this push:
new e233575 fix travis config (#16)
e233575 is described below
commit e2335758330bb76a3aad82d515fbe4bde215ddd7
Author: Rui Wang
AuthorDate: Thu Apr 2 16:53:03 2020 -0700
fix travis config (#16)
---
.travis.yml | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index f18943d..22d813b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,9 +1,10 @@
language: python
python: 3.5
+sudo: required
dist: trusty
services: docker
osx_image: xcode9.4
-os: linux
+
env:
global:
- VERSION=$(echo ${TRAVIS_BRANCH} | sed 's/release-//')
@@ -15,7 +16,7 @@ env:
- BUILD_DEPENDS="Cython"
- TEST_DEPENDS=
-jobs:
+matrix:
include:
- os: osx
language: generic
@@ -92,9 +93,9 @@ deploy:
access_key_id: ${ACCESS_KEY_ID}
secret_access_key: ${SECRET_ACCESS_KEY}
bucket: "beam-wheels-staging"
- cleanup: true
+ skip_cleanup: true
acl: public-read
- local_dir: wheelhouse
+ local-dir: wheelhouse
edge:
branch: master
on:
[beam-wheels] branch release-2.20.0 created (now 0c52ab9)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam-wheels.git. at 0c52ab9 update travis cofig to remove some warnning messages and fix no job matrix problem. (#15) No new revisions were added by this update.
[beam-wheels] branch master updated: update travis cofig to remove some warnning messages and fix no job matrix problem. (#15)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam-wheels.git
The following commit(s) were added to refs/heads/master by this push:
new 0c52ab9 update travis cofig to remove some warnning messages and fix
no job matrix problem. (#15)
0c52ab9 is described below
commit 0c52ab9b1d4c10c8fe44882743685fc935d41070
Author: Rui Wang
AuthorDate: Thu Apr 2 15:15:33 2020 -0700
update travis cofig to remove some warnning messages and fix no job matrix
problem. (#15)
---
.travis.yml | 12
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 3df74f9..f18943d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,10 +1,9 @@
language: python
python: 3.5
-sudo: required
dist: trusty
services: docker
osx_image: xcode9.4
-
+os: linux
env:
global:
- VERSION=$(echo ${TRAVIS_BRANCH} | sed 's/release-//')
@@ -16,10 +15,7 @@ env:
- BUILD_DEPENDS="Cython"
- TEST_DEPENDS=
-matrix:
- exclude:
-# Exclude the default Python 3.5 build
-- python: 3.5
+jobs:
include:
- os: osx
language: generic
@@ -96,9 +92,9 @@ deploy:
access_key_id: ${ACCESS_KEY_ID}
secret_access_key: ${SECRET_ACCESS_KEY}
bucket: "beam-wheels-staging"
- skip_cleanup: true
+ cleanup: true
acl: public-read
- local-dir: wheelhouse
+ local_dir: wheelhouse
edge:
branch: master
on:
[beam-wheels] branch release-2.20.0 created (now bafcd5d)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch release-2.20.0 in repository https://gitbox.apache.org/repos/asf/beam-wheels.git. at bafcd5d Update user guide with helpful links (#14) No new revisions were added by this update.
