[beam] branch nightly-refs/heads/master updated (6e735c1 -> b33bebc)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 6e735c1 Merge pull request #16341: [adhoc] Forbid to import guava and
others from org.testcontainers.shaded
add f8b5d01 [BEAM-13526] Kafka.IO: make DeserializerProvider a public
interface (#16339)
add 1096956 [BEAM-13521] flink pvr batch precommit
add 5b06fa3 Merge pull request #16349 from ibzib/flink-pvr
add f2f1bdf Exclude UsesOnWindowExpiration by category from Dataflow v2
streaming (#16346)
add acff5d1 [BEAM-13052] Increment pubsub python version and fix
breakages. (#16126)
add 2d458bc [BEAM-13498] [BEAM-13573] exclude new tests on Flink
add 9e07ab0 Merge pull request #16376 from ibzib/BEAM-13572
add 8b50ba2 [BEAM-13052] Add Pub/Sub Lite xlang transforms in python
(#15727)
add 0712b0a [BEAM-13571] Fix ClassNotFound exception in Flink tests
add 9ac567a Merge pull request #16375 from ibzib/flink-class1
add d7ccd0f [BEAM-13402] Version bump Pub/Sub Lite and implement changes
to ensure client and channel reuse (#16358)
add 7dbbd09 [BEAM-13402] Simplify PubsubLiteSink
add b33bebc Merge pull request #16215 from
dpcollins-google/publish-no-desync
No new revisions were added by this update.
Summary of changes:
...Java_PortableValidatesRunner_Flink_Batch.groovy | 44 ---
...ava_PortableValidatesRunner_Flink_Batch.groovy} | 22 ++--
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
runners/flink/flink_runner.gradle | 4 +
runners/flink/job-server/flink_job_server.gradle | 1 +
runners/google-cloud-dataflow-java/build.gradle| 6 +-
.../internal/ExternalTransformConfig.java | 108 ++
.../internal/ExternalTransformRegistrarImpl.java} | 21 ++--
.../pubsublite/internal/InitialOffsetReader.java | 5 +-
.../internal/InitialOffsetReaderImpl.java | 13 +--
.../internal/PerSubscriptionPartitionSdf.java | 7 +-
.../gcp/pubsublite/internal/ProtoFromBytes.java} | 45
.../io/gcp/pubsublite/internal/ProtoToBytes.java | 33 +++---
.../pubsublite/internal/PublisherAssembler.java| 57 ++
.../io/gcp/pubsublite/internal/PublisherCache.java | 5 +
.../gcp/pubsublite/internal/PublisherOrError.java | 49
.../io/gcp/pubsublite/internal/PubsubLiteSink.java | 107 -
.../pubsublite/internal/SubscribeTransform.java| 11 +-
.../pubsublite/internal/SubscriberAssembler.java | 105 ++---
.../SubscriptionPartitionProcessorImpl.java| 2 +-
.../internal/TopicBacklogReaderImpl.java | 22 +---
.../internal/TopicBacklogReaderSettings.java | 45 ++--
.../pubsublite/internal/PubsubLiteSinkTest.java| 46
.../beam/sdk/io/kafka/DeserializerProvider.java| 2 +-
.../examples/complete/game/game_stats_it_test.py | 7 +-
.../examples/complete/game/leader_board_it_test.py | 7 +-
.../streaming_wordcount_debugging_it_test.py | 14 ++-
.../examples/streaming_wordcount_it_test.py| 14 ++-
sdks/python/apache_beam/io/gcp/bigquery_test.py| 7 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 15 ++-
.../apache_beam/io/gcp/pubsub_integration_test.py | 21 ++--
.../apache_beam/io/gcp/pubsub_io_perf_test.py | 12 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 33 +++---
.../{tools => io/gcp/pubsublite}/__init__.py | 8 +-
.../apache_beam/io/gcp/pubsublite/external.py | 126 +
.../apache_beam/io/gcp/pubsublite/proto_api.py | 112 ++
.../apache_beam/io/gcp/tests/pubsub_matcher.py | 5 +-
sdks/python/apache_beam/io/gcp/tests/utils.py | 2 +-
sdks/python/apache_beam/io/gcp/tests/utils_test.py | 8 +-
...low_exercise_streaming_metrics_pipeline_test.py | 14 ++-
.../runners/direct/transform_evaluator.py | 15 +--
sdks/python/apache_beam/testing/test_utils.py | 16 ++-
sdks/python/apache_beam/testing/test_utils_test.py | 4 +-
sdks/python/apache_beam/utils/timestamp.py | 2 +-
sdks/python/setup.py | 3 +-
45 files changed, 689 insertions(+), 518 deletions(-)
delete mode 100644
.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
copy .test-infra/jenkins/{job_PreCommit_Java_Examples_Dataflow.groovy =>
job_PreCommit_Java_PortableValidatesRunner_Flink_Batch.groovy} (66%)
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ExternalTransformConfig.java
copy
sdks/java/io/{snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/crosslanguage/SnowflakeTransformRegistrar.java
=>
google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ExternalTransformRegistrarImpl.java}
[beam] branch master updated: [BEAM-13402] Simplify PubsubLiteSink
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7dbbd09 [BEAM-13402] Simplify PubsubLiteSink
new b33bebc Merge pull request #16215 from
dpcollins-google/publish-no-desync
7dbbd09 is described below
commit 7dbbd097585017c7a655afc041435123745086dc
Author: Daniel Collins
AuthorDate: Sun Dec 12 23:54:02 2021 -0500
[BEAM-13402] Simplify PubsubLiteSink
This also makes it not afflicted by futures never terminating by avoiding
indefinite waits
---
.../io/gcp/pubsublite/internal/PublisherCache.java | 5 +
.../gcp/pubsublite/internal/PublisherOrError.java | 49 --
.../io/gcp/pubsublite/internal/PubsubLiteSink.java | 107 +
.../pubsublite/internal/PubsubLiteSinkTest.java| 46 -
4 files changed, 29 insertions(+), 178 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherCache.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherCache.java
index 76b69b3..ac85ba9 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherCache.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherCache.java
@@ -27,9 +27,13 @@ import
com.google.errorprone.annotations.concurrent.GuardedBy;
import java.util.HashMap;
import org.apache.beam.sdk.io.gcp.pubsublite.PublisherOptions;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** A map of working publishers by PublisherOptions. */
class PublisherCache implements AutoCloseable {
+ private final Logger logger = LoggerFactory.getLogger(PublisherCache.class);
+
@GuardedBy("this")
private final HashMap>
livePublishers =
new HashMap<>();
@@ -49,6 +53,7 @@ class PublisherCache implements AutoCloseable {
new Listener() {
@Override
public void failed(State s, Throwable t) {
+logger.warn("Publisher failed.", t);
evict(options);
}
},
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherOrError.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherOrError.java
deleted file mode 100644
index 7eb1c66..000
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PublisherOrError.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.io.gcp.pubsublite.internal;
-
-import com.google.auto.value.AutoOneOf;
-import com.google.cloud.pubsublite.MessageMetadata;
-import com.google.cloud.pubsublite.internal.CheckedApiException;
-import com.google.cloud.pubsublite.internal.Publisher;
-
-/** A helper representing either a Publisher or an error. */
-@AutoOneOf(PublisherOrError.Kind.class)
-@SuppressWarnings({
- "nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
-})
-abstract class PublisherOrError {
- enum Kind {
-PUBLISHER,
-ERROR
- }
-
- abstract Kind getKind();
-
- abstract Publisher publisher();
-
- abstract CheckedApiException error();
-
- static PublisherOrError ofPublisher(Publisher p) {
-return AutoOneOf_PublisherOrError.publisher(p);
- }
-
- static PublisherOrError ofError(CheckedApiException e) {
-return AutoOneOf_PublisherOrError.error(e);
- }
-}
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PubsubLiteSink.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PubsubLiteSink.java
index 4b666d2..f370919 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/PubsubLiteSink.java
+++
[beam] branch master updated: [BEAM-13402] Version bump Pub/Sub Lite and implement changes to ensure client and channel reuse (#16358)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d7ccd0f [BEAM-13402] Version bump Pub/Sub Lite and implement changes
to ensure client and channel reuse (#16358)
d7ccd0f is described below
commit d7ccd0fe39958c4cd33fd1da850f8f13b2874495
Author: dpcollins-google <40498610+dpcollins-goo...@users.noreply.github.com>
AuthorDate: Tue Dec 28 15:59:56 2021 -0500
[BEAM-13402] Version bump Pub/Sub Lite and implement changes to ensure
client and channel reuse (#16358)
* [BEAM-13402] Version bump Pub/Sub Lite and implement changes to ensure
client (and channel) reuse
* revert TopicBacklogReader changes- ManagedBacklogReader already handles
this
* reformat
* revert TopicBacklogReaderSettings changes
---
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
.../pubsublite/internal/InitialOffsetReader.java | 5 +-
.../internal/InitialOffsetReaderImpl.java | 13 +--
.../internal/PerSubscriptionPartitionSdf.java | 7 +-
.../pubsublite/internal/PublisherAssembler.java| 57 +++
.../pubsublite/internal/SubscribeTransform.java| 11 ++-
.../pubsublite/internal/SubscriberAssembler.java | 105 +
.../SubscriptionPartitionProcessorImpl.java| 2 +-
.../internal/TopicBacklogReaderImpl.java | 22 +
.../internal/TopicBacklogReaderSettings.java | 45 ++---
10 files changed, 136 insertions(+), 133 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 76d7b47..a6c53d1 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -452,7 +452,7 @@ class BeamModulePlugin implements Plugin {
def errorprone_version = "2.3.4"
def google_clients_version = "1.32.1"
def google_cloud_bigdataoss_version = "2.2.4"
-def google_cloud_pubsublite_version = "1.4.6"
+def google_cloud_pubsublite_version = "1.4.7"
def google_code_gson_version = "2.8.9"
def google_oauth_clients_version = "1.32.1"
// Try to keep grpc_version consistent with gRPC version in
google_cloud_platform_libraries_bom
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReader.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReader.java
index 402a37b..292d3e1 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReader.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReader.java
@@ -20,9 +20,6 @@ package org.apache.beam.sdk.io.gcp.pubsublite.internal;
import com.google.api.gax.rpc.ApiException;
import com.google.cloud.pubsublite.Offset;
-interface InitialOffsetReader extends AutoCloseable {
+interface InitialOffsetReader {
Offset read() throws ApiException;
-
- @Override
- void close();
}
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReaderImpl.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReaderImpl.java
index 5972b13..ddee9b7 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReaderImpl.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/InitialOffsetReaderImpl.java
@@ -18,6 +18,7 @@
package org.apache.beam.sdk.io.gcp.pubsublite.internal;
import static com.google.cloud.pubsublite.internal.ExtractStatus.toCanonical;
+import static java.util.concurrent.TimeUnit.MINUTES;
import com.google.api.gax.rpc.ApiException;
import com.google.cloud.pubsublite.Offset;
@@ -31,8 +32,9 @@ class InitialOffsetReaderImpl implements InitialOffsetReader {
private final SubscriptionPath subscription;
private final Partition partition;
- InitialOffsetReaderImpl(CursorClient client, SubscriptionPath subscription,
Partition partition) {
-this.client = client;
+ InitialOffsetReaderImpl(
+ CursorClient unownedCursorClient, SubscriptionPath subscription,
Partition partition) {
+this.client = unownedCursorClient;
this.subscription = subscription;
this.partition = partition;
}
@@ -40,15 +42,10 @@ class InitialOffsetReaderImpl implements
InitialOffsetReader {
@Override
public Offset read() throws ApiException {
try {
- Map results =
client.listPartitionCursors(subscription).get();
+
[beam] branch master updated: [BEAM-13571] Fix ClassNotFound exception in Flink tests
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0712b0a [BEAM-13571] Fix ClassNotFound exception in Flink tests
new 9ac567a Merge pull request #16375 from ibzib/flink-class1
0712b0a is described below
commit 0712b0a12c4885d5af9b78b7e4bd3bfa626defc1
Author: Kyle Weaver
AuthorDate: Tue Dec 28 09:15:37 2021 -0800
[BEAM-13571] Fix ClassNotFound exception in Flink tests
---
runners/flink/job-server/flink_job_server.gradle | 1 +
1 file changed, 1 insertion(+)
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
index 07ce308..1dcb837 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -80,6 +80,7 @@ dependencies {
implementation project(flinkRunnerProject)
permitUnusedDeclared project(flinkRunnerProject) // BEAM-11761
runtimeOnly group: "org.slf4j", name: "jcl-over-slf4j", version:
dependencies.create(project.library.java.slf4j_api).getVersion()
+ validatesPortableRunner project(path: flinkRunnerProject)
validatesPortableRunner project(path: flinkRunnerProject, configuration:
"testRuntimeMigration")
validatesPortableRunner project(path: ":sdks:java:core", configuration:
"shadowTest")
validatesPortableRunner project(path: ":runners:core-java", configuration:
"testRuntimeMigration")
[beam] branch master updated (9e07ab0 -> 8b50ba2)
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 9e07ab0 Merge pull request #16376 from ibzib/BEAM-13572
add 8b50ba2 [BEAM-13052] Add Pub/Sub Lite xlang transforms in python
(#15727)
No new revisions were added by this update.
Summary of changes:
.../internal/ExternalTransformConfig.java | 108 ++
.../internal/ExternalTransformRegistrarImpl.java} | 21 ++--
.../gcp/pubsublite/internal/ProtoFromBytes.java} | 45
.../io/gcp/pubsublite/internal/ProtoToBytes.java | 33 +++---
.../{tools => io/gcp/pubsublite}/__init__.py | 8 +-
.../apache_beam/io/gcp/pubsublite/external.py | 126 +
.../apache_beam/io/gcp/pubsublite/proto_api.py | 112 ++
sdks/python/setup.py | 1 +
8 files changed, 400 insertions(+), 54 deletions(-)
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ExternalTransformConfig.java
copy
sdks/java/io/{snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/crosslanguage/SnowflakeTransformRegistrar.java
=>
google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ExternalTransformRegistrarImpl.java}
(70%)
copy
sdks/java/{testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
=>
io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ProtoFromBytes.java}
(51%)
copy learning/katas/java/Triggers/Early Triggers/Early
Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java
=>
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/ProtoToBytes.java
(60%)
copy sdks/python/apache_beam/{tools => io/gcp/pubsublite}/__init__.py (86%)
create mode 100644 sdks/python/apache_beam/io/gcp/pubsublite/external.py
create mode 100644 sdks/python/apache_beam/io/gcp/pubsublite/proto_api.py
[beam] branch master updated: [BEAM-13498] [BEAM-13573] exclude new tests on Flink
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2d458bc [BEAM-13498] [BEAM-13573] exclude new tests on Flink
new 9e07ab0 Merge pull request #16376 from ibzib/BEAM-13572
2d458bc is described below
commit 2d458bc75aa2940531cb78578a9bf7a938f7b75a
Author: Kyle Weaver
AuthorDate: Tue Dec 28 12:29:20 2021 -0500
[BEAM-13498] [BEAM-13573] exclude new tests on Flink
---
runners/flink/flink_runner.gradle | 4
1 file changed, 4 insertions(+)
diff --git a/runners/flink/flink_runner.gradle
b/runners/flink/flink_runner.gradle
index 19204a5..c224852 100644
--- a/runners/flink/flink_runner.gradle
+++ b/runners/flink/flink_runner.gradle
@@ -237,6 +237,8 @@ def createValidatesRunnerTask(Map m) {
excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedPCollections'
excludeCategories
'org.apache.beam.sdk.testing.UsesUnboundedSplittableParDo'
excludeCategories 'org.apache.beam.sdk.testing.UsesTestStream'
+excludeCategories
'org.apache.beam.sdk.testing.UsesPerKeyOrderedDelivery'
+excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderInBundle'
}
filter {
// https://issues.apache.org/jira/browse/BEAM-12039
@@ -245,6 +247,8 @@ def createValidatesRunnerTask(Map m) {
excludeTestsMatching
'org.apache.beam.sdk.testing.TestStreamTest.testFirstElementLate'
// https://issues.apache.org/jira/browse/BEAM-12038
excludeTestsMatching
'org.apache.beam.sdk.testing.TestStreamTest.testLateDataAccumulating'
+// TODO(BEAM-13573)
+excludeTestsMatching
'org.apache.beam.sdk.transforms.ParDoTest$TimestampTests.testOnWindowTimestampSkew'
}
}
}
[beam] branch master updated (f2f1bdf -> acff5d1)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f2f1bdf Exclude UsesOnWindowExpiration by category from Dataflow v2 streaming (#16346) add acff5d1 [BEAM-13052] Increment pubsub python version and fix breakages. (#16126) No new revisions were added by this update. Summary of changes: .../examples/complete/game/game_stats_it_test.py | 7 +++-- .../examples/complete/game/leader_board_it_test.py | 7 +++-- .../streaming_wordcount_debugging_it_test.py | 14 + .../examples/streaming_wordcount_it_test.py| 14 + sdks/python/apache_beam/io/gcp/bigquery_test.py| 7 +++-- sdks/python/apache_beam/io/gcp/pubsub.py | 15 +- .../apache_beam/io/gcp/pubsub_integration_test.py | 21 +- .../apache_beam/io/gcp/pubsub_io_perf_test.py | 12 sdks/python/apache_beam/io/gcp/pubsub_test.py | 33 +- .../apache_beam/io/gcp/tests/pubsub_matcher.py | 5 ++-- sdks/python/apache_beam/io/gcp/tests/utils.py | 2 +- sdks/python/apache_beam/io/gcp/tests/utils_test.py | 8 -- ...low_exercise_streaming_metrics_pipeline_test.py | 14 + .../runners/direct/transform_evaluator.py | 15 +- sdks/python/apache_beam/testing/test_utils.py | 16 +++ sdks/python/apache_beam/testing/test_utils_test.py | 4 +-- sdks/python/apache_beam/utils/timestamp.py | 2 +- sdks/python/setup.py | 2 +- 18 files changed, 107 insertions(+), 91 deletions(-)
[beam] branch master updated: Exclude UsesOnWindowExpiration by category from Dataflow v2 streaming (#16346)
This is an automated email from the ASF dual-hosted git repository.
emilyye pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f2f1bdf Exclude UsesOnWindowExpiration by category from Dataflow v2
streaming (#16346)
f2f1bdf is described below
commit f2f1bdf624ca06d372f6ec227d9ff6637f11baf3
Author: emily
AuthorDate: Tue Dec 28 10:13:26 2021 -0800
Exclude UsesOnWindowExpiration by category from Dataflow v2 streaming
(#16346)
---
runners/google-cloud-dataflow-java/build.gradle | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index bec6de6..0387fe2 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -423,6 +423,7 @@ task validatesRunnerV2Streaming {
'org.apache.beam.sdk.testing.UsesCommittedMetrics',
'org.apache.beam.sdk.testing.UsesStrictTimerOrdering' /* BEAM-8543 */,
'org.apache.beam.sdk.testing.UsesTestStream',
+ 'org.apache.beam.sdk.testing.UsesOnWindowExpiration',
],
excludedTests: [
'org.apache.beam.sdk.transforms.windowing.WindowTest.testMergingCustomWindows',
@@ -436,11 +437,6 @@ task validatesRunnerV2Streaming {
'org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.testTriggeredTumble',
'org.apache.beam.sdk.transforms.ReshuffleTest.testReshuffleWithTimestampsStreaming',
-
'org.apache.beam.sdk.transforms.ParDoTest$OnWindowExpirationTests.testOnWindowExpirationSimpleBounded',
-
'org.apache.beam.sdk.transforms.ParDoTest$OnWindowExpirationTests.testOnWindowExpirationSimpleBoundedGlobal',
-
'org.apache.beam.sdk.transforms.ParDoTest$OnWindowExpirationTests.testOnWindowExpirationSimpleUnbounded',
-
'org.apache.beam.sdk.transforms.ParDoTest$OnWindowExpirationTests.testOnWindowExpirationSimpleUnboundedGlobal',
-
// TODO(BEAM-11858) reading a side input twice fails
'org.apache.beam.sdk.transforms.ParDoTest$MultipleInputsAndOutputTests.testSameSideInputReadTwice',
'org.apache.beam.sdk.transforms.CombineFnsTest.testComposedCombineWithContext',
[beam] branch master updated: [BEAM-13521] flink pvr batch precommit
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 1096956 [BEAM-13521] flink pvr batch precommit
new 5b06fa3 Merge pull request #16349 from ibzib/flink-pvr
1096956 is described below
commit 1096956c2ed9e379dc914d961d7b3e6131ed59ca
Author: Kyle Weaver
AuthorDate: Thu Dec 23 11:53:31 2021 -0800
[BEAM-13521] flink pvr batch precommit
---
...Java_PortableValidatesRunner_Flink_Batch.groovy | 44 --
...Java_PortableValidatesRunner_Flink_Batch.groovy | 40
2 files changed, 40 insertions(+), 44 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
deleted file mode 100644
index 9857320..000
---
a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-import CommonTestProperties
-import PostcommitJobBuilder
-
-// This job runs the suite of ValidatesRunner tests against the Flink runner.
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_PVR_Flink_Batch',
-'Run Java Flink PortableValidatesRunner Batch', 'Java Flink
PortableValidatesRunner Batch Tests', this) {
- description('Runs the Java PortableValidatesRunner suite on the Flink
runner.')
-
- // Set common parameters.
- commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 240)
-
- // Publish all test results to Jenkins
- publishers {
-archiveJunit('**/build/test-results/**/*.xml')
- }
-
- // Gradle goals for this job.
- steps {
-gradle {
- rootBuildScriptDir(commonJobProperties.checkoutDir)
-
tasks(":runners:flink:${CommonTestProperties.getFlinkVersion()}:job-server:validatesPortableRunnerBatch")
- commonJobProperties.setGradleSwitches(delegate)
-}
- }
-}
diff --git
a/.test-infra/jenkins/job_PreCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
b/.test-infra/jenkins/job_PreCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
new file mode 100644
index 000..8efb098
--- /dev/null
+++
b/.test-infra/jenkins/job_PreCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonTestProperties
+import PrecommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Flink runner.
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+scope: this,
+nameBase: 'Java_PVR_Flink_Batch',
+gradleTask:
":runners:flink:${CommonTestProperties.getFlinkVersion()}:job-server:validatesPortableRunnerBatch",
+timeoutMins: 240,
+triggerPathPatterns: [
+ '^sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/.*$',
+ '^runners/flink/.*$',
+ '^runners/java-fn-execution/.*$',
+],
+)
+builder.build {
+ previousNames('beam_PostCommit_Java_PVR_Flink_Batch')
+ // Publish all test results to Jenkins.
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+ }
+}
[beam] branch master updated: [BEAM-13526] Kafka.IO: make DeserializerProvider a public interface (#16339)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f8b5d01 [BEAM-13526] Kafka.IO: make DeserializerProvider a public
interface (#16339)
f8b5d01 is described below
commit f8b5d01077de9eee19a033b82494f7168c43ae94
Author: Arkadiusz Gasiński
AuthorDate: Tue Dec 28 10:27:54 2021 +0100
[BEAM-13526] Kafka.IO: make DeserializerProvider a public interface (#16339)
---
.../main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
index db3ccbe..08fa07f 100644
---
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
+++
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
@@ -27,7 +27,7 @@ import org.apache.kafka.common.serialization.Deserializer;
/** Provides a configured {@link Deserializer} instance and its associated
{@link Coder}. */
@Experimental(Kind.SOURCE_SINK)
-interface DeserializerProvider extends Serializable {
+public interface DeserializerProvider extends Serializable {
Deserializer getDeserializer(Map configs, boolean isKey);
