[beam] branch master updated (3de6bee -> cac6b50)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3de6bee Merge pull request #15544 from udim/master add dd37ac8 fixing test link add 853fa36 fix linter issues add cac6b50 Merge pull request #15504: Fixing test link in TestPipeline.java No new revisions were added by this update. Summary of changes: .../core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (0111cff -> 2254f09)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0111cff [BEAM-12906] Add a `dataframe` extra for installing a pandas version supported by the DataFrame API (#15528) add 0c87b49 [BEAM-12628] make useReflectApi default to true add 2254f09 Merge pull request #15616: [BEAM-12628] make AvroCoder#useReflectApi default to true No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/coders/AvroCoder.java | 24 +++--- .../org/apache/beam/sdk/coders/AvroCoderTest.java | 7 --- 2 files changed, 21 insertions(+), 10 deletions(-)
[beam] branch master updated (e79750d -> 49dd708)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e79750d [BEAM-12957] Add support for pyarrow 5.x (#15588) add 0aa25df Switching to innerText and removing alerts add 49dd708 Merge pull request #15624: [BEAM-12607] - Copy Code Snippet copies html tags No new revisions were added by this update. Summary of changes: website/www/site/static/js/copy-to-clipboard.js | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-)
[beam] branch master updated (ebf2aac -> 8e3c1de)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ebf2aac Merge pull request #15537 from [BEAM-12908] Add a sleep to the IT after topic creation to wait for propagation add 531b29c Fix 2.32.0 release notes. new 8e3c1de Merge pull request #15443 Fix 2.32.0 release notes The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #15443 Fix 2.32.0 release notes
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8e3c1deb105f179f6f109ed37ad5e29412914302 Merge: ebf2aac 531b29c Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Sep 22 11:58:03 2021 +0200 Merge pull request #15443 Fix 2.32.0 release notes CHANGES.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (b150ace -> 18fa2a1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b150ace [BEAM-13434] Bump google pubsublite on master. (#16265) add 4efb3d6 [adhoc] Fix guava imports in tests new 18fa2a1 Merge pull request #16270: [adhoc] Fix guava imports in tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 2 +- .../test/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #16270: [adhoc] Fix guava imports in tests
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 18fa2a1d81f91ab35b2c93f957d00b943eb0350e Merge: b150ace 4efb3d6 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Dec 17 10:33:43 2021 +0100 Merge pull request #16270: [adhoc] Fix guava imports in tests .../test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 2 +- .../test/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (67bcf1e -> cdc9065)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 67bcf1e Merge pull request #15973 from mosche/BEAM-13233-aws2-list-shards add cd74e9f [BEAM-13443] Avoid blocking put to Kinesis records queue to shutdown readers faster add cdc9065 Merge pull request #16286: [BEAM-13443] Avoid blocking put to Kinesis records queue to shutdown readers faster No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws2/kinesis/ShardReadersPool.java | 16 +--- .../org/apache/beam/sdk/io/kinesis/ShardReadersPool.java | 16 +--- 2 files changed, 26 insertions(+), 6 deletions(-)
[beam] branch master updated (6d35bf9 -> 6e735c1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6d35bf9 Merge pull request #16285 from [BEAM-13492][Playground] Update backend README file add d2bd03b [adhoc] Forbid to import guava and others from org.testcontainers.shaded add 6e735c1 Merge pull request #16341: [adhoc] Forbid to import guava and others from org.testcontainers.shaded No new revisions were added by this update. Summary of changes: sdks/java/build-tools/src/main/resources/beam/checkstyle.xml| 2 +- .../beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableIT.java | 2 +- .../sdk/extensions/sql/meta/provider/kafka/BeamKafkaTableJsonTest.java | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated: [BEAM-13526] Kafka.IO: make DeserializerProvider a public interface (#16339)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f8b5d01 [BEAM-13526] Kafka.IO: make DeserializerProvider a public
interface (#16339)
f8b5d01 is described below
commit f8b5d01077de9eee19a033b82494f7168c43ae94
Author: Arkadiusz Gasiński
AuthorDate: Tue Dec 28 10:27:54 2021 +0100
[BEAM-13526] Kafka.IO: make DeserializerProvider a public interface (#16339)
---
.../main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
index db3ccbe..08fa07f 100644
---
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
+++
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/DeserializerProvider.java
@@ -27,7 +27,7 @@ import org.apache.kafka.common.serialization.Deserializer;
/** Provides a configured {@link Deserializer} instance and its associated
{@link Coder}. */
@Experimental(Kind.SOURCE_SINK)
-interface DeserializerProvider extends Serializable {
+public interface DeserializerProvider extends Serializable {
Deserializer getDeserializer(Map configs, boolean isKey);
[beam] branch master updated (0ef6d4b -> 0d0278f)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0ef6d4b Merge pull request #16316 Benchmark and optimize Python RowCoder add 9e87d11 [BEAM-13494] Avoid draining Kinesis API limits & CPU when reading from empty or low throughput shard add 0d0278f Merge pull request #16288: [BEAM-13494] Avoid draining Kinesis API limits & CPU when reading from empty or low throughput shard No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 2 +- .../io/aws2/kinesis/RateLimitPolicyFactory.java| 76 .../aws2/kinesis/RateLimitPolicyFactoryTest.java | 81 ++ .../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 2 +- .../sdk/io/kinesis/RateLimitPolicyFactory.java | 76 .../sdk/io/kinesis/RateLimitPolicyFactoryTest.java | 64 +++-- 6 files changed, 295 insertions(+), 6 deletions(-) create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/RateLimitPolicyFactoryTest.java
[beam] branch master updated (c4fc398 -> 17b62ad)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c4fc398 Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source. add 17b62ad Merge pull request #16404: [BEAM-13586] Fix NPE in DefaultS3ClientBuilderFactory No new revisions were added by this update. Summary of changes: .../io/aws2/s3/DefaultS3ClientBuilderFactory.java | 27 --- .../aws2/s3/DefaultS3ClientBuilderFactoryTest.java | 92 ++ 2 files changed, 108 insertions(+), 11 deletions(-) create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/s3/DefaultS3ClientBuilderFactoryTest.java
[beam] branch master updated (7487bf0 -> 2cd5013)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7487bf0 [BEAM-13498, BEAM-13522] Sickbay broken test and test category for Spark (#16411) add df8b83b [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions new 2cd5013 Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/io/aws/options/AwsOptions.java | 17 +++-- .../org/apache/beam/sdk/io/aws2/options/AwsOptions.java | 17 +++-- 2 files changed, 30 insertions(+), 4 deletions(-)
[beam] 01/01: Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2cd50137ad229c3c33fdca1f391bd9f3bc18cffd Merge: 7487bf0 df8b83b Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Jan 3 15:32:29 2022 +0100 Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions .../org/apache/beam/sdk/io/aws/options/AwsOptions.java | 17 +++-- .../org/apache/beam/sdk/io/aws2/options/AwsOptions.java | 17 +++-- 2 files changed, 30 insertions(+), 4 deletions(-)
[beam] branch master updated (c9e8104 -> 0f06ad6)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c9e8104 Merge pull request #16336: [BEAM-13517] Unable to write nulls to columns with logical types add 4fabbda [BEAM-7559] Add integration test for DynamoDBIO (SDK v2) add 0f06ad6 Merge pull request #15965: [BEAM-7559] Add integration test for DynamoDBIO (SDK v2) No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 14 + .../beam/sdk/io/aws2/dynamodb/DynamoDBIOIT.java| 225 +++ .../sdk/io/aws2/dynamodb/DynamoDBIOReadTest.java | 24 ++ .../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 444 - .../sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java | 214 -- 5 files changed, 263 insertions(+), 658 deletions(-) create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOIT.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java
[beam] branch master updated (0f06ad6 -> b0556e5)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 0f06ad6 Merge pull request #15965: [BEAM-7559] Add integration test
for DynamoDBIO (SDK v2)
add bc66f6a [BEAM-7559] Add integration test for DynamoDBIO (SDK v1)
add b0556e5 Merge pull request #16242: [BEAM-7559] Add integration test
for DynamoDBIO (SDK v1)
No new revisions were added by this update.
Summary of changes:
.../sdk/io/aws/dynamodb/BasicDynamoDBProvider.java | 19 +-
.../org/apache/beam/sdk/io/aws/ITEnvironment.java | 14 +
.../beam/sdk/io/aws}/dynamodb/DynamoDBIOIT.java| 99 ---
.../sdk/io/aws/dynamodb/DynamoDBIOReadTest.java| 23 ++
.../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 305 -
.../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java | 172
6 files changed, 98 insertions(+), 534 deletions(-)
copy
sdks/java/io/{amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2 =>
amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws}/dynamodb/DynamoDBIOIT.java
(66%)
delete mode 100644
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java
delete mode 100644
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java
[beam] branch master updated (d8784ac -> fe456b7)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d8784ac Merge pull request #16390: [adhoc] Use predefined min/max combiners in SQS reader to gather stats add 52715c0 [BEAM-13009] Fix tests to ensure absence of duplicates only per request to avoid issues when running with low targetParellelism new fe456b7 Merge pull request #16367: [BEAM-13009] Fix tests to ensure absence of duplicates only per request The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/io/aws/dynamodb/DynamoDBIOWriteTest.java | 56 +++--- .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 54 +++-- 2 files changed, 78 insertions(+), 32 deletions(-)
[beam] 01/01: Merge pull request #16367: [BEAM-13009] Fix tests to ensure absence of duplicates only per request
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit fe456b79419d1a67ebf13d7d4b6695fa1aa6204d Merge: d8784ac 52715c0 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Dec 30 16:00:18 2021 +0100 Merge pull request #16367: [BEAM-13009] Fix tests to ensure absence of duplicates only per request .../sdk/io/aws/dynamodb/DynamoDBIOWriteTest.java | 56 +++--- .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 54 +++-- 2 files changed, 78 insertions(+), 32 deletions(-)
[beam] 01/01: Merge pull request #16390: [adhoc] Use predefined min/max combiners in SQS reader to gather stats
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d8784ac7889c6e33da54ec76ddab9eb54692c4ab Merge: 0af3689 e2189b3 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Dec 30 15:12:49 2021 +0100 Merge pull request #16390: [adhoc] Use predefined min/max combiners in SQS reader to gather stats .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 30 +++--- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 30 +++--- 2 files changed, 8 insertions(+), 52 deletions(-)
[beam] branch master updated (0af3689 -> d8784ac)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0af3689 [BEAM-13430] Re-enable checkerframework for the project excluding the two classes it started to crash on. (#16389) add e2189b3 [adhoc] Use predefined min/max combiners in SQS reader to gather stats new d8784ac Merge pull request #16390: [adhoc] Use predefined min/max combiners in SQS reader to gather stats The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 30 +++--- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 30 +++--- 2 files changed, 8 insertions(+), 52 deletions(-)
[beam] 01/01: Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source.
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c4fc3987f31a57a31d2081297fa639084b35d391 Merge: 2cd5013 0fd6504 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Jan 3 18:17:42 2022 +0100 Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source. .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 34 +++- .../beam/sdk/io/aws/sqs/SqsUnboundedSource.java| 20 +- .../sdk/io/aws/sqs/SqsUnboundedReaderTest.java | 2 +- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 45 -- .../beam/sdk/io/aws2/sqs/SqsUnboundedSource.java | 12 -- .../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java| 2 +- 6 files changed, 53 insertions(+), 62 deletions(-)
[beam] branch master updated (2cd5013 -> c4fc398)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2cd5013 Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions add 0fd6504 [BEAM-13509] Stop sharing SQS client between readers of same source. new c4fc398 Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 34 +++- .../beam/sdk/io/aws/sqs/SqsUnboundedSource.java| 20 +- .../sdk/io/aws/sqs/SqsUnboundedReaderTest.java | 2 +- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 45 -- .../beam/sdk/io/aws2/sqs/SqsUnboundedSource.java | 12 -- .../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java| 2 +- 6 files changed, 53 insertions(+), 62 deletions(-)
[beam] branch master updated (986cffa -> 692227d)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 986cffa Merge pull request #15935: [BEAM-8727] Bump software.amazon.awssdk to 2.17.77 add 120a4ef [BEAM-4868] Bump com.amazonaws to 1.12.106 new 692227d Merge pull request #15932: [BEAM-4868] Bump com.amazonaws to 1.12.106 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #15932: [BEAM-4868] Bump com.amazonaws to 1.12.106
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 692227d4d034e2c13a68e5a863f26f9a68579aec Merge: 986cffa 120a4ef Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Nov 12 15:48:20 2021 +0100 Merge pull request #15932: [BEAM-4868] Bump com.amazonaws to 1.12.106 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (692227d -> 4c0479b)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 692227d Merge pull request #15932: [BEAM-4868] Bump com.amazonaws to 1.12.106 add 4c0479b [BEAM-13147] Fix nullability issues in aws (SDK v1 + v2) for AwsModule (#15843) No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/aws/options/AwsModule.java | 181 +++-- .../beam/sdk/io/aws/options/AwsModuleTest.java | 32 ++-- .../apache/beam/sdk/io/aws2/options/AwsModule.java | 112 + .../beam/sdk/io/kinesis/serde/AwsModule.java | 19 +-- 4 files changed, 183 insertions(+), 161 deletions(-)
[beam] branch master updated (4351c49 -> 986cffa)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4351c49 [BEAM-13001] collect DoFn metrics for Combine (#15911) add e462607 [BEAM-8727] Bump software.amazon.awssdk to 2.17.77 add 986cffa Merge pull request #15935: [BEAM-8727] Bump software.amazon.awssdk to 2.17.77 No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- .../aws2/kinesis/DynamicCheckpointGeneratorTest.java | 18 +- 2 files changed, 10 insertions(+), 10 deletions(-)
[beam] branch master updated (be22467 -> cf2a715)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from be22467 [BEAM-13052] Implement ProtoPlusCoder and add it to the
default options (#15817)
add b655cbe [BEAM-13009] Bugfix: Prevent dataloss in DynamoDBIO write if
deduplication keys are not set (aws2)
add cf2a715 Merge pull request #15895: [BEAM-13009] Bugfix: Prevent
dataloss in DynamoDBIO write if deduplication keys are not set (aws SDK v2)
No new revisions were added by this update.
Summary of changes:
.../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 52 ++--
.../sdk/io/aws2/dynamodb/DynamoDBIOReadTest.java | 199
.../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 38 ++-
.../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 343 +
...Mock.java => StaticDynamoDBClientProvider.java} | 34 +-
5 files changed, 622 insertions(+), 44 deletions(-)
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOReadTest.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/{DynamoDbClientProviderMock.java
=> StaticDynamoDBClientProvider.java} (52%)
[beam] branch master updated (cf2a715 -> 61c9d1e)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from cf2a715 Merge pull request #15895: [BEAM-13009] Bugfix: Prevent
dataloss in DynamoDBIO write if deduplication keys are not set (aws SDK v2)
add 82443b6 [BEAM-13009] Bugfix: Prevent dataloss in DynamoDBIO write if
deduplication keys are not set (aws sdk v1)
add 61c9d1e Merge pull request #15898: [BEAM-13009] Bugfix: Prevent
dataloss in DynamoDBIO write if deduplication keys are not set (aws SDK v1)
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 2 +
.../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 54 +-
.../sdk/io/aws}/dynamodb/DynamoDBIOReadTest.java | 58 ---
.../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 70 +++---
.../sdk/io/aws}/dynamodb/DynamoDBIOWriteTest.java | 85 +++---
...iderMock.java => StaticAwsClientsProvider.java} | 37 +++---
6 files changed, 166 insertions(+), 140 deletions(-)
copy
sdks/java/io/{amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2 =>
amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws}/dynamodb/DynamoDBIOReadTest.java
(74%)
copy
sdks/java/io/{amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2 =>
amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws}/dynamodb/DynamoDBIOWriteTest.java
(77%)
rename
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/{AwsClientsProviderMock.java
=> StaticAwsClientsProvider.java} (54%)
[beam] branch master updated (59bd9a0 -> 2654502)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 59bd9a0 Merge pull request #15853 from [BEAM-13145][Playground] Validation error add 0bb6c14 [BEAM-12070] ParquetIO: use splittable reading by default add 2654502 Merge pull request #15811: [BEAM-12070] Make ParquetIO splittable by default No new revisions were added by this update. Summary of changes: CHANGES.md | 2 + .../org/apache/beam/sdk/io/parquet/ParquetIO.java | 91 -- .../apache/beam/sdk/io/parquet/ParquetIOTest.java | 23 -- 3 files changed, 102 insertions(+), 14 deletions(-)
[beam] branch master updated (643825b -> 67ef9ac)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 643825b Merge pull request #15801 from [BEAM-12970][Playground]
Implement gRPC API - GetCompileOutput
add 67ef9ac [BEAM-8958] Use AWS credentials provider with
BasicKinesisProvider (kinesis v1) (#15816)
No new revisions were added by this update.
Summary of changes:
sdks/java/io/kinesis/build.gradle | 3 +
.../beam/sdk/io/kinesis/BasicKinesisProvider.java | 46 +++--
.../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 140 +++--
.../beam/sdk/io/kinesis/serde/AwsModule.java | 222 +
.../io/kinesis/serde}/AwsSerializableUtils.java| 12 +-
.../beam/sdk/io/kinesis/serde}/package-info.java | 4 +-
.../kinesis/BasicKinesisClientProviderTest.java| 19 +-
.../beam/sdk/io/kinesis/KinesisIOReadTest.java | 138 +
.../beam/sdk/io/kinesis/KinesisIOWriteTest.java| 130
.../beam/sdk/io/kinesis/serde/AwsModuleTest.java | 174
.../io/kinesis/serde/AwsSerializableUtilsTest.java | 174
11 files changed, 1014 insertions(+), 48 deletions(-)
create mode 100644
sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/serde/AwsModule.java
copy
sdks/java/io/{amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options
=>
kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/serde}/AwsSerializableUtils.java
(79%)
copy
sdks/java/io/{amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs
=>
kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/serde}/package-info.java
(90%)
copy
sdks/java/io/{amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2 =>
kinesis/src/test/java/org/apache/beam/sdk/io}/kinesis/BasicKinesisClientProviderTest.java
(72%)
create mode 100644
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOReadTest.java
create mode 100644
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOWriteTest.java
create mode 100644
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/serde/AwsModuleTest.java
create mode 100644
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/serde/AwsSerializableUtilsTest.java
[beam] branch master updated (e892496 -> 290d994)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e892496 Merge pull request #15294 from [BEAM-11986] Spanner write metric add af925d8 [BEAM-13138] Update code to retrieve container endpoint add 290d994 Merge pull request #15823: [BEAM-13138] Update code to retrieve container endpoint in DynamoDBIOTest (aws2) No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java | 9 - 1 file changed, 9 deletions(-)
[beam] branch master updated (0b9fb71 -> 6220e47)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 0b9fb71 Merge pull request #15606 from ihji/BEAM-12978
add c4f0434 [BEAM-8958] Use AWS credentials provider with
BasicKinesisProvider (AWS sdk v2) rather than supporting basic credentials only.
add 6220e47 Merge pull request #15788: [BEAM-8958] Use AWS credentials
provider with BasicKinesisProvider (AWS sdk v2)
No new revisions were added by this update.
Summary of changes:
.../sdk/io/aws2/kinesis/BasicKinesisProvider.java | 41 +++---
.../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 48 +--
.../BasicKinesisClientProviderTest.java} | 21 ++---
.../sdk/io/aws2/kinesis/KinesisIOReadTest.java | 92 ++
4 files changed, 177 insertions(+), 25 deletions(-)
copy
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/{dynamodb/BasicDynamoDbClientProviderTest.java
=> kinesis/BasicKinesisClientProviderTest.java} (74%)
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIOReadTest.java
[beam] branch master updated (cb5913a -> 90a48ee)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cb5913a [BEAM-13081] Fixes a compatible issue of decoding null-value bitmap between JVM coder and Python Coder (#15829) add 90a48ee [adhoc] Speedup slow tests for AWS IO modules (#15899) No new revisions were added by this update. Summary of changes: sdks/java/io/amazon-web-services/build.gradle| 1 + .../java/org/apache/beam/sdk/io/aws/sqs/EmbeddedSqsServer.java | 6 -- sdks/java/io/amazon-web-services2/build.gradle | 3 +-- .../java/org/apache/beam/sdk/io/aws2/sqs/EmbeddedSqsServer.java | 9 - sdks/java/io/kinesis/build.gradle| 3 +-- 5 files changed, 11 insertions(+), 11 deletions(-)
[beam] branch master updated (56a4e58 -> f37b6c7)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 56a4e58 [BEAM-12566] Implement set_axis for DataFrame and Series (#15773) add f37b6c7 [BEAM-13192] Fix buggy retry tests for AWS SnsIO (#15910) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/aws/sns/SnsIO.java | 23 + .../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 20 ++- .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 24 + .../beam/sdk/io/aws2/sns/SnsClientMockErrors.java | 12 +++-- .../beam/sdk/io/aws2/sns/SnsClientMockSuccess.java | 27 +++ .../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 30 +++--- 6 files changed, 78 insertions(+), 58 deletions(-)
[beam] branch master updated (0faa3b9 -> 0262dd6)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0faa3b9 Merge pull request #15885 from ibzib/wc-twister2 add ec99053 [BEAM-11440] Add integration test for KinesisIO using Localstack on javaPostCommit add aa5f06f Bump localstack container version for more robust version with respect to HTTP/HTTPS and simplify KinesisIOIT in kinesis module. add 0831b75 Bump localstack container version for DynamoDBIOTest to version used in KinesisIOIT add b60f732 Apply format add aa67b0f Apply format add f829123 Use vendored guava deps add 0262dd6 Merge pull request #15841: [BEAM-11440] Add integration test for KinesisIO using Localstack (aws2) No new revisions were added by this update. Summary of changes: build.gradle.kts | 1 + .../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java | 2 +- .../beam/sdk/io/aws2/kinesis/KinesisIOIT.java | 136 +++-- .../sdk/io/aws2/kinesis/KinesisTestOptions.java| 25 .../apache/beam/sdk/io/kinesis/KinesisIOIT.java| 115 + .../beam/sdk/io/kinesis/KinesisTestOptions.java| 6 + 6 files changed, 217 insertions(+), 68 deletions(-)
[beam] branch master updated (79496b2 -> 84d57ca)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 79496b2 Merge pull request #15913: [BEAM-2791] remove spammy log statement add 2a94534 [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse new 84d57ca Merge pull request #15914: [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/io/parquet/ParquetIOTest.java | 28 ++ 1 file changed, 28 insertions(+)
[beam] 01/01: Merge pull request #15914: [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 84d57ca36294f1bc29d22a4ac4282a45846c Merge: 79496b2 2a94534 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Nov 8 15:44:49 2021 +0100 Merge pull request #15914: [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse .../apache/beam/sdk/io/parquet/ParquetIOTest.java | 28 ++ 1 file changed, 28 insertions(+)
[beam] branch master updated (13a8746 -> 918f345)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 13a8746 Merge pull request #16136 from [BEAM-13365] [Playground] Add Pipelines options text field to the frontend add 918f345 [BEAM-13244] Support STS Assume role credentials provider for AWS SDK v2 (#16116) No new revisions were added by this update. Summary of changes: sdks/java/io/amazon-web-services2/build.gradle | 2 +- .../apache/beam/sdk/io/aws2/options/AwsModule.java | 62 +-- .../beam/sdk/io/aws2/options/AwsModuleTest.java| 179 + 3 files changed, 164 insertions(+), 79 deletions(-)
[beam] branch master updated (6e2d39f -> 3985d8d)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6e2d39f [BEAM-13015] Plumb the cache through contexts and transform executors. (#16188) add 2547dbe [BEAM-13441] Use quiet delete for S3 batch deletes. In quiet mode only errors will be returned in the service response. add 3985d8d Merge pull request #16197: [BEAM-13441] Use S3 quiet mode for deletes No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java | 3 ++- .../src/main/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystem.java | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (3b74958 -> 9fce7f6)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3b74958 Merge pull request #16168: [BEAM-13410][BEAM-11788] Integration test for S3Filesystem using Localstack add 4600532 [BEAM-13434] Bump up Apache log4j2 vulnerability to 2.15.0 add 9fce7f6 Merge pull request #16191: [BEAM-13434] Bump up Apache log4j2 to 2.15.0 due to the vulnerability No new revisions were added by this update. Summary of changes: sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/build.gradle | 2 +- .../java/io/elasticsearch-tests/elasticsearch-tests-common/build.gradle | 2 +- sdks/java/io/hcatalog/build.gradle | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-)
[beam] branch master updated (d842e4e -> 3b74958)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d842e4e Merge pull request #16194: [BEAM-13171] Fix Duration ambiguity add f57973b [BEAM-13410][BEAM-11788] Integration test for S3Filesystem using Localstack. This overrides the AWS service endpoint and invalidates [BEAM-11788]. add 3b74958 Merge pull request #16168: [BEAM-13410][BEAM-11788] Integration test for S3Filesystem using Localstack No new revisions were added by this update. Summary of changes: build.gradle.kts | 1 + sdks/java/io/amazon-web-services/build.gradle | 3 + .../org/apache/beam/sdk/io/aws/ITEnvironment.java | 143 .../apache/beam/sdk/io/aws/s3/S3FileSystemIT.java | 133 ++ .../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 148 + .../apache/beam/sdk/io/aws2/s3/S3FileSystemIT.java | 119 + 6 files changed, 547 insertions(+) create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/ITEnvironment.java create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemIT.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/ITEnvironment.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/s3/S3FileSystemIT.java
[beam] branch master updated (626e7a9 -> e2ac013)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 626e7a9 Merge pull request #16208 from [BEAM-13442][Playground] Filepath to log error add f969a7b [BEAM-13445] Correctly set data limit when flushing S3 upload buffer and avoid temporary copy to array when writing source buffer to upload buffer add e2ac013 Merge pull request #16205: [BEAM-13445] Correctly set data limit when flushing S3 upload buffer No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws/s3/S3WritableByteChannel.java | 25 ++ .../sdk/io/aws/s3/S3WritableByteChannelTest.java | 39 +- .../beam/sdk/io/aws2/s3/S3WritableByteChannel.java | 25 ++ .../apache/beam/sdk/io/aws2/s3/S3FileSystemIT.java | 18 +- .../sdk/io/aws2/s3/S3WritableByteChannelTest.java | 17 ++ 5 files changed, 72 insertions(+), 52 deletions(-)
[beam] branch master updated (ffb18c7 -> 124a8c2)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ffb18c7 [BEAM-13015] Allow decoding a set of elements until we hit the block boundary. (#16220) add 124a8c2 [BEAM-13159] Add Redis Stream (XADD) Write Support (#15858) No new revisions were added by this update. Summary of changes: CHANGES.md | 1 + sdks/java/io/redis/build.gradle| 2 +- .../java/org/apache/beam/sdk/io/redis/RedisIO.java | 204 + .../org/apache/beam/sdk/io/redis/RedisIOTest.java | 79 4 files changed, 285 insertions(+), 1 deletion(-)
[beam] branch master updated (fbe386e -> 665a542)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fbe386e Merge pull request #15964: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v2) add 9b212bc [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v1) new 665a542 Merge pull request #16239: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v1) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 128 - .../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 16 ++- .../sdk/io/aws/dynamodb/DynamoDBIOWriteTest.java | 98 +--- 3 files changed, 168 insertions(+), 74 deletions(-)
[beam] 01/01: Merge pull request #16239: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 665a542e277e27df11830bb0785fc51293139e77 Merge: fbe386e 9b212bc Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Dec 15 13:00:45 2021 +0100 Merge pull request #16239: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v1) .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 128 - .../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 16 ++- .../sdk/io/aws/dynamodb/DynamoDBIOWriteTest.java | 98 +--- 3 files changed, 168 insertions(+), 74 deletions(-)
[beam] branch master updated (665a542 -> bc82f6f)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 665a542 Merge pull request #16239: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v1) add bc82f6f [BEAM-13341] Serialize reflection parameter in AvroCoder cloud object (#16080) No new revisions were added by this update. Summary of changes: .../dataflow/util/AvroCoderCloudObjectTranslator.java| 5 - .../beam/runners/dataflow/util/CloudObjectsTest.java | 13 - .../main/java/org/apache/beam/sdk/coders/AvroCoder.java | 15 --- .../java/org/apache/beam/sdk/coders/AvroCoderTest.java | 16 +++- 4 files changed, 43 insertions(+), 6 deletions(-)
[beam] branch master updated (ba49cd9 -> d842e4e)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ba49cd9 [BEAM-13402] Add another workaround for https://github.com/googleapis/gax-java/issues/1577 and bump Pub/Sub Lite version to pick up another similar workaround (#16190) add 9cc8ca6 [BEAM-13171] Fix Duration ambiguity add d842e4e Merge pull request #16194: [BEAM-13171] Fix Duration ambiguity No new revisions were added by this update. Summary of changes: .../test/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[beam] branch master updated (431ee4d -> ca50055)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 431ee4d Merge pull request #16131 from lukecwik/beam13015.4 add ca50055 [BEAM-13236] Properly close kinesis producer on teardown (#15955) No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 87 ++ 1 file changed, 55 insertions(+), 32 deletions(-)
[beam] branch master updated (f9b0d6d -> fbe386e)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f9b0d6d [BEAM-13434] Bump log4j to 2.16.0. (#16237) add 62c34bd [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success with unprocessed items remaining. new fbe386e Merge pull request #15964: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v2) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 124 + .../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 13 ++- .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 101 ++--- 3 files changed, 172 insertions(+), 66 deletions(-)
[beam] 01/01: Merge pull request #15964: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v2)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit fbe386e8275e52d6367b6c455930a408ebf224e9 Merge: f9b0d6d 62c34bd Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Dec 15 10:28:22 2021 +0100 Merge pull request #15964: [BEAM-13209] Fix DynamoDBIO.write to properly handle partial success (SDK v2) .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 124 + .../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 13 ++- .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 101 ++--- 3 files changed, 172 insertions(+), 66 deletions(-)
[beam] branch master updated (39c40f2 -> ab20d7a)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 39c40f2 Merge pull request #16203 from [BEAM-12164] Add Spanner Change Stream Mappers add 55d16a6 [adhoc] Fix BigTableIO description add ab20d7a Merge pull request #16453: [adhoc] Fix BigTableIO description No new revisions were added by this update. Summary of changes: website/www/site/data/io_matrix.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (e208b9d -> 7681258)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e208b9d Merge pull request #16433 from danthev/py_overflow_fix add b4ccfc6 [BEAM-8727] Bump software.amazon.awssdk to 2.17.106 add 7681258 Merge pull request #16456: [BEAM-8727] Bump software.amazon.awssdk to 2.17.106 No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (1688ac1 -> a5d5b64)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1688ac1 [BEAM-13616] Initial files for vendored gRPC 1.43.2 (#16460) add a2db81d [BEAM-13638] Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public new a5d5b64 Merge pull request #16491: [BEAM-13638] Fix visibility of fields in SqsMessage (AWS Sdk v2) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 1 + .../apache/beam/sdk/io/aws2/sqs/SqsMessage.java| 36 +- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 28 ++--- 3 files changed, 25 insertions(+), 40 deletions(-)
[beam] 01/01: Merge pull request #16491: [BEAM-13638] Fix visibility of fields in SqsMessage (AWS Sdk v2)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a5d5b64b4b0c140965f4d25c374a7b61872ebeb3 Merge: 1688ac1 a2db81d Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Jan 12 15:11:09 2022 +0100 Merge pull request #16491: [BEAM-13638] Fix visibility of fields in SqsMessage (AWS Sdk v2) CHANGES.md | 1 + .../apache/beam/sdk/io/aws2/sqs/SqsMessage.java| 36 +- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 28 ++--- 3 files changed, 25 insertions(+), 40 deletions(-)
[beam] branch master updated (5201237 -> 5ad247b)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 5201237 Merge pull request #16479: [adhoc] Run spotlessApply on java
examples to fix master
add 562346b [BEAM-13243][BEAM-8374] Add support for missing
PublishResponse fields in SnsIO.Write (AWS SDK v2)
add 5ad247b Merge pull request #16128: [BEAM-13243][BEAM-8374] Add
support for missing PublishResponse fields in SnsIO.Write (AWS SDK v2)
No new revisions were added by this update.
Summary of changes:
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 1 +
sdks/java/io/amazon-web-services2/build.gradle | 1 +
.../apache/beam/sdk/io/aws2/coders/AwsCoders.java | 135 +
.../beam/sdk/io/aws2}/coders/package-info.java | 2 +-
.../beam/sdk/io/aws2/sns/PublishResponseCoder.java | 48
.../sdk/io/aws2/sns/PublishResponseCoders.java | 129
.../sdk/io/aws2/sns/SnsCoderProviderRegistrar.java | 3 +-
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 42 ++-
.../beam/sdk/io/aws2/coders/AwsCodersTest.java | 69 +++
.../sdk/io/aws2/sns/PublishResponseCodersTest.java | 92 ++
.../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 40 ++
11 files changed, 510 insertions(+), 52 deletions(-)
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/coders/AwsCoders.java
copy
sdks/java/io/{amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws =>
amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2}/coders/package-info.java
(95%)
delete mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoders.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/coders/AwsCodersTest.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCodersTest.java
[beam] branch master updated (5f3c17b -> 5201237)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5f3c17b Loosen typing extensions bound #16466 add a263858 [adhoc] Run spotlessApply on java examples to fix master add 5201237 Merge pull request #16479: [adhoc] Run spotlessApply on java examples to fix master No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/examples/cookbook/DistinctExample.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (8c2e1fd -> 181403a)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8c2e1fd Merge pull request #16507: [BEAM-13137] Fixes ES utest size flakiness with _flush api and index.store.stats_refresh_interval=0 add ed10912 [BEAM-8806] Integration test for SqsIO using Localstack new 181403a Merge pull request #16468: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v2) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 2 +- .../org/apache/beam/sdk/io/aws2/sqs/SqsIOIT.java | 121 + 2 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/SqsIOIT.java
[beam] 01/01: Merge pull request #16468: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v2)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 181403ab0d22f6fed46cde9f0d7458f745b26a27 Merge: 8c2e1fd ed10912 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Jan 17 12:12:55 2022 +0100 Merge pull request #16468: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v2) .../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 2 +- .../org/apache/beam/sdk/io/aws2/sqs/SqsIOIT.java | 121 + 2 files changed, 122 insertions(+), 1 deletion(-)
[beam] branch master updated (7681258 -> 2d3afbf)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7681258 Merge pull request #16456: [BEAM-8727] Bump software.amazon.awssdk to 2.17.106 add 8d01b1e [BEAM-4868] Bump com.amazonaws to 1.12.135 add 2d3afbf Merge pull request #16454: [BEAM-4868] Bump com.amazonaws to 1.12.135 No new revisions were added by this update. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- .../org/apache/beam/sdk/io/kinesis/AmazonKinesisMock.java | 11 +-- 2 files changed, 10 insertions(+), 3 deletions(-)
[beam] branch master updated (2d3afbf -> 928da54)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2d3afbf Merge pull request #16454: [BEAM-4868] Bump com.amazonaws to 1.12.135 add 928da54 [BEAM-12092] Bump jedis to version 4.0.1 (#16287) No new revisions were added by this update. Summary of changes: CHANGES.md | 2 + sdks/java/io/redis/build.gradle| 2 +- .../java/org/apache/beam/sdk/io/redis/RedisIO.java | 74 -- .../org/apache/beam/sdk/io/redis/RedisIOTest.java | 10 +-- 4 files changed, 49 insertions(+), 39 deletions(-)
[beam] branch master updated (150e311 -> 31332b8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 150e311 Merge pull request #16840: Add tests for processing time continuation trigger add 1e05bc9 [BEAM-13854] Document casting trick for Avro value serializer in KafkaIO add 31332b8 Merge pull request #16858: [BEAM-13854] Document casting trick for Avro value serializer in KafkaIO No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 13 + 1 file changed, 13 insertions(+)
[beam] branch master updated (2aa4da0 -> 80d63c0)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2aa4da0 [BEAM-13857] Add K:V flags for expansion service jars and addresses to Go ITs. (#16908) add 133f569 BEAM-14011 fix s3 filesystem multipart copy add 80d63c0 Merge pull request #16987: [BEAM-14011] fix s3 filesystem multipart copy No new revisions were added by this update. Summary of changes: CHANGES.md | 2 +- .../apache/beam/sdk/io/aws/s3/S3FileSystem.java| 10 +-- .../apache/beam/sdk/io/aws2/s3/S3FileSystem.java | 32 + .../beam/sdk/io/aws2/s3/S3FileSystemTest.java | 82 -- 4 files changed, 87 insertions(+), 39 deletions(-)
[beam] branch master updated (eea81f4 -> c1db5fe)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from eea81f4 Merge pull request #17073: [BEAM-13981] Remove Spark Runner specific code for event logging add 48dcb7a [BEAM-2766] Support null key/values in HadoopFormatIO add 32aed30 [BEAM-2766] Fix checkstyle add c1db5fe Merge pull request #17074: [BEAM-2766] Support null key/values in HadoopFormatIO No new revisions were added by this update. Summary of changes: .../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 2 +- .../io/hadoop/format/HadoopFormatIOReadTest.java | 49 ++ 2 files changed, 50 insertions(+), 1 deletion(-)
[beam] branch master updated (05f9b45 -> 5345214)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 05f9b45 [BEAM-14051] Add a warning to gradle.properties that many Beam modules override the project version value defined there. (#17022) add db6e34d [adhoc] Update owners for aws modules add 5345214 Merge pull request #17083: [adhoc] Update owners for aws modules No new revisions were added by this update. Summary of changes: sdks/java/io/amazon-web-services/OWNERS | 1 + sdks/java/io/amazon-web-services2/OWNERS | 1 + sdks/java/io/kinesis/OWNERS | 1 + 3 files changed, 3 insertions(+)
[beam] branch master updated (ae06110 -> 7bc28f7)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ae06110 Merge pull request #16077: [BEAM-13175] Add KinesisIO.write for AWS SDK v2. add 9215856 [BEAM-13175] Document changes to IOs in amazon-web-services2 for AWS Java SDK v2 including new write support in KinesisIO. add 7bc28f7 Merge pull request #17082: [BEAM-13175] Document changes to IOs in amazon-web-services2 for AWS Java SDK v2 No new revisions were added by this update. Summary of changes: CHANGES.md | 7 +++ 1 file changed, 7 insertions(+)
[beam] branch master updated (f779a3f -> ae06110)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f779a3f Merge pull request #17054 from [BEAM-14075] [SnowflakeIO] Change a output type in WriteToSnowflake add d324e9e [BEAM-13175] Add KinesisIO.write for AWS SDK v2. add ae06110 Merge pull request #16077: [BEAM-13175] Add KinesisIO.write for AWS SDK v2. No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- .../src/main/resources/beam/suppressions.xml | 1 + sdks/java/io/amazon-web-services2/build.gradle | 41 +- .../apache/beam/sdk/io/aws2/common/ClientPool.java | 123 .../io/aws2/kinesis/AsyncPutRecordsHandler.java| 271 .../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 768 - .../sdk/io/aws2/kinesis/KinesisPartitioner.java| 87 +++ .../sdk/io/aws2/kinesis/RecordsAggregator.java | 182 + .../beam/sdk/io/aws2/common/ClientPoolTest.java| 146 .../aws2/kinesis/AsyncPutRecordsHandlerTest.java | 192 ++ .../beam/sdk/io/aws2/kinesis/KinesisIOIT.java | 69 +- .../sdk/io/aws2/kinesis/KinesisIOWriteTest.java| 450 .../sdk/io/aws2/kinesis/PutRecordsHelpers.java | 96 +++ .../sdk/io/aws2/kinesis/RecordsAggregatorTest.java | 159 + .../beam/sdk/io/aws2/options/AwsModuleTest.java| 12 +- 15 files changed, 2517 insertions(+), 82 deletions(-) create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientPool.java create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/AsyncPutRecordsHandler.java create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisPartitioner.java create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/RecordsAggregator.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/common/ClientPoolTest.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/AsyncPutRecordsHandlerTest.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIOWriteTest.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/PutRecordsHelpers.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/RecordsAggregatorTest.java
[beam] branch master updated (87dba3a -> adcde0e)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 87dba3a Case studies page improvements (#16702)
add 53f5a3c [BEAM-13563] Introducing common AWS ClientBuilderFactory to
unify configuration of sync / async AWS clients across all IOs (AWS Sdk v2).
[BEAM-13206] Additionally, retry configuration is pushed down into clients to
delegate retries to the AWS Sdk.
add adcde0e Merge pull request #16760: [BEAM-13563] Introducing common
AWS ClientBuilderFactory (AWS Sdk v2)
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 3 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 1 +
sdks/java/io/amazon-web-services2/build.gradle | 2 +-
.../sdk/io/aws2/common/ClientBuilderFactory.java | 250 +++
.../sdk/io/aws2/common/ClientConfiguration.java| 189 +
.../io/aws2/common/HttpClientConfiguration.java| 170
.../sdk/io/aws2/common/RetryConfiguration.java | 118 ++
.../io/aws2/{kinesis => common}/package-info.java | 4 +-
.../aws2/dynamodb/BasicDynamoDbClientProvider.java | 48 +--
.../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 464 +++--
.../io/aws2/dynamodb/DynamoDbClientProvider.java | 5 +
.../sdk/io/aws2/kinesis/AWSClientsProvider.java| 6 +-
.../sdk/io/aws2/kinesis/BasicKinesisProvider.java | 61 +--
.../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 276
.../beam/sdk/io/aws2/kinesis/KinesisSource.java| 29 +-
.../io/aws2/kinesis/SimplifiedKinesisClient.java | 7 +-
.../apache/beam/sdk/io/aws2/options/AwsModule.java | 161 +++
.../beam/sdk/io/aws2/options/AwsOptions.java | 149 ---
.../sdk/io/aws2/options/AwsSerializableUtils.java | 35 +-
.../io/aws2/s3/DefaultS3ClientBuilderFactory.java | 24 +-
.../io/aws2/sns/BasicSnsAsyncClientProvider.java | 39 +-
.../sdk/io/aws2/sns/BasicSnsClientProvider.java| 48 +--
.../beam/sdk/io/aws2/sns/SnsClientProvider.java| 5 +
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 314 ++
.../sdk/io/aws2/sqs/BasicSqsClientProvider.java| 48 +--
.../beam/sdk/io/aws2/sqs/SqsClientProvider.java| 5 +
.../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java | 219 ++
.../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 26 +-
.../beam/sdk/io/aws2/sqs/SqsUnboundedSource.java | 3 +-
.../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 9 +-
.../beam/sdk/io/aws2/MockClientBuilderFactory.java | 96 +
.../io/aws2/common/ClientBuilderFactoryTest.java | 398 ++
.../ClientConfigurationTest.java} | 37 +-
.../sdk/io/aws2/common/RetryConfigurationTest.java | 50 +++
.../beam/sdk/io/aws2/dynamodb/DynamoDBIOIT.java| 17 +-
.../sdk/io/aws2/dynamodb/DynamoDBIOReadTest.java | 126 --
.../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 164
.../sdk/io/aws2/kinesis/AmazonKinesisMock.java | 397 --
.../beam/sdk/io/aws2/kinesis/KinesisIOIT.java | 9 +-
.../sdk/io/aws2/kinesis/KinesisIOReadTest.java | 234 +--
.../sdk/io/aws2/kinesis/KinesisMockReadTest.java | 110 -
.../sdk/io/aws2/kinesis/KinesisServiceMock.java| 84
.../beam/sdk/io/aws2/options/AwsModuleTest.java| 66 +--
.../beam/sdk/io/aws2/options/AwsOptionsTest.java | 73 ++--
.../aws2/s3/DefaultS3ClientBuilderFactoryTest.java | 24 +-
.../beam/sdk/io/aws2/s3/S3ResourceIdTest.java | 3 +-
.../apache/beam/sdk/io/aws2/s3/S3TestUtils.java| 7 +-
.../org/apache/beam/sdk/io/aws2/sns/SnsIOIT.java | 19 +-
.../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 171 +---
.../sdk/io/aws2/sns/StaticSnsClientProvider.java | 34 --
.../org/apache/beam/sdk/io/aws2/sqs/SqsIOIT.java | 15 +-
.../apache/beam/sdk/io/aws2/sqs/SqsIOReadTest.java | 164
.../org/apache/beam/sdk/io/aws2/sqs/SqsIOTest.java | 88
.../beam/sdk/io/aws2/sqs/SqsIOWriteTest.java | 108 +
.../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java| 19 +-
55 files changed, 3090 insertions(+), 2141 deletions(-)
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientBuilderFactory.java
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientConfiguration.java
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/HttpClientConfiguration.java
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/RetryConfiguration.java
copy
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/{kinesis
=> common}/package-info.java (88%)
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/be
[beam] branch master updated (84353a7 -> 89fa0aa)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 84353a7 Merge pull request #16757 from [BEAM-13655] [Playground] Persist the dark/light mode selection add 1150fc6 Update 2.36.0 blog post to mention ARM64 support add 89fa0aa Merge pull request #16839: Update 2.36.0 blog post to mention ARM64 support No new revisions were added by this update. Summary of changes: website/www/site/content/en/blog/beam-2.36.0.md | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (096aeef -> f7de08e)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 096aeef [BEAM-14122] Upgrade pip-licenses dependency (#17132)
add ca2d91e [adhoc] Move aws IT tests to testing package according to
best practices
new f7de08e Merge pull request #17118: [adhoc] Move aws IT tests to
testing package according to best practices
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 2 +-
.../io/aws2/dynamodb/{ => testing}/DynamoDBIOIT.java | 5 +++--
.../sdk/io/aws2/kinesis/{ => testing}/KinesisIOIT.java | 6 --
.../sdk/io/aws2/s3/{ => testing}/S3FileSystemIT.java | 4 ++--
.../beam/sdk/io/aws2/sns/{ => testing}/SnsIOIT.java| 18 +-
.../beam/sdk/io/aws2/sqs/{ => testing}/SqsIOIT.java| 17 -
6 files changed, 43 insertions(+), 9 deletions(-)
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/{
=> testing}/DynamoDBIOIT.java (97%)
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/{
=> testing}/KinesisIOIT.java (96%)
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/s3/{
=> testing}/S3FileSystemIT.java (97%)
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/{
=> testing}/SnsIOIT.java (90%)
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/{
=> testing}/SqsIOIT.java (87%)
[beam] 01/01: Merge pull request #17118: [adhoc] Move aws IT tests to testing package according to best practices
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit f7de08eb86b77794195e74f90763703ce7cb88d7
Merge: 096aeef ca2d91e
Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com>
AuthorDate: Mon Mar 21 13:59:16 2022 +0100
Merge pull request #17118: [adhoc] Move aws IT tests to testing package
according to best practices
.../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 2 +-
.../io/aws2/dynamodb/{ => testing}/DynamoDBIOIT.java | 5 +++--
.../sdk/io/aws2/kinesis/{ => testing}/KinesisIOIT.java | 6 --
.../sdk/io/aws2/s3/{ => testing}/S3FileSystemIT.java | 4 ++--
.../beam/sdk/io/aws2/sns/{ => testing}/SnsIOIT.java| 18 +-
.../beam/sdk/io/aws2/sqs/{ => testing}/SqsIOIT.java| 17 -
6 files changed, 43 insertions(+), 9 deletions(-)
[beam] 01/01: Merge pull request #17140: [BEAM-13232] Close clients properly in KinesisSource.
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1d6aacf18176230f94efa3f7b6a82b9edcc088de Merge: 7ef6560 cdf99be Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Mar 23 11:37:19 2022 +0100 Merge pull request #17140: [BEAM-13232] Close clients properly in KinesisSource. .../beam/sdk/io/aws2/kinesis/KinesisReader.java| 10 ++- .../beam/sdk/io/aws2/kinesis/KinesisSource.java| 30 + .../io/aws2/kinesis/SimplifiedKinesisClient.java | 75 -- .../aws2/kinesis/SimplifiedKinesisClientTest.java | 47 ++ 4 files changed, 116 insertions(+), 46 deletions(-)
[beam] branch master updated (7ef6560 -> 1d6aacf)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7ef6560 [BEAM-13015] Clarify ownership of the list for state caching across bundles (#16982) add cdf99be [BEAM-13232] Close clients properly in KinesisSource. Also use lazy init for AWS clients to avoid unnecessary allocations for some use cases. new 1d6aacf Merge pull request #17140: [BEAM-13232] Close clients properly in KinesisSource. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/aws2/kinesis/KinesisReader.java| 10 ++- .../beam/sdk/io/aws2/kinesis/KinesisSource.java| 30 + .../io/aws2/kinesis/SimplifiedKinesisClient.java | 75 -- .../aws2/kinesis/SimplifiedKinesisClientTest.java | 47 ++ 4 files changed, 116 insertions(+), 46 deletions(-)
[beam] branch master updated (b9c4919 -> cc0b2c5)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b9c4919 Edit license script for Java, add manual licenses for xz (#16692) add 1663d6d [BEAM-13563] Restructure Kinesis Source for Aws 2 internally to prepare for ClientBuilderFactory add cc0b2c5 Merge pull request #16657: [BEAM-13563] Restructure Kinesis Source for AWS 2 No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 15 ++-- .../beam/sdk/io/aws2/kinesis/KinesisReader.java| 71 +-- .../beam/sdk/io/aws2/kinesis/KinesisSource.java| 100 - .../beam/sdk/io/aws2/kinesis/ShardReadersPool.java | 23 ++--- .../sdk/io/aws2/kinesis/KinesisReaderTest.java | 22 +++-- .../sdk/io/aws2/kinesis/ShardReadersPoolTest.java | 73 +-- 6 files changed, 89 insertions(+), 215 deletions(-)
[beam] branch master updated (318ce2a -> f79363c)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 318ce2a Merge pull request #16712: [BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1) add 6bb9e857 [BEAM-13203] Deprecate SnsIO.writeAsync for AWS Sdk v2 due to risk of data loss. add f79363c Merge pull request #16711: [BEAM-13203] Deprecate SnsIO.writeAsync for AWS Sdk v2 due to risk of data loss. No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 51 +++--- .../beam/sdk/io/aws2/sns/MockSnsAsyncClient.java | 15 --- .../io/aws2/sns/MockSnsAsyncExceptionClient.java | 8 ++-- .../beam/sdk/io/aws2/sns/SnsIOWriteTest.java | 34 ++- 4 files changed, 43 insertions(+), 65 deletions(-)
[beam] branch master updated (aab61e9 -> 318ce2a)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aab61e9 Merge pull request #16730: Fixed CSS for Case study page add 9d0cbbe [BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1). add 318ce2a Merge pull request #16712: [BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1) No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/aws/options/AwsOptions.java | 28 -- 1 file changed, 28 deletions(-)
[beam] branch master updated (368baf6 -> fd73461)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 368baf6 [BEAM-13537] Fix NPE in kafkatopubsub example (#16625) add e19742e [adhoc] Test S3Options and AwsOptions for Sdk v2 add fd73461 Merge pull request #16628: [adhoc] Test S3Options and AwsOptions for Sdk v2 No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws2/options/AwsOptionsTest.java | 161 + .../beam/sdk/io/aws2/options/S3OptionsTest.java| 119 +++ 2 files changed, 280 insertions(+) create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsOptionsTest.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/S3OptionsTest.java
[beam] branch master updated (56e5c2c -> 8c0fa37)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 56e5c2c [BEAM-13647] Use role for Go worker binary. (#16729) add 13a8001 [BEAM-13840] Fix usage of legacy rawtypes in AWS modules add 8c0fa37 Merge pull request #16756: [BEAM-13840] Fix usage of legacy rawtypes in AWS modules No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 29 +++--- .../java/org/apache/beam/sdk/io/aws/sns/SnsIO.java | 7 ++ .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 29 +++--- .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 5 ++-- 4 files changed, 34 insertions(+), 36 deletions(-)
[beam] branch master updated (8c0fa37 -> d1b8e56)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8c0fa37 Merge pull request #16756: [BEAM-13840] Fix usage of legacy rawtypes in AWS modules add c691d12 [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 new d1b8e56 Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d1b8e569fd651975f08823a3db49dbee56d491b5 Merge: 8c0fa37 c691d12 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Feb 8 11:49:27 2022 +0100 Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (6e050bc -> 963c04a)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6e050bc Merge pull request #16768 from Output successful rows from BQ Streaming Inserts add d192b62 [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2). add 963c04a Merge pull request #16714: [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/sdk/io/aws2/options/S3Options.java | 7 +++ .../apache/beam/sdk/io/aws2/s3/S3FileSystemConfiguration.java| 9 + .../org/apache/beam/sdk/io/aws2/s3/S3WritableByteChannel.java| 1 + .../test/java/org/apache/beam/sdk/io/aws2/s3/S3TestUtils.java| 7 ++- .../apache/beam/sdk/io/aws2/s3/S3WritableByteChannelTest.java| 5 + 5 files changed, 28 insertions(+), 1 deletion(-)
[beam] branch master updated (3c6fb0d -> 8bf68ad)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3c6fb0d [BEAM-13193] Allow BeamFnDataOutboundObserver to flush elements. (#16778) add 256e338 [adhoc] Remove remaining usage of Powermock from aws2. new 8bf68ad Merge pull request #16777: [adhoc] Remove remaining usage of Powermock from aws2. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/amazon-web-services2/build.gradle | 2 - .../sdk/io/aws2/kinesis/WatermarkPolicyTest.java | 58 +++--- 2 files changed, 28 insertions(+), 32 deletions(-)
[beam] 01/01: Merge pull request #16777: [adhoc] Remove remaining usage of Powermock from aws2.
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8bf68ad5023c30cd56ec63d2084017b4a547fe48 Merge: 3c6fb0d 256e338 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Feb 9 11:20:12 2022 +0100 Merge pull request #16777: [adhoc] Remove remaining usage of Powermock from aws2. sdks/java/io/amazon-web-services2/build.gradle | 2 - .../sdk/io/aws2/kinesis/WatermarkPolicyTest.java | 58 +++--- 2 files changed, 28 insertions(+), 32 deletions(-)
[beam] branch master updated (fee1528 -> 0e057fd)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fee1528 Minor: Add 2.38.0 section to CHANGES.md (#16804) add bcd3913 BEAM-13159 Update embedded-redis dependency add 93d7bc9 address comments add 0e057fd Merge pull request #16798: [BEAM-13159] Update embedded-redis dependency No new revisions were added by this update. Summary of changes: sdks/java/io/redis/build.gradle | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-)
[beam] branch master updated (6c9c208 -> e638c11)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 6c9c208 [BEAM-10206] Add Go Vet to Github Actions (#16612)
add 08a5651 Generalize S3FileSystem to support multiple URI schemes.
add e638c11 Merge pull request #16607: [BEAM-13245] Generalize
S3FileSystem to support multiple URI schemes.
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 1 +
.../s3/DefaultS3FileSystemSchemeRegistrar.java | 7 +-
.../apache/beam/sdk/io/aws2/s3/S3FileSystem.java | 88 ++---
.../sdk/io/aws2}/s3/S3FileSystemConfiguration.java | 54 ++-
.../beam/sdk/io/aws2/s3/S3FileSystemRegistrar.java | 22 +-
.../io/aws2}/s3/S3FileSystemSchemeRegistrar.java | 2 +-
.../io/aws2/s3/S3ReadableSeekableByteChannel.java | 13 +-
.../apache/beam/sdk/io/aws2/s3/S3ResourceId.java | 60 +++-
.../beam/sdk/io/aws2/s3/S3WritableByteChannel.java | 41 ++-
.../beam/sdk/io/aws2/s3/S3FileSystemTest.java | 383 +
.../beam/sdk/io/aws2/s3/S3ResourceIdTest.java | 92 -
.../apache/beam/sdk/io/aws2/s3/S3TestUtils.java| 77 -
.../sdk/io/aws2/s3/S3WritableByteChannelTest.java | 89 -
13 files changed, 706 insertions(+), 223 deletions(-)
copy
sdks/java/io/{amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws =>
amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2}/s3/DefaultS3FileSystemSchemeRegistrar.java
(89%)
copy
sdks/java/io/{amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws =>
amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2}/s3/S3FileSystemConfiguration.java
(64%)
copy
sdks/java/io/{amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws =>
amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2}/s3/S3FileSystemSchemeRegistrar.java
(98%)
[beam] branch master updated (3c8cd42 -> a3bf36e)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3c8cd42 [BEAM-12572] Run java examples on multiple runners (#16450) add cdd6f71 [BEAM-13400] JDBC IO does not support UUID and JSONB PostgreSQL types and OTHER JDBC types in general new a3bf36e Merge pull request #16284: [BEAM-13400] JdbcIO should support UUID and JSONB PostgreSQL types and OTHER JDBC types in general The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/schemas/logicaltypes/UuidLogicalType.java | 80 +++ .../sdk/schemas/logicaltypes/LogicalTypesTest.java | 16 +++ .../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 13 ++ .../org/apache/beam/sdk/io/jdbc/LogicalTypes.java | 9 ++ .../org/apache/beam/sdk/io/jdbc/SchemaUtil.java| 34 - .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 37 +- .../apache/beam/sdk/io/jdbc/OtherJdbcTypesIT.java | 147 + 7 files changed, 330 insertions(+), 6 deletions(-) create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/UuidLogicalType.java create mode 100644 sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/OtherJdbcTypesIT.java
[beam] 01/01: Merge pull request #16284: [BEAM-13400] JdbcIO should support UUID and JSONB PostgreSQL types and OTHER JDBC types in general
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a3bf36eae08a7e31ec862674180466166b9cf501 Merge: 3c8cd42 cdd6f71 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Jan 14 18:24:58 2022 +0100 Merge pull request #16284: [BEAM-13400] JdbcIO should support UUID and JSONB PostgreSQL types and OTHER JDBC types in general .../sdk/schemas/logicaltypes/UuidLogicalType.java | 80 +++ .../sdk/schemas/logicaltypes/LogicalTypesTest.java | 16 +++ .../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 13 ++ .../org/apache/beam/sdk/io/jdbc/LogicalTypes.java | 9 ++ .../org/apache/beam/sdk/io/jdbc/SchemaUtil.java| 34 - .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 37 +- .../apache/beam/sdk/io/jdbc/OtherJdbcTypesIT.java | 147 + 7 files changed, 330 insertions(+), 6 deletions(-)
[beam] branch master updated (5187008 -> b3e9c7d)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 5187008 Merge pull request #16480: [BEAM-13631] Add deterministic SQS
message coder to fix reading from SQS in batch mode (AWS Sdk v1)
add 66be723 [BEAM-8806] Integration test for SqsIO
new b3e9c7d Merge pull request #16481: [BEAM-8806] Integration test for
SqsIO using Localstack (AWS Sdk v1)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/beam/sdk/io/aws/ITEnvironment.java | 2 +-
.../org/apache/beam/sdk/io/aws}/sqs/SqsIOIT.java | 39 +-
2 files changed, 16 insertions(+), 25 deletions(-)
copy
sdks/java/io/{amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2 =>
amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws}/sqs/SqsIOIT.java
(75%)
[beam] 01/01: Merge pull request #16481: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit b3e9c7d5fc2cbf6308f526768c79f11f40d206a1 Merge: 5187008 66be723 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Jan 18 13:23:06 2022 +0100 Merge pull request #16481: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v1) .../org/apache/beam/sdk/io/aws/ITEnvironment.java | 2 +- .../org/apache/beam/sdk/io/aws/sqs/SqsIOIT.java| 112 + 2 files changed, 113 insertions(+), 1 deletion(-)
[beam] branch master updated (1838347 -> 5187008)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1838347 [BEAM-13616] Update com.google.cloud:libraries-bom to 24.2.0 (#16509) add db28e19 [BEAM-13631] Add deterministic SQS message coder to fix reading from SQS in batch mode add 5187008 Merge pull request #16480: [BEAM-13631] Add deterministic SQS message coder to fix reading from SQS in batch mode (AWS Sdk v1) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java | 24 ++- .../beam/sdk/io/aws/sqs/SqsMessageCoder.java | 82 ++ .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 38 -- .../beam/sdk/io/aws/sqs/SqsUnboundedSource.java| 9 ++- .../beam/sdk/io/aws/sqs/SqsMessageCoderTest.java | 62 .../sdk/io/aws/sqs/SqsUnboundedReaderTest.java | 6 +- .../sdk/io/aws/sqs/SqsUnboundedSourceTest.java | 3 +- 7 files changed, 192 insertions(+), 32 deletions(-) create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsMessageCoder.java create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsMessageCoderTest.java
[beam] branch master updated (737be25 -> b521b3f)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 737be25 [BEAM-13685] Enable users to specify cache directory under
Interactive Beam options
add 976b14c [BEAM-13653] Make SnsIO.write topicArn optional. If provided,
validate it and force usage on the Sns publish request.
add b521b3f Merge pull request #16503: [BEAM-13653] Fix usage of
SnsIO.write topicArn (AWS Sdk v2)
No new revisions were added by this update.
Summary of changes:
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 96 ++-
.../apache/beam/sdk/io/aws2/StaticSupplier.java} | 36 ++--
.../dynamodb/StaticDynamoDBClientProvider.java | 32 +---
.../beam/sdk/io/aws2/sns/SnsClientMockErrors.java | 52 --
.../beam/sdk/io/aws2/sns/SnsClientMockSuccess.java | 58 ---
.../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 189 ++---
.../sdk/io/aws2/sns/StaticSnsClientProvider.java} | 21 ++-
.../org/apache/beam/sdk/io/aws2/sqs/SqsIOTest.java | 2 +-
.../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java| 4 +-
.../sdk/io/aws2/sqs/SqsUnboundedSourceTest.java| 2 +-
...viderMock.java => StaticSqsClientProvider.java} | 18 +-
11 files changed, 230 insertions(+), 280 deletions(-)
copy
sdks/java/{extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/testkit/accumulators/LongHistogram.java
=>
io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/StaticSupplier.java}
(53%)
delete mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
delete mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
copy
sdks/java/io/amazon-web-services2/src/{main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
=> test/java/org/apache/beam/sdk/io/aws2/sns/StaticSnsClientProvider.java}
(66%)
rename
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/{SqsClientProviderMock.java
=> StaticSqsClientProvider.java} (69%)
[beam] branch master updated (81006dd -> 41d585f)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 81006dd Merge pull request #16594 from [BEAM-13710][Playground] Update the Apache Beam privacy policy with Playground note add 7b06fa7 [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2) new 41d585f Merge pull request #16595З: [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/io/aws/dynamodb/AwsClientsProvider.java| 8 +- .../sdk/io/aws/dynamodb/BasicDynamoDBProvider.java | 15 -- .../beam/sdk/io/aws/sns/AwsClientsProvider.java| 8 +- .../beam/sdk/io/aws/sns/BasicSnsProvider.java | 16 +-- .../org/apache/beam/sdk/io/aws/ITEnvironment.java | 31 ++-- .../io/aws/dynamodb/StaticAwsClientsProvider.java | 6 - .../apache/beam/sdk/io/aws/s3/S3FileSystemIT.java | 12 +- .../org/apache/beam/sdk/io/aws/sns/SnsIOIT.java| 159 + .../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 6 - .../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 31 ++-- .../org/apache/beam/sdk/io/aws2/sns/SnsIOIT.java | 158 11 files changed, 359 insertions(+), 91 deletions(-) create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/SnsIOIT.java create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOIT.java
[beam] 01/01: Merge pull request #16595З: [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 41d585f82b10195f758d14e3a54076ea1f05aa75 Merge: 81006dd 7b06fa7 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Jan 26 12:52:13 2022 +0100 Merge pull request #16595З: [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2) .../sdk/io/aws/dynamodb/AwsClientsProvider.java| 8 +- .../sdk/io/aws/dynamodb/BasicDynamoDBProvider.java | 15 -- .../beam/sdk/io/aws/sns/AwsClientsProvider.java| 8 +- .../beam/sdk/io/aws/sns/BasicSnsProvider.java | 16 +-- .../org/apache/beam/sdk/io/aws/ITEnvironment.java | 31 ++-- .../io/aws/dynamodb/StaticAwsClientsProvider.java | 6 - .../apache/beam/sdk/io/aws/s3/S3FileSystemIT.java | 12 +- .../org/apache/beam/sdk/io/aws/sns/SnsIOIT.java| 159 + .../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 6 - .../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 31 ++-- .../org/apache/beam/sdk/io/aws2/sns/SnsIOIT.java | 158 11 files changed, 359 insertions(+), 91 deletions(-)
[beam] branch master updated (04a6125 -> ed2f1fb)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 04a6125 [BEAM-7928] Allow users to specify worker disk type for Dataflow runner (#16622) add f556c60 [BEAM-13746] Fix deserialization of SSECustomerKey for AWS Sdk v2 add ed2f1fb Merge pull request #16623: [BEAM-13746] Fix deserialization of SSECustomerKey for AWS Sdk v2 No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/aws2/options/AwsModule.java | 33 -- .../apache/beam/sdk/io/aws2/s3/SSECustomerKey.java | 6 ++-- .../beam/sdk/io/aws2/options/AwsModuleTest.java| 11 3 files changed, 19 insertions(+), 31 deletions(-)
[beam] 01/01: Merge pull request #27991: Refactor ConvertHelpers in "core" to not depend on Avro
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8037b06b613e2226aefbccd3078e9229a346d2d3 Merge: 61f0184c528 96f3dfba4ce Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Sep 7 14:00:03 2023 +0200 Merge pull request #27991: Refactor ConvertHelpers in "core" to not depend on Avro .../beam/sdk/schemas/utils/ConvertHelpers.java | 34 ++ .../utils/RowSchemaInformationProvider.java| 41 + .../schemas/utils/SchemaInformationProvider.java | 32 + .../utils/AvroSchemaInformationProvider.java | 42 ++ 4 files changed, 134 insertions(+), 15 deletions(-)
[beam] branch master updated (61f0184c528 -> 8037b06b613)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 61f0184c528 Retag 2.50.0 RC2 to publish multi-arch containers (#28343)
add 888fe9128d2 Refactor ConvertHelpers in "core" to not depend on Avro
add 96f3dfba4ce Address review comments
new 8037b06b613 Merge pull request #27991: Refactor ConvertHelpers in
"core" to not depend on Avro
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../beam/sdk/schemas/utils/ConvertHelpers.java | 34 --
.../utils/RowSchemaInformationProvider.java} | 30 ++-
.../SchemaInformationProvider.java}| 20 ++---
.../utils/AvroSchemaInformationProvider.java} | 30 ++-
4 files changed, 62 insertions(+), 52 deletions(-)
copy
sdks/java/{io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderProviderRegistrar.java
=>
core/src/main/java/org/apache/beam/sdk/schemas/utils/RowSchemaInformationProvider.java}
(53%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{io/payloads/PayloadSerializerProvider.java
=> utils/SchemaInformationProvider.java} (65%)
copy
sdks/java/{io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java
=>
extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroSchemaInformationProvider.java}
(50%)
[beam] 01/01: Merge pull request #28250: [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 967e366e1beef0d1d7064a7092a9aedd8d96deb6 Merge: 741e8c66b0d 0a6a6210f3e Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Sep 6 18:37:48 2023 +0200 Merge pull request #28250: [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries sdks/java/testing/tpcds/build.gradle | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (741e8c66b0d -> 967e366e1be)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 741e8c66b0d Better error messages on missing or malformed yaml providers. (#28311) add 0a6a6210f3e [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries (closes #28249) new 967e366e1be Merge pull request #28250: [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/testing/tpcds/build.gradle | 1 + 1 file changed, 1 insertion(+)
(beam) branch master updated (f3636f32b3f -> 4f8e83d1ad6)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from f3636f32b3f A couple small fixes to the pypi deploy action (#29321) add 26acfaefc8c [Spark Runner] Add Spark 3.5.0 version for compatibility tests add 4f8e83d1ad6 Merge pull request #29327: [Spark Runner] Add Spark 3.5.0 version for compatibility tests No new revisions were added by this update. Summary of changes: runners/spark/3/build.gradle | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (9efdf6be61e -> 0a560fc09d8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9efdf6be61e Fix SpannerChangeStreamErrorTest.java and stop disabling tests (#28751) add a9684c5563d TPC-DS: Fix CSV format add 0a560fc09d8 Merge pull request #28819: TPC-DS: Fix CSV format No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java| 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (695dd4116ef -> 5fd1245d5d1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 695dd4116ef Fix comment phrase and name (#29052) add 0850a6c9523 Remove Avro-related code from Java SDK "core" module add 5fd1245d5d1 Merge pull request #27851: Remove Avro-related code from Java SDK "core" module No new revisions were added by this update. Summary of changes: CHANGES.md | 10 +- sdks/java/core/build.gradle|6 +- .../java/org/apache/beam/sdk/coders/AvroCoder.java | 820 .../apache/beam/sdk/coders/AvroGenericCoder.java | 40 - .../main/java/org/apache/beam/sdk/io/AvroIO.java | 2031 .../apache/beam/sdk/io/AvroSchemaIOProvider.java | 157 -- .../main/java/org/apache/beam/sdk/io/AvroSink.java | 167 -- .../java/org/apache/beam/sdk/io/AvroSource.java| 773 .../beam/sdk/io/ConstantAvroDestination.java | 156 -- .../org/apache/beam/sdk/io/CountingSource.java | 30 +- .../beam/sdk/io/DynamicAvroDestinations.java | 61 - .../beam/sdk/io/SerializableAvroCodecFactory.java | 119 -- .../apache/beam/sdk/schemas/AvroRecordSchema.java | 64 - .../org/apache/beam/sdk/schemas/io/Providers.java | 27 +- .../io/payloads/AvroPayloadSerializerProvider.java | 47 - .../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 149 -- .../apache/beam/sdk/schemas/utils/AvroUtils.java | 1396 -- .../org/apache/beam/sdk/coders/AvroCoderTest.java | 1106 --- .../apache/beam/sdk/coders/AvroCoderTestPojo.java | 51 - .../apache/beam/sdk/coders/CoderRegistryTest.java |4 +- .../apache/beam/sdk/coders/DefaultCoderTest.java |6 +- .../apache/beam/sdk/coders/MockDefaultCoder.java} | 42 +- .../java/org/apache/beam/sdk/io/AvroIOTest.java| 1627 .../beam/sdk/io/AvroSchemaIOProviderTest.java | 173 -- .../org/apache/beam/sdk/io/AvroSourceTest.java | 843 .../test/java/org/apache/beam/sdk/io/ReadTest.java |3 +- .../sdk/io/SerializableAvroCodecFactoryTest.java | 93 - .../apache/beam/sdk/schemas/AvroSchemaTest.java| 496 - .../apache/beam/sdk/schemas/SchemaCoderTest.java | 55 - .../io/AvroPayloadSerializerProviderTest.java | 64 - .../beam/sdk/schemas/transforms/ConvertTest.java | 14 - .../beam/sdk/schemas/utils/AvroGenerators.java | 220 --- .../beam/sdk/schemas/utils/AvroUtilsTest.java | 914 - .../beam/sdk/schemas/utils/TestJavaBeans.java | 12 +- .../extensions/avro/coders/DefaultCoderTest.java | 61 - sdks/java/extensions/timeseries/build.gradle |1 + sdks/java/extensions/zetasketch/build.gradle |1 + sdks/java/harness/build.gradle |6 + sdks/java/io/clickhouse/build.gradle |1 + sdks/java/io/contextualtextio/build.gradle |1 + .../pubsub/PubsubReadSchemaTransformProvider.java |2 +- .../pubsub/PubsubWriteSchemaTransformProvider.java |2 +- .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java |2 +- sdks/java/io/kafka/build.gradle|2 +- .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java |2 +- .../test/unit/read/SnowflakeIOReadTest.java|2 +- sdks/java/io/thrift/build.gradle |1 + 47 files changed, 97 insertions(+), 11763 deletions(-) delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroGenericCoder.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSchemaIOProvider.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/ConstantAvroDestination.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicAvroDestinations.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/payloads/AvroPayloadSerializerProvider.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java delete mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java delete mode 100644 sdks/java/core/src/test/java/org/ap
