(beam) 01/01: Merge pull request #31711: Set SchemaCoder for key in WithKeys transform
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bf8afb8d58083f3372ab826eb83402b69db72e95 Merge: 3b8ddda10a0 bacb9ec48b5 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Jul 2 10:03:34 2024 +0200 Merge pull request #31711: Set SchemaCoder for key in WithKeys transform .../org/apache/beam/sdk/transforms/WithKeys.java | 17 +++ .../apache/beam/sdk/transforms/WithKeysTest.java | 52 ++ 2 files changed, 69 insertions(+)
(beam) branch master updated (3b8ddda10a0 -> bf8afb8d580)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 3b8ddda10a0 Solace Read connector: adding Basic Authentication support (#31541) add bacb9ec48b5 Set SchemaCoder for key in WithKeys transform new bf8afb8d580 Merge pull request #31711: Set SchemaCoder for key in WithKeys transform The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/transforms/WithKeys.java | 17 +++ .../apache/beam/sdk/transforms/WithKeysTest.java | 52 ++ 2 files changed, 69 insertions(+)
(beam) 01/01: Merge pull request #31464: Updating Commons Compress to 1.26.2
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit f410b71c076dc4d7b75fc7b018bbebce8c35782c Merge: b50ad0fe8fc 6911a1af720 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri May 31 14:20:25 2024 +0200 Merge pull request #31464: Updating Commons Compress to 1.26.2 .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-)
(beam) branch master updated (b50ad0fe8fc -> f410b71c076)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b50ad0fe8fc [ManagedIO] pass underlying transform URN as an annotation (#31398) add 6911a1af720 Updating Commons Compress to 1.26.2 new f410b71c076 Merge pull request #31464: Updating Commons Compress to 1.26.2 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-)
(beam) branch master updated (5fb4db31994 -> e1ce548cea7)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 5fb4db31994 Bump actions/upload-artifact from 3 to 4 (#29777) add 860bc30a110 Add SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow add e1ce548cea7 Merge pull request #29781: Add SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-)
(beam) 01/01: Merge pull request #29716:Add InfluxDB credentials as env variables
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8ecfaa64fa81a536665153973049bfb4e0e732c7 Merge: eff6c203991 4431479fd90 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Dec 12 14:33:23 2023 +0100 Merge pull request #29716:Add InfluxDB credentials as env variables .github/workflows/beam_Java_JMH.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Direct.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Flink.yml| 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Spark.yml| 2 ++ .github/workflows/beam_PostCommit_Java_Tpcds_Dataflow.yml | 4 +++- .github/workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 2 ++ 10 files changed, 21 insertions(+), 1 deletion(-)
(beam) branch master updated (eff6c203991 -> 8ecfaa64fa8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from eff6c203991 Bump actions/setup-java from 3 to 4 (#29573) add 4431479fd90 Add InfluxDB credentials as env variables where they are missing new 8ecfaa64fa8 Merge pull request #29716:Add InfluxDB credentials as env variables The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/beam_Java_JMH.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Direct.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Flink.yml| 2 ++ .github/workflows/beam_PostCommit_Java_Nexmark_Spark.yml| 2 ++ .github/workflows/beam_PostCommit_Java_Tpcds_Dataflow.yml | 4 +++- .github/workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 ++ .github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 2 ++ 10 files changed, 21 insertions(+), 1 deletion(-)
(beam) branch aromanenko-dev-patch-1 created (now 5542d5927c4)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch aromanenko-dev-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git at 5542d5927c4 Update beam_PostCommit_Java_Tpcds_Spark.yml This branch includes the following new commits: new 5542d5927c4 Update beam_PostCommit_Java_Tpcds_Spark.yml The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Update beam_PostCommit_Java_Tpcds_Spark.yml
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch aromanenko-dev-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 5542d5927c48716ae6af70778e12167f834ad497
Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com>
AuthorDate: Tue Dec 12 12:46:04 2023 +0100
Update beam_PostCommit_Java_Tpcds_Spark.yml
---
.github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml
b/.github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml
index 370c9c406d4..580cfc83132 100644
--- a/.github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml
+++ b/.github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml
@@ -54,6 +54,8 @@ env:
--dataDirectory=gs://beam-tpcds/datasets/parquet/nonpartitioned
--resultsDirectory=gs://beam-tpcds/results/spark3-rdd/
--tpcParallel=1
+ INFLUXDB_USER: ${{ secrets.INFLUXDB_USER }}
+ INFLUXDB_USER_PASSWORD: ${{ secrets.INFLUXDB_USER_PASSWORD }}
tpcdsBigQueryArgs: |
--bigQueryTable=tpcds
--bigQueryDataset=tpcds
(beam) branch master updated (8ac8b20bcd1 -> a59fe16d619)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 8ac8b20bcd1 Gcsio migration (#29360) add 481b434dccc [changes.md] #29451 notice about deprecation of Euphoria DSL new a59fe16d619 Merge pull request #29454: [changes.md] #29451 notice about deprecation of Euphoria DSL The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) 01/01: Merge pull request #29454: [changes.md] #29451 notice about deprecation of Euphoria DSL
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a59fe16d619a91b12571c4f01217dbb11ab0f659 Merge: 8ac8b20bcd1 481b434dccc Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Nov 16 17:02:06 2023 +0100 Merge pull request #29454: [changes.md] #29451 notice about deprecation of Euphoria DSL CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) 01/01: Merge pull request #29330: TPC-DS query processing should fail if output is empty
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 38db0e58438456e26d4480d22819e592c4e9ccbe Merge: 7fabc1267a0 570f41ffca9 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Nov 14 15:52:52 2023 +0100 Merge pull request #29330: TPC-DS query processing should fail if output is empty .../apache/beam/sdk/tpcds/SqlTransformRunner.java | 18 +++ .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 36 ++ 2 files changed, 54 insertions(+)
(beam) branch master updated (7fabc1267a0 -> 38db0e58438)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 7fabc1267a0 Fix the test 'test_user_agent_passed' without skipping it (#29422) add 570f41ffca9 TPC-DS query processing should fail if output is empty new 38db0e58438 Merge pull request #29330: TPC-DS query processing should fail if output is empty The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/beam/sdk/tpcds/SqlTransformRunner.java | 18 +++ .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 36 ++ 2 files changed, 54 insertions(+)
(beam) branch master updated (f3636f32b3f -> 4f8e83d1ad6)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from f3636f32b3f A couple small fixes to the pypi deploy action (#29321) add 26acfaefc8c [Spark Runner] Add Spark 3.5.0 version for compatibility tests add 4f8e83d1ad6 Merge pull request #29327: [Spark Runner] Add Spark 3.5.0 version for compatibility tests No new revisions were added by this update. Summary of changes: runners/spark/3/build.gradle | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (695dd4116ef -> 5fd1245d5d1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 695dd4116ef Fix comment phrase and name (#29052) add 0850a6c9523 Remove Avro-related code from Java SDK "core" module add 5fd1245d5d1 Merge pull request #27851: Remove Avro-related code from Java SDK "core" module No new revisions were added by this update. Summary of changes: CHANGES.md | 10 +- sdks/java/core/build.gradle|6 +- .../java/org/apache/beam/sdk/coders/AvroCoder.java | 820 .../apache/beam/sdk/coders/AvroGenericCoder.java | 40 - .../main/java/org/apache/beam/sdk/io/AvroIO.java | 2031 .../apache/beam/sdk/io/AvroSchemaIOProvider.java | 157 -- .../main/java/org/apache/beam/sdk/io/AvroSink.java | 167 -- .../java/org/apache/beam/sdk/io/AvroSource.java| 773 .../beam/sdk/io/ConstantAvroDestination.java | 156 -- .../org/apache/beam/sdk/io/CountingSource.java | 30 +- .../beam/sdk/io/DynamicAvroDestinations.java | 61 - .../beam/sdk/io/SerializableAvroCodecFactory.java | 119 -- .../apache/beam/sdk/schemas/AvroRecordSchema.java | 64 - .../org/apache/beam/sdk/schemas/io/Providers.java | 27 +- .../io/payloads/AvroPayloadSerializerProvider.java | 47 - .../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 149 -- .../apache/beam/sdk/schemas/utils/AvroUtils.java | 1396 -- .../org/apache/beam/sdk/coders/AvroCoderTest.java | 1106 --- .../apache/beam/sdk/coders/AvroCoderTestPojo.java | 51 - .../apache/beam/sdk/coders/CoderRegistryTest.java |4 +- .../apache/beam/sdk/coders/DefaultCoderTest.java |6 +- .../apache/beam/sdk/coders/MockDefaultCoder.java} | 42 +- .../java/org/apache/beam/sdk/io/AvroIOTest.java| 1627 .../beam/sdk/io/AvroSchemaIOProviderTest.java | 173 -- .../org/apache/beam/sdk/io/AvroSourceTest.java | 843 .../test/java/org/apache/beam/sdk/io/ReadTest.java |3 +- .../sdk/io/SerializableAvroCodecFactoryTest.java | 93 - .../apache/beam/sdk/schemas/AvroSchemaTest.java| 496 - .../apache/beam/sdk/schemas/SchemaCoderTest.java | 55 - .../io/AvroPayloadSerializerProviderTest.java | 64 - .../beam/sdk/schemas/transforms/ConvertTest.java | 14 - .../beam/sdk/schemas/utils/AvroGenerators.java | 220 --- .../beam/sdk/schemas/utils/AvroUtilsTest.java | 914 - .../beam/sdk/schemas/utils/TestJavaBeans.java | 12 +- .../extensions/avro/coders/DefaultCoderTest.java | 61 - sdks/java/extensions/timeseries/build.gradle |1 + sdks/java/extensions/zetasketch/build.gradle |1 + sdks/java/harness/build.gradle |6 + sdks/java/io/clickhouse/build.gradle |1 + sdks/java/io/contextualtextio/build.gradle |1 + .../pubsub/PubsubReadSchemaTransformProvider.java |2 +- .../pubsub/PubsubWriteSchemaTransformProvider.java |2 +- .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java |2 +- sdks/java/io/kafka/build.gradle|2 +- .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java |2 +- .../test/unit/read/SnowflakeIOReadTest.java|2 +- sdks/java/io/thrift/build.gradle |1 + 47 files changed, 97 insertions(+), 11763 deletions(-) delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroCoder.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/coders/AvroGenericCoder.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSchemaIOProvider.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSink.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/ConstantAvroDestination.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/DynamicAvroDestinations.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/payloads/AvroPayloadSerializerProvider.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java delete mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/coders/AvroCoderTest.java delete mode 100644 sdks/java/core/src/test/java/org/ap
[beam] branch master updated (9efdf6be61e -> 0a560fc09d8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9efdf6be61e Fix SpannerChangeStreamErrorTest.java and stop disabling tests (#28751) add a9684c5563d TPC-DS: Fix CSV format add 0a560fc09d8 Merge pull request #28819: TPC-DS: Fix CSV format No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java| 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (61f0184c528 -> 8037b06b613)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 61f0184c528 Retag 2.50.0 RC2 to publish multi-arch containers (#28343)
add 888fe9128d2 Refactor ConvertHelpers in "core" to not depend on Avro
add 96f3dfba4ce Address review comments
new 8037b06b613 Merge pull request #27991: Refactor ConvertHelpers in
"core" to not depend on Avro
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../beam/sdk/schemas/utils/ConvertHelpers.java | 34 --
.../utils/RowSchemaInformationProvider.java} | 30 ++-
.../SchemaInformationProvider.java}| 20 ++---
.../utils/AvroSchemaInformationProvider.java} | 30 ++-
4 files changed, 62 insertions(+), 52 deletions(-)
copy
sdks/java/{io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderProviderRegistrar.java
=>
core/src/main/java/org/apache/beam/sdk/schemas/utils/RowSchemaInformationProvider.java}
(53%)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{io/payloads/PayloadSerializerProvider.java
=> utils/SchemaInformationProvider.java} (65%)
copy
sdks/java/{io/amqp/src/main/java/org/apache/beam/sdk/io/amqp/AmqpMessageCoderProviderRegistrar.java
=>
extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroSchemaInformationProvider.java}
(50%)
[beam] 01/01: Merge pull request #27991: Refactor ConvertHelpers in "core" to not depend on Avro
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8037b06b613e2226aefbccd3078e9229a346d2d3 Merge: 61f0184c528 96f3dfba4ce Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Sep 7 14:00:03 2023 +0200 Merge pull request #27991: Refactor ConvertHelpers in "core" to not depend on Avro .../beam/sdk/schemas/utils/ConvertHelpers.java | 34 ++ .../utils/RowSchemaInformationProvider.java| 41 + .../schemas/utils/SchemaInformationProvider.java | 32 + .../utils/AvroSchemaInformationProvider.java | 42 ++ 4 files changed, 134 insertions(+), 15 deletions(-)
[beam] branch master updated (741e8c66b0d -> 967e366e1be)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 741e8c66b0d Better error messages on missing or malformed yaml providers. (#28311) add 0a6a6210f3e [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries (closes #28249) new 967e366e1be Merge pull request #28250: [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/testing/tpcds/build.gradle | 1 + 1 file changed, 1 insertion(+)
[beam] 01/01: Merge pull request #28250: [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 967e366e1beef0d1d7064a7092a9aedd8d96deb6 Merge: 741e8c66b0d 0a6a6210f3e Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Sep 6 18:37:48 2023 +0200 Merge pull request #28250: [Spark, TPCDS] Disable AQE with Spark 3.2 in TPCDS benchmarks to reduce overhead for complex queries sdks/java/testing/tpcds/build.gradle | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (a0a297dddaa -> 0019cd554b0)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a0a297dddaa Try a second time to delete old images (#27905) add 5a7e8c7 Update changelog to mention new SqsIO.writeBatches (relates to #21429). new 0019cd554b0 Merge pull request #27921: Update changelog to mention new SqsIO.writeBatches The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 1 + 1 file changed, 1 insertion(+)
[beam] 01/01: Merge pull request #27921: Update changelog to mention new SqsIO.writeBatches
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0019cd554b0883813ff13ec715412c6f6cf63d54 Merge: a0a297dddaa 5a7e8c7 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Aug 9 11:25:35 2023 +0200 Merge pull request #27921: Update changelog to mention new SqsIO.writeBatches CHANGES.md | 1 + 1 file changed, 1 insertion(+)
[beam] 01/01: Merge pull request #27849: Move changes from #27635 to the right section in CHANGES.md
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 887560b35ffd800d5e64e0ba177c1851f69f7f93 Merge: 7e365d78c79 5289d6bfbcb Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Aug 4 16:11:09 2023 +0200 Merge pull request #27849: Move changes from #27635 to the right section in CHANGES.md CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (7e365d78c79 -> 887560b35ff)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 7e365d78c79 Update settings.gradle.kts to include project `arm` (#27831) add 5289d6bfbcb Move changes from #27635 to the right section in CHANGES.md new 887560b35ff Merge pull request #27849: Move changes from #27635 to the right section in CHANGES.md The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (fede6d4df7b -> 966bb2fe0a8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from fede6d4df7b Bump go.mongodb.org/mongo-driver from 1.12.0 to 1.12.1 in /sdks (#27826) add 966bb2fe0a8 Upgrade Kryo coder from Kryo v4 to Kryo v5 (#27803) No new revisions were added by this update. Summary of changes: CHANGES.md | 2 +- sdks/java/extensions/kryo/build.gradle | 2 +- .../main/java/org/apache/beam/sdk/extensions/kryo/KryoCoder.java| 6 +++--- .../main/java/org/apache/beam/sdk/extensions/kryo/KryoState.java| 3 ++- 4 files changed, 7 insertions(+), 6 deletions(-)
[beam] 01/01: Merge pull request #26694: [Java] Track pipeline options revision for idempotent initialization of file systems
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 663bd529a470a8b8870e94b45b91d27033fdb871 Merge: e0b6d835cc0 f3bdc8d9594 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Aug 2 10:13:00 2023 +0200 Merge pull request #26694: [Java] Track pipeline options revision for idempotent initialization of file systems .../java/org/apache/beam/sdk/io/FileSystems.java | 30 +- .../apache/beam/sdk/options/PipelineOptions.java | 6 + .../beam/sdk/options/PipelineOptionsFactory.java | 1 + .../beam/sdk/options/ProxyInvocationHandler.java | 27 ++- .../sdk/options/PipelineOptionsFactoryTest.java| 12 + 5 files changed, 63 insertions(+), 13 deletions(-)
[beam] branch master updated (e0b6d835cc0 -> 663bd529a47)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from e0b6d835cc0 Update website with DicomIO (#27791) add 2f55b2598b5 Track pipeline options revision add f3bdc8d9594 Safe initialization of file systems using pipeline options revision. new 663bd529a47 Merge pull request #26694: [Java] Track pipeline options revision for idempotent initialization of file systems The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/beam/sdk/io/FileSystems.java | 30 +- .../apache/beam/sdk/options/PipelineOptions.java | 6 + .../beam/sdk/options/PipelineOptionsFactory.java | 1 + .../beam/sdk/options/ProxyInvocationHandler.java | 27 ++- .../sdk/options/PipelineOptionsFactoryTest.java| 12 + 5 files changed, 63 insertions(+), 13 deletions(-)
[beam] branch master updated (df6964aac62 -> 881338ed7ff)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from df6964aac62 Bump github.com/fsouza/fake-gcs-server from 1.45.2 to 1.46.0 in /sdks (#27751) add 05fb24492c6 [Spark DS Runner] Test compatibility against more Spark versions including latest 3.4.x (closes #27576) add 881338ed7ff Merge pull request #27597: [Spark DS Runner] Test compatibility against more Spark versions No new revisions were added by this update. Summary of changes: runners/spark/3/build.gradle | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-)
[beam] branch master updated: Bump Spark3 version to 3.2.2 (#23805)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0cd9b90467b Bump Spark3 version to 3.2.2 (#23805)
0cd9b90467b is described below
commit 0cd9b90467b43f37898b077687b931db1634c842
Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com>
AuthorDate: Thu Jul 20 14:05:58 2023 +0200
Bump Spark3 version to 3.2.2 (#23805)
* Bump Spark3 version to 3.2.0
* [23804] Add Spark 3.2.0 constructors in EncoderFactory
* Update Hadoop deps
* Bump Spark3 version to 3.2.2
* Add Spark 3.1.1 for compatibility testing
* Update CHANGES.md on Spark version bump
* Fix whitespace check
* Add Spark 3.1.2 for compatibility testing
* Address review comments
---
CHANGES.md| 1 +
.../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
runners/spark/3/build.gradle | 2 +-
runners/spark/3/job-server/build.gradle | 11 +--
.../translation/helpers/EncoderFactory.java | 10 ++
runners/spark/spark_runner.gradle | 2 +-
website/www/site/content/en/documentation/runners/spark.md| 8
7 files changed, 19 insertions(+), 17 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 7ed75c7ee05..dff482b6b74 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -56,6 +56,7 @@
* New highly anticipated feature X added to Python SDK
([#X](https://github.com/apache/beam/issues/X)).
* New highly anticipated feature Y added to Java SDK
([#Y](https://github.com/apache/beam/issues/Y)).
+* Spark 3.2.2 is used as default version for Spark runner
([#23804](https://github.com/apache/beam/issues/23804)).
## I/Os
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 9637f17dbf8..852edfe62e4 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -559,7 +559,7 @@ class BeamModulePlugin implements Plugin {
def singlestore_jdbc_version = "1.1.4"
def slf4j_version = "1.7.30"
def spark2_version = "2.4.8"
-def spark3_version = "3.1.2"
+def spark3_version = "3.2.2"
def spotbugs_version = "4.0.6"
def testcontainers_version = "1.17.3"
def arrow_version = "5.0.0"
diff --git a/runners/spark/3/build.gradle b/runners/spark/3/build.gradle
index 494d367131b..cb34a1fd972 100644
--- a/runners/spark/3/build.gradle
+++ b/runners/spark/3/build.gradle
@@ -35,7 +35,7 @@ createJavaExamplesArchetypeValidationTask(type: 'Quickstart',
runner: 'Spark')
// Additional supported Spark versions (used in compatibility tests)
def sparkVersions = [
"330": "3.3.0",
-"321": "3.2.1"
+"312": "3.1.2"
]
sparkVersions.each { kv ->
diff --git a/runners/spark/3/job-server/build.gradle
b/runners/spark/3/job-server/build.gradle
index d11a1a8edb1..68bb8d9a10e 100644
--- a/runners/spark/3/job-server/build.gradle
+++ b/runners/spark/3/job-server/build.gradle
@@ -28,13 +28,4 @@ project.ext {
}
// Load the main build script which contains all build logic.
-apply from: "$basePath/spark_job_server.gradle"
-
-
-configurations.runtimeClasspath {
- resolutionStrategy {
-// Downgrade the Scala version of the job-server to match the Scala
version of a Spark 3.1.2 cluster to prevent
-// a Scala bug (InvalidClassException when deserializing WrappedArray),
see https://github.com/apache/beam/issues/21092
-force "org.scala-lang:scala-library:2.12.10"
- }
-}
\ No newline at end of file
+apply from: "$basePath/spark_job_server.gradle"
\ No newline at end of file
diff --git
a/runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
b/runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
index e70cc7253f8..ceafc1642ba 100644
---
a/runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
+++
b/runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
@@ -69,6 +69,10 @@ public class EncoderFactory {
// Spark 3.1.x
return STATIC_INVOKE_CONSTRUCTOR.newInstance(
cls, type, fun, seqOf(args), propagateNull, true);
+case 7:
+ /
[beam] branch master updated (bac98cf2075 -> e8d8043f441)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from bac98cf2075 add gcloud auth and push to registry on merge PR (#27341) add e8d8043f441 Upgrade obsolete Avro plugin to 0.16.0 (#27232) No new revisions were added by this update. Summary of changes: buildSrc/build.gradle.kts | 7 ++- ownership/JAVA_DEPENDENCY_OWNERS.yaml | 10 +- sdks/java/build-tools/src/main/resources/beam/suppressions.xml | 1 + 3 files changed, 12 insertions(+), 6 deletions(-)
[beam] branch fixQuickstart deleted (was 09159ad46e2)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch fixQuickstart in repository https://gitbox.apache.org/repos/asf/beam.git was 09159ad46e2 [#27207] Simplify Go quickstart. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] branch master updated (7890bdcee84 -> 98886285a54)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 7890bdcee84 Merge pull request #27277: Use requirements file as a constraint when building the expansion service container add 09159ad46e2 [#27207] Simplify Go quickstart. new 98886285a54 Merge pull request #27299: [#27207] Update Go quickstart. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../site/content/en/get-started/quickstart-go.md | 28 +++--- 1 file changed, 8 insertions(+), 20 deletions(-)
[beam] 01/01: Merge pull request #27299: [#27207] Update Go quickstart.
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 98886285a54fe0f5b7bcaad54ab868908290c5ac Merge: 7890bdcee84 09159ad46e2 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Jun 30 14:17:29 2023 +0200 Merge pull request #27299: [#27207] Update Go quickstart. .../site/content/en/get-started/quickstart-go.md | 28 +++--- 1 file changed, 8 insertions(+), 20 deletions(-)
[beam] branch master updated (9d0fc05d004 -> 5223ab466e7)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9d0fc05d004 Fix sklearn requirement file in Py311 (#27152) add 5223ab466e7 Select dedicated avro datum reader and writer (#26320) No new revisions were added by this update. Summary of changes: CHANGES.md | 1 + .../util/AvroCoderCloudObjectTranslator.java | 29 ++- .../runners/dataflow/util/CloudObjectsTest.java| 10 +- sdks/java/extensions/avro/build.gradle | 69 +++-- .../beam/sdk/extensions/avro/coders/AvroCoder.java | 257 +- .../extensions/avro/coders/AvroGenericCoder.java | 5 +- .../sdk/extensions/avro/io/AvroDatumFactory.java | 196 ++ .../apache/beam/sdk/extensions/avro/io/AvroIO.java | 194 +++--- .../beam/sdk/extensions/avro/io/AvroSink.java | 40 ++- .../beam/sdk/extensions/avro/io/AvroSource.java| 123 + .../extensions/avro/schemas/AvroRecordSchema.java | 5 +- .../schemas/utils/AvroJavaTimeConversions.java | 288 + .../schemas/utils/AvroJodaTimeConversions.java | 154 +++ .../extensions/avro/schemas/utils/AvroUtils.java | 38 ++- .../beam/sdk/extensions/avro/schemas/test.avsc | 71 ++--- .../sdk/extensions/avro/coders/AvroCoderTest.java | 148 +++ .../beam/sdk/extensions/avro/io/AvroIOTest.java| 4 +- .../sdk/extensions/avro/io/AvroSourceTest.java | 40 +-- .../avro/schemas/TestAvroConversionFactory.java| 41 +++ .../extensions/avro/schemas/TestAvroFactory.java | 66 +++-- .../avro/schemas/utils/AvroUtilsTest.java | 34 ++- .../AvroWriteSchemaTransformFormatProvider.java| 4 +- .../ParquetWriteSchemaTransformFormatProvider.java | 4 +- .../AvroReadSchemaTransformFormatProviderTest.java | 8 +- ...AvroWriteSchemaTransformFormatProviderTest.java | 4 +- ...rquetReadSchemaTransformFormatProviderTest.java | 8 +- .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java| 7 +- .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 4 +- 28 files changed, 1418 insertions(+), 434 deletions(-) create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroDatumFactory.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroJavaTimeConversions.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroJodaTimeConversions.java create mode 100644 sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/schemas/TestAvroConversionFactory.java
[beam] branch master updated (113a8757c70 -> bc5a8907dad)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 113a8757c70 Bump cloud.google.com/go/pubsub from 1.30.1 to 1.31.0 in /sdks (#26888) add 1441c0db308 Add startPollTimeoutSec parameter. Fix splitting of restriction add bc5a8907dad Merge pull request #26862: [SparkReceiverIO] Add startPollTimeoutSec parameter. Fix splitting of restriction No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/sdk/io/cdap/CdapIO.java | 18 +- .../ReadFromSparkReceiverWithOffsetDoFn.java | 15 +++ .../beam/sdk/io/sparkreceiver/SparkReceiverIO.java | 14 +- .../ReadFromSparkReceiverWithOffsetDoFnTest.java | 9 + .../beam/sdk/io/sparkreceiver/SparkReceiverIOTest.java | 14 ++ 5 files changed, 60 insertions(+), 10 deletions(-)
[beam] branch master updated: #25232 bump log4j2 to 2.20.0 (#25231)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new dcef836ce61 #25232 bump log4j2 to 2.20.0 (#25231)
dcef836ce61 is described below
commit dcef836ce617c76678437657ccf03477d066649a
Author: Masato Nakamura
AuthorDate: Wed May 24 00:24:58 2023 +0900
#25232 bump log4j2 to 2.20.0 (#25231)
* bump log4j2 to 2.19.0
* #25232 bump log4j2 to 2.20.0
---
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 949dd6b4737..9e9bb9bad11 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -546,7 +546,7 @@ class BeamModulePlugin implements Plugin {
def jsr305_version = "3.0.2"
def everit_json_version = "1.14.1"
def kafka_version = "2.4.1"
-def log4j2_version = "2.17.2"
+def log4j2_version = "2.20.0"
def nemo_version = "0.1"
def netty_version = "4.1.77.Final"
def postgres_version = "42.2.16"
[beam] 01/01: Merge pull request #26682: Remove deprecated AWS 2 client providers
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5df8c9eb1aa5b87adda9b440ca216aaf5b5427d7 Merge: 05a15079846 ccc7695ac0a Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon May 15 18:34:39 2023 +0200 Merge pull request #26682: Remove deprecated AWS 2 client providers CHANGES.md | 1 + .../aws2/dynamodb/BasicDynamoDbClientProvider.java | 69 .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 188 + .../io/aws2/dynamodb/DynamoDbClientProvider.java | 36 .../sdk/io/aws2/kinesis/AWSClientsProvider.java| 36 .../sdk/io/aws2/kinesis/BasicKinesisProvider.java | 76 - .../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 70 +--- .../beam/sdk/io/aws2/kinesis/KinesisSource.java| 42 ++--- .../sdk/io/aws2/sns/BasicSnsClientProvider.java| 69 .../beam/sdk/io/aws2/sns/SnsClientProvider.java| 36 .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 104 +--- .../sdk/io/aws2/sqs/BasicSqsClientProvider.java| 69 .../beam/sdk/io/aws2/sqs/SqsClientProvider.java| 36 .../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java | 117 ++--- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 9 +- .../beam/sdk/io/aws2/MockClientBuilderFactory.java | 25 +-- .../dynamodb/BasicDynamoDbClientProviderTest.java | 51 -- .../sdk/io/aws2/dynamodb/DynamoDBIOReadTest.java | 61 +-- .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 73 +--- .../dynamodb/StaticDynamoDBClientProvider.java | 35 .../kinesis/BasicKinesisClientProviderTest.java| 52 -- .../sdk/io/aws2/kinesis/KinesisIOReadTest.java | 89 -- .../sdk/io/aws2/kinesis/KinesisSourceTest.java | 36 ++-- .../io/aws2/sns/BasicSnsClientProviderTest.java| 51 -- .../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 103 +-- .../io/aws2/sqs/BasicSqsClientProviderTest.java| 51 -- .../apache/beam/sdk/io/aws2/sqs/SqsIOReadTest.java | 41 + .../beam/sdk/io/aws2/sqs/SqsIOWriteTest.java | 42 + .../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java| 7 +- .../sdk/io/aws2/sqs/StaticSqsClientProvider.java | 35 30 files changed, 89 insertions(+), 1621 deletions(-)
[beam] branch master updated (05a15079846 -> 5df8c9eb1aa)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 05a15079846 Add run_inference windowing notebook (#26404) add ccc7695ac0a Remove deprecated AWS 2 client providers (closes #26681) new 5df8c9eb1aa Merge pull request #26682: Remove deprecated AWS 2 client providers The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 1 + .../aws2/dynamodb/BasicDynamoDbClientProvider.java | 69 .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 188 + .../io/aws2/dynamodb/DynamoDbClientProvider.java | 36 .../sdk/io/aws2/kinesis/AWSClientsProvider.java| 36 .../sdk/io/aws2/kinesis/BasicKinesisProvider.java | 76 - .../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 70 +--- .../beam/sdk/io/aws2/kinesis/KinesisSource.java| 42 ++--- .../sdk/io/aws2/sns/BasicSnsClientProvider.java| 69 .../beam/sdk/io/aws2/sns/SnsClientProvider.java| 36 .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 104 +--- .../sdk/io/aws2/sqs/BasicSqsClientProvider.java| 69 .../beam/sdk/io/aws2/sqs/SqsClientProvider.java| 36 .../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java | 117 ++--- .../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 9 +- .../beam/sdk/io/aws2/MockClientBuilderFactory.java | 25 +-- .../dynamodb/BasicDynamoDbClientProviderTest.java | 51 -- .../sdk/io/aws2/dynamodb/DynamoDBIOReadTest.java | 61 +-- .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 73 +--- .../dynamodb/StaticDynamoDBClientProvider.java | 35 .../kinesis/BasicKinesisClientProviderTest.java| 52 -- .../sdk/io/aws2/kinesis/KinesisIOReadTest.java | 89 -- .../sdk/io/aws2/kinesis/KinesisSourceTest.java | 36 ++-- .../io/aws2/sns/BasicSnsClientProviderTest.java| 51 -- .../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 103 +-- .../io/aws2/sqs/BasicSqsClientProviderTest.java| 51 -- .../apache/beam/sdk/io/aws2/sqs/SqsIOReadTest.java | 41 + .../beam/sdk/io/aws2/sqs/SqsIOWriteTest.java | 42 + .../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java| 7 +- .../sdk/io/aws2/sqs/StaticSqsClientProvider.java | 35 30 files changed, 89 insertions(+), 1621 deletions(-) delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/AWSClientsProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/BasicKinesisProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/BasicSqsClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProviderTest.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/StaticDynamoDBClientProvider.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/BasicKinesisClientProviderTest.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProviderTest.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/BasicSqsClientProviderTest.java delete mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/StaticSqsClientProvider.java
[beam] branch master updated (71af99f80f1 -> 38e43d06132)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 71af99f80f1 Support configuration of maxCapacityPerShard when reading from Kinesis with EFO (closes #26257) (#2) add f9a788bca39 [Avro] Update Beam documentation add 38e43d06132 Merge pull request #26464: [Avro] Update Beam documentation No new revisions were added by this update. Summary of changes: website/www/site/content/en/documentation/io/connectors.md | 2 +- website/www/site/content/en/documentation/io/developing-io-java.md | 4 ++-- website/www/site/content/en/documentation/io/io-standards.md | 2 +- website/www/site/content/en/documentation/ml/orchestration.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-)
[beam] branch master updated (7a060c5c95f -> a14f98ade74)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 7a060c5c95f [AWS2] Bump AWS v2 SDK (closes #26323) (#26380) add a14f98ade74 Support Dataflow runner v2 in SparkRecieverIO (#26051) No new revisions were added by this update. Summary of changes: .../job_PerformanceTests_SparkReceiverIO_IT.groovy | 2 +- examples/java/cdap/hubspot/build.gradle| 10 ++ examples/java/cdap/salesforce/build.gradle | 11 +- examples/java/cdap/servicenow/build.gradle | 8 ++ examples/java/cdap/zendesk/build.gradle| 8 ++ .../beam/sdk/io/sparkreceiver/HasOffset.java | 6 + .../ReadFromSparkReceiverWithOffsetDoFn.java | 121 + .../sparkreceiver/RabbitMqReceiverWithOffset.java | 81 +- .../ReadFromSparkReceiverWithOffsetDoFnTest.java | 12 +- 9 files changed, 152 insertions(+), 107 deletions(-)
[beam] branch master updated: Make `withDesiredBundleSizeBytes` methods public (#25448)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f1e95970ca4 Make `withDesiredBundleSizeBytes` methods public (#25448)
f1e95970ca4 is described below
commit f1e95970ca4cb9f4ed9099761ba6a757099d22d8
Author: Israel Herraiz
AuthorDate: Thu Apr 20 13:26:08 2023 +0200
Make `withDesiredBundleSizeBytes` methods public (#25448)
* Make `withDesiredBundleSizeBytes` methods public
With these methods being private, the bundle size is 64 MB, which is too
large for streaming applications using these classes.
This changes the method visibility to public so a smaller bundle size can
be used for streaming.
Fixes #25447
* Add Javadoc to new public methods
* Make withDesiredBundleSizeBytes public, useful for streaming applications
---
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 28 +++---
.../apache/beam/sdk/extensions/avro/io/AvroIO.java | 28 +++---
2 files changed, 40 insertions(+), 16 deletions(-)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index c8ccf753229..42fab212576 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -813,8 +813,11 @@ public class AvroIO {
abstract ReadFiles build();
}
-@VisibleForTesting
-ReadFiles withDesiredBundleSizeBytes(long desiredBundleSizeBytes) {
+/**
+ * Set a value for the bundle size for parallel reads. Default is 64 MB.
You may want to use a
+ * lower value (e.g. 1 MB) for streaming applications.
+ */
+public ReadFiles withDesiredBundleSizeBytes(long
desiredBundleSizeBytes) {
return
toBuilder().setDesiredBundleSizeBytes(desiredBundleSizeBytes).build();
}
@@ -927,8 +930,11 @@ public class AvroIO {
getMatchConfiguration().continuously(pollInterval,
terminationCondition));
}
-@VisibleForTesting
-ReadAll withDesiredBundleSizeBytes(long desiredBundleSizeBytes) {
+/**
+ * Set a value for the bundle size for parallel reads. Default is 64 MB.
You may want to use a
+ * lower value (e.g. 1 MB) for streaming applications.
+ */
+public ReadAll withDesiredBundleSizeBytes(long desiredBundleSizeBytes) {
return
toBuilder().setDesiredBundleSizeBytes(desiredBundleSizeBytes).build();
}
@@ -1169,8 +1175,11 @@ public class AvroIO {
return toBuilder().setFileExceptionHandler(exceptionHandler).build();
}
-@VisibleForTesting
-ParseFiles withDesiredBundleSizeBytes(long desiredBundleSizeBytes) {
+/**
+ * Set a value for the bundle size for parallel reads. Default is 64 MB.
You may want to use a
+ * lower value (e.g. 1 MB) for streaming applications.
+ */
+public ParseFiles withDesiredBundleSizeBytes(long
desiredBundleSizeBytes) {
return
toBuilder().setDesiredBundleSizeBytes(desiredBundleSizeBytes).build();
}
@@ -1278,8 +1287,11 @@ public class AvroIO {
return toBuilder().setCoder(coder).build();
}
-@VisibleForTesting
-ParseAll withDesiredBundleSizeBytes(long desiredBundleSizeBytes) {
+/**
+ * Set a value for the bundle size for parallel reads. Default is 64 MB.
You may want to use a
+ * lower value (e.g. 1 MB) for streaming applications.
+ */
+public ParseAll withDesiredBundleSizeBytes(long desiredBundleSizeBytes)
{
return
toBuilder().setDesiredBundleSizeBytes(desiredBundleSizeBytes).build();
}
diff --git
a/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java
b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java
index b159d38c603..f35fc66e654 100644
---
a/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java
+++
b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java
@@ -816,8 +816,11 @@ public class AvroIO {
abstract ReadFiles build();
}
-@VisibleForTesting
-ReadFiles withDesiredBundleSizeBytes(long desiredBundleSizeBytes) {
+/**
+ * Set a value for the bundle size for parallel reads. Default is 64 MB.
You may want to use a
+ * lower value (e.g. 1 MB) for streaming applications.
+ */
+public ReadFiles withDesiredBundleSizeBytes(long
desiredBundleSizeBytes) {
return
toBuilder().setDesiredBundleSizeBytes(desiredBundleSizeBytes).build();
}
@@ -930,8 +933,11 @@ public class AvroIO {
getMatchConfiguration().continuously(pollInterval,
terminationCondition));
}
-@VisibleForTesting
-ReadAll withDesiredBundleSizeBytes(long desiredBundleSizeBytes
[beam] branch master updated: Test Avro extension against multiple Avro versions (#25216)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7919c3fdf07 Test Avro extension against multiple Avro versions (#25216)
7919c3fdf07 is described below
commit 7919c3fdf07c7802f62061e5e222027df5e02ce5
Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com>
AuthorDate: Thu Apr 13 18:28:05 2023 +0200
Test Avro extension against multiple Avro versions (#25216)
---
.../job_PostCommit_Java_Avro_Versions.groovy | 47 +
.test-infra/jenkins/job_PreCommit_Java.groovy | 1 +
build.gradle.kts | 4 +
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 5 +
.../beam/sdk/schemas/utils/SchemaTestUtils.java| 2 +
sdks/java/extensions/avro/build.gradle | 91 -
.../beam/sdk/extensions/avro/coders/AvroCoder.java | 2 +-
.../extensions/avro/schemas/utils/AvroUtils.java | 105 ++--
.../sdk/extensions/avro/coders/AvroCoderTest.java | 22 -
.../avro/io/AvroGeneratedUserFactory.java | 49 +
.../beam/sdk/extensions/avro/io/AvroIOTest.java| 6 +-
.../extensions/avro/schemas/AvroSchemaTest.java| 10 +-
.../extensions/avro/schemas/TestAvroFactory.java | 109 +
.../avro/schemas/utils/AvroUtilsTest.java | 36 ---
14 files changed, 451 insertions(+), 38 deletions(-)
diff --git a/.test-infra/jenkins/job_PostCommit_Java_Avro_Versions.groovy
b/.test-infra/jenkins/job_PostCommit_Java_Avro_Versions.groovy
new file mode 100644
index 000..ea07cb26392
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Java_Avro_Versions.groovy
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+
+// This job runs the Java tests that depends on Avro against different Avro
API versions
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_Avro_Versions', 'Run
PostCommit_Java_Avro_Versions',
+'Java Avro Versions Post Commit Tests', this) {
+
+ description('Java Avro Versions Post Commit Tests')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 240)
+
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(":javaAvroVersionsTest")
+ commonJobProperties.setGradleSwitches(delegate)
+ // Specify maven home on Jenkins, needed by Maven archetype
integration tests.
+ switches('-Pmaven_home=/home/jenkins/tools/maven/apache-maven-3.5.4')
+}
+ }
+}
diff --git a/.test-infra/jenkins/job_PreCommit_Java.groovy
b/.test-infra/jenkins/job_PreCommit_Java.groovy
index ad2a4f1ce94..ddeb05506cb 100644
--- a/.test-infra/jenkins/job_PreCommit_Java.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Java.groovy
@@ -20,6 +20,7 @@ import PrecommitJobBuilder
// exclude paths with their own PreCommit tasks
def excludePaths = [
+ 'extensions/avro',
'extensions/sql',
'io/amazon-web-services',
'io/amazon-web-services2',
diff --git a/build.gradle.kts b/build.gradle.kts
index 613d4097b6b..7cef53c275c 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -393,6 +393,10 @@ tasks.register("javaHadoopVersionsTest") {
dependsOn(":runners:spark:3:hadoopVersionsTest")
}
+tasks.register("javaAvroVersionsTest") {
+ dependsOn(":sdks:java:extensions:avro:avroVersionsTest")
+}
+
tasks.register("sqlPostCommit") {
dependsOn(":sdks:java:extensions:sql:postCommit")
dependsOn(":sdks:java:extensions:sql:jdbc:postCommit")
diff --git
a/sdks/java/core/sr
[beam] 01/01: Merge pull request #26114: [AWS2] Fix performance issue of S3 filesystem when reading large files
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 36c9c738f47a214241d3b847e18e817219db6e77 Merge: 485822beeef db656bfec92 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Apr 7 17:27:01 2023 +0200 Merge pull request #26114: [AWS2] Fix performance issue of S3 filesystem when reading large files .../io/aws2/s3/S3ReadableSeekableByteChannel.java | 34 +++--- 1 file changed, 23 insertions(+), 11 deletions(-)
[beam] branch master updated (485822beeef -> 36c9c738f47)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 485822beeef Fix Jacoco not working for a few modules (#26125) add db656bfec92 (AWS S3 FS) Fix performance issue of S3 filesystem when reading large files (fixes #25991) new 36c9c738f47 Merge pull request #26114: [AWS2] Fix performance issue of S3 filesystem when reading large files The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../io/aws2/s3/S3ReadableSeekableByteChannel.java | 34 +++--- 1 file changed, 23 insertions(+), 11 deletions(-)
[beam] 01/01: Merge pull request #25618: Cherry Pick: Prioritise Avro providers from "extensions/core" (#25611)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch release-2.46.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 8414cada7869a8eab715b5678ce648cc0253fa50 Merge: 695e40d2ce9 5f870679918 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Feb 24 11:04:00 2023 +0100 Merge pull request #25618: Cherry Pick: Prioritise Avro providers from "extensions/core" (#25611) .../org/apache/beam/sdk/schemas/io/Providers.java | 11 ++-- .../ExternalSchemaIOTransformRegistrar.java| 60 ++ 2 files changed, 53 insertions(+), 18 deletions(-)
[beam] branch release-2.46.0 updated (695e40d2ce9 -> 8414cada786)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch release-2.46.0 in repository https://gitbox.apache.org/repos/asf/beam.git from 695e40d2ce9 Set Dataflow container to release version. add 5f870679918 Prioritise Avro providers from "extensions/core" (#25611) new 8414cada786 Merge pull request #25618: Cherry Pick: Prioritise Avro providers from "extensions/core" (#25611) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/schemas/io/Providers.java | 11 ++-- .../ExternalSchemaIOTransformRegistrar.java| 60 ++ 2 files changed, 53 insertions(+), 18 deletions(-)
[beam] branch master updated (85817b6b343 -> 5b59377ff61)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 85817b6b343 Add partitioned reads to JDBC SchemaIO (#25577) add 42827acbb64 Deprecate Avro API classes in "sdks/java/core" new 5b59377ff61 Merge pull request #25534: [Avro] Deprecate Avro API classes in "sdks/java/core" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 4 +++- .../src/main/java/org/apache/beam/sdk/coders/AvroCoder.java| 5 + .../main/java/org/apache/beam/sdk/coders/AvroGenericCoder.java | 10 +- .../java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java | 6 ++ .../main/java/org/apache/beam/sdk/io/AvroSchemaIOProvider.java | 6 ++ .../core/src/main/java/org/apache/beam/sdk/io/AvroSink.java| 10 +- .../core/src/main/java/org/apache/beam/sdk/io/AvroSource.java | 5 + .../java/org/apache/beam/sdk/io/ConstantAvroDestination.java | 10 +- .../java/org/apache/beam/sdk/io/DynamicAvroDestinations.java | 6 ++ .../org/apache/beam/sdk/io/SerializableAvroCodecFactory.java | 7 +++ .../java/org/apache/beam/sdk/schemas/AvroRecordSchema.java | 6 ++ .../sdk/schemas/io/payloads/AvroPayloadSerializerProvider.java | 8 .../org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 8 .../main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java | 6 ++ sdks/java/extensions/avro/build.gradle | 2 +- 15 files changed, 94 insertions(+), 5 deletions(-)
[beam] 01/01: Merge pull request #25534: [Avro] Deprecate Avro API classes in "sdks/java/core"
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5b59377ff6113ac7c26ddeb0c62547daa255 Merge: 85817b6b343 42827acbb64 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Feb 22 18:24:18 2023 +0100 Merge pull request #25534: [Avro] Deprecate Avro API classes in "sdks/java/core" CHANGES.md | 4 +++- .../src/main/java/org/apache/beam/sdk/coders/AvroCoder.java| 5 + .../main/java/org/apache/beam/sdk/coders/AvroGenericCoder.java | 10 +- .../java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java | 6 ++ .../main/java/org/apache/beam/sdk/io/AvroSchemaIOProvider.java | 6 ++ .../core/src/main/java/org/apache/beam/sdk/io/AvroSink.java| 10 +- .../core/src/main/java/org/apache/beam/sdk/io/AvroSource.java | 5 + .../java/org/apache/beam/sdk/io/ConstantAvroDestination.java | 10 +- .../java/org/apache/beam/sdk/io/DynamicAvroDestinations.java | 6 ++ .../org/apache/beam/sdk/io/SerializableAvroCodecFactory.java | 7 +++ .../java/org/apache/beam/sdk/schemas/AvroRecordSchema.java | 6 ++ .../sdk/schemas/io/payloads/AvroPayloadSerializerProvider.java | 8 .../org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 8 .../main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java | 6 ++ sdks/java/extensions/avro/build.gradle | 2 +- 15 files changed, 94 insertions(+), 5 deletions(-)
[beam] branch master updated (d8e5047dad3 -> 95a6566107e)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from d8e5047dad3 Bump github.com/testcontainers/testcontainers-go in /sdks (#25517) add 95a6566107e [Avro] Use "extensions/avro" instead of avro from"core" in Java SDK modules (#24992) No new revisions were added by this update. Summary of changes: CHANGES.md | 1 + build.gradle.kts | 1 + examples/java/build.gradle | 1 + .../beam/examples/complete/AutoComplete.java | 2 +- .../beam/examples/complete/TrafficMaxLaneFlow.java | 2 +- .../beam/examples/complete/TrafficRoutes.java | 2 +- .../beam/examples/complete/game/UserScore.java | 2 +- .../complete/kafkatopubsub/avro/AvroDataClass.java | 2 +- .../kafkatopubsub/transforms/FormatTransform.java | 2 +- .../apache/beam/examples/snippets/Snippets.java| 2 +- .../transforms/io/gcp/bigquery/BigQueryMyData.java | 2 +- .../examples/subprocess/utils/ExecutableFile.java | 2 +- .../examples/complete/game/LeaderBoardTest.java| 2 +- .../complete/game/StatefulTeamScoreTest.java | 2 +- examples/kotlin/build.gradle | 1 + .../beam/examples/kotlin/snippets/Snippets.kt | 2 +- runners/core-construction-java/build.gradle| 2 + .../construction/AvroGenericCoderRegistrar.java| 2 +- .../construction/AvroGenericCoderTranslator.java | 2 +- .../core/construction/CoderTranslationTest.java| 2 +- runners/direct-java/build.gradle | 4 +- runners/google-cloud-dataflow-java/build.gradle| 2 + .../util/AvroCoderCloudObjectTranslator.java | 2 +- .../runners/dataflow/util/CloudObjectsTest.java| 2 +- .../google-cloud-dataflow-java/worker/build.gradle | 1 + .../runners/dataflow/worker/AvroByteReader.java| 4 +- .../translation/SparkSessionFactory.java | 16 +- runners/spark/spark_runner.gradle | 2 + .../beam/runners/spark/io/AvroPipelineTest.java| 2 +- .../org/apache/beam/sdk/coders/DefaultCoder.java | 2 +- .../org/apache/beam/sdk/schemas/io/Providers.java | 33 +++- .../org/apache/beam/sdk/io/TextIOWriteTest.java| 30 ++- .../org/apache/beam/sdk/transforms/LatestTest.java | 4 +- sdks/java/extensions/avro/build.gradle | 2 +- .../extensions/avro/coders/DefaultCoderTest.java | 61 ++ .../extensions/avro}/schemas/SchemaCoderTest.java | 205 + .../avro}/schemas/transforms/ConvertTest.java | 119 +--- sdks/java/extensions/sketching/build.gradle| 1 + .../sketching/ApproximateDistinctTest.java | 2 +- .../sketching/SketchFrequenciesTest.java | 2 +- sdks/java/extensions/sql/build.gradle | 1 + .../sql/meta/provider/avro/AvroTableProvider.java | 4 +- .../sql/meta/provider/parquet/ParquetTable.java| 2 +- .../provider/kafka/BeamKafkaTableAvroTest.java | 4 +- .../meta/provider/kafka/KafkaTableProviderIT.java | 2 +- .../provider/pubsub/PubsubTableProviderIT.java | 4 +- sdks/java/harness/build.gradle | 1 + sdks/java/io/amazon-web-services/build.gradle | 1 + .../sdk/io/aws/dynamodb/DynamoDBIOWriteTest.java | 2 +- sdks/java/io/amazon-web-services2/build.gradle | 1 + .../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 2 +- sdks/java/io/cdap/build.gradle | 1 + .../apache/beam/sdk/io/cdap/TestRowDBWritable.java | 2 +- sdks/java/io/file-based-io-tests/build.gradle | 1 + .../java/org/apache/beam/sdk/io/avro/AvroIOIT.java | 4 +- .../apache/beam/sdk/io/parquet/ParquetIOIT.java| 2 +- sdks/java/io/file-schema-transform/build.gradle| 1 + .../AvroWriteSchemaTransformFormatProvider.java| 6 +- .../FileWriteSchemaTransformFormatProviders.java | 2 +- .../ParquetWriteSchemaTransformFormatProvider.java | 2 +- ...FileWriteSchemaTransformFormatProviderTest.java | 6 +- ...FileWriteSchemaTransformFormatProviderTest.java | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 2 + .../AvroGenericRecordToStorageApiProto.java| 2 +- .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 2 +- .../sdk/io/gcp/bigquery/BigQueryQuerySource.java | 2 +- .../io/gcp/bigquery/BigQueryQuerySourceDef.java| 2 +- .../sdk/io/gcp/bigquery/BigQuerySourceBase.java| 2 +- .../sdk/io/gcp/bigquery/BigQuerySourceDef.java | 2 +- .../gcp/bigquery/BigQueryStorageArrowReader.java | 2 +- .../io/gcp/bigquery/BigQueryStorageSourceBase.java | 2 +- .../sdk/io/gcp/bigquery/BigQueryTableSource.java | 2 +- .../io/gcp/bigquery/BigQueryTableSourceDef.java| 2 +- ...StorageApiDynamicDestinationsGenericRecord.java | 2 +- .../be
[beam] branch master updated (40838f76447 -> 20c8d3a2ba1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 40838f76447 Support Avro GenericRecord as a valid format for StorageWrite API on BigQueryIO (#24274) add ee9244d6261 complete examples links fixed new 20c8d3a2ba1 Merge pull request #25527: [CdapIO][SparkReceiverIO] Complete examples links in READMEs and CdapIO website page fixed The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/cdap/README.md | 2 +- sdks/java/io/sparkreceiver/2/README.md| 2 +- website/www/site/content/en/documentation/io/built-in/cdap.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #25527: [CdapIO][SparkReceiverIO] Complete examples links in READMEs and CdapIO website page fixed
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 20c8d3a2ba11f3965022e2ba5e12a241a4ec3442 Merge: 40838f76447 ee9244d6261 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Fri Feb 17 15:30:22 2023 +0100 Merge pull request #25527: [CdapIO][SparkReceiverIO] Complete examples links in READMEs and CdapIO website page fixed sdks/java/io/cdap/README.md | 2 +- sdks/java/io/sparkreceiver/2/README.md| 2 +- website/www/site/content/en/documentation/io/built-in/cdap.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (9f7c55f40d3 -> 869150cdd2e)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9f7c55f40d3 Bump github.com/dustin/go-humanize from 1.0.0 to 1.0.1 in /sdks (#25017) add 39a348ef536 BEAM-13261 added max connections setting add 351f40393e8 BEAM-13261 updated comment for max connection add 869150cdd2e Merge pull request #24851: BEAM-13261 added max connections setting No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 20 .../beam/sdk/io/jdbc/JdbcSchemaIOProvider.java | 6 ++ 2 files changed, 26 insertions(+)
[beam] 01/01: Merge pull request #24854: Support SqlTypes Date in AvroUtils (sync)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 37358ff0196fc19b8f517d52df8c81099b78c219 Merge: 0f423dd6cc1 0d835cdc72f Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Jan 2 17:04:54 2023 +0100 Merge pull request #24854: Support SqlTypes Date in AvroUtils (sync) .../sdk/extensions/avro/schemas/utils/AvroUtils.java | 10 +- .../extensions/avro/schemas/utils/AvroUtilsTest.java | 20 2 files changed, 29 insertions(+), 1 deletion(-)
[beam] branch master updated (0f423dd6cc1 -> 37358ff0196)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 0f423dd6cc1 Merge pull request #24747: [Website] update links from absolute to relative in md files add 0d835cdc72f Support SqlTypes Date in AvroUtils (sync) new 37358ff0196 Merge pull request #24854: Support SqlTypes Date in AvroUtils (sync) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/extensions/avro/schemas/utils/AvroUtils.java | 10 +- .../extensions/avro/schemas/utils/AvroUtilsTest.java | 20 2 files changed, 29 insertions(+), 1 deletion(-)
[beam] branch master updated (3ee5b86a3f1 -> 0f423dd6cc1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 3ee5b86a3f1 Retry create database in InfluxDbIOIT (#24800) add 587e41eccef [Website] update links from absolute to relative in md files new 0f423dd6cc1 Merge pull request #24747: [Website] update links from absolute to relative in md files The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../www/site/content/en/blog/ApachePlayground.md | 8 ++--- .../content/en/blog/adding-data-sources-to-sql.md | 2 +- website/www/site/content/en/blog/beam-2.21.0.md| 4 +-- website/www/site/content/en/blog/beam-2.25.0.md| 4 +-- website/www/site/content/en/blog/beam-2.32.0.md| 6 ++-- website/www/site/content/en/blog/beam-2.38.0.md| 2 +- website/www/site/content/en/blog/beam-2.42.0.md| 2 +- website/www/site/content/en/blog/beam-2.8.0.md | 2 +- .../content/en/blog/beam-katas-kotlin-release.md | 4 +-- .../content/en/blog/beam-sql-with-notebooks.md | 8 ++--- .../site/content/en/blog/beam-starter-projects.md | 2 +- .../content/en/blog/beam-summit-europe-2019.md | 4 +-- .../en/blog/dataframe-api-preview-available.md | 6 ++-- website/www/site/content/en/blog/go-2.40.md| 12 website/www/site/content/en/blog/gsoc-19.md| 4 +-- website/www/site/content/en/blog/hop-web-cloud.md | 6 ++-- .../content/en/blog/kafka-to-pubsub-example.md | 6 ++-- website/www/site/content/en/blog/ml-resources.md | 8 ++--- .../content/en/blog/python-improved-annotations.md | 4 +-- .../python-performance-runtime-type-checking.md| 2 +- website/www/site/content/en/blog/python-typing.md | 4 +-- website/www/site/content/en/blog/season-of-docs.md | 6 ++-- .../en/blog/splittable-do-fn-is-available.md | 4 +-- website/www/site/content/en/case-studies/hop.md| 2 +- .../www/site/content/en/case-studies/paloalto.md | 14 - .../www/site/content/en/case-studies/ricardo.md| 4 +-- website/www/site/content/en/case-studies/seznam.md | 8 ++--- .../www/site/content/en/case-studies/snowflake.md | 2 +- website/www/site/content/en/community/in-person.md | 2 +- website/www/site/content/en/contribute/_index.md | 16 +- .../en/contribute/get-started-contributing.md | 34 +++--- .../site/content/en/contribute/release-guide.md| 4 +-- .../dsls/dataframes/differences-from-pandas.md | 8 ++--- .../en/documentation/dsls/dataframes/overview.md | 8 ++--- .../dsls/sql/extensions/create-external-table.md | 2 +- .../en/documentation/io/built-in/snowflake.md | 16 +- .../content/en/documentation/ml/data-processing.md | 4 +-- .../en/documentation/ml/large-language-modeling.md | 2 +- .../en/documentation/ml/multi-model-pipelines.md | 6 ++-- .../en/documentation/ml/online-clustering.md | 4 +-- .../content/en/documentation/ml/orchestration.md | 2 +- .../site/content/en/documentation/ml/overview.md | 6 ++-- .../en/documentation/ml/runinference-metrics.md| 2 +- .../site/content/en/documentation/patterns/bqml.md | 2 +- ...lements-for-efficient-external-service-calls.md | 2 +- .../content/en/documentation/programming-guide.md | 6 ++-- .../content/en/documentation/runners/direct.md | 2 +- .../site/content/en/documentation/runners/spark.md | 2 +- .../site/content/en/documentation/runtime/model.md | 2 +- .../sdks/java-multi-language-pipelines.md | 4 +-- .../documentation/sdks/python-machine-learning.md | 4 +-- .../sdks/python-pipeline-dependencies.md | 2 +- .../en/documentation/sdks/python-streaming.md | 4 +-- .../java/aggregation/approximatequantiles.md | 2 +- .../java/aggregation/approximateunique.md | 2 +- .../transforms/java/aggregation/cogroupbykey.md| 2 +- .../transforms/java/aggregation/combine.md | 2 +- .../java/aggregation/combinewithcontext.md | 2 +- .../transforms/java/aggregation/count.md | 2 +- .../transforms/java/aggregation/distinct.md| 2 +- .../transforms/java/aggregation/groupbykey.md | 2 +- .../java/aggregation/groupintobatches.md | 2 +- .../transforms/java/aggregation/hllcount.md| 2 +- .../transforms/java/aggregation/latest.md | 2 +- .../transforms/java/aggregation/max.md | 2 +- .../transforms/java/aggregation/mean.md| 2 +- .../transforms/java/aggregation/min.md | 2 +- .../transforms/java/aggregation/sample.md | 2 +- .../transforms/java/aggregation/sum.md | 2 +- .../transforms/java/aggregation/top
[beam] 01/01: Merge pull request #24747: [Website] update links from absolute to relative in md files
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0f423dd6cc1d17d7fb4286de59bd4a38370594bc Merge: 3ee5b86a3f1 587e41eccef Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Jan 2 15:46:47 2023 +0100 Merge pull request #24747: [Website] update links from absolute to relative in md files .../www/site/content/en/blog/ApachePlayground.md | 8 ++--- .../content/en/blog/adding-data-sources-to-sql.md | 2 +- website/www/site/content/en/blog/beam-2.21.0.md| 4 +-- website/www/site/content/en/blog/beam-2.25.0.md| 4 +-- website/www/site/content/en/blog/beam-2.32.0.md| 6 ++-- website/www/site/content/en/blog/beam-2.38.0.md| 2 +- website/www/site/content/en/blog/beam-2.42.0.md| 2 +- website/www/site/content/en/blog/beam-2.8.0.md | 2 +- .../content/en/blog/beam-katas-kotlin-release.md | 4 +-- .../content/en/blog/beam-sql-with-notebooks.md | 8 ++--- .../site/content/en/blog/beam-starter-projects.md | 2 +- .../content/en/blog/beam-summit-europe-2019.md | 4 +-- .../en/blog/dataframe-api-preview-available.md | 6 ++-- website/www/site/content/en/blog/go-2.40.md| 12 website/www/site/content/en/blog/gsoc-19.md| 4 +-- website/www/site/content/en/blog/hop-web-cloud.md | 6 ++-- .../content/en/blog/kafka-to-pubsub-example.md | 6 ++-- website/www/site/content/en/blog/ml-resources.md | 8 ++--- .../content/en/blog/python-improved-annotations.md | 4 +-- .../python-performance-runtime-type-checking.md| 2 +- website/www/site/content/en/blog/python-typing.md | 4 +-- website/www/site/content/en/blog/season-of-docs.md | 6 ++-- .../en/blog/splittable-do-fn-is-available.md | 4 +-- website/www/site/content/en/case-studies/hop.md| 2 +- .../www/site/content/en/case-studies/paloalto.md | 14 - .../www/site/content/en/case-studies/ricardo.md| 4 +-- website/www/site/content/en/case-studies/seznam.md | 8 ++--- .../www/site/content/en/case-studies/snowflake.md | 2 +- website/www/site/content/en/community/in-person.md | 2 +- website/www/site/content/en/contribute/_index.md | 16 +- .../en/contribute/get-started-contributing.md | 34 +++--- .../site/content/en/contribute/release-guide.md| 4 +-- .../dsls/dataframes/differences-from-pandas.md | 8 ++--- .../en/documentation/dsls/dataframes/overview.md | 8 ++--- .../dsls/sql/extensions/create-external-table.md | 2 +- .../en/documentation/io/built-in/snowflake.md | 16 +- .../content/en/documentation/ml/data-processing.md | 4 +-- .../en/documentation/ml/large-language-modeling.md | 2 +- .../en/documentation/ml/multi-model-pipelines.md | 6 ++-- .../en/documentation/ml/online-clustering.md | 4 +-- .../content/en/documentation/ml/orchestration.md | 2 +- .../site/content/en/documentation/ml/overview.md | 6 ++-- .../en/documentation/ml/runinference-metrics.md| 2 +- .../site/content/en/documentation/patterns/bqml.md | 2 +- ...lements-for-efficient-external-service-calls.md | 2 +- .../content/en/documentation/programming-guide.md | 6 ++-- .../content/en/documentation/runners/direct.md | 2 +- .../site/content/en/documentation/runners/spark.md | 2 +- .../site/content/en/documentation/runtime/model.md | 2 +- .../sdks/java-multi-language-pipelines.md | 4 +-- .../documentation/sdks/python-machine-learning.md | 4 +-- .../sdks/python-pipeline-dependencies.md | 2 +- .../en/documentation/sdks/python-streaming.md | 4 +-- .../java/aggregation/approximatequantiles.md | 2 +- .../java/aggregation/approximateunique.md | 2 +- .../transforms/java/aggregation/cogroupbykey.md| 2 +- .../transforms/java/aggregation/combine.md | 2 +- .../java/aggregation/combinewithcontext.md | 2 +- .../transforms/java/aggregation/count.md | 2 +- .../transforms/java/aggregation/distinct.md| 2 +- .../transforms/java/aggregation/groupbykey.md | 2 +- .../java/aggregation/groupintobatches.md | 2 +- .../transforms/java/aggregation/hllcount.md| 2 +- .../transforms/java/aggregation/latest.md | 2 +- .../transforms/java/aggregation/max.md | 2 +- .../transforms/java/aggregation/mean.md| 2 +- .../transforms/java/aggregation/min.md | 2 +- .../transforms/java/aggregation/sample.md | 2 +- .../transforms/java/aggregation/sum.md | 2 +- .../transforms/java/aggregation/top.md | 2 +- .../transforms/java/elementwise/filter.md | 2 +- .../transforms/java/elementwise/flatmapelements.md | 2 +- .../transforms/java/elementwise/keys.md| 2 +- .../transforms/java/elementwise/kvs
[beam] branch master updated (994f733a409 -> bab773fdcf1)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 994f733a409 Merge pull request #1: [Website] update check-links add 873e0d0978d [#24292] Create Avro extension for Java SDK add 8b532db7db0 Address the review comments add 150877e6494 Apply #24454 changes new bab773fdcf1 Merge pull request #24294: Create Avro extension for Java SDK The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/io/BlockBasedSource.java |4 +- .../apache/beam/sdk/io/DefaultFilenamePolicy.java |2 +- .../beam/sdk/io/ReadAllViaFileBasedSource.java |2 +- sdks/java/extensions/avro/build.gradle | 56 + .../beam/sdk/extensions/avro/coders/AvroCoder.java | 820 .../extensions/avro/coders/AvroGenericCoder.java | 32 + .../sdk/extensions/avro/coders/package-info.java | 29 + .../apache/beam/sdk/extensions/avro/io/AvroIO.java | 2043 .../extensions/avro/io/AvroSchemaIOProvider.java | 150 ++ .../beam/sdk/extensions/avro/io/AvroSink.java | 161 ++ .../beam/sdk/extensions/avro/io/AvroSource.java| 777 .../avro/io/ConstantAvroDestination.java | 148 ++ .../avro/io/DynamicAvroDestinations.java | 55 + .../avro/io/SerializableAvroCodecFactory.java | 112 ++ .../beam/sdk/extensions/avro/io/package-info.java | 26 + .../extensions/avro/schemas/AvroRecordSchema.java | 64 + .../io/payloads/AvroPayloadSerializerProvider.java | 44 + .../avro/schemas/io/payloads/package-info.java | 27 + .../sdk/extensions/avro/schemas/package-info.java | 29 + .../avro/schemas/utils/AvroByteBuddyUtils.java | 142 ++ .../extensions/avro/schemas/utils/AvroUtils.java | 1374 + .../avro/schemas/utils/package-info.java | 26 + .../apache/beam/sdk/extensions/avro/io/user.avsc | 10 + .../beam/sdk/extensions/avro/schemas/test.avsc | 30 + .../sdk/extensions/avro/coders/AvroCoderTest.java | 1108 +++ .../extensions/avro/coders/AvroCoderTestPojo.java | 51 + .../beam/sdk/extensions/avro/io/AvroIOTest.java| 1587 +++ .../avro/io/AvroSchemaIOProviderTest.java | 174 ++ .../sdk/extensions/avro/io/AvroSourceTest.java | 846 .../avro/io/SerializableAvroCodecFactoryTest.java | 93 + .../extensions/avro/schemas/AvroSchemaTest.java| 497 + .../io/AvroPayloadSerializerProviderTest.java | 64 + .../avro/schemas/utils/AvroGenerators.java | 220 +++ .../avro/schemas/utils/AvroUtilsTest.java | 895 + settings.gradle.kts|1 + 35 files changed, 11695 insertions(+), 4 deletions(-) create mode 100644 sdks/java/extensions/avro/build.gradle create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/coders/AvroCoder.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/coders/AvroGenericCoder.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/coders/package-info.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSchemaIOProvider.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSink.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSource.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/ConstantAvroDestination.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/DynamicAvroDestinations.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/SerializableAvroCodecFactory.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/package-info.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/AvroRecordSchema.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/io/payloads/AvroPayloadSerializerProvider.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/io/payloads/package-info.java create mode 100644 sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/package-info.java
[beam] 01/01: Merge pull request #24294: Create Avro extension for Java SDK
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bab773fdcf1abe96a7f912a1f4a2795014a22e36 Merge: 994f733a409 150877e6494 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Dec 28 15:53:31 2022 +0100 Merge pull request #24294: Create Avro extension for Java SDK .../org/apache/beam/sdk/io/BlockBasedSource.java |4 +- .../apache/beam/sdk/io/DefaultFilenamePolicy.java |2 +- .../beam/sdk/io/ReadAllViaFileBasedSource.java |2 +- sdks/java/extensions/avro/build.gradle | 56 + .../beam/sdk/extensions/avro/coders/AvroCoder.java | 820 .../extensions/avro/coders/AvroGenericCoder.java | 32 + .../sdk/extensions/avro/coders/package-info.java | 29 + .../apache/beam/sdk/extensions/avro/io/AvroIO.java | 2043 .../extensions/avro/io/AvroSchemaIOProvider.java | 150 ++ .../beam/sdk/extensions/avro/io/AvroSink.java | 161 ++ .../beam/sdk/extensions/avro/io/AvroSource.java| 777 .../avro/io/ConstantAvroDestination.java | 148 ++ .../avro/io/DynamicAvroDestinations.java | 55 + .../avro/io/SerializableAvroCodecFactory.java | 112 ++ .../beam/sdk/extensions/avro/io/package-info.java | 26 + .../extensions/avro/schemas/AvroRecordSchema.java | 64 + .../io/payloads/AvroPayloadSerializerProvider.java | 44 + .../avro/schemas/io/payloads/package-info.java | 27 + .../sdk/extensions/avro/schemas/package-info.java | 29 + .../avro/schemas/utils/AvroByteBuddyUtils.java | 142 ++ .../extensions/avro/schemas/utils/AvroUtils.java | 1374 + .../avro/schemas/utils/package-info.java | 26 + .../apache/beam/sdk/extensions/avro/io/user.avsc | 10 + .../beam/sdk/extensions/avro/schemas/test.avsc | 30 + .../sdk/extensions/avro/coders/AvroCoderTest.java | 1108 +++ .../extensions/avro/coders/AvroCoderTestPojo.java | 51 + .../beam/sdk/extensions/avro/io/AvroIOTest.java| 1587 +++ .../avro/io/AvroSchemaIOProviderTest.java | 174 ++ .../sdk/extensions/avro/io/AvroSourceTest.java | 846 .../avro/io/SerializableAvroCodecFactoryTest.java | 93 + .../extensions/avro/schemas/AvroSchemaTest.java| 497 + .../io/AvroPayloadSerializerProviderTest.java | 64 + .../avro/schemas/utils/AvroGenerators.java | 220 +++ .../avro/schemas/utils/AvroUtilsTest.java | 895 + settings.gradle.kts|1 + 35 files changed, 11695 insertions(+), 4 deletions(-)
[beam] branch master updated (e28f6f79f2b -> 994f733a409)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from e28f6f79f2b [Spark Dataset runner] Trigger evaluation using write noop rather than using foreach (closes #24797) (#24565) add a43b834681f [Website] update check links, catch prod & staging links add 994f733a409 Merge pull request #1: [Website] update check-links No new revisions were added by this update. Summary of changes: website/www/check-links.sh | 91 ++ 1 file changed, 76 insertions(+), 15 deletions(-)
[beam] branch master updated (f80ac005bfd -> eb23b0a123e)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from f80ac005bfd [Release Tasks Migration] - Git Tag Workflow (#24418) add 46c77783b1e Fix POM of beam-sdks-java-core (closes #24675) new eb23b0a123e Merge pull request #24760: Fix POM of beam-sdks-java-core The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #24760: Fix POM of beam-sdks-java-core
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit eb23b0a123ea94429f34385194e8eca7299d66cd Merge: f80ac005bfd 46c77783b1e Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Dec 22 16:48:06 2022 +0100 Merge pull request #24760: Fix POM of beam-sdks-java-core buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #24635: [Website] update links with 404 status code
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 3f67b2c07697e1f7ec8f8d8bdf017099638d4d02 Merge: ce5fcac379f 4764f0e2043 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Dec 22 16:29:17 2022 +0100 Merge pull request #24635: [Website] update links with 404 status code website/www/site/content/en/blog/review-input-streaming-connectors.md | 2 +- website/www/site/content/en/contribute/postcommits-policies.md| 2 +- website/www/site/content/en/contribute/precommit-triage-guide.md | 4 ++-- website/www/site/content/en/contribute/runner-guide.md| 2 +- website/www/site/content/en/documentation/programming-guide.md| 2 +- 5 files changed, 6 insertions(+), 6 deletions(-)
[beam] branch master updated (ce5fcac379f -> 3f67b2c0769)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ce5fcac379f [Website] delete 404 links #24745 (#24744) add 4764f0e2043 [Website] update links with 404 status code new 3f67b2c0769 Merge pull request #24635: [Website] update links with 404 status code The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/content/en/blog/review-input-streaming-connectors.md | 2 +- website/www/site/content/en/contribute/postcommits-policies.md| 2 +- website/www/site/content/en/contribute/precommit-triage-guide.md | 4 ++-- website/www/site/content/en/contribute/runner-guide.md| 2 +- website/www/site/content/en/documentation/programming-guide.md| 2 +- 5 files changed, 6 insertions(+), 6 deletions(-)
[beam] 01/01: Merge pull request #24594: [Website] Fix join beam page mobile layout
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 506c8f9cab83caba4cf93e4ecd7025c9a60be11f Merge: 77af3237521 d5d19e54b9c Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Dec 14 13:19:16 2022 +0100 Merge pull request #24594: [Website] Fix join beam page mobile layout website/www/site/assets/scss/_lists.scss | 9 +++-- website/www/site/assets/scss/_local.scss | 8 website/www/site/content/en/community/join-beam.md | 3 +++ 3 files changed, 14 insertions(+), 6 deletions(-)
[beam] branch master updated (77af3237521 -> 506c8f9cab8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 77af3237521 Merge pull request #24624: Eliminate rawtypes from GCP IO add d5d19e54b9c [Website] fix join beam page mobile layout #24611 new 506c8f9cab8 Merge pull request #24594: [Website] Fix join beam page mobile layout The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/assets/scss/_lists.scss | 9 +++-- website/www/site/assets/scss/_local.scss | 8 website/www/site/content/en/community/join-beam.md | 3 +++ 3 files changed, 14 insertions(+), 6 deletions(-)
[beam] branch master updated (83679216cce -> 3f2aecfc5bc)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 83679216cce Merge pull request #24380:[Spark RDD runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators add f0c7f86c990 [Spark Dataset runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators from the runner (closes #24379) add 3f2aecfc5bc Merge pull request #24381: [Spark Dataset runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators No new revisions were added by this update. Summary of changes: .../SparkStructuredStreamingRunner.java| 28 ++--- .../aggregators/AggregatorsAccumulator.java| 73 - .../aggregators/NamedAggregators.java | 113 - .../aggregators/NamedAggregatorsAccumulator.java | 63 .../aggregators/package-info.java | 20 .../metrics/AggregatorMetric.java | 74 -- .../metrics/AggregatorMetricSource.java| 49 - .../metrics/CompositeSource.java | 45 .../metrics/SparkBeamMetricSource.java | 4 +- .../metrics/sink/CodahaleCsvSink.java | 3 +- .../metrics/sink/CodahaleGraphiteSink.java | 3 +- 11 files changed, 10 insertions(+), 465 deletions(-) delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/AggregatorsAccumulator.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregators.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregatorsAccumulator.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/package-info.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetric.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetricSource.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/CompositeSource.java
[beam] 01/01: Merge pull request #24380:[Spark RDD runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 83679216cce2d52dbeb7e837f06ca1d57b31d509 Merge: 59849d6609b 5a60769700d Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Dec 12 15:41:31 2022 +0100 Merge pull request #24380:[Spark RDD runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators .../beam/runners/spark/SparkPipelineRunner.java| 5 - .../org/apache/beam/runners/spark/SparkRunner.java | 20 +--- .../apache/beam/runners/spark/TestSparkRunner.java | 2 - .../spark/aggregators/AggregatorsAccumulator.java | 133 - .../spark/aggregators/NamedAggregators.java| 113 - .../aggregators/NamedAggregatorsAccumulator.java | 63 -- .../runners/spark/aggregators/package-info.java| 20 .../runners/spark/metrics/AggregatorMetric.java| 74 .../spark/metrics/AggregatorMetricSource.java | 49 .../runners/spark/metrics/CompositeSource.java | 45 --- .../beam/runners/spark/metrics/sink/CsvSink.java | 3 +- .../runners/spark/metrics/sink/GraphiteSink.java | 3 +- .../ResumeFromCheckpointStreamingTest.java | 2 - 13 files changed, 5 insertions(+), 527 deletions(-)
[beam] branch master updated (59849d6609b -> 83679216cce)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 59849d6609b Merge pull request #24195: Website update keywords table add 5a60769700d [Spark RDD runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators from the runner (closes #24379) new 83679216cce Merge pull request #24380:[Spark RDD runner] Remove obsolete unusable AggregatorsAccumulator / NamedAggregators The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/spark/SparkPipelineRunner.java| 5 - .../org/apache/beam/runners/spark/SparkRunner.java | 20 +--- .../apache/beam/runners/spark/TestSparkRunner.java | 2 - .../spark/aggregators/AggregatorsAccumulator.java | 133 - .../spark/aggregators/NamedAggregators.java| 113 - .../aggregators/NamedAggregatorsAccumulator.java | 63 -- .../runners/spark/aggregators/package-info.java| 20 .../runners/spark/metrics/AggregatorMetric.java| 74 .../spark/metrics/AggregatorMetricSource.java | 49 .../runners/spark/metrics/CompositeSource.java | 45 --- .../beam/runners/spark/metrics/sink/CsvSink.java | 3 +- .../runners/spark/metrics/sink/GraphiteSink.java | 3 +- .../ResumeFromCheckpointStreamingTest.java | 2 - 13 files changed, 5 insertions(+), 527 deletions(-) delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/AggregatorsAccumulator.java delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/NamedAggregators.java delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/NamedAggregatorsAccumulator.java delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/package-info.java delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/AggregatorMetric.java delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/AggregatorMetricSource.java delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/CompositeSource.java
[beam] branch master updated (6ed0cf57c58 -> 59849d6609b)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6ed0cf57c58 Bump google.golang.org/api from 0.103.0 to 0.104.0 in /sdks (#24586) add a845bbab505 [Website] add table-wrapper--equal-p table-wrapper--pr classes, use them in md files with tables #24193 #24194 add 59849d6609b Merge pull request #24195: Website update keywords table No new revisions were added by this update. Summary of changes: website/www/site/assets/scss/_table-wrapper.scss | 15 ++ .../documentation/dsls/sql/calcite/data-types.md | 2 +- .../dsls/sql/calcite/lexical-structure.md | 4 +-- .../dsls/sql/extensions/create-external-table.md | 4 ++- .../dsls/sql/zetasql/conditional-expressions.md| 4 +-- .../dsls/sql/zetasql/conversion-rules.md | 6 ++-- .../documentation/dsls/sql/zetasql/data-types.md | 8 ++--- .../en/documentation/dsls/sql/zetasql/lexical.md | 6 ++-- .../dsls/sql/zetasql/math-functions.md | 4 +-- .../en/documentation/dsls/sql/zetasql/operators.md | 18 ++-- .../documentation/dsls/sql/zetasql/query-syntax.md | 12 .../content/en/documentation/programming-guide.md | 34 +- .../www/site/content/en/get-started/from-spark.md | 4 +-- website/www/site/layouts/shortcodes/table.html | 2 ++ 14 files changed, 74 insertions(+), 49 deletions(-)
[beam] branch master updated (f1fcf0a96c6 -> 63b8ad22b03)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from f1fcf0a96c6 Composite Action to activate SA in GitHub Actions Runners (#24626) add f228dce7a17 Remove 'examples' dependency from HadoopFormatIO new 63b8ad22b03 Merge pull request #24564: [CdapIO] Remove 'examples' dependency from HadoopFormatIO The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/hadoop-format/build.gradle| 2 - .../format/HadoopFormatIOSequenceFileTest.java | 61 -- 2 files changed, 58 insertions(+), 5 deletions(-)
[beam] 01/01: Merge pull request #24564: [CdapIO] Remove 'examples' dependency from HadoopFormatIO
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 63b8ad22b03e00e983b19698640f331a24ba5b47 Merge: f1fcf0a96c6 f228dce7a17 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Dec 12 13:32:03 2022 +0100 Merge pull request #24564: [CdapIO] Remove 'examples' dependency from HadoopFormatIO sdks/java/io/hadoop-format/build.gradle| 2 - .../format/HadoopFormatIOSequenceFileTest.java | 61 -- 2 files changed, 58 insertions(+), 5 deletions(-)
[beam] branch master updated: [CdapIO] Add CdapIO and SparkReceiverIO documentation in website (#23338)
This is an automated email from the ASF dual-hosted git repository.
aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ac29569ae1a [CdapIO] Add CdapIO and SparkReceiverIO documentation in
website (#23338)
ac29569ae1a is described below
commit ac29569ae1a6681cfaee466034fdfa8f9184f49f
Author: Vitaly Terentyev
AuthorDate: Tue Dec 6 20:56:57 2022 +0400
[CdapIO] Add CdapIO and SparkReceiverIO documentation in website (#23338)
* Add CdapIO and SparkReceiverIO to the connectors table
* Add Cdap documentation
* Add SparkReceiver documentation
* Fix whitespace
* Resolve comments
---
.../content/en/documentation/io/built-in/cdap.md | 396 +
.../en/documentation/io/built-in/sparkreceiver.md | 103 ++
.../site/content/en/documentation/io/connectors.md | 26 ++
.../partials/section-menu/en/documentation.html| 2 +
4 files changed, 527 insertions(+)
diff --git a/website/www/site/content/en/documentation/io/built-in/cdap.md
b/website/www/site/content/en/documentation/io/built-in/cdap.md
new file mode 100644
index 000..3ed4fd06ed0
--- /dev/null
+++ b/website/www/site/content/en/documentation/io/built-in/cdap.md
@@ -0,0 +1,396 @@
+---
+title: "Cdap IO"
+---
+
+
+# Cdap IO
+
+A `CdapIO` is a transform for reading data from source or writing data to sink
CDAP plugin.
+
+## Batch plugins support
+
+`CdapIO` currently supports the following CDAP Batch plugins by referencing
`CDAP plugin` class name:
+- [Hubspot Batch
Source](https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/batch/HubspotBatchSource.java)
+- [Hubspot Batch
Sink](https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/sink/batch/HubspotBatchSink.java)
+- [Salesforce Batch
Source](https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/source/batch/SalesforceBatchSource.java)
+- [Salesforce Batch
Sink](https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/sink/batch/SalesforceBatchSink.java)
+- [ServiceNow Batch
Source](https://github.com/data-integrations/servicenow-plugins/blob/develop/src/main/java/io/cdap/plugin/servicenow/source/ServiceNowSource.java)
+- [Zendesk Batch
Source](https://github.com/data-integrations/zendesk/blob/develop/src/main/java/io/cdap/plugin/zendesk/source/batch/ZendeskBatchSource.java)
+
+Also, any other CDAP Batch plugin based on Hadoop's `InputFormat` or
`OutputFormat` can be used. They can be easily added to the list of supported
by class name plugins, for more details please see [CdapIO
readme](https://github.com/apache/beam/blob/master/sdks/java/io/cdap/README.md).
+
+## Streaming plugins support
+
+`CdapIO` currently supports CDAP Streaming plugins based on [Apache Spark
Receiver](https://spark.apache.org/docs/2.4.0/streaming-custom-receivers.html).
+
+Requirements for CDAP Streaming plugins:
+- CDAP Streaming plugin should be based on `Spark Receiver` (Spark 2.4).
+- CDAP Streaming plugin should support work with offsets.
+- Corresponding Spark Receiver should implement
[HasOffset](https://github.com/apache/beam/blob/master/sdks/java/io/sparkreceiver/src/main/java/org/apache/beam/sdk/io/sparkreceiver/HasOffset.java)
interface.
+- Records should have the numeric field that represents record offset.
+
+## Batch reading using CdapIO
+
+In order to read from CDAP plugin you will need to pass:
+- `Key` and `Value` classes. You will need to check if these classes have a
Beam Coder available.
+- `PluginConfig` object with parameters for certain CDAP plugin.
+
+You can easily build `PluginConfig` object using `ConfigWrapper` class by
specifying:
+
+- Class of the needed `PluginConfig`.
+- `Map` parameters map for corresponding CDAP plugin.
+
+For example:
+
+{{< highlight java >}}
+Map myPluginConfigParams = new HashMap<>();
+// Read plugin parameters (e.g. from PipelineOptions) and put them into
'myPluginConfigParams' map.
+myPluginConfigParams.put(MyPluginConstants.USERNAME_PARAMETER_NAME,
pipelineOptions.getUsername());
+// ...
+MyPluginConfig pluginConfig =
+ new
ConfigWrapper<>(MyPluginConfig.class).withParams(myPluginConfigParams).build();
+{{< /highlight >}}
+
+### Read data by plugin class name
+
+Some CDAP plugins are already supported and can be used just by plugin class
name.
+
+For example:
+
+{{< highlight java >}}
+CdapIO.Read readTransform =
+ CdapIO.read()
+.withCdapPluginClass(HubspotBatchSource.class)
+.withPluginConfig(pluginConfig)
+.withKeyClass(NullWritable.class)
+.withValueClass(JsonElement.class);
+p.apply("read", readTransform);
+{{< /highlight >}}
+
+### Read data with building Ba
[beam] branch master updated (cceac3b55d9 -> 4fd73625660)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from cceac3b55d9 Merge pull request #24399: [Website] Update copy icon positioning add b4d6b6652bf [Spark Dataset runner] Broadcast pipeline options new 4fd73625660 Merge pull request #24523: [Spark Dataset runner] Broadcast pipeline options The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../io/BoundedDatasetFactory.java | 10 +++--- .../translation/PipelineTranslator.java| 39 +++--- .../translation/SparkSessionFactory.java | 5 +++ .../translation/TransformTranslator.java | 9 ++--- .../batch/DoFnMapPartitionsFactory.java| 7 ++-- .../batch/GroupByKeyTranslatorBatch.java | 2 +- .../translation/batch/ParDoTranslatorBatch.java| 2 +- .../batch/ReadSourceTranslatorBatch.java | 5 +-- .../GroupAlsoByWindowViaOutputBufferFn.java| 7 ++-- 9 files changed, 62 insertions(+), 24 deletions(-)
[beam] 01/01: Merge pull request #24523: [Spark Dataset runner] Broadcast pipeline options
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4fd73625660a4c27cf36c6d639cba5c6e5644d15 Merge: cceac3b55d9 b4d6b6652bf Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Dec 6 11:28:52 2022 +0100 Merge pull request #24523: [Spark Dataset runner] Broadcast pipeline options .../io/BoundedDatasetFactory.java | 10 +++--- .../translation/PipelineTranslator.java| 39 +++--- .../translation/SparkSessionFactory.java | 5 +++ .../translation/TransformTranslator.java | 9 ++--- .../batch/DoFnMapPartitionsFactory.java| 7 ++-- .../batch/GroupByKeyTranslatorBatch.java | 2 +- .../translation/batch/ParDoTranslatorBatch.java| 2 +- .../batch/ReadSourceTranslatorBatch.java | 5 +-- .../GroupAlsoByWindowViaOutputBufferFn.java| 7 ++-- 9 files changed, 62 insertions(+), 24 deletions(-)
[beam] branch master updated (e52039c5ada -> cceac3b55d9)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from e52039c5ada Merge pull request #24427: [Website] center the main content of about page add 895eb19ad78 [Website] update copy icon positioning #24426 add cceac3b55d9 Merge pull request #24399: [Website] Update copy icon positioning No new revisions were added by this update. Summary of changes: website/www/site/assets/scss/_table-wrapper.scss | 10 ++ 1 file changed, 10 insertions(+)
[beam] 01/01: Merge pull request #24427: [Website] center the main content of about page
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e52039c5ada9cb4758cd2179ba71bbee220aad35 Merge: ad92087babb ff60ac3147a Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Dec 6 11:25:07 2022 +0100 Merge pull request #24427: [Website] center the main content of about page website/www/site/layouts/about/baseof.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (ad92087babb -> e52039c5ada)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ad92087babb Update dataflow container version for Pandas upgrade (#24532) add ff60ac3147a [Website] center the main content #24521 new e52039c5ada Merge pull request #24427: [Website] center the main content of about page The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/layouts/about/baseof.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (a9dcf95453d -> 66db2d81811)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) add c197579e9f2 [Spark Dataset runner] Add @Experimental and reduce visibility where possible. new 66db2d81811 Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../SparkStructuredStreamingPipelineOptions.java | 2 ++ .../spark/structuredstreaming/SparkStructuredStreamingRunner.java | 2 ++ .../spark/structuredstreaming/translation/batch/Aggregators.java | 8 +--- .../translation/batch/CombineGloballyTranslatorBatch.java | 2 +- .../translation/batch/CombineGroupedValuesTranslatorBatch.java| 2 +- .../translation/batch/GroupByKeyTranslatorBatch.java | 4 ++-- .../translation/batch/ImpulseTranslatorBatch.java | 3 +-- .../translation/batch/ReshuffleTranslatorBatch.java | 4 ++-- 8 files changed, 16 insertions(+), 11 deletions(-)
[beam] 01/01: Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 66db2d81811dc26e624f2472415d45e879c076c0 Merge: a9dcf95453d c197579e9f2 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Dec 5 14:08:01 2022 +0100 Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible .../SparkStructuredStreamingPipelineOptions.java | 2 ++ .../spark/structuredstreaming/SparkStructuredStreamingRunner.java | 2 ++ .../spark/structuredstreaming/translation/batch/Aggregators.java | 8 +--- .../translation/batch/CombineGloballyTranslatorBatch.java | 2 +- .../translation/batch/CombineGroupedValuesTranslatorBatch.java| 2 +- .../translation/batch/GroupByKeyTranslatorBatch.java | 4 ++-- .../translation/batch/ImpulseTranslatorBatch.java | 3 +-- .../translation/batch/ReshuffleTranslatorBatch.java | 4 ++-- 8 files changed, 16 insertions(+), 11 deletions(-)
[beam] branch master updated: [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) a9dcf95453d is described below commit a9dcf95453d43621ef3ce63782f7cd6c47399bb6 Author: Vitaly Terentyev AuthorDate: Mon Dec 5 14:45:31 2022 +0400 [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) * Add README for CdapIO. Update README for SparkReceiverIO. * Set export javadoc true for cdap and sparkreceiver * Updates to CdapIO and SparkReceiverIO readmes * Add manual how to add support for Batch and Streaming Cdap plugins. * Add manual how to add support for Spark Receiver. * Fix whitespace * updates to CDAP and SparkReceiver readme files * Updated CDAP readme * Fix links in readme Co-authored-by: Alex Kosolapov --- sdks/java/io/cdap/README.md | 145 ++ sdks/java/io/cdap/build.gradle| 1 - sdks/java/io/sparkreceiver/2/README.md| 40 +++-- sdks/java/io/sparkreceiver/2/build.gradle | 1 - 4 files changed, 180 insertions(+), 7 deletions(-) diff --git a/sdks/java/io/cdap/README.md b/sdks/java/io/cdap/README.md new file mode 100644 index 000..5269204c1d1 --- /dev/null +++ b/sdks/java/io/cdap/README.md @@ -0,0 +1,145 @@ + + +# CdapIO +CdapIO provides I/O transforms for [CDAP](https://cdap.io/) plugins. + +## What is CDAP? + +[CDAP](https://cdap.io/) is an application platform for building and managing data applications in hybrid and multi-cloud environments. +It enables developers, business analysts, and data scientists to use a visual rapid development environment and utilize common patterns, +data, and application abstractions to accelerate the development of data applications, addressing a broader range of real-time and batch use cases. + +[CDAP plugins](https://github.com/data-integrations) types: +- Batch source +- Batch sink +- Streaming source + +To learn more about CDAP plugins please see [io.cdap.cdap.api.annotation.Plugin](https://javadoc.io/static/io.cdap.cdap/cdap-api/6.7.2/io/cdap/cdap/api/annotation/Plugin.html) and [Data Integrations](https://github.com/data-integrations) plugins repository. + +## CDAP Batch plugins support in CDAP IO + +CdapIO supports CDAP Batch plugins based on Hadoop [InputFormat](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/InputFormat.html) and [OutputFormat](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/OutputFormat.html). +CDAP batch plugins support is implemented using [HadoopFormatIO](https://beam.apache.org/documentation/io/built-in/hadoop/). + +CdapIO currently supports the following CDAP Batch plugins by referencing `CDAP plugin` class: +* [Hubspot Batch Source](https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/batch/HubspotBatchSource.java) +* [Hubspot Batch Sink](https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/sink/batch/HubspotBatchSink.java) +* [Salesforce Batch Source](https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/source/batch/SalesforceBatchSource.java) +* [Salesforce Batch Sink](https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/sink/batch/SalesforceBatchSink.java) +* [ServiceNow Batch Source](https://github.com/data-integrations/servicenow-plugins/blob/develop/src/main/java/io/cdap/plugin/servicenow/source/ServiceNowSource.java) +* [Zendesk Batch Source](https://github.com/data-integrations/zendesk/blob/develop/src/main/java/io/cdap/plugin/zendesk/source/batch/ZendeskBatchSource.java) + +It means that all these plugins can be used like this: +``CdapIO.withCdapPluginClass(HubspotBatchSource.class)`` + +### Requirements for Cdap Batch plugins + +CDAP Batch plugin should be based on `HadoopFormat` implementation. + +### How to add support for a new CDAP Batch plugin + +To add CdapIO support for a new CDAP Batch [Plugin](src/main/java/org/apache/beam/sdk/io/cdap/Plugin.java) perform the following steps: +1. Find CDAP plugin artifacts in the Maven Central repository. *Example:* [Hubspot plugin Maven repository](https://mvnrepository.com/artifact/io.cdap/hubspot-plugins/1.0.0). *Note:* To add a custom CDAP plugin, please follow [Sonatype publishing guidelines](https://central.sonatype.org/publish/). +2. Add the CDAP plugin Maven dependency to the `build.gradle` file. *Example:* ``implementation "io.cdap:hubspot-plugins:1.0.0"``. +3. Here are two ways of using CDAP batch plugin with CdapIO: + 1. Using `Plugin.createBatch()` method. Pass C
[beam] branch master updated (fd5ad1ec633 -> 46919b7bbf7)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from fd5ad1ec633 Reduce version matrix of build wheel on pull request event (#24448) add 46919b7bbf7 [Spark dataset runner] Make sure PCollection views get only broadcasted once if reused (#24288) No new revisions were added by this update. Summary of changes: .../translation/PipelineTranslator.java| 27 ++- .../translation/SparkSessionFactory.java | 148 .../translation/TransformTranslator.java | 24 ++- .../batch/DoFnMapPartitionsFactory.java| 31 +--- .../translation/batch/ParDoTranslatorBatch.java| 99 ++- .../batch/functions/CachedSideInputReader.java | 176 +++ .../batch/functions/SideInputValues.java | 189 .../batch/functions/SparkSideInputReader.java | 193 - .../translation/helpers/SideInputBroadcast.java| 49 -- .../translation/utils/CachedSideInputReader.java | 93 -- .../translation/utils/SideInputStorage.java| 107 .../batch/functions/SideInputValuesTest.java | 129 ++ runners/spark/spark_runner.gradle | 8 + 13 files changed, 825 insertions(+), 448 deletions(-) create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/CachedSideInputReader.java create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SideInputValues.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SideInputBroadcast.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/CachedSideInputReader.java delete mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/SideInputStorage.java create mode 100644 runners/spark/3/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SideInputValuesTest.java
[beam] branch master updated (3bd76cc130f -> 4ab557bf987)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 3bd76cc130f Deleted initialNumReaders paramter. (#24355) add 466db663a9b [Website] update copy-to-clipboard.js #24372 new 4ab557bf987 Merge pull request #24370: [Website] update copy-to-clipboard.js The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/assets/js/copy-to-clipboard.js | 2 ++ website/www/site/assets/scss/_global.sass | 3 +++ 2 files changed, 5 insertions(+)
[beam] 01/01: Merge pull request #24370: [Website] update copy-to-clipboard.js
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4ab557bf98784e51b6f8dca7d89ba5bcea9f04ef Merge: 3bd76cc130f 466db663a9b Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Wed Nov 30 12:18:07 2022 +0100 Merge pull request #24370: [Website] update copy-to-clipboard.js website/www/site/assets/js/copy-to-clipboard.js | 2 ++ website/www/site/assets/scss/_global.sass | 3 +++ 2 files changed, 5 insertions(+)
[beam] branch master updated (b9088ba62a2 -> db54c84e170)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b9088ba62a2 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24348) add 60719d1b4d7 [Spark Dataset runner] Fix support for Java 11 (closes #24392) add 36b89946883 fix spotless add db54c84e170 Merge pull request #24393: [Spark Dataset runner] Fix support for Java 11 No new revisions were added by this update. Summary of changes: .../structuredstreaming/translation/PipelineTranslator.java | 11 --- .../translation/helpers/EncoderHelpers.java | 3 ++- 2 files changed, 10 insertions(+), 4 deletions(-)
[beam] branch master updated (00828db664f -> 84d47ef4a2f)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 00828db664f Adds GCP core dependency to the test expansion service (#24308) add e0d0a72f559 [Spark dataset runner] Fix translation to run in the evaluation thread as well so exceptions are handled properly (closes #24330) add 84d47ef4a2f Merge pull request #24331: [Spark dataset runner] Fix handling of exceptions during translation No new revisions were added by this update. Summary of changes: .../SparkStructuredStreamingPipelineResult.java| 7 +-- .../SparkStructuredStreamingRunner.java| 58 -- 2 files changed, 35 insertions(+), 30 deletions(-)
[beam] branch master updated (a440fd600fc -> c0a328ffa77)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a440fd600fc Merge pull request #23462: [Website] Update pages with table content overflow add 9bfffb78b09 [Metrics] Add 'performance tests' tag to JMH dashboard (related to #22238) add c0a328ffa77 Merge pull request #24335: [Metrics] Add 'performance tests' tag to JMH dashboard No new revisions were added by this update. Summary of changes: .../grafana/dashboards/perftests_metrics/Java_JMH_benchmarks.json | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
[beam] branch master updated (8baae457744 -> a440fd600fc)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 8baae457744 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks (#24318) add 24f10b5888d [Website] update table text content overflow #23460 add a440fd600fc Merge pull request #23462: [Website] Update pages with table content overflow No new revisions were added by this update. Summary of changes: .../content/en/community/presentation-materials.md | 2 +- .../dsls/sql/calcite/lexical-structure.md | 2 ++ .../documentation/dsls/sql/zetasql/data-types.md | 24 +- .../en/documentation/dsls/sql/zetasql/lexical.md | 10 - .../en/documentation/dsls/sql/zetasql/operators.md | 12 ++- .../documentation/dsls/sql/zetasql/query-syntax.md | 12 ++- .../shortcodes/community/table_with_icons.html | 2 ++ 7 files changed, 59 insertions(+), 5 deletions(-)
[beam] 01/01: Merge pull request #24251: [Website] add lazy loading attr to images
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4d5937b9a3d268f6ec7ce919ca9cef8d1db994b3 Merge: 79b8c154b58 be346e619b0 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Nov 22 10:58:21 2022 +0100 Merge pull request #24251: [Website] add lazy loading attr to images website/www/site/layouts/case-studies/list.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (79b8c154b58 -> 4d5937b9a3d)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 79b8c154b58 Merge pull request #24023 Add a Dataflow smoke test. add be346e619b0 [Website] add lazy loading attr to images #24250 new 4d5937b9a3d Merge pull request #24251: [Website] add lazy loading attr to images The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/layouts/case-studies/list.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (33bcf89ac2f -> 2a8842b7dad)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 33bcf89ac2f Merge pull request #24211: Unify test parameters for certain IOs based on test row and Grafana fixes add 2dc6e2f8ca3 [Metrics] Add new performance dashboard for Java JMH benchmarks (closes #22238) add 2a8842b7dad Merge pull request #24032: [Metrics] Add new performance dashboard for Java JMH benchmarks No new revisions were added by this update. Summary of changes: .../perftests_metrics/Java_JMH_benchmarks.json | 1640 1 file changed, 1640 insertions(+) create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/Java_JMH_benchmarks.json
[beam] branch master updated (8cd94c5c439 -> 33bcf89ac2f)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 8cd94c5c439 Add enableGzipHttpCompression option to SplunkIO (#24197) add fb5418b1ef5 Unify test parameters for certain IOs based on test row and grafana fixes add 33bcf89ac2f Merge pull request #24211: Unify test parameters for certain IOs based on test row and Grafana fixes No new revisions were added by this update. Summary of changes: .../jenkins/job_PerformanceTests_CdapIO.groovy | 4 +- .../job_PerformanceTests_HadoopFormat.groovy | 4 +- .../jenkins/job_PerformanceTests_JDBC.groovy | 2 +- .../job_PerformanceTests_SparkReceiverIO_IT.groovy | 4 +- .../Java_IO_IT_Tests_Dataflow.json | 137 +++-- .../Python_IO_IT_Tests_Dataflow.json | 6 +- 6 files changed, 140 insertions(+), 17 deletions(-)
[beam] 01/01: Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bff4919ff00ec3b5d7186efde41c884dfc4c8344 Merge: 79ba69483a8 ce8d618c77d Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Thu Nov 17 10:34:02 2022 +0100 Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../io/elasticsearch/ElasticsearchIOTestUtils.java | 35 +++--- .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 17 --- 6 files changed, 69 insertions(+), 15 deletions(-)
[beam] branch master updated (79ba69483a8 -> bff4919ff00)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 79ba69483a8 Merge pull request #24218: Update Python wheel format for RC validation add a0146371069 Uses _all to follow alias/datastreams when estimating index size add ce8d618c77d Adds test for following aliases when estimating index size new bff4919ff00 Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 8 + .../io/elasticsearch/ElasticsearchIOTestUtils.java | 35 +++--- .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 17 --- 6 files changed, 69 insertions(+), 15 deletions(-)
[beam] branch master updated (e3b9bdb2e60 -> 2ee809fa0ca)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from e3b9bdb2e60 [Dockerized Jenkins] Fix build of dockerized jenkins (fixes #24053) (#24054) add 563c66d6fd3 [Website] update pre tag copy link styles #23064 new 2ee809fa0ca Merge pull request #23065: [Website] Update copy icon styles The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/assets/scss/_global.sass | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #23065: [Website] Update copy icon styles
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2ee809fa0ca7689dd0279e186ebc02d9569a8429 Merge: e3b9bdb2e60 563c66d6fd3 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Nov 15 11:01:14 2022 +0100 Merge pull request #23065: [Website] Update copy icon styles website/www/site/assets/scss/_global.sass | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
[beam] branch master updated (4e39ef20410 -> 2bb03d62e2d)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 4e39ef20410 Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links add 623083cd0a7 [Website] change headers size from h4,h3 to h2 #24082 add 2bb03d62e2d Merge pull request #24000: [Website] Change headers size from h4,h3 to h2 No new revisions were added by this update. Summary of changes: .../www/site/content/en/documentation/basics.md| 26 +++--- website/www/site/content/en/get-started/_index.md | 10 - 2 files changed, 18 insertions(+), 18 deletions(-)
[beam] 01/01: Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4e39ef20410ee51c6040317bcd60171e64c5171f Merge: 223768f782f 105ed6fedcb Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Nov 14 10:55:33 2022 +0100 Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links website/www/site/content/en/documentation/programming-guide.md | 6 +++--- website/www/site/content/en/documentation/sdks/go-dependencies.md | 4 ++-- website/www/site/content/en/documentation/sdks/java-dependencies.md | 2 +- .../www/site/content/en/documentation/sdks/python-dependencies.md | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-)
[beam] branch master updated (223768f782f -> 4e39ef20410)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 223768f782f Merge pull request #24136: Fix checkArgument format string in ExecutionStateTracker add 105ed6fedcb [Website] update go-dependencies.md java-dependencies.md programming-guide.md links #24084 new 4e39ef20410 Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/content/en/documentation/programming-guide.md | 6 +++--- website/www/site/content/en/documentation/sdks/go-dependencies.md | 4 ++-- website/www/site/content/en/documentation/sdks/java-dependencies.md | 2 +- .../www/site/content/en/documentation/sdks/python-dependencies.md | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-)
[beam] branch master updated (95c121b2549 -> 858f5048f57)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 95c121b2549 Fix dependency mismatch in Playground Java runner (#24059) add 1929968891d [Spark Dataset runner] Enable projection pushdown for Spark dataset runner. add 858f5048f57 Merge pull request #24057: [Spark Dataset runner] Enable projection pushdown for Spark dataset runner No new revisions were added by this update. Summary of changes: .../spark/structuredstreaming/SparkStructuredStreamingRunner.java| 5 + 1 file changed, 5 insertions(+)
[beam] branch master updated (23676a9ec17 -> c4218e572aa)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 23676a9ec17 [Tour of Beam] Learning content for "Introduction" module (#23085) add a5ee669a439 [Website] change case-study-card width on mobile add c4218e572aa Merge pull request #23934: [Website] change case studies page mobile layout No new revisions were added by this update. Summary of changes: website/www/site/assets/scss/_case_study.scss | 3 +++ 1 file changed, 3 insertions(+)
