This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a change to branch users/damccorm/adkHandler
in repository https://gitbox.apache.org/repos/asf/beam.git
from 5d0706cc5ad Apply suggestions from code review
add 181995db2c6 Upgrade Docker setup actions to version 4 (#37914)
add 5139ca67b64 Sync clickhouse resource manager client version with IO
(#37918)
add 0e7dd4f3cf7 Add version to beam-sdks-java-io-components
add 866d6f73ddb Merge pull request #37920 from apache/fix-maven-archetypes
add d9a6159d97f Pin docker actions to sha for now
add 0cd35ec0927 Merge pull request #37922 from apache/users/damccorm/shas
add 982c9bd8044 Pin version for setup-qemu-action (#37924)
add ca3421606ee Add Quick Start section to README (#37909)
add 65a896b9780 Fix #37894: Update yaml doc for composite transform
changes (#37901)
add 600b7d9050e Pin gcp actions (#37926)
add 45c2863c0a3 Pin to SHA for GCP GitHub Actions
add 90a7db1c094 Merge pull request #37927 from apache/gcp-actions
add 3316ef93182 [Website] Add missing I/O connectors to built-in
connectors table (#37903)
add 7c82e73bb1e [Python][Java] Add support for record headers in
WriteToKafka (Fixes #27033) (#37458)
add 79c52affb1d Updates YAML Database transforms to use managed I/O by
default (#37923)
add cf3d6ed3e52 [ErrorProne] Enable UnnecessaryParentheses validation and
fix all violations (#37913)
add 4b448b069f4 Bump Java dev containers (#37931)
add 7187ff67ee8 Handle Flink 2 job server tag in releases and SDK snapshot
(#37939)
add 3aad50b99e8 [Security] Bump ActiveMQ from 5.14.5 to 5.19.2 (#37944)
add 012f825245d [Java] Optimize beam pubsub message to serialized pubsub
protobuf message (#37896)
add 8e90c36ff1b [Java] Optimize StreamUtils to avoid copying bytes (#37912)
add b6d8f428e29 Iceberg Add Files (#37701)
add 683ecebe3af Fix typo in arc readme (#37948)
add 7431dea675b [#37930][Runners] Reap child processes in ProcessManager
to prevent zombie accumulation (#37932)
add 08685524465 Pin setuptools below 82 for TensorFlow Hub pkg_resources
compatibility (#37929)
add 40951024bfe Iceberg AddFiles -- handle FileNotFound exceptions (#37952)
add 97da2d69121 Update changes.md (#37936)
add dcb1b756e7d Add OrderedListState support in Go SDK (#37629)
add 2eb71e989b4 Bump picomatch from 2.2.2 to 2.3.2 in /website/www (#37955)
add 64d75fa8f57 Merge in master
No new revisions were added by this update.
Summary of changes:
.../gh-actions-self-hosted-runners/arc/README.md | 2 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
.github/workflows/beam_CleanUpGCPResources.yml | 2 +-
.../beam_Infrastructure_PolicyEnforcer.yml | 2 +-
.../beam_Infrastructure_SecurityLogging.yml | 2 +-
.../beam_Infrastructure_ServiceAccountKeys.yml | 2 +-
.../beam_Infrastructure_UsersPermissions.yml | 2 +-
.github/workflows/beam_Playground_Precommit.yml | 2 +-
.github/workflows/beam_PostCommit_Go.yml | 2 +-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 6 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 4 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 4 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 2 +-
.../beam_PostCommit_Yaml_Xlang_Direct.yml | 2 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 2 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 2 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 2 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 6 +-
.github/workflows/beam_Publish_Website.yml | 2 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 6 +-
.github/workflows/build_release_candidate.yml | 4 +-
.github/workflows/build_runner_image.yml | 2 +-
.github/workflows/build_wheels.yml | 2 +-
.github/workflows/finalize_release.yml | 8 +-
.github/workflows/refresh_looker_metrics.yml | 4 +-
.../republish_released_docker_containers.yml | 6 +-
.../run_rc_validation_python_mobile_gaming.yml | 2 +-
.../workflows/run_rc_validation_python_yaml.yml | 2 +-
CHANGES.md | 14 +-
README.md | 19 +-
.../org/apache/beam/gradle/BeamDockerPlugin.groovy | 7 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 3 +-
it/clickhouse/build.gradle | 7 +-
.../apache/beam/it/gcp/bigquery/BigQueryIOLT.java | 2 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 2 +-
.../flink_job_server_container.gradle | 3 +
.../unbounded/FlinkUnboundedSourceReader.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 2 +-
.../dataflow/worker/DataflowExecutionContext.java | 2 +-
.../worker/util/common/worker/ShuffleEntry.java | 8 +-
.../client/getdata/ApplianceGetDataClient.java | 4 +-
.../worker/windmill/state/WindmillMultimap.java | 4 +-
.../worker/CombineValuesFnFactoryTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 4 +-
.../dataflow/worker/TestShuffleReaderTest.java | 4 +-
.../fnexecution/environment/ProcessManager.java | 20 +-
.../ordered_list_state/ordered_list_state.go | 93 +++
sdks/go/examples/snippets/04transforms.go | 32 +
sdks/go/pkg/beam/core/graph/fn.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/data.go | 6 +
.../pkg/beam/core/runtime/exec/sideinput_test.go | 15 +
sdks/go/pkg/beam/core/runtime/exec/translate.go | 2 +
sdks/go/pkg/beam/core/runtime/exec/userstate.go | 150 +++++
.../pkg/beam/core/runtime/exec/userstate_test.go | 25 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 16 +-
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 82 +++
sdks/go/pkg/beam/core/state/state.go | 133 ++++
sdks/go/pkg/beam/core/state/state_test.go | 243 ++++++-
sdks/go/test/integration/integration.go | 6 +
sdks/go/test/integration/primitives/state.go | 38 ++
sdks/go/test/integration/primitives/state_test.go | 5 +
.../apache/beam/sdk/jmh/util/VarIntBenchmark.java | 2 +-
.../java/org/apache/beam/sdk/io/TextSource.java | 2 +-
.../org/apache/beam/sdk/io/range/ByteKeyRange.java | 4 +-
.../beam/sdk/transforms/reflect/DoFnSignature.java | 2 +-
.../transforms/resourcehints/ResourceHints.java | 2 +-
.../org/apache/beam/sdk/util/HistogramData.java | 2 +-
.../java/org/apache/beam/sdk/util/StreamUtils.java | 47 +-
.../apache/beam/sdk/util/UnownedInputStream.java | 4 +
.../main/java/org/apache/beam/sdk/util/VarInt.java | 2 +-
.../java/org/apache/beam/sdk/values/RowUtils.java | 2 +-
.../apache/beam/sdk/testing/TestPipelineTest.java | 4 +-
.../org/apache/beam/sdk/util/CoderUtilsTest.java | 10 +
.../org/apache/beam/sdk/util/StreamUtilsTest.java | 95 +++
.../ExpansionServiceSchemaTransformProvider.java | 2 +-
.../schemas/utils/AvroJavaTimeConversions.java | 4 +-
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 2 +-
.../sql/impl/rule/BeamSideInputLookupJoinRule.java | 7 +-
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../sdk/io/aws2/options/SerializationTestUtil.java | 2 +-
sdks/java/io/amqp/build.gradle | 4 +-
.../AvroGenericRecordToStorageApiProto.java | 4 +-
.../io/gcp/bigquery/BeamRowToStorageApiProto.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 2 +-
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 4 +-
.../io/gcp/bigtable/BigtableConfigTranslator.java | 2 +-
.../BigtableWriteSchemaTransformProvider.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessages.java | 52 +-
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 2 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 4 +-
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 2 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 2 +-
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 2 +-
.../ReadChangeStreamPartitionActionTest.java | 10 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessagesTest.java | 176 +++++
.../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 2 +-
.../apache/beam/sdk/io/gcp/storage/GcsMatchIT.java | 2 +-
.../java/org/apache/beam/sdk/io/hbase/HBaseIO.java | 4 +-
sdks/java/io/iceberg/build.gradle | 7 +-
.../org/apache/beam/sdk/io/iceberg/AddFiles.java | 706 +++++++++++++++++++++
.../iceberg/AddFilesSchemaTransformProvider.java | 190 ++++++
.../apache/beam/sdk/io/iceberg/PartitionUtils.java | 5 +-
.../org/apache/beam/sdk/io/iceberg/ReadUtils.java | 30 +
.../org/apache/beam/sdk/io/iceberg/AddFilesIT.java | 536 ++++++++++++++++
.../apache/beam/sdk/io/iceberg/AddFilesTest.java | 678 ++++++++++++++++++++
.../io/jdbc/JdbcReadSchemaTransformProvider.java | 4 +-
.../io/jdbc/JdbcWriteSchemaTransformProvider.java | 4 +-
sdks/java/io/jms/build.gradle | 2 +-
.../jms/MockNonSerializableConnectionFactory.java | 21 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 143 ++++-
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 2 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 86 +++
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 8 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 4 +-
.../apache/beam/sdk/io/pulsar/PulsarIOTest.java | 6 +-
.../src/main/resources/archetype-resources/pom.xml | 1 +
.../src/main/resources/archetype-resources/pom.xml | 1 +
sdks/python/apache_beam/io/kafka.py | 25 +-
.../ml/inference/tensorflow_tests_requirements.txt | 3 +
sdks/python/apache_beam/yaml/standard_io.yaml | 22 +-
.../site/content/en/documentation/io/connectors.md | 162 ++++-
.../content/en/documentation/programming-guide.md | 4 +
.../content/en/documentation/sdks/yaml-schema.md | 48 ++
website/www/yarn.lock | 13 +-
128 files changed, 4030 insertions(+), 213 deletions(-)
create mode 100644 sdks/go/examples/ordered_list_state/ordered_list_state.go
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagesTest.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/AddFilesIT.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/AddFilesTest.java
