This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a change to branch users/damccorm/redisExtra
in repository https://gitbox.apache.org/repos/asf/beam.git
from f0319b4e7aa guard missing imports
add 5ffd998ed80 Bump golang.org/x/oauth2 from 0.32.0 to 0.33.0 in /sdks
(#36767)
add ad73e89987e Fix PostCommit XVR GoUsingJava Dataflow job (#36757)
add 8a69cddcbf1 update container java version
add c236996a455 Merge pull request #36762 from
derrickaw/updateJavaTagging_20251104
add 659201612f7 Move DataflowRunnerTest.java references to Python 3.10
containers (#36774)
add 11b0579d0c7 Dont use platform separator for docker build paths.
(#36777)
add 82fe92a2ba2 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#36783)
add 2cd7a75a86c Adds SchemaFieldNumber annotations to Iceberg classes that
use SchemaCoder
add 296421d0203 Merge pull request #36780: Adds SchemaFieldNumber
annotations to Iceberg classes that use the SchemaCoder
add 8e5b84527e1 Bump github.com/tetratelabs/wazero from 1.9.0 to 1.10.1 in
/sdks (#36797)
add 4eff1798314 fix CVE-2025-59250 (#36800)
add 4a59cb778e6 Fix Post Commit Transform Service (#36799)
add 83ebe731133 [1/3] sdks/python: refactor Milvus-related utilities as
preparation step for Milvus Sink I/O integration (#35708)
add addc06e8200 [Dataflow Streaming] Reuse ByteStringOutputStream buffers
in WindmillBag (#36742)
add 8c44a9ee80e Support custom id function in async_dofn (#36779)
add d134f242c78 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#36804)
add de918a653f2 fix(iceberg): Incorrect $partition Metadata in Trino for
Iceberg Tables Written via IcebergIO.writeRows with Timestamp Partitioning
(#36562)
add ce1b1dcbc59 Fix flaky GCS bucket deletion in
test_create_default_bucket (#36786)
add 1aca8519a47 Fix
PostCommit_Java_ValidatesRunner_SparkStructuredStreaming (#36810)
add 13949036151 Bring the Python Version update docs into the main repo
(#36811)
add 55c5854e6a3 Remove remaining ununsed Python 3.9 artifacts (#36812)
add 49de281005b Convert pipeline.py type comments to type hints (#36801)
add 61b8f416e61 Changes multimap entries Iterable to make a deep copy of
pending adds and deletes (#36759)
add db08b7c8c0b Type inference tests (#36776)
add e51998b00ae Fix test_create_default_bucket: check None instead of
NotFound (#36814)
add 6ba393a26e9 Added function for deterministic ID for class definitions
by hashing. (#36793)
add 63a8be9f023 Handle query returned with empty rows gracefully on
bigquery enrichmement (#36791)
add 92888144601 optimize grpc settings (#36528)
add ae2d3b80a4c Bump golang.org/x/net from 0.46.0 to 0.47.0 in /sdks
(#36817)
add c8d7ca02867 Fix DinD Action failure (#36813)
add 57320861aa2 Ensure that Operations are aborted when MapTaskExecutor is
closed. Add tests around setup/teardown of DoFns (#36631)
add 715a482bf5a Update REVIEWERS.yml (#36828)
add 3af78a7a19d Fix DinD startup failures (#36835)
add d4f5372a67c Improve error message for decoding input stream (#36823)
add ce58563ffd9 update container tag (#36826)
add 53a0d3c31b7 Update BOM to 26.71.0 (#36663)
add 3721c17265e Adds proto fields for overwriting fetch/write state
functionality (#36708)
add 914c16eb1d1 Provide more contextual error message for PyCoder (#36825)
add b843f4af3eb Increase the GZip DeflaterOutputStream buffer size to
increase performance (#36818)
add a799d2b76ea Bump google.golang.org/grpc from 1.76.0 to 1.77.0 in /sdks
(#36843)
add 050369f39d1 make use of generic type info in *Utils classes
add b369dfa3814 make use of generic type info in *Utils classes (#36581)
add 87d204f6038 Fix inconsistent data type in GenericRecord and AvroSchema
for AvroWriter (#36839)
add 6d9df8cbc81 Remove 3.9-specific branches in typehinting module (#36732)
add eadbc6e74fc Kerberos auth python (#36211)
add f41cbde78c1 Allow users to specify trusted Avro serializable classes
to Dataflow worker (#36809)
add 534e1cb0dbf Add artifactregistry.writer to service account (#36756)
add 0ab2d039f8d Add @Pure annotations to Preconditions
add e84b318ad3c Add @Pure annotations to DoFnSignature
add a33d59420c7 Add @Pure annotations to DoFnSchemaInformation
add 8f5bc796213 Add @Pure annotations to TimerSpec
add edad6545dad Add @Pure annotations to SideInputReader
add 6c99f7ceb71 Add @Pure annotations to LateDataUtils
add f07ca33e355 Fix nullness errors in SimpleDoFnRunner and DoFnInvoker
add 87530a369fb Trivial change to KafkaSourceConsumerFnTest to eliminate
warning
add be65cfe8d78 Fix nullness of currentRecordId and currentRecordOffset
add 2a06694cc5e Merge pull request #35029: Fix nullness errors in
SimpleDoFnRunner and DoFnInvoker
add f5b4b6d2429 Add back sys import. (#36862)
add 5806b6e4b93 Support Beam MicrosInstant conversion to Avro Timestamp
(#36605)
add 614e742a9d0 Bump com.gradle.common-custom-user-data-gradle-plugin
(#36856)
add 108225d3797 Add Iceberg CDC support to YAML (#36641)
add 8e8fd1aefc1 Bump golang.org/x/crypto from 0.35.0 to 0.45.0 in
/.test-infra/mock-apis (#36865)
add e336419205c Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#36867)
add 9250fb36b76 Merge branch 'master' of https://github.com/apache/beam
into users/damccorm/redisExtra
No new revisions were added by this update.
Summary of changes:
.github/REVIEWERS.yml | 7 +-
.github/actions/dind-up-action/action.yml | 24 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 +-
...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 2 -
.../beam_PostCommit_Python_Xlang_Gcp_Direct.json | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 4 -
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +
.test-infra/mock-apis/go.mod | 10 +-
.test-infra/mock-apis/go.sum | 20 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 16 +-
.../updating-supported-python-versions.md | 82 ++++
examples/java/build.gradle | 1 +
infra/iam/users.yml | 1 +
.../beam/model/fn_execution/v1/beam_fn_api.proto | 9 +
.../apache/beam/runners/core/LateDataUtils.java | 3 +
...TimeBoundedSplittableProcessElementInvoker.java | 4 +-
.../apache/beam/runners/core/SideInputReader.java | 13 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 169 ++++++--
runners/google-cloud-dataflow-java/build.gradle | 151 +++++---
.../beam/runners/dataflow/DataflowRunnerTest.java | 12 +-
.../worker/IntrinsicMapTaskExecutorFactory.java | 53 ++-
.../util/ThreadLocalByteStringOutputStream.java | 103 +++++
.../worker/util/common/worker/MapTaskExecutor.java | 71 +++-
.../worker/windmill/state/WindmillBag.java | 25 +-
.../windmill/state/WindmillStateTagUtil.java | 58 +--
.../work/processing/StreamingWorkScheduler.java | 13 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 137 ++++++-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 12 +-
.../worker/StreamingDataflowWorkerTest.java | 114 +++++-
.../worker/testing/TestCountingSource.java | 5 +
.../ThreadLocalByteStringOutputStreamTest.java | 68 ++++
.../util/common/worker/MapTaskExecutorTest.java | 39 ++
runners/spark/spark_runner.gradle | 2 +-
sdks/go.mod | 78 ++--
sdks/go.sum | 172 ++++-----
sdks/java/container/boot.go | 22 ++
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../java/org/apache/beam/sdk/io/Compression.java | 4 +-
.../apache/beam/sdk/options/SdkHarnessOptions.java | 12 +
.../beam/sdk/schemas/utils/AutoValueUtils.java | 16 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 59 +--
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 28 +-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 138 ++-----
.../java/org/apache/beam/sdk/state/TimerSpec.java | 2 +
.../java/org/apache/beam/sdk/transforms/DoFn.java | 4 +-
.../beam/sdk/transforms/DoFnSchemaInformation.java | 6 +
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 28 +-
.../beam/sdk/transforms/reflect/DoFnSignature.java | 108 ++++++
.../org/apache/beam/sdk/util/Preconditions.java | 31 ++
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 80 ----
.../extensions/avro/schemas/utils/AvroUtils.java | 25 ++
.../avro/schemas/utils/AvroUtilsTest.java | 34 ++
.../beam/fn/harness/state/MultimapUserState.java | 16 +-
.../io/debezium/KafkaSourceConsumerFnTest.java | 2 +-
sdks/java/io/expansion-service/build.gradle | 2 +
.../providers/PortableBigQueryDestinations.java | 2 +-
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 5 +-
...gQueryFileLoadsSchemaTransformProviderTest.java | 23 +-
sdks/java/io/iceberg/hive/build.gradle | 9 +-
.../org/apache/beam/sdk/io/iceberg/ReadTask.java | 2 +
.../beam/sdk/io/iceberg/ReadTaskDescriptor.java | 2 +
.../beam/sdk/io/iceberg/RecordWriterManager.java | 5 +-
.../apache/beam/sdk/io/iceberg/SnapshotInfo.java | 10 +
.../IcebergReadSchemaTransformProviderTest.java | 64 +++
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 40 ++
sdks/python/apache_beam/coders/coder_impl.py | 17 +-
.../transforms/elementwise/enrichment_test.py | 60 +--
.../internal/cloudpickle/cloudpickle.py | 28 +-
.../io/external/xlang_bigqueryio_it_test.py | 35 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 24 +-
sdks/python/apache_beam/io/kafka.py | 47 ++-
.../apache_beam/ml/rag/enrichment/milvus_search.py | 133 ++-----
.../ml/rag/enrichment/milvus_search_it_test.py | 429 ++-------------------
.../ml/rag/ingestion/postgres_common.py | 56 +--
sdks/python/apache_beam/ml/rag/test_utils.py | 413 ++++++++++++++++++++
sdks/python/apache_beam/ml/rag/utils.py | 224 +++++++++++
sdks/python/apache_beam/pipeline.py | 425 ++++++++++----------
.../apache_beam/runners/dataflow/internal/names.py | 3 +-
.../runners/portability/sdk_container_builder.py | 3 +-
.../portability/sdk_container_builder_test.py | 26 ++
.../apache_beam/runners/worker/bundle_processor.py | 6 +-
.../apache_beam/runners/worker/channel_factory.py | 10 +-
sdks/python/apache_beam/transforms/async_dofn.py | 42 +-
.../apache_beam/transforms/async_dofn_test.py | 34 ++
.../transforms/enrichment_handlers/bigquery.py | 38 +-
.../enrichment_handlers/bigquery_it_test.py | 141 +++++++
.../apache_beam/transforms/ptransform_test.py | 99 +++++
.../typehints/native_type_compatibility.py | 5 +-
.../apache_beam/typehints/trivial_inference.py | 8 +-
sdks/python/apache_beam/typehints/typehints.py | 6 +-
.../python/apache_beam/typehints/typehints_test.py | 12 +-
sdks/python/apache_beam/utils/subprocess_server.py | 16 +-
.../yaml/extended_tests/databases/iceberg.yaml | 54 ++-
sdks/python/apache_beam/yaml/standard_io.yaml | 28 ++
sdks/python/container/distroless/py39/build.gradle | 28 --
.../container/ml/py39/base_image_requirements.txt | 228 -----------
sdks/python/container/ml/py39/build.gradle | 28 --
.../container/py39/base_image_requirements.txt | 194 ----------
sdks/python/container/py39/build.gradle | 28 --
sdks/python/expansion-service-container/Dockerfile | 3 +-
.../expansion-service-container/build.gradle | 4 +-
sdks/python/test-suites/dataflow/py39/build.gradle | 24 --
sdks/python/test-suites/direct/py39/build.gradle | 24 --
sdks/python/test-suites/portable/py39/build.gradle | 26 --
sdks/python/test-suites/tox/py39/build.gradle | 236 ------------
settings.gradle.kts | 51 ++-
108 files changed, 3224 insertions(+), 2343 deletions(-)
create mode 100644 contributor-docs/updating-supported-python-versions.md
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/ThreadLocalByteStringOutputStream.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/ThreadLocalByteStringOutputStreamTest.java
create mode 100644 sdks/python/apache_beam/ml/rag/test_utils.py
create mode 100644 sdks/python/apache_beam/ml/rag/utils.py
delete mode 100644 sdks/python/container/distroless/py39/build.gradle
delete mode 100644 sdks/python/container/ml/py39/base_image_requirements.txt
delete mode 100644 sdks/python/container/ml/py39/build.gradle
delete mode 100644 sdks/python/container/py39/base_image_requirements.txt
delete mode 100644 sdks/python/container/py39/build.gradle
delete mode 100644 sdks/python/test-suites/dataflow/py39/build.gradle
delete mode 100644 sdks/python/test-suites/direct/py39/build.gradle
delete mode 100644 sdks/python/test-suites/portable/py39/build.gradle
delete mode 100644 sdks/python/test-suites/tox/py39/build.gradle
