This is an automated email from the ASF dual-hosted git repository.
chamikara pushed a change to branch java_expansion_service_container_snapshots
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 05c7091625a Publish Java Expansion Service SNAPSHOT containers
add 3cc00799c76 Fix non-utf8 encoding problem in ReadFromCsv and
WritetoCsv.
add f81c475f02a Apply yapf
add 8088ecfffb3 Set default file mode to read for _TruncatingFileHandle.
add ec37750639b Flush the underlying file-like object.
add 4b8cc5fe082 Merge branch 'master' into readfromcsv-encoding
add fc4db69dea6 Merge pull request #32463 from
shunping/readfromcsv-encoding
add 8690ddd6bfa Update python SDK container (#32487)
add 70dd6473abd Bump github.com/docker/docker in /sdks (#32495)
add 019efe131f9 [Dataflow Streaming] Add support to read user worker
settings from backend (#32408)
add 3105260ce58 Warn when using Beam with Python 3.8 (#32484)
add ef532179d32 Fix JmsIO NPE (#32489)
add 7daeadff032 Restore release-essential references to 3.8 (#32499)
add 2cd6fb8d758 Better error message for incorrect pipeline options flags.
add e3f8c20eeea More conservative args parsing.
add e5f454b04b3 Fix mypy strictness.
add 475c98c8779 Merge pull request Better error message for incorrect
pipeline options flags.
add eb8639b013e Add throttling counter in gcsio and refactor retrying
(#32428)
add 06e2170c1ab Update python SDK container (#32501)
add aeead3f7cd3 Disable a permared suite. (#32493)
add 83411302794 add test to verify kafka sdf commit expansion (#32438)
add bb96ac0456c Publish Java Expansion Service SNAPSHOT containers (#32491)
add ec307a52496 Emit warning when Mqtt waiting for connection for extended
period of time (#32500)
add 75a463790d7 Support Managed Iceberg streaming writes (#32451)
add 7581eb1667b Add runners/prism/java to java PreCommit (#32507)
add e1e6bdb2947 Change PeriodicSequence to report backlog accurately
(#32505)
add f2d0558ad58 [Python] Fix bug where WorkerHandlerManager.close_all
re-initialized state with incorrect dict type (#32517)
add bbf72c2cf92 Include jamm as runtime dependency in expansion-service
(#32521)
add 678104c55ec Remove debug leftover (#32524)
add 7474e6a7427 Rollback Bigtable throttling counter (#32442)
add 6a095456aa3 [Python] Add a couple quality-of-life improvemenets to
`testing.util.assert_that` (#30771)
add 818966c6163 [#32498][prism] Add split / progress back off + catch-up.
(#32526)
add 986ee964858 [Prism] Support BundleFinalization DoFn parameter (#32425)
add 6adb1453890 Add schrodinger logo (#32525)
add 855bd54bba1 Move assert into pipeline
add 50dd0487724 Update beam_PostCommit_Python.json
add 149f355cc84 Fix elements
add 7c382df7d18 Merge pull request #32530 from apache/users/damccorm/assert
add 42efefd9892 [GCSIO] Fix internal unit test failure (#32518)
add 878ccceb038 ReadFromKafkaDoFn: handle failed seek (#32456)
add 65b29361eb1 Set np printoptions (#32539)
add c3be9f096f0 Add RowFilter utility (#32366)
add 97cb452e4b9 Vllm model handler (#32410)
add cfcccbb4718 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#32542)
add d312e959fa0 Bump com.pswidersk.terraform-plugin from 1.0.0 to 1.1.1
(#32541)
add 2d5e72c615d Bump go.mongodb.org/mongo-driver from 1.16.1 to 1.17.0 in
/sdks (#32497)
add 5d8b0418cef Update build.gradle to use 3.12 for metrics report (#32547)
add c503ec4fda5 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#32543)
add a79e7103450 Set a UUID when building a Schema object. (#32399)
add 72d46ce0caf Revert "Bump com.pswidersk.terraform-plugin from 1.0.0 to
1.1.1 (#32541)" (#32551)
add d93f93abf0e Fix backlog reported by periodicsequence in python and go
sdks to not include future outputs (#32513)
add faf884a7a1b Update Python Inference Benchmark Test to Python 3.10
(#32550)
add e3c6f4747be Add RowStringInterpolator utility (#32367)
add 01c7caf92af SolaceIO: deduplicate records in the integration test
(#32557)
add 380ed7b7f74 fix: add topic existing validation (#32465)
add 80960d19b61 Remove unused code (#32414)
add aabae2729c0 fix NPE and other exceptions thrown when work fails to
process before work execution (#32566)
add c7fb9a0d5da [Dataflow Streaming] Use separate heartbeat streams based
on job settings (#32511)
add 8e8c9143ca5 [Dataflow Streaming] Use isolated windmill streams based
on job settings (#32503)
add 9053dbf72e9 [website][Go SDK] Add Go testing advice to test your
pipeline. (#32549)
add 11318ae4211 Bump cloud.google.com/go/bigquery from 1.62.0 to 1.63.0 in
/sdks (#32555)
add 1eddbdca4c5 Tour of Beam markdown touchups (#32536)
add 91c9be4e9ce Log warning if process function return None (#32545)
add 7c565c38c3c Add info log between executing work failed and cleanup
(#32570)
add 160dffd88e5 Revert "fix: add topic existing validation (#32465)"
(#32572)
add c2c640f8c33 Update CHANGES for recent fixes, improvements. (#32568)
add 7f849eb49a9 Add test case
add 4661b14017f reverse ordering of test
add 52b86c52a29 Update is_consistent_with
add ad55edc3249 Update failing unit test to use str instead of int
add 22ce9feddb9 update a couple more tests
add 4309675e266 Merge pull request #32516 from hjtran/permissive_num_types
add 271ea431156 BigQuery trim schema with selected fields (#32514)
add ee604a5eb8a Improve BQ <-> Avro conversions (#32482)
add 9b28982fe4b Update python KafkaIO docstring to add use_deprecated_read
option
add e705e28ecfd Merge pull request #32589 from jaehyeon-kim/master
add ac01dccfd28 represenation->representation (#32588)
add 48f836aae06 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#32595)
add e640b2578fb Force BQIO to output elements in the correct row (#32584)
add 9487927adce [#32562] Incorporate Prism into the Beam Website. (#32563)
add 7da6c89592b [Go] Update Go version used by Beam repo to go1.23.1
(#32575)
add 543d2ce97de Keep string FQN as a way to add lineage information
(#32585)
add 9b85f718cb8 [yaml] Preserve windowing for windowed input when using
FileIO Java providers (#32586)
add 40a5524d65d Update environments.py
add 111f4c34ab2 Merge pull request #32602 from apache/tvalentyn-patch-2
add deb4b4ada3a Managed Iceberg dynamic destinations (#32565)
add aadb695348d Fix a bug in _get_function_body_without_inners for module
sdks.python.transforms.core (#32591)
add 301286fd6a8 default null array to empty array (#32604)
add c8c674e1c04 Add support for dynamic write in `MqttIO` (#32470)
add dcc0bd2fb22 Build release candidate with Java 11 (#32573)
add 4d254a0863b Fixes a transform upgrade compatibility issue related to
BigqueryIO
add 5aed5850619 Merge pull request #32567: Fixes a transform upgrade
compatibility issue related to BigqueryIO
add 3dee026a2ab Update staticcheck version to fix breakage. (#32614)
add 6a7ffa587df Update Release guide with new github release guidance.
(#32576)
add 2438f43d604 [YAML] - Remove warning message (#32607)
add 987c4c9a903 Bump dataflow java container version to
beam-master-20240930 (#32615)
add eaf53e531b9 Deepcopy combine_fn in PrecombineFn and PostCombineFn.
(#32598)
add 00445adda91 Support string FQN as a way to add lineage information
(#32613)
add a0c9245e752 Fix writing raw messages to pubsub
add 3d90cffe6b0 Add case for exisiting bytes object
add 68263fcf507 Require bytes or strings.
add 76a600d25b8 Fix formatting typo.
add 79b534c8941 fix typo
add 0ca3f19555e Merge pull request #32342 Fix writing raw messages to
pubsub.
add 1bb357159d6 [prism][Java] Register option types (#32616)
add 415fdd32e7d Report Lineage metrics for SpannerIO (#32561)
add 36a8d39a3de [prism][java] Update Prism locator to match Python SDK
semantics. (#32619)
add a92eb38db67 Bump github.com/docker/docker in /sdks (#32554)
add 6a09ca9f091 Bump cloud.google.com/go/bigtable from 1.31.0 to 1.33.0 in
/sdks (#32556)
add d9ffc23ae13 Call out breaking assert_that change more explicitly
(#32624)
add 322667368c4 BigQuey fix invalid null checks in io translation (#32515)
add 721cbba4707 add virtualenv to bypass system packages requirement with
pip. Bump pyYAML (#32625)
add 8bdea3df66e Bump google.golang.org/api from 0.197.0 to 0.199.0 in
/sdks (#32605)
add e91882f010c Fix pull_licenses_java script retry broken for tenacity
8.5 (#32626)
add 2829c140c63 Update building steps to use go 1.23.2 (#32629)
add 6314775b2a3 Move remaining reference of python3.8 docker image to
python 3.9 (#32630)
add 3f79e30996e [yaml] package kafka_clients 3.1.2 in Kafka Provider jar
(#32623)
add fbeba899071 Revert "Deepcopy combine_fn in PrecombineFn and
PostCombineFn." (#32634)
add d5b910c750d Remove unused testinfra pipelines module (#32560)
add bb09de7c183 Add Java documentation to IcebergIO (#32621)
add 9095547bcd3 Update direct_runner.py (#32325)
add 3f76d11350e Optimized SparkRunner ParDo Operation (#32546)
add 2fb9efc6e52 Moving to 2.61.0-SNAPSHOT on master branch.
add 1d7b0bcde9e Rag opensearch usecase with Beam's MLTransform (#32018)
add eb84ff6b962 Bump go.mongodb.org/mongo-driver from 1.17.0 to 1.17.1 in
/sdks (#32641)
add 6561abf7beb Update groupbykey.py (#32359)
add 001ac59aa58 [#32601][prism] Initial Deep Dive Documentation (#32143)
add 0a71499f3f5 Fix counter metrics for
ParDo#with_exception_handling(timeout). (#32571)
add d84cfffd017 [Managed Iceberg] add GiB autosharding (#32612)
add a2710ed16e6 Spark Runner: Change to use partitioner in
GroupNonMergingWindowsFunctions#groupByKeyInGlobalWindow (#32610)
add 2a917c71e20 Update CHANGES.md add 2.61.0 section and 2.59.0 known
issue (#32664)
add decc997ea57 Improve wheels job name (#32644)
add d4dd58b2c4c sdks/python/apache_beam/io: add note on `output_fn` ctor
arg to indicate that It is unused and only retained for backward compatibility
(#32635)
new ac1f0239644 Tags Java SNAPSHOT containers with the '.dev' version
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (05c7091625a)
\
N -- N -- N
refs/heads/java_expansion_service_container_snapshots (ac1f0239644)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 1 +
.../test-properties.json | 2 +-
.../actions/setup-environment-action/action.yml | 2 +-
.github/autolabeler.yml | 7 +-
.github/build.gradle | 4 +-
.../arc/images/Dockerfile | 2 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 3 +-
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
...tCommit_Python_ValidatesContainer_Dataflow.json | 3 +-
.github/workflows/README.md | 4 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 12 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 11 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 10 +-
.github/workflows/build_release_candidate.yml | 24 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/go_tests.yml | 4 +-
...hon_Combine_Flink_Batch_2GB_10_byte_records.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_4.txt | 2 +-
.../python_Combine_Flink_Batch_2GB_Fanout_8.txt | 2 +-
.../python_GBK_Flink_Batch_2GB_of_10B_records.txt | 2 +-
.github/workflows/update_python_dependencies.yml | 1 +
.test-infra/jenkins/build.gradle | 3 -
.test-infra/pipelines/README.md | 34 -
.test-infra/pipelines/build.gradle | 85 -
.test-infra/pipelines/go.mod | 21 -
.test-infra/pipelines/go.sum | 2 -
.../infrastructure/01.setup/.terraform.lock.hcl | 40 -
.../pipelines/infrastructure/01.setup/README.md | 83 -
.../01.setup/apache-beam-testing.tfbackend | 19 -
.../01.setup/apache-beam-testing.tfvars | 19 -
.../infrastructure/01.setup/artifactregistry.tf | 30 -
.../infrastructure/01.setup/common.tfvars | 21 -
.../pipelines/infrastructure/01.setup/iam.tf | 37 -
.../pipelines/infrastructure/01.setup/provider.tf | 22 -
.../pipelines/infrastructure/01.setup/services.tf | 35 -
.../pipelines/infrastructure/01.setup/state.tf | 26 -
.../pipelines/infrastructure/01.setup/storage.tf | 43 -
.../pipelines/infrastructure/01.setup/variables.tf | 37 -
.../infrastructure/02.network/.terraform.lock.hcl | 22 -
.../pipelines/infrastructure/02.network/README.md | 81 -
.../02.network/apache-beam-testing.tfbackend | 19 -
.../02.network/apache-beam-testing.tfvars | 19 -
.../infrastructure/02.network/common.tfvars | 22 -
.../pipelines/infrastructure/02.network/data.tf | 22 -
.../pipelines/infrastructure/02.network/network.tf | 46 -
.../pipelines/infrastructure/02.network/state.tf | 26 -
.../infrastructure/02.network/variables.tf | 42 -
.../pipelines/infrastructure/03.io/README.md | 24 -
.../03.io/api-overuse-study/02.redis/README.md | 46 -
.../03.io/api-overuse-study/02.redis/common.tfvars | 19 -
.../03.io/api-overuse-study/02.redis/data.tf | 24 -
.../03.io/api-overuse-study/02.redis/provider.tf | 27 -
.../03.io/api-overuse-study/02.redis/redis.tf | 35 -
.../03.io/api-overuse-study/02.redis/variables.tf | 22 -
.../03.io/api-overuse-study/README.md | 46 -
.../03.io/api-overuse-study/common.tfvars | 19 -
.../03.io/api-overuse-study/namespace.tf | 24 -
.../03.io/api-overuse-study/provider.tf | 21 -
.../03.io/api-overuse-study/variables.tf | 22 -
.../03.io/dataflow-to-bigquery/.terraform.lock.hcl | 21 -
.../03.io/dataflow-to-bigquery/README.md | 86 -
.../apache-beam-testing.tfbackend | 19 -
.../apache-beam-testing.tfvars | 19 -
.../03.io/dataflow-to-bigquery/bigquery.tf | 30 -
.../03.io/dataflow-to-bigquery/common.tfvars | 22 -
.../03.io/dataflow-to-bigquery/iam.tf | 41 -
.../03.io/dataflow-to-bigquery/provider.tf | 21 -
.../03.io/dataflow-to-bigquery/pubsub.tf | 39 -
.../03.io/dataflow-to-bigquery/state.tf | 26 -
.../03.io/dataflow-to-bigquery/variables.tf | 42 -
.../03.io/dataflow-to-bigquery/workflow.tf | 67 -
.../04.template/dataflow-to-bigquery/README.md | 66 -
.../apache-beam-testing.tfvars | 20 -
.../04.template/dataflow-to-bigquery/common.tfvars | 24 -
.../04.template/dataflow-to-bigquery/data.tf | 53 -
.../dataflow-to-bigquery/dataflow-template.json | 20 -
.../04.template/dataflow-to-bigquery/output.tf | 22 -
.../04.template/dataflow-to-bigquery/provider.tf | 21 -
.../04.template/dataflow-to-bigquery/template.tf | 58 -
.../04.template/dataflow-to-bigquery/variables.tf | 57 -
.test-infra/pipelines/infrastructure/README.md | 65 -
.../src/main/go/internal/environment/variable.go | 81 -
.../main/go/internal/environment/variable_test.go | 358 ----
.../pipelines/ReadDataflowApiWriteBigQuery.java | 362 -----
.../pipelines/bigquery/BigQueryWriteOptions.java | 35 -
.../pipelines/bigquery/BigQueryWrites.java | 166 --
.../bigquery/DatasetReferenceOptionValue.java | 58 -
.../testinfra/pipelines/bigquery/package-info.java | 20 -
.../pipelines/conversions/ConversionError.java | 73 -
.../pipelines/conversions/EventarcConversions.java | 116 --
.../testinfra/pipelines/conversions/JobsToRow.java | 89 -
.../pipelines/conversions/RowConversionResult.java | 81 -
.../conversions/WithAppendedDetailsToRow.java | 265 ---
.../pipelines/conversions/package-info.java | 20 -
.../pipelines/dataflow/DataflowClientFactory.java | 102 --
.../DataflowClientFactoryConfiguration.java | 54 -
.../dataflow/DataflowFilterEventarcJobs.java | 110 --
.../dataflow/DataflowGetJobExecutionDetails.java | 160 --
.../pipelines/dataflow/DataflowGetJobMetrics.java | 139 --
.../pipelines/dataflow/DataflowGetJobs.java | 121 --
.../dataflow/DataflowGetStageExecutionDetails.java | 166 --
.../pipelines/dataflow/DataflowReadResult.java | 90 -
.../pipelines/dataflow/DataflowRequestError.java | 95 --
.../pipelines/dataflow/DataflowRequests.java | 52 -
.../dataflow/JobMetricsWithAppendedDetails.java | 84 -
.../dataflow/StageSummaryWithAppendedDetails.java | 84 -
.../testinfra/pipelines/dataflow/Throttle.java | 74 -
.../dataflow/WorkerDetailsWithAppendedDetails.java | 84 -
.../testinfra/pipelines/dataflow/package-info.java | 20 -
.../beam/testinfra/pipelines/package-info.java | 19 -
.../pubsub/SubscriptionPathOptionValue.java | 45 -
.../testinfra/pipelines/pubsub/package-info.java | 20 -
.../schemas/DependencyDrivenDescriptorQueue.java | 110 --
.../schemas/DescriptorSchemaRegistry.java | 153 --
.../schemas/GeneratedMessageV3RowBuilder.java | 280 ----
.../testinfra/pipelines/schemas/package-info.java | 20 -
.../conversions/EventarcConversionsTest.java | 271 ---
.../JobMetricsWithAppendedDetailsTest.java | 143 --
.../StageSummaryWithAppendedDetailsTest.java | 146 --
.../conversions/WithAppendedDetailsToRowTest.java | 153 --
.../WorkerDetailsWithAppendedDetailsTest.java | 153 --
.../dataflow/DataflowFilterEventarcJobsTest.java | 145 --
.../AbstractGeneratedMessageV3RowBuilderTest.java | 409 -----
.../DependencyDrivenDescriptorQueueTest.java | 56 -
.../schemas/DescriptorSchemaRegistryTest.java | 79 -
.../schemas/EnvironmentRowBuilderTest.java | 140 --
.../schemas/ExecutionStageSummaryTest.java | 84 -
.../schemas/TransformSummaryRowBuilderTest.java | 83 -
.../has_extra_data_payload_foo_property.json | 1 -
.../job_state_canceled_streaming.json | 1 -
.../job_state_canceling_streaming.json | 1 -
.../eventarc_data/job_state_done_batch.json | 1 -
.../eventarc_data/job_state_pending_batch.json | 1 -
.../eventarc_data/job_state_pending_streaming.json | 1 -
.../eventarc_data/job_state_queued_streaming.json | 1 -
.../eventarc_data/job_state_running_batch.json | 1 -
.../eventarc_data/job_state_running_streaming.json | 1 -
CHANGES.md | 55 +-
README.md | 1 +
build.gradle.kts | 2 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 +-
contributor-docs/release-guide.md | 36 +-
dev-support/docker/Dockerfile | 2 +-
examples/multi-language/README.md | 6 +-
.../beam-ml/rag_usecase/opensearch_connector.py | 372 +++++
.../beam-ml/rag_usecase/opensearch_enrichment.py | 134 ++
.../rag_usecase/opensearch_rag_pipeline.ipynb | 1719 ++++++++++++++++++++
gradle.properties | 6 +-
.../overview-pipeline/description.md | 2 +-
.../runner-concepts/description.md | 6 +-
.../introduction/introduction-guide/description.md | 35 +-
local-env-setup.sh | 4 +-
.../cloudbuild/playground_cd_examples.sh | 3 +
playground/infrastructure/requirements.txt | 2 +-
release/src/main/Dockerfile | 7 +-
.../python-release/python_release_automation.sh | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../options/DataflowStreamingPipelineOptions.java | 10 +-
.../runners/dataflow/worker/OperationalLimits.java | 39 +-
.../dataflow/worker/StreamingDataflowWorker.java | 137 +-
.../worker/StreamingModeExecutionContext.java | 37 +-
.../worker/streaming/ComputationWorkExecutor.java | 5 +-
.../worker/streaming/config/ComputationConfig.java | 3 +-
.../streaming/config/FakeGlobalConfigHandle.java | 52 +
.../streaming/config/FixedGlobalConfigHandle.java | 37 +-
...StreamingApplianceComputationConfigFetcher.java | 10 +-
.../StreamingEngineComputationConfigFetcher.java | 69 +-
...elineConfig.java => StreamingGlobalConfig.java} | 41 +-
.../config/StreamingGlobalConfigHandle.java | 25 +-
.../config/StreamingGlobalConfigHandleImpl.java | 113 ++
.../harness/StreamingWorkerStatusPages.java | 22 +-
.../windmill/client/grpc/GrpcDispatcherClient.java | 74 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 16 +-
.../grpc/stubs/WindmillStubFactoryFactory.java | 12 +-
.../grpc/stubs/WindmillStubFactoryFactoryImpl.java | 54 +
.../processing/ComputationWorkExecutorFactory.java | 7 +-
.../work/processing/StreamingWorkScheduler.java | 26 +-
.../work/refresh/StreamPoolHeartbeatSender.java | 45 +-
.../worker/StreamingDataflowWorkerTest.java | 45 +-
.../worker/StreamingModeExecutionContextTest.java | 8 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 11 +-
.../streaming/ComputationWorkExecutorTest.java | 72 +
.../config/FixedGlobalConfigHandleTest.java | 84 +
...amingApplianceComputationConfigFetcherTest.java | 4 +-
...treamingEngineComputationConfigFetcherTest.java | 102 +-
.../StreamingGlobalConfigHandleImplTest.java | 293 ++++
.../FanOutStreamingEngineWorkerHarnessTest.java | 9 +-
.../client/grpc/GrpcDispatcherClientTest.java | 154 ++
.../client/grpc/GrpcWindmillServerTest.java | 11 +-
.../testing/FakeWindmillStubFactoryFactory.java} | 24 +-
.../refresh/StreamPoolHeartbeatSenderTest.java | 132 ++
.../worker/windmill/src/main/proto/windmill.proto | 9 +
runners/prism/java/build.gradle | 5 -
.../beam/runners/prism/PrismArtifactResolver.java | 110 --
.../beam/runners/prism/PrismArtifactStager.java | 173 --
.../apache/beam/runners/prism/PrismJobManager.java | 160 --
.../apache/beam/runners/prism/PrismLocator.java | 69 +-
.../beam/runners/prism/PrismPipelineOptions.java | 7 +-
...ismRunnerRegistrar.java => PrismRegistrar.java} | 30 +-
.../apache/beam/runners/prism/StateListener.java | 27 -
.../apache/beam/runners/prism/StateWatcher.java | 146 --
.../apache/beam/runners/prism/WorkerService.java | 116 --
.../runners/prism/PrismArtifactResolverTest.java | 45 -
.../runners/prism/PrismArtifactStagerTest.java | 143 --
.../beam/runners/prism/PrismJobManagerTest.java | 211 ---
.../beam/runners/prism/PrismLocatorTest.java | 87 +-
.../beam/runners/prism/PrismRegistrarTest.java | 72 +
.../beam/runners/prism/StateWatcherTest.java | 136 --
.../beam/runners/prism/WorkerServiceTest.java | 85 -
.../runners/spark/SparkTransformOverrides.java | 2 +-
.../GroupNonMergingWindowsFunctions.java | 25 +-
.../spark/translation/TransformTranslator.java | 28 +-
.../GroupNonMergingWindowsFunctionsTest.java | 57 +
.../runners/spark/translation/PassThrough.java | 91 ++
.../beam/runners/spark/translation/RDDNode.java | 65 +
.../runners/spark/translation/RDDTreeParser.java | 55 +
.../spark/translation/TransformTranslatorTest.java | 96 +-
sdks/go.mod | 66 +-
sdks/go.sum | 132 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 1 +
sdks/go/pkg/beam/core/runtime/harness/harness.go | 2 +-
sdks/go/pkg/beam/core/typex/special.go | 12 +-
sdks/go/pkg/beam/forward.go | 1 +
sdks/go/pkg/beam/runners/prism/internal/README.md | 492 +++++-
.../pkg/beam/runners/prism/internal/handlepardo.go | 6 +-
.../beam/runners/prism/internal/jobservices/job.go | 1 +
.../pkg/beam/runners/prism/internal/preprocess.go | 1 +
.../beam/runners/prism/internal/preprocess_test.go | 6 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 58 +-
.../runners/prism/internal/unimplemented_test.go | 1 +
.../beam/runners/prism/internal/worker/bundle.go | 11 +
sdks/go/pkg/beam/transforms/periodic/periodic.go | 13 +-
.../pkg/beam/transforms/periodic/periodic_test.go | 36 +
sdks/go/run_with_go_version.sh | 2 +-
sdks/go/test/integration/integration.go | 15 +
sdks/go/test/integration/primitives/pardo.go | 80 +
sdks/go/test/integration/primitives/pardo_test.go | 43 +
.../license_scripts/pull_licenses_java.py | 12 +-
.../java/org/apache/beam/sdk/metrics/Lineage.java | 15 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 75 +-
.../apache/beam/sdk/schemas/SchemaTranslation.java | 67 +-
.../beam/sdk/schemas/logicaltypes/OneOfType.java | 3 +-
.../beam/sdk/transforms/PeriodicSequence.java | 34 +-
.../java/org/apache/beam/sdk/util/RowFilter.java | 421 +++++
.../beam/sdk/util/RowStringInterpolator.java | 167 ++
.../beam/sdk/schemas/SchemaTranslationTest.java | 46 +-
.../sdk/schemas/logicaltypes/LogicalTypesTest.java | 10 +
.../beam/sdk/transforms/PeriodicSequenceTest.java | 22 +
.../org/apache/beam/sdk/util/RowFilterTest.java | 353 ++++
.../beam/sdk/util/RowStringInterpolatorTest.java | 205 +++
sdks/java/expansion-service/build.gradle | 1 +
.../extensions/python/PythonExternalTransform.java | 3 -
.../csv/providers/CsvWriteTransformProvider.java | 17 +-
sdks/java/io/expansion-service/build.gradle | 6 +-
.../io/gcp/bigquery/BeamRowToStorageApiProto.java | 3 +
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 464 +++---
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 33 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 38 +-
.../io/gcp/bigquery/BigQueryQuerySourceDef.java | 6 +-
.../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 3 +-
.../sdk/io/gcp/bigquery/BigQuerySourceDef.java | 7 +-
.../io/gcp/bigquery/BigQueryStorageSourceBase.java | 46 +-
.../io/gcp/bigquery/BigQueryTableSourceDef.java | 6 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 103 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 107 +-
.../sdk/io/gcp/bigtable/BigtableWriteOptions.java | 5 -
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 1 -
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 24 +
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 26 +-
.../beam/sdk/io/gcp/spanner/ReadOperation.java | 27 +
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 26 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 46 +-
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 23 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 213 ++-
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 12 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 15 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 25 +
...QueryDirectReadSchemaTransformProviderTest.java | 3 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 15 -
.../sdk/io/gcp/spanner/FakeServiceFactory.java | 24 +
.../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 195 ++-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 97 +-
sdks/java/io/iceberg/hive/exec/build.gradle | 15 +-
.../beam/sdk/io/iceberg/AppendFilesToTables.java | 19 +-
.../beam/sdk/io/iceberg/AssignDestinations.java | 45 +-
.../beam/sdk/io/iceberg/DynamicDestinations.java | 13 +-
.../org/apache/beam/sdk/io/iceberg/IcebergIO.java | 267 ++-
.../IcebergWriteSchemaTransformProvider.java | 116 +-
.../io/iceberg/OneTableDynamicDestinations.java | 31 +-
.../io/iceberg/PortableIcebergDestinations.java | 81 +
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 22 +-
.../beam/sdk/io/iceberg/RecordWriterManager.java | 29 +-
.../apache/beam/sdk/io/iceberg/SnapshotInfo.java | 5 +-
.../sdk/io/iceberg/WriteGroupedRowsToFiles.java | 16 +-
.../beam/sdk/io/iceberg/WriteToDestinations.java | 140 +-
.../sdk/io/iceberg/WriteUngroupedRowsToFiles.java | 62 +-
.../apache/beam/sdk/io/iceberg/IcebergIOIT.java | 240 ++-
.../beam/sdk/io/iceberg/IcebergIOWriteTest.java | 105 +-
.../IcebergWriteSchemaTransformProviderTest.java | 219 ++-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 35 +-
.../json/providers/JsonWriteTransformProvider.java | 13 +-
sdks/java/io/kafka/build.gradle | 1 +
.../java/io/kafka/kafka-312/build.gradle | 14 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 33 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 6 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 96 ++
.../java/org/apache/beam/sdk/io/mqtt/MqttIO.java | 161 +-
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 227 ++-
.../apache/beam/sdk/io/solace/it/SolaceIOIT.java | 1 +
.../ManagedSchemaTransformTranslationTest.java | 49 +-
sdks/python/apache_beam/__init__.py | 6 +
sdks/python/apache_beam/dataframe/doctests.py | 5 +-
sdks/python/apache_beam/dataframe/io.py | 16 +-
.../apache_beam/examples/inference/README.md | 80 +
.../examples/inference/vllm_text_completion.py | 162 ++
.../transforms/aggregation/groupby_expr.py | 5 +-
.../aggregation/groupby_global_aggregate.py | 5 +-
.../aggregation/groupby_simple_aggregate.py | 5 +-
.../transforms/aggregation/groupby_test.py | 125 +-
.../snippets/transforms/aggregation/groupbykey.py | 8 +-
sdks/python/apache_beam/io/fileio.py | 2 +
sdks/python/apache_beam/io/gcp/bigquery.py | 59 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 1 +
sdks/python/apache_beam/io/gcp/gcsio.py | 73 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 39 +-
sdks/python/apache_beam/io/gcp/gcsio_retry.py | 71 +
sdks/python/apache_beam/io/gcp/gcsio_retry_test.py | 84 +
sdks/python/apache_beam/io/gcp/gcsio_test.py | 16 +-
sdks/python/apache_beam/io/kafka.py | 12 +
sdks/python/apache_beam/io/textio_test.py | 42 +
sdks/python/apache_beam/metrics/metric.py | 26 +-
sdks/python/apache_beam/metrics/metric_test.py | 11 +
sdks/python/apache_beam/ml/gcp/visionml_test_it.py | 23 +-
.../ml/inference/test_resources/vllm.dockerfile | 47 +
.../apache_beam/ml/inference/vllm_inference.py | 312 ++++
.../python/apache_beam/options/pipeline_options.py | 31 +-
.../apache_beam/options/pipeline_options_test.py | 47 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/runners/direct/direct_runner.py | 4 +-
.../portability/fn_api_runner/worker_handlers.py | 2 +-
.../fn_api_runner/worker_handlers_test.py | 52 +
sdks/python/apache_beam/testing/util.py | 22 +
sdks/python/apache_beam/testing/util_test.py | 13 +
sdks/python/apache_beam/transforms/core.py | 49 +-
sdks/python/apache_beam/transforms/core_test.py | 38 +
sdks/python/apache_beam/transforms/environments.py | 6 +-
.../apache_beam/transforms/periodicsequence.py | 21 +-
.../transforms/periodicsequence_test.py | 20 +
.../apache_beam/transforms/ptransform_test.py | 32 +-
sdks/python/apache_beam/transforms/trigger_test.py | 11 +-
sdks/python/apache_beam/transforms/util_test.py | 8 +-
.../apache_beam/typehints/typed_pipeline_test.py | 10 +-
sdks/python/apache_beam/typehints/typehints.py | 6 +
.../python/apache_beam/typehints/typehints_test.py | 10 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/yaml_io.py | 17 +-
sdks/python/setup.py | 6 +
.../tensorrt_runinference/tensor_rt.dockerfile | 6 +-
sdks/python/test-suites/dataflow/common.gradle | 39 +
sdks/typescript/package.json | 2 +-
settings.gradle.kts | 2 -
website/www/site/content/en/blog/beam-2.59.0.md | 4 +-
.../site/content/en/blog/validate-beam-release.md | 31 +
.../site/content/en/case-studies/schrodinger.md | 19 +
.../documentation/pipelines/test-your-pipeline.md | 119 +-
.../content/en/documentation/runners/direct.md | 1 +
.../site/content/en/documentation/runners/prism.md | 145 ++
website/www/site/content/en/roadmap/_index.md | 4 +-
.../www/site/content/en/roadmap/prism-runner.md | 35 +
.../www/site/data/en/documentation_runners.yaml | 3 +
.../layouts/partials/section-menu/en/roadmap.html | 1 +
.../layouts/partials/section-menu/en/runners.html | 1 +
.../static/images/logos/powered-by/schrodinger.png | Bin 0 -> 145837 bytes
377 files changed, 11309 insertions(+), 10890 deletions(-)
delete mode 100644 .test-infra/pipelines/README.md
delete mode 100644 .test-infra/pipelines/build.gradle
delete mode 100644 .test-infra/pipelines/go.mod
delete mode 100644 .test-infra/pipelines/go.sum
delete mode 100644
.test-infra/pipelines/infrastructure/01.setup/.terraform.lock.hcl
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/01.setup/apache-beam-testing.tfbackend
delete mode 100644
.test-infra/pipelines/infrastructure/01.setup/apache-beam-testing.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/01.setup/artifactregistry.tf
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/common.tfvars
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/iam.tf
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/provider.tf
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/services.tf
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/state.tf
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/storage.tf
delete mode 100644 .test-infra/pipelines/infrastructure/01.setup/variables.tf
delete mode 100644
.test-infra/pipelines/infrastructure/02.network/.terraform.lock.hcl
delete mode 100644 .test-infra/pipelines/infrastructure/02.network/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/02.network/apache-beam-testing.tfbackend
delete mode 100644
.test-infra/pipelines/infrastructure/02.network/apache-beam-testing.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/02.network/common.tfvars
delete mode 100644 .test-infra/pipelines/infrastructure/02.network/data.tf
delete mode 100644 .test-infra/pipelines/infrastructure/02.network/network.tf
delete mode 100644 .test-infra/pipelines/infrastructure/02.network/state.tf
delete mode 100644 .test-infra/pipelines/infrastructure/02.network/variables.tf
delete mode 100644 .test-infra/pipelines/infrastructure/03.io/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/common.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/data.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/provider.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/redis.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/02.redis/variables.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/common.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/namespace.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/provider.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/api-overuse-study/variables.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/.terraform.lock.hcl
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/apache-beam-testing.tfbackend
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/apache-beam-testing.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/bigquery.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/common.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/iam.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/provider.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/pubsub.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/state.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/variables.tf
delete mode 100644
.test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/workflow.tf
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/README.md
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/apache-beam-testing.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/common.tfvars
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/data.tf
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/dataflow-template.json
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/output.tf
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/provider.tf
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/template.tf
delete mode 100644
.test-infra/pipelines/infrastructure/04.template/dataflow-to-bigquery/variables.tf
delete mode 100644 .test-infra/pipelines/infrastructure/README.md
delete mode 100644
.test-infra/pipelines/src/main/go/internal/environment/variable.go
delete mode 100644
.test-infra/pipelines/src/main/go/internal/environment/variable_test.go
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/ReadDataflowApiWriteBigQuery.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/BigQueryWriteOptions.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/BigQueryWrites.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/DatasetReferenceOptionValue.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/bigquery/package-info.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/ConversionError.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/EventarcConversions.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/JobsToRow.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/RowConversionResult.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/WithAppendedDetailsToRow.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/conversions/package-info.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowClientFactory.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowClientFactoryConfiguration.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowFilterEventarcJobs.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetJobExecutionDetails.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetJobMetrics.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetJobs.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowGetStageExecutionDetails.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowReadResult.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowRequestError.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowRequests.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/JobMetricsWithAppendedDetails.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/StageSummaryWithAppendedDetails.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/Throttle.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/WorkerDetailsWithAppendedDetails.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/package-info.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/package-info.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/pubsub/SubscriptionPathOptionValue.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/pubsub/package-info.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/DependencyDrivenDescriptorQueue.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/DescriptorSchemaRegistry.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/GeneratedMessageV3RowBuilder.java
delete mode 100644
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/schemas/package-info.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/EventarcConversionsTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/JobMetricsWithAppendedDetailsTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/StageSummaryWithAppendedDetailsTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/WithAppendedDetailsToRowTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/conversions/WorkerDetailsWithAppendedDetailsTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowFilterEventarcJobsTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/AbstractGeneratedMessageV3RowBuilderTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/DependencyDrivenDescriptorQueueTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/DescriptorSchemaRegistryTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/EnvironmentRowBuilderTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/ExecutionStageSummaryTest.java
delete mode 100644
.test-infra/pipelines/src/test/java/org/apache/beam/testinfra/pipelines/schemas/TransformSummaryRowBuilderTest.java
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/has_extra_data_payload_foo_property.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_canceled_streaming.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_canceling_streaming.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_done_batch.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_pending_batch.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_pending_streaming.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_queued_streaming.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_running_batch.json
delete mode 100644
.test-infra/pipelines/src/test/resources/eventarc_data/job_state_running_streaming.json
create mode 100644
examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py
create mode 100644
examples/notebooks/beam-ml/rag_usecase/opensearch_enrichment.py
create mode 100644
examples/notebooks/beam-ml/rag_usecase/opensearch_rag_pipeline.ipynb
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/FakeGlobalConfigHandle.java
rename
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/pubsub/PubsubReadOptions.java
=>
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/FixedGlobalConfigHandle.java
(53%)
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/{StreamingEnginePipelineConfig.java
=> StreamingGlobalConfig.java} (56%)
rename
.test-infra/pipelines/src/main/java/org/apache/beam/testinfra/pipelines/dataflow/DataflowJobsOptions.java
=>
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandle.java
(58%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandleImpl.java
rename .test-infra/pipelines/infrastructure/03.io/dataflow-to-bigquery/data.tf
=>
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactory.java
(78%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationWorkExecutorTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/FixedGlobalConfigHandleTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingGlobalConfigHandleImplTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClientTest.java
copy
runners/{prism/java/src/main/java/org/apache/beam/runners/prism/PrismRunnerRegistrar.java
=>
google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/testing/FakeWindmillStubFactoryFactory.java}
(55%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/work/refresh/StreamPoolHeartbeatSenderTest.java
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismArtifactResolver.java
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismArtifactStager.java
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismJobManager.java
rename
runners/prism/java/src/main/java/org/apache/beam/runners/prism/{PrismRunnerRegistrar.java
=> PrismRegistrar.java} (50%)
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/StateListener.java
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/StateWatcher.java
delete mode 100644
runners/prism/java/src/main/java/org/apache/beam/runners/prism/WorkerService.java
delete mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismArtifactResolverTest.java
delete mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismArtifactStagerTest.java
delete mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismJobManagerTest.java
create mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismRegistrarTest.java
delete mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/StateWatcherTest.java
delete mode 100644
runners/prism/java/src/test/java/org/apache/beam/runners/prism/WorkerServiceTest.java
create mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/PassThrough.java
create mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/RDDNode.java
create mode 100644
runners/spark/src/test/java/org/apache/beam/runners/spark/translation/RDDTreeParser.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowFilter.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowStringInterpolator.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowFilterTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowStringInterpolatorTest.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/PortableIcebergDestinations.java
rename .test-infra/pipelines/infrastructure/02.network/provider.tf =>
sdks/java/io/kafka/kafka-312/build.gradle (72%)
create mode 100644
sdks/python/apache_beam/examples/inference/vllm_text_completion.py
create mode 100644 sdks/python/apache_beam/io/gcp/gcsio_retry.py
create mode 100644 sdks/python/apache_beam/io/gcp/gcsio_retry_test.py
create mode 100644
sdks/python/apache_beam/ml/inference/test_resources/vllm.dockerfile
create mode 100644 sdks/python/apache_beam/ml/inference/vllm_inference.py
create mode 100644
sdks/python/apache_beam/runners/portability/fn_api_runner/worker_handlers_test.py
create mode 100644 website/www/site/content/en/case-studies/schrodinger.md
create mode 100644 website/www/site/content/en/documentation/runners/prism.md
create mode 100644 website/www/site/content/en/roadmap/prism-runner.md
create mode 100644
website/www/site/static/images/logos/powered-by/schrodinger.png
