This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a change to branch ml_transform_criteo
in repository https://gitbox.apache.org/repos/asf/beam.git
from ea3faa479c5 Refactor pipeline
add b2073c6c986 Fix typo in notebook (#29728)
add c8deda71743 Upgrade go version to resolve vulnerabilities (#29743)
add 5d42cb654c0 revert kafka host change (#29725)
add 86720287c21 Deep-copy the process bundle descriptor when creating a
new bundle processor. (#29742)
add 06c3340e338 Add logs to confirm JvmInitializer completed running and
Spanner successfully connected. (#29727)
add 8f90fdd62d3 Upgrade Debian Bullseye to Bookworm (#29721)
add 951b3b1a81b Per DoFn latency instrumentation (#29592)
add fa47640888c Remove comment body from action (#29750)
add ad2d8dc6e30 Update Bigtable python client version (#29753)
add e37fa77c3b6 Pass getPort method through JDBC Resource Managers (#29732)
add 8d41b62e828 Moving to 2.54.0-SNAPSHOT on master branch.
add c49e9ec7dc7 test revert _beam_schema_proto (#29754)
add db0bb446aa0 Add outputWindowedValue capability to Java SDK
add ef0ee769e8f Merge pull request #29616: Add outputWindowedValue
capability to Java SDK
add 19858e990ad [#29760] Only respond to sampling request while data
sampling is enabled (#29761)
add a04a4ad4cd2 [yaml] Normalize drop on MapToFields
add c124e473fa4 Merge pull request #29755 from Polber/jkinard/mapping-drop
add a9f5ab14d06 Add support to use side inputs with
Combine.PerKeyWithHotKeyFanout (#28867)
add def026bfe11 Bump google.golang.org/api from 0.153.0 to 0.154.0 in
/sdks (#29734)
add 921e40a12f4 Dynamic SchemaTransform wrapper provider (#29561)
add 4264c2c3e25 Fixes breakages of the upgrade feature (#29731)
add 75746c6fe4a Bump cloud.google.com/go/storage from 1.35.1 to 1.36.0 in
/sdks (#29767)
add 00462b13cc1 Bump google.golang.org/grpc from 1.59.0 to 1.60.0 in /sdks
(#29714)
add 1ce707c3865 Add user-agent string for Beam in the GCS request header.
(#29765)
add bc42a637d56 [#28543][prism] Implement State API (#29712)
add 9e8a310f0c0 Bump Precommit Java Examples Timeouts (#29771)
add c4c9bb82723 Add 2.54.0 release section to CHANGES.md (#29757)
add 543b013a548 Bump com.gradle.enterprise from 3.15.1 to 3.16 (#29740)
add a81c9e8e38b Bump actions/setup-python from 4 to 5 (#29654)
add 970a0bd9a92 Bump actions/stale from 8 to 9 (#29682)
add d297cd531ee Bump org.javacc.javacc from 3.0.0 to 3.0.2 (#29739)
add 1c7d178dd2b Bump github.com/google/uuid from 1.4.0 to 1.5.0 in /sdks
(#29733)
add 52f4fc0c378 [Go SDK] Decrease sampling frequency for streaming jobs to
avoid oversampling (#29774)
add 3bace64d16a Ignore
testOnNewWorkerMetadata_correctlyRemovesStaleWindmillServers (#29744)
add 5fb4db31994 Bump actions/upload-artifact from 3 to 4 (#29777)
add 860bc30a110 Add SparkStructuredStreamingRunner to
beam_PostCommit_Java_Tpcds_Spark workflow
add e1ce548cea7 Merge pull request #29781: Add
SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow
add 539f04233c6 add alts to grpc vendored lib (#29763)
add 1033b71aff6 pin dind image temoporarily due to issues
https://github.com/actions/actions-runner-controller/issues/3159 (#29796)
add fb8ae575eeb getProcessingTimesByStep returns a modifiable map (#29786)
add 7db8c274dd5 make DataSample external (#29788)
add f02bf88fd15 Upgrade download-artifact to v4 (#29801)
add c3d130c8ebf Update python base image dependencies (#29800)
add add34385719 Fix GenerateSequenceSchemaTransformProvider formatting
(#29798)
add 4cfe5498850 Fix nil pointer access in logRuntimeDependencies (#29804)
add 2e38af69cf2 Adds Kafka upgrade module to the I/O precommit (#29773)
add 87794aaef03 Update dataflow container tag (#29790)
add 2baea8b65eb Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in
/.test-infra/mock-apis (#29809)
add d6426c4fdfc Bump google.golang.org/grpc from 1.60.0 to 1.60.1 in /sdks
(#29814)
add 60f04081240 Update changes.md with known fixes. (#29817)
add 54d2c2df5cf Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sdks
(#29810)
add f841098c54d Support DatabaseID in Datastore beam connector (#29815)
add 1a7174c68bd Build and publish multi-arch wheels separately from main
wheels (#29821)
add b9a62f0f793 [YAML] Properly deligate methods for renaming provider.
add 56d177713ed Merge pull request #29827 [YAML] Properly deligate methods
for renaming provider.
add c5d53d61e78 Revert to non-schema-based Sql transform.
add 0bbf2c3351b Merge pull request #29826 Revert to non-schema-based Sql
transform.
add 1986517d38a Enable User State tests for Prism (#29822)
add 91aeba43273 Remove failing multimap key iteration tests.
add fa11c0f29d1 Merge pull request #29836 Remove failing multimap key
iteration tests.
add 47ccb8a5ee5 [YAML] Better IO documentation.
add 192f7655e2e Merge pull request #29687 [YAML] Better IO documentation.
add 8de029a412a Bump github.com/fsouza/fake-gcs-server from 1.47.6 to
1.47.7 in /sdks (#29833)
add 140dd1458eb Bump transformers (#29847)
add 79d679db218 Bump transformers from 4.30.0 to 4.36.0 in
/sdks/python/apache_beam/examples/inference/large_language_modeling (#29846)
add 273008534af Fix javadoc action in build-release-candidate (#29841)
add 05068ead3ea Bump transformers (#29844)
add 840178b91b5 Fix input issue and quoting (#29840)
add 6345c8bf681 Fix logging/quoting (#29854)
add a0081221f1d Fix tour of beam co-group-by-key python example (#29775)
add 6a12e8b88df fix contructor names (#29855)
add 61c53676a7b Dataflow Streaming: Add a pipeline option
`--desiredNumUnboundedSourceSplits` to overide the desired number of splits for
CustomSources (#29704)
add 7e4dc66ba20 Change pubsub message cap size from 10MiB to 10MB (#29791)
add 6499c654c4a Fix typos
add 78e231a9a53 Merge pull request #29785: Fix typos
add 44515bf2a80 Update google-api-services-storage to match libraries-bom
(#29864)
add 304f0cbe768 Bump objsize upper bound (#29562)
add 6a036699dc8 DuetAI knowledge lookup prompts (#29507)
add 2d1df39c01a Fix uploadArtifact@4 naming conflict (#29867)
add 32660e60a66 Add docs on updating the starter repos as a post release
task (#29870)
add 989a2198a31 Allow large timestamp skew for at-least-once streaming
(#29858)
add 43e79080b44 update container tag (#29872)
add cfa4fe72c00 add new vendored grpc version (#29866)
add dda2ffae602 Fix ItFramework extractJobName (#29876)
add 8bc7b376622 Added some learning doc links (#29878)
add 2f17526bd4e Add ComputeAndApplyVocabulary and Scale data notebook
(#29857)
add 41bd171a181 Fix race condition in SpannerAccessor (#29820)
add 50daf7dc438 Use RemoveSafeDeltaCounterCell for PerWorker counters
(#29877)
add 8f58ad12594 fix: Fix typo s/commiting/committing/ (#29880)
add 783c72a6dce [#29772][Go SDK] Add EventTime Timer tests. (#29829)
add f303d6ae7eb Add Error Handlers to File IO and related IOs (TextIO,
AvroIO) (#29670)
add 7342b03ff4e Creating a Fully Managed Beam Streaming System with Flink
Runner on Kubernetes - Part 2 (#29794)
add 18f1b9a72ef Make Neo4JIO streamed (#29853)
add 62de5b89c40 Revert "Bump com.gradle.enterprise from 3.15.1 to 3.16
(#29740)" (#29891)
add 9f7ab3a6e8e [#29848][Go SDK] Fix exec side transform identification.
(#29842)
add a8f0586b6e2 Record mean for underflow/overflow values in
HistogramData. (#29808)
add 9da9b8dd0e0 Clean up settings.kts (#29883)
add 1c1cfa85a9e Add support for tuple ClickHouse (#29715)
add 6751a1737c1 [#29895][Go SDK] Update timer test function names. (#29896)
add 80450399472 Add `svetakvsundhar` to Healthcare reviewer pool (#29899)
add 6d534fdd802 Bump github.com/aws/aws-sdk-go-v2/config from 1.26.1 to
1.26.2 in /sdks (#29850)
add 36bdff42e8e Support key iteration in Python ULR. (#29839)
add 7827d1d1809 Fix Reshuffle implementation in Java SDK
add 8bb66105689 Merge pull request #28853: Fix Reshuffle implementation in
Java SDK
add 3f3f64429fc fix indent (#29905)
add f227175a477 Allow pulling default beam services from an environment
variable.
add 881ca8c212a fix import order
add 7e95776a8d0 Exapnd docstring.
add b4ffbfe79b9 Merge pull request #29828 Also pull default beam services
from an environment variable.
add ec3fe3145e8 Clean up java PreCommit (#29894)
add 2d7586b3526 update bq docs on stable schemas (#29904)
add fe1627db947 [Blog Post] Scaling a Dataflow streaming workload (#29619)
add e6e180b6896 Switch to use self hosted runners for linux unit tests
(#29907)
add 02e26c1828f raise error if transforms are provided while read mode in
MLTransform (#29909)
add a5f0cda4f3b Fix: creating templates with the Typescript SDK (#29887)
add 308223f447f [release] 2.53.0 Website Updates (#29856)
add 24e5f3a6c01 [release] Add extra context for release branch cut action
input (#29920)
add f88c78b79ca [release] Update Release Guide Instructions (#29919)
add 8bd4a1e525b [Typescript] stop service when done (#29929)
add c6f0d9c462a Add notebooks for text embeddings (#29893)
add a6948ad71e3 Install beam from pypi instead of github in mltransform
notebooks (#29934)
add b9ae59a9978 Remove TODO in the notebook (#29935)
add 0b0d97325c1 Update pmc finalization in release-guide.md (#29936)
add f79eaddf574 Preparing the data processing notebooks for import to
devsite (#29937)
add 3260a7bf25a Final edit on data preprocessing notebooks (#29940)
add 4cf1b15024d Updated the doc with Security Fixes for 2.53.0 (#29941)
add 66f98a0670c Update finalize_release.yml (#29918)
add 75cfbee1591 Update MLTransform docs (#29910)
add c9e036e40e4 [Python BQ] Retry get_table for quota errors (#28820)
add 992e4fd5019 upgrade prettier
add 63503391562 Merge pull request #29942 [Typescript] upgrade prettier
add 094eb7dc5d7 Increase granularity of DICOM IO Unit Tests (#29956)
add b666c6434dc [Dataflow Streaming] Make SideInputCache bytes and expiry
configurable (#29871)
add 97ca4430d97 remove threadlocal for DESERIALIZATION_CONTEXT and use
copy instead (#29952)
add f311d327aac Bump org.checkerframework:checkerframework-gradle-plugin
add bb0a2037970 Merge pull request #29811: Bump
org.checkerframework:checkerframework-gradle-plugin from 0.6.34 to 0.6.37
add 8fb06da4ce8 [RRIO] Build RequestResponseIO and related dependency
changes (#29710)
add 5e8fef44939 Bump timeout to 120 minutes (#29964)
add 8aa16df9bd1 Bump runner v2 beam-master to 20240109 (#29965)
add 6066af3dbd5 [YAML] - Kafka Proto String schema (#29835)
add a3e065366e9 Bump GCP Libraries BOM to v26.29.0 (#29922)
add 2eb375be82f Downgrade artifact version in build_wheels (#29960)
add 0a368056904 Add support for pandas 2.0 (#28636)
add ee3d0a3c5ca [YAML] Add AssignTimestamps transform. (#29969)
add eb7addc1a5b [yaml] Replace pipeline_spec with yaml_pipeline in main.py
add 0c955dfdb7e Merge pull request #29916 [YAML] Replace pipeline_spec
with yaml_pipeline in main.py
add a80522ef5e7 update dependencies (#29974)
add 0bc1556d5ed Allows Java SDK to pass through external coders during
proto translation without fully interpretting them (#29859)
add 9697ae818d2 Update container tag (#29977)
add ba73001d687 Feature/bigtable error handling (#29885)
add efd50ede0dd update CHANGES.md with error handling for fileIO and
bigtableIO (#29983)
add 96d092b31ec Only auth and push on master branch (#29995)
add 812684f3cef [#29772][prism] Handle EventTime Timers. (#29900)
add 68849914cfc Fixes to environment cloning.
add 0fb42e5ee32 [YAML] Better support for inline PyTransforms.
add cfec51224e4 Merge pull request #29958 [YAML] Better support for inline
PyTransforms.
add d70f516ce20 fix: Spanner PostgreSQL should filter on 'public' schema
(#29984)
add db0a4deea2f [IT] Pub/Sub RM should delete subscriptions that were
created by Beam (#29957)
add c38dc77283f Upgrade Parquet to 0.13.1 (#29953)
add b0d74c3c6f3 Generate per transform schema for validation.
add 48650f12894 Conditional validation.
add e50b7cc89d2 Conditionally validate transform schemas.
add cb825565a9d Better line numbers for messages.
add 1c7f8392c3b Fix imports.
add 7cb559f2677 Merge pull request #29830 [YAML] More complete schema
validation.
add 0d58d23f2d9 [#29180][prism] Return total element count to progress
loop. Split less aggressively. (#29968)
add 057c1d56f88 Change colab links (#30006)
add 6cd53fad82a Support DLQ in MLTransform (#29938)
add 308c38e243c [Typescript] Worker improvements (#29903)
add dd5bbb2d77d Add a read timeout and cache BigQueryIOMetadata (#29662)
add c11b7c1f7cb Update scaling-streaming-workload.md (#29921)
add cf201198df9 sickbay new reshuffle test on runners that do not support
it
add 96a9830ac27 Merge pull request #29994: sickbay new reshuffle test on
runners that do not support it
add 534015c0b8e [Typescript] creating dataflow job template fix (#29928)
add a41fd2cba03 Remove INFO from playground python logger
add 7f5c4b9b72d Merge pull request #29948 Remove INFO from playground
python logger
add c62c24dda4c [yaml] Revise yaml user label (#29989)
add fc626b8b280 typo in docs
add f1c154778f5 Merge pull request #30021: typo in docs
add 5a4cbe2074f Additional test for windmill OrderedListState
implementation. (#29970)
add d1a17fa3cbd Update `google_api_services_healthcare` [01/24] (#30027)
add 4b909912ae8 Vendor grpc 1.60.1 (#29954)
add 730f1c5edb3 [YAML] Support comment and delimiter attributes on CSV IO.
add 87145a8c2c1 Merge pull request #30013 [YAML] Support comment and
delimiter attributes on CSV IO.
add b5dc728b677 Fix an edge case of getting duplicated records when using
TextIO. (#30026)
add 3faa8729677 add #29987 to 2.53 known issues (#29988)
add 52b4a9cb58e Update to fix endpoint connectivity (#30022)
add 79b9de283fe GCS autosharding flag (#29886)
add e867ed7cdce Worker message plumbing (#29879)
add b4c23b32f2b Correct Examples of utilizing `UploadToDicomStore` (#30008)
add 65811eed6ea Upgrade checkerframework to 3.42.0
add 8772389d0c2 Merge pull request #29961: Upgrade checkerframework to
3.42.0
add ba05f0f0343 Add a enum for the new value capture type
NEW_ROW_AND_OLD_VALUES. Also add related unit tests.
add e019de89307 Merge pull request #30012 Support spanner
NEW_ROW_AND_OLD_VALUES.
add 02b369d87a7 ReadFromTsv example.
add 2ff551eeb9e Merge pull request #29997 Add a ReadFromTsv example.
add 1de8454ddd2 [Python] Enrichment Transform with BigTable handler
(#30001)
add 0bfd1a47ff4 Update to libraries-bom 26.30.0 (#30028)
add 81688eb57b3 [YAML] add time unit suffix to windowing (#30035)
add e594f57a2d7 Undeprecate reshuffle (#30049)
add 195d8dcb367 Add logging to help understand errors from #28131. (#30052)
add ef8d3faa7df Switch back to use official jamm (#30034)
add b4be68e2404 [Python BQ] Substitute final destination schema when no
input schema is specified (#30015)
add 23e83092345 Fix buildx always been true (#30051)
add 774082f57a9 Exclude newly added failing validation runner tests
(#30030)
add d5a7fc92cdf Centralize exec stage urn string (#29947)
add e4b81809c13 fix config translator. If config is set on BigtableIO
directly, it should override configs in BigtableOptions (#30039)
add 175346e3507 Automatically enable Dataflow Runner v2 for pipeliens that
use the transform upgrade feature
add c0cc7c4db1b Merge pull request #30061: Automatically enable Dataflow
Runner v2 for pipeliens that use the transform upgrade feature
add 5eeffc7715f Address Feedback from Beam Website Feedback Form. (#30064)
add d014a98936e Time at max threads fix (#30041)
add 173d8340b20 split write rename and temp table deletion into to dofns
with a shuffle (#30023)
add 821a1694c5c Fix Cassandra Test (#30075)
add 5e7edc45598 Heartbeats (#29963)
add 0a813b99657 Plumbing remaining autoscaling metrics (#30070)
add 93339bcf061 Fixes an upgrade compatibility breakage for the BQ write
transform (#30032)
add e8e0ccfdd7f Bump pillow (#30076)
add cffeff78f62 Fix Dicom IO Integration Tests (#30073)
add 58650378fe7 [flink] Fix watermark generation for empty UnboundedSource
(#30050)
add 90e7c9e8f6b Merge pull request #30066: [flink] Fix watermark
generation for empty UnboundedSource
add 77285589fe3 Use vendored grpc 1.60.1 (#30069)
add 0b06cd897d2 Fix BigQuerySinkMetrics constants and increment metrics in
more places. (#30067)
add 0c2b1a512db Add converter class to convert metrics to
PerStepNamespaceMetrics (#30084)
add 9e5b112be9f Use FileSystems.exist to check the remote the path (#30082)
add 405e6c35bb8 [YAML] Use default logging level for main.py
add 330a632fefb Merge pull request #30087 [YAML] Use default logging level
for main.py
add 9dd30765476 Bump pillow from 10.0.1 to 10.2.0 in
/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing
(#30071)
add 7ff25d89625 [YAML] Allow windowing to be done in Java or Python.
(#30055)
add b22fe9287b0 [YAML] LogForTesting improvements. (#30005)
add 0eefb797765 [YAML] Expose flatten implementation from Java. (#30057)
add 8f3887bb512 [YAML] Require numStreams for unbounded
BigQueryStorageWriteApiSchemaTransform xlang transform
add 11646ca7da2 Merge pull request #30059 [YAML] Require numStreams for
unbounded BigQueryStorageWrite
add 582d5c03569 fix for #30078 (#30079)
add 98a86b11cc7 [MLTransform]Throw an error when specifiend columns are
not present in the input data (#30085)
add f7da756ac6d Update container build version to Go 1.21.6 (#30090)
add e85d070a32b Ignore DicomMetadataRead tests (#30096)
add 27214149d34 Support dynamic destinations with Python Storage API
(#30045)
add 674fe77df47 [Python] Add an option to retain timestamp returned from
the BigTable row (#30088)
add e3f45249620 Moving to 2.55.0-SNAPSHOT on master branch.
add d759f970139 Add StreamingStepMetricsContainer::extractPerWorkerMetrics
method. (#30089)
add d62ae0144cc Add section for 2.55.0, cleanup 2.54.0 due to cut. (#30101)
add 61a62e19c4a rename ExternalSchemaTransform to ExternalTransform
(#30102)
add 93073e74e11 Decommission all remaining jenkins jobs (#30113)
add 772cf0a5c01 Don't run DicomIO Integration Tests on GCP DF Runner
(#30103)
add 3ae851876e2 Fix failing python BQ test (#30099)
add 6b6fc98ec93 Don't depend on the go 1.21 standard lib yet (#30120)
add 445151694fe Check the issue number type (#30127)
add 3fd7bf29dcd Bump Dataflow dev container version (#30124)
add 1ab504c34a6 Update update_python_dependencies.yml with all python
versions (#30126)
add 4981c0c3abc Upgrade numpy dependency (#29645)
add 1d4413d8e95 Update project root documentations reflecting Jenkins
shutdown (#30130)
add e0e20a10e99 [#28142][Go SDK] Evict closed readers from the cache.
(#30119)
add 8485ab40e12 Allow JSON type in TableSchema for BigQuery FILE_LOAD
(#29923)
add 3172736aaa3 Update Python Dependencies (#30140)
add dde81384ef2 Fix clean up stale container workflow (#30132)
add c9135e00041 Add info on getting started contributing (#30143)
add b72eacfba37 make getProcessingDistributionsForWorkId threadsafe
(#29979)
add c1c81b36187 Respect gcloud active configuration, if any, rather than
always using "default."
add ee4f8cb33f9 Merge pull request #30134 Respect gcloud active
configuration.
add 847a8f4486e [YAML] Improve docs on create.
add 8521d2768e0 [YAML] Better document naming parameter.
add ba29583d46c [YAML] Move pipeline running instructions to the top.
add e70a28acbd4 [YAML] Link to additonal resources.
add b85e1ebaea3 Some clarification on the mapping docs.
add 09aacf1003c Another note useful for trying things out.
add 799c1041473 Avoid SQL keywords for SQL table names.
add afd8b79e8e0 Clarification on aggregating transform based on user
feedback.
add 88bf2fc94fe Update sdks/python/apache_beam/yaml/yaml_provider.py
add b835111d66f Make note about requiring Graphviz for rendering pipelines.
add 3970b618371 Merge branch 'master' into yaml-create-doc
add b5dc54da26f Also mention JavaScript.
add e199e42a341 Fix whitespace in create suggestion.
add ce28c33940c Merge pull request #30007 [YAML] Several improvements to
the documentation.
add f9029616a91 Update environments.md (#30149)
add 097e6e27bfe Bump actions/cache from 3 to 4 (#30033)
add f9c9788815e Update 02_io_pubsub.md (#29986)
add 5a9a27cbafa Volatile heartbeat (#30148)
add b9fd39cb16b Fix Dataproc cleanup race condition (#30154)
add 881a6cf1cc9 Add criteo benchmark for MLTransform
add b498e4ab3bb Skip few tests
add 30041ad9b97 Refactor pipeline
add cebdefe0309 Merge branch 'ml_transform_criteo' of
https://github.com/apache/beam into ml_transform_criteo
add 22fefebacab BigQuery: Decouple clustering from time partitioning when
writing (#30094)
add 415b1658f50 Merge pull request #30137: Fix a misspelling in zetasql
overview page
add 1df9d6bba91 Bumping timeout for beam_PostCommit_Java_DataflowV2
(#30160)
add 4c5ee349f9b [sdks-java-core] Test for Reiterator outside loop in
CoGbkResult
add 36cab4ffe76 Merge pull request #30163: [sdks-java-core] Test for
Reiterator outside loop in CoGbkResult
add 41dee464db4 Make ReadFromBigQueryRequest id more randomized (#30156)
add 81513f10d60 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks
(#30162)
add 89d1c06e1ea Fix Fhir IO Search IT (#30158)
add a221f98a5f4 fix: fix error handling in segment reader and add an
integration test (#30147)
add ddb93272ab6 Fix git trusted directory errors in website build. (#30171)
add 30917ff7b3e Don't use '-' as a delimiter in BigQuerySinkMetrics
(#30118)
add fb0762352c2 Periodically send perWorkerMetrics on the WorkerMessages
API (#30135)
add 4f09e62f520 Improve varint encoding throughput with unrolled loop
(#29689)
add 5188e37ed44 Bump github.com/opencontainers/runc from 1.1.5 to 1.1.12
in /sdks (#30175)
add 96cf695a101 Fix JdbcIO PreCommit mysql breaking change (#30179)
add 824e06cd967 Allow stale_bq_datasets_cleaner.sh to run to completion,
even when there are failures to grab a dataset (#30153)
add 1d9f6044c76 Remove constraint introduced in #30059 (#30186)
add bdbbef9b54d [YAML] - PubSubLite proto (#30129)
add 1a5466b7df7 Support dict as the input of delete_batch (#30188)
add 574e7bfbc1d Fix script typos for validating docker licenses. (#30176)
add 7fb05cbb5e1 Move the code from sdks-java-fn-execution to sdks-java-core
add e8df60381a3 Merge pull request #30170: Move the code from
sdks-java-fn-execution to sdks-java-core
add 124880bc8ce Fix typo affecting
DataflowPipelineDebugOptions.setStreamingSideInputCacheExpirationMillis (#30195)
add 1f69271044d [Dataflow Streaming] Add a option to use multiple commit
threads (#30194)
add 1183b5d5274 Knowledge lookup prompts: (#29747)
add 801682930e2 Bump google.golang.org/protobuf from 1.31.0 to 1.32.0 in
/sdks (#29913)
add 30a778b9cbe Bump github.com/containerd/containerd from 1.7.7 to 1.7.11
in /sdks (#29823)
add f03b115830b Remove extraneous mocking that was causing NPEs in
DataflowWorkUnitClientTest
add 00f07864784 Remove sdks/java/fn-execution
add 8ec51f181a0 Merge pull request #30193: Remove sdks/java/fn-execution
add 9f8dda274dc Split standalone transform service application from
launcher jar (#30190)
add f2056e72dea Fix stale dataflow prebuilt image cleaner (#30168)
add 34bbbda9f98 Try a different approach for cibuildwheel flake. (#30204)
add 1a1ff756f86 Bump golang.org/x/sync from 0.5.0 to 0.6.0 in /sdks
(#30209)
add c1b3a27ee0c Bump github.com/nats-io/nats-server/v2 from 2.10.7 to
2.10.10 in /sdks (#30210)
add a47b1faa527 [Prism] Implement jobservices.Server Cancel (#30178)
add 7f91baadcf2 Upgrade website node to LTS (#30213)
add d5aa44c9ba9 Handle a user whose account was deleted. (#30192)
add a341eb69abe Added default watermark generation interval (apache#30141)
add cd5f2711c83 Merge pull request #30141: Adding a default watermark emit
interval for FlinkUnboundedSourceReader
add b0f2eebb024 When failing work items during commit, make sure to call
completeWork… (#30215)
add 23dcb7ec1d5 Revert "When failing work items during commit, make sure
to call completeWork…" (#30228)
add 5494f114382 [Dataflow Streaming] Invalidate caches and remove work on
failure before commit (#30229)
add 27f1c0774fd Bump github.com/tetratelabs/wazero from 1.5.0 to 1.6.0 in
/sdks (#30222)
add 78d08235c72 Add grpc-services, grpc-utils to vendored grpc (#30196)
add 3412b4f7a5b fix markdown links (#30227)
add c006c5e2d53 Add Default method for OutputReceiver.outputWindowedValue
(#30220)
add 08a717dc5c2 Creating a Fully Managed Beam Streaming System with Flink
Runner on Kubernetes - Part 3 (#29860)
add 7fa4c99535f Fix to take StreamingDataflowWorkerOptions from external
options (#30232)
add f08058ca75c Bump cloud.google.com/go/spanner from 1.53.1 to 1.56.0 in
/sdks (#30224)
add 7a46686577d Fix a naming conflict of the TransformService test suite
(#30243)
add 4645a76ee2a gcsio: reduce number of get requests in function calls
(#30205)
add f250a843230 Adjust transform-service module names and deps with
sdks/java/core
add 13a70844abb Merge pull request #30237: Adjust transform-service module
names and deps with sdks/java/core
add 4f3963f7878 Bump google.golang.org/grpc from 1.60.1 to 1.61.0 in /sdks
(#30249)
add e031888c818 Merge remote-tracking branch 'origin/master' into
ml_transform_criteo
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 2 +
.github/REVIEWERS.yml | 4 +
.github/actions/setup-action/action.yml | 6 +-
.github/autolabeler.yml | 1 +
.../gh-actions-self-hosted-runners/arc/locals.tf | 2 +
.../beam_PostCommit_Java_DataflowV2.json | 0
.../trigger_files/beam_PostCommit_Python.json | 0
.../beam_PostCommit_Python_Xlang_Gcp_Direct.json | 1 +
.github/workflows/README.md | 1 +
.github/workflows/beam_CancelStaleDataflowJobs.yml | 2 +
.../workflows/beam_CleanUpDataprocResources.yml | 2 +-
.github/workflows/beam_CleanUpGCPResources.yml | 2 +
.../workflows/beam_CleanUpPrebuiltSDKImages.yml | 8 +-
...eam_LoadTests_Java_CoGBK_Dataflow_Streaming.yml | 2 +-
.github/workflows/beam_Metrics_Report.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Avro.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Json.yml | 2 +-
..._PerformanceTests_BigQueryIO_Streaming_Java.yml | 2 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 2 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 2 +-
.github/workflows/beam_PostCommit_Java.yml | 2 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 2 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV1.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV2.yml | 6 +-
.../beam_PostCommit_Java_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
...beam_PostCommit_Java_Examples_Dataflow_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_V2.yml | 2 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 2 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java21.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Spark_Java11.yml | 2 +-
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 2 +-
.../workflows/beam_PostCommit_Java_PVR_Samza.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.yml | 4 +-
.github/workflows/beam_PostCommit_Java_Sickbay.yml | 2 +-
.../workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 10 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 2 +-
...it_Java_ValidatesRunner_Direct_JavaVersions.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +-
...ostCommit_Java_ValidatesRunner_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Samza.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 2 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 2 +-
...ostCommit_Java_ValidatesRunner_Spark_Java11.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 2 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 2 +-
.../beam_PostCommit_PortableJar_Flink.yml | 2 +-
.../beam_PostCommit_PortableJar_Spark.yml | 2 +-
.github/workflows/beam_PostCommit_Python.yml | 4 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 4 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_SQL.yml | 2 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 2 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 2 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 2 +-
.github/workflows/beam_PreCommit_Java.yml | 6 +-
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 4 +-
...reCommit_Java_Amazon-Web-Services_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 4 +-
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 4 +-
...eam_PreCommit_Java_Examples_Dataflow_Java21.yml | 6 +-
...Commit_Java_File-schema-transform_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 2 +-
.../beam_PreCommit_Java_GCP_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Google-ads_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 4 +-
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 4 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 8 +-
.../beam_PreCommit_Java_Kinesis_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 4 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 2 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 4 +-
...am_PreCommit_Java_RequestResponse_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Spark3_Versions.yml | 2 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 4 +-
.github/workflows/beam_PreCommit_Python.yml | 4 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 4 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 4 +-
.../beam_PreCommit_Python_Integration.yml | 4 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 4 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 4 +-
.github/workflows/beam_PreCommit_SQL.yml | 4 +-
.github/workflows/beam_PreCommit_SQL_Java11.yml | 6 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 6 +-
.github/workflows/beam_PreCommit_Spotless.yml | 2 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 8 +-
.github/workflows/build_release_candidate.yml | 26 +-
.github/workflows/build_runner_image.yml | 4 +-
.github/workflows/build_wheels.yml | 24 +-
.github/workflows/code_completion_plugin_tests.yml | 2 +-
.github/workflows/cut_release_branch.yml | 2 +-
.github/workflows/dask_runner_tests.yml | 8 +-
.../workflows/deploy_release_candidate_pypi.yaml | 7 +-
.github/workflows/finalize_release.yml | 26 +-
.github/workflows/go_tests.yml | 2 +-
.github/workflows/java_tests.yml | 18 +-
.github/workflows/local_env_tests.yml | 4 +-
.github/workflows/playground_frontend_test.yml | 2 +-
.github/workflows/python_dependency_tests.yml | 2 +-
.github/workflows/python_tests.yml | 22 +-
.github/workflows/run_perf_alert_tool.yml | 2 +-
.github/workflows/run_rc_validation.yml | 14 +-
.github/workflows/stale.yml | 2 +-
.github/workflows/tour_of_beam_frontend_test.yml | 2 +-
.github/workflows/typescript_tests.yml | 4 +-
.github/workflows/update_python_dependencies.yml | 15 +-
.test-infra/dataproc/cleanup.sh | 4 +-
.test-infra/jenkins/README.md | 149 +----
.test-infra/jenkins/job_00_seed.groovy | 115 ----
.test-infra/jenkins/job_Inventory.groovy | 83 ---
...mmit_CrossLanguageValidatesRunner_Direct.groovy | 51 --
...uageValidatesRunner_GoUsingJava_Dataflow.groovy | 44 --
...ValidatesRunner_PythonUsingJava_Dataflow.groovy | 49 --
.../jenkins/job_PostCommit_Java_DataflowV2.groovy | 48 --
.../job_PostCommit_Java_InfluxDBIO_IT.groovy | 55 --
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 45 --
...Commit_Java_ValidatesRunner_Flink_Java11.groovy | 61 --
.test-infra/jenkins/job_PostCommit_Python.groovy | 50 --
...tCommit_Python_CrossLanguage_IO_Dataflow.groovy | 55 --
.../job_PostCommit_Python_Examples_Direct.groovy | 43 --
.../job_PostCommit_Python_Examples_Flink.groovy | 43 --
..._PostCommit_Python_ValidatesRunner_Flink.groovy | 38 --
..._PostCommit_Python_ValidatesRunner_Spark.groovy | 43 --
.../jenkins/job_PostRelease_NightlySnapshot.groovy | 59 --
.test-infra/jenkins/job_seed_standalone.groovy | 115 ----
.test-infra/jupyter/README.md | 2 +-
.test-infra/metrics/sync/github/sync_workflows.py | 12 +-
.test-infra/mock-apis/go.mod | 6 +-
.test-infra/mock-apis/go.sum | 12 +-
.../google-kubernetes-engine/README.md | 14 +-
.../apache-beam-testing.tfvars | 22 -
.../google-kubernetes-engine/cluster.tf | 19 +-
.../us-central1.apache-beam-testing.tfvars | 22 +
.../us-west1.apache-beam-testing.tfvars | 22 +
.test-infra/tools/stale_bq_datasets_cleaner.sh | 34 +-
.../tools/stale_dataflow_prebuilt_image_cleaner.sh | 103 ++-
CHANGES.md | 93 ++-
CI.md | 37 +-
README.md | 3 -
build.gradle.kts | 45 +-
buildSrc/build.gradle.kts | 2 +-
.../org/apache/beam/gradle/BeamDockerPlugin.groovy | 3 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 52 +-
.../apache/beam/gradle/GrpcVendoring_1_54_0.groovy | 204 ------
.../apache/beam/gradle/GrpcVendoring_1_60_1.groovy | 225 +++++++
.../org/apache/beam/gradle/VendorJavaPlugin.groovy | 9 +-
contributor-docs/release-guide.md | 35 +-
examples/java/build.gradle | 2 +-
.../org/apache/beam/examples/KafkaStreaming.java | 2 +-
.../transforms/DataProtectors.java | 6 +-
.../kafkatopubsub/kafka/consumer/Utils.java | 4 +-
examples/notebooks/beam-ml/README.md | 19 +-
.../beam-ml/automatic_model_refresh.ipynb | 2 +-
.../compute_and_apply_vocab.ipynb | 419 ++++++++++++
.../huggingface_text_embeddings.ipynb | 399 +++++++++++
.../beam-ml/data_preprocessing/scale_data.ipynb | 504 ++++++++++++++
.../vertex_ai_text_embeddings.ipynb | 308 +++++++++
.../notebooks/beam-ml/run_custom_inference.ipynb | 2 +-
gradle.properties | 4 +-
.../apache/beam/it/common/utils/PipelineUtils.java | 10 +-
.../beam/it/common/utils/PipelineUtilsTest.java | 6 +
.../it/gcp/dataflow/AbstractPipelineLauncher.java | 17 +-
.../it/gcp/dataflow/DefaultPipelineLauncher.java | 4 +-
.../beam/it/gcp/pubsub/PubsubResourceManager.java | 14 +-
.../it/gcp/pubsub/PubsubResourceManagerTest.java | 28 +-
.../beam/it/jdbc/AbstractJDBCResourceManager.java | 6 +-
learning/prompts/README.md | 76 +++
learning/prompts/code-explanation/01_io_kafka.md | 65 ++
learning/prompts/code-explanation/02_io_pubsub.md | 33 +
.../prompts/code-explanation/03_io_bigquery.md | 39 ++
.../prompts/code-explanation/04_io_bigtable.md | 44 ++
learning/prompts/code-explanation/05_io_spanner.md | 96 +++
.../prompts/code-explanation/06_io_tfrecord.md | 63 ++
learning/prompts/code-explanation/07_io_json.md | 50 ++
learning/prompts/code-explanation/08_io_csv.md | 44 ++
learning/prompts/code-explanation/09_io_avro.md | 41 ++
learning/prompts/code-explanation/10_io_parquet.md | 39 ++
learning/prompts/code-explanation/11_io_jdbc.md | 105 +++
learning/prompts/code-generation/01_io_kafka.md | 46 ++
learning/prompts/code-generation/02_io_pubsub.md | 51 ++
learning/prompts/code-generation/03_io_bigquery.md | 54 ++
learning/prompts/code-generation/04_io_bigtable.md | 66 ++
learning/prompts/code-generation/05_io_spanner.md | 63 ++
learning/prompts/code-generation/06_io_tfrecord.md | 60 ++
learning/prompts/code-generation/07_io_json.md | 48 ++
learning/prompts/code-generation/08_io_csv.md | 47 ++
learning/prompts/code-generation/09_io_avro.md | 42 ++
learning/prompts/code-generation/10_io_parquet.md | 39 ++
learning/prompts/code-generation/11_io_jdbc.md | 75 +++
.../01_basic_learning_apache_beam.md | 14 +
.../documentation-lookup/02_basic_pipelines.md | 19 +
.../03_basic_configuring_pipelines.md | 29 +
.../documentation-lookup/04_basic_pcollections.md | 25 +
.../documentation-lookup/05_basic_ptransforms.md | 34 +
.../documentation-lookup/06_basic_schema.md | 25 +
.../documentation-lookup/07_basic_runners.md | 23 +
.../documentation-lookup/08_basic_windowing.md | 29 +
.../documentation-lookup/09_basic_triggers.md | 32 +
.../documentation-lookup/10_basic_metrics.md | 24 +
.../prompts/documentation-lookup/11_basic_state.md | 20 +
.../documentation-lookup/12_basic_timers.md | 12 +
.../13_advanced_splittable_dofn.md | 18 +
.../14_advanced_pipeline_patterns.md | 13 +
.../documentation-lookup/15_advanced_xlang.md | 15 +
.../16_advanced_pipeline_lifecycle.md | 36 +
.../documentation-lookup/17_advanced_ai_ml.md | 18 +
.../prompts/documentation-lookup/18_io_kafka.md | 27 +
.../prompts/documentation-lookup/19_io_pubsub.md | 27 +
.../prompts/documentation-lookup/20_io_biguery.md | 39 ++
.../prompts/documentation-lookup/21_io_bigtable.md | 28 +
.../prompts/documentation-lookup/22_io_spanner.md | 31 +
learning/prompts/documentation-lookup/23_io_gcs.md | 23 +
.../prompts/documentation-lookup/24_io_json.md | 32 +
learning/prompts/documentation-lookup/25_io_csv.md | 32 +
.../prompts/documentation-lookup/26_io_avro.md | 31 +
.../prompts/documentation-lookup/27_io_parquet.md | 31 +
.../prompts/documentation-lookup/28_io_jdbc.md | 52 ++
.../map/co-group-by-key/description.md | 52 +-
.../beam/model/pipeline/v1/beam_runner_api.proto | 4 +
playground/infrastructure/logger.py | 2 +-
release/go-licenses/Dockerfile | 2 +-
release/src/main/Dockerfile | 2 +-
runners/core-construction-java/build.gradle | 3 +-
.../beam/runners/core/construction/BeamUrns.java | 2 +-
.../core/construction/CoderTranslation.java | 32 +-
.../core/construction/CoderTranslators.java | 2 +-
.../core/construction/CombineTranslation.java | 2 +-
.../CreatePCollectionViewTranslation.java | 2 +-
.../DefaultExpansionServiceClientFactory.java | 2 +-
.../core/construction/DisplayDataTranslation.java | 2 +-
.../runners/core/construction/Environments.java | 4 +-
.../beam/runners/core/construction/External.java | 6 +-
.../runners/core/construction/ModelCoders.java | 2 +-
.../construction/PCollectionViewTranslation.java | 2 +-
.../core/construction/PTransformTranslation.java | 5 +-
.../core/construction/ParDoTranslation.java | 4 +-
.../construction/PipelineOptionsTranslation.java | 6 +-
.../core/construction/PipelineTranslation.java | 17 +-
.../runners/core/construction/ReadTranslation.java | 4 +-
.../runners/core/construction/SdkComponents.java | 2 +-
.../construction/SplittableParDoNaiveBounded.java | 38 ++
.../core/construction/TestStreamTranslation.java | 2 +-
.../core/construction/TransformUpgrader.java | 68 +-
.../core/construction/UnknownCoderWrapper.java | 69 ++
.../core/construction/WindowIntoTranslation.java | 2 +-
.../construction/WindowingStrategyTranslation.java | 8 +-
.../core/construction/WriteFilesTranslation.java | 25 +-
.../graph/GreedyPCollectionFusers.java | 2 +-
.../core/construction/graph/QueryablePipeline.java | 2 +-
.../core/construction/BaseExternalTest.java | 6 +-
.../core/construction/CoderTranslationTest.java | 1 +
.../PipelineOptionsTranslationTest.java | 6 +-
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/construction/TransformUpgraderTest.java | 9 +-
.../construction/WindowIntoTranslationTest.java | 2 +-
.../construction/WriteFilesTranslationTest.java | 6 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
runners/core-java/build.gradle | 3 +-
...TimeBoundedSplittableProcessElementInvoker.java | 25 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 79 ++-
.../core/metrics/ExecutionStateSampler.java | 13 +-
.../core/metrics/ExecutionStateTracker.java | 9 +-
.../runners/core/metrics/MetricsContainerImpl.java | 2 +-
.../core/metrics/MetricsContainerStepMap.java | 6 +-
.../core/metrics/MonitoringInfoEncodings.java | 2 +-
.../beam/runners/core/metrics/ShortIdMap.java | 2 +-
.../runners/core/metrics/SimpleExecutionState.java | 2 +-
.../runners/core/metrics/SimpleStateRegistry.java | 2 +-
.../core/metrics/MonitoringInfoEncodingsTest.java | 2 +-
runners/direct-java/build.gradle | 7 +-
runners/flink/flink_runner.gradle | 5 +-
.../flink_job_server_container.gradle | 5 +-
runners/flink/job-server/flink_job_server.gradle | 2 +
.../FlinkBatchPortablePipelineTranslator.java | 2 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 2 +-
.../beam/runners/flink/FlinkPipelineRunner.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 2 +-
.../apache/beam/runners/flink/metrics/Metrics.java | 3 +-
.../utils/FlinkPortableRunnerUtils.java | 2 +-
.../streaming/ExecutableStageDoFnOperator.java | 4 +-
.../wrappers/streaming/FlinkKeyUtils.java | 2 +-
.../streaming/io/source/FlinkSourceReaderBase.java | 1 +
.../unbounded/FlinkUnboundedSourceReader.java | 49 +-
.../FlinkExecutableStageFunctionTest.java | 2 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 4 +-
.../wrappers/streaming/FlinkKeyUtilsTest.java | 2 +-
.../streaming/io/source/EmptyUnboundedSource.java | 101 +++
.../io/source/FlinkSourceReaderTestBase.java | 8 +
.../unbounded/FlinkUnboundedSourceReaderTest.java | 33 +
runners/google-cloud-dataflow-java/build.gradle | 8 +-
.../beam/runners/dataflow/DataflowMetrics.java | 8 +-
.../dataflow/DataflowPipelineTranslator.java | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 30 +-
.../runners/dataflow/internal/CustomSources.java | 4 +
.../options/DataflowPipelineDebugOptions.java | 32 +-
.../util/RowCoderCloudObjectTranslator.java | 2 +-
.../util/SchemaCoderCloudObjectTranslator.java | 2 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 2 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 35 +-
.../google-cloud-dataflow-java/worker/build.gradle | 9 +-
.../dataflow/worker/ActiveMessageMetadata.java | 32 +
.../runners/dataflow/worker/ByteArrayReader.java | 4 +-
.../runners/dataflow/worker/ByteStringCoder.java | 2 +-
.../worker/ChunkingShuffleBatchReader.java | 2 +-
.../worker/DataflowBatchWorkerHarness.java | 2 +-
.../dataflow/worker/DataflowExecutionContext.java | 61 +-
.../worker/DataflowExecutionStateSampler.java | 133 ++++
.../dataflow/worker/DataflowWorkUnitClient.java | 60 ++
.../worker/DataflowWorkerHarnessHelper.java | 10 +-
.../worker/GroupAlsoByWindowParDoFnFactory.java | 2 +-
.../worker/MetricTrackingWindmillServerStub.java | 32 +-
.../MetricsToPerStepNamespaceMetricsConverter.java | 196 ++++++
.../runners/dataflow/worker/PubsubDynamicSink.java | 2 +-
.../beam/runners/dataflow/worker/PubsubReader.java | 8 +
.../beam/runners/dataflow/worker/PubsubSink.java | 2 +-
.../worker/RemoveSafeDeltaCounterCell.java | 117 ++++
.../dataflow/worker/StreamingDataflowWorker.java | 290 +++++---
.../worker/StreamingModeExecutionContext.java | 16 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 2 +-
.../worker/StreamingStepMetricsContainer.java | 139 +++-
.../dataflow/worker/UngroupedShuffleReader.java | 2 +-
.../dataflow/worker/UngroupedWindmillReader.java | 8 +
.../dataflow/worker/WindmillComputationKey.java | 4 +-
.../dataflow/worker/WindmillNamespacePrefix.java | 2 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 2 +-
.../dataflow/worker/WindmillTimerInternals.java | 2 +-
.../worker/WorkItemCancelledException.java | 39 ++
.../runners/dataflow/worker/WorkUnitClient.java | 28 +
.../dataflow/worker/WorkerCustomSources.java | 5 +-
.../worker/WorkerPipelineOptionsFactory.java | 15 +-
.../dataflow/worker/streaming/ActiveWorkState.java | 92 ++-
.../worker/streaming/ComputationState.java | 16 +-
.../dataflow/worker/streaming/ShardedKey.java | 2 +-
.../dataflow/worker/streaming/StageInfo.java | 9 +
.../runners/dataflow/worker/streaming/Work.java | 85 ++-
.../worker/streaming/sideinput/SideInputCache.java | 11 +-
.../streaming/sideinput/SideInputStateFetcher.java | 6 +-
.../dataflow/worker/util/BoundedQueueExecutor.java | 4 +-
.../common/worker/ByteArrayShufflePosition.java | 4 +-
.../worker/GroupingShuffleEntryIterator.java | 2 +-
.../common/worker/KeyGroupedShuffleEntries.java | 2 +-
.../worker/util/common/worker/ShuffleEntry.java | 2 +-
.../worker/windmill/WindmillServerStub.java | 6 +
.../windmill/client/AbstractWindmillStream.java | 6 +-
.../worker/windmill/client/WindmillStream.java | 5 +-
.../client/grpc/AppendableInputStream.java | 2 +-
.../windmill/client/grpc/GrpcCommitWorkStream.java | 4 +-
.../client/grpc/GrpcDeadlineClientInterceptor.java | 10 +-
.../client/grpc/GrpcDirectGetWorkStream.java | 4 +-
.../windmill/client/grpc/GrpcGetDataStream.java | 109 ++-
.../windmill/client/grpc/GrpcGetWorkStream.java | 4 +-
.../client/grpc/GrpcGetWorkerMetadataStream.java | 2 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 41 +-
.../client/grpc/GrpcWindmillStreamFactory.java | 16 +-
.../grpc/auth/VendoredCredentialsAdapter.java | 4 +-
.../VendoredRequestMetadataCallbackAdapter.java | 4 +-
.../grpc/observers/DirectStreamObserver.java | 4 +-
.../ForwardingClientResponseObserver.java | 6 +-
.../grpc/observers/StreamObserverFactory.java | 4 +-
.../client/grpc/stubs/WindmillChannelFactory.java | 12 +-
.../client/grpc/stubs/WindmillStubFactory.java | 4 +-
.../dataflow/worker/windmill/state/StateTag.java | 2 +-
.../worker/windmill/state/WindmillBag.java | 2 +-
.../worker/windmill/state/WindmillMap.java | 2 +-
.../worker/windmill/state/WindmillMultimap.java | 2 +-
.../worker/windmill/state/WindmillOrderedList.java | 2 +-
.../worker/windmill/state/WindmillStateCache.java | 7 +-
.../worker/windmill/state/WindmillStateReader.java | 20 +-
.../worker/windmill/state/WindmillStateUtil.java | 2 +-
.../worker/windmill/state/WindmillValue.java | 2 +-
.../windmill/state/WindmillWatermarkHold.java | 2 +-
.../runners/dataflow/harness/test/TestStreams.java | 4 +-
.../worker/DataflowExecutionContextTest.java | 76 +++
.../worker/DataflowExecutionStateSamplerTest.java | 148 +++++
.../runners/dataflow/worker/DataflowMatchers.java | 2 +-
.../worker/DataflowWorkUnitClientTest.java | 154 ++++-
.../worker/DataflowWorkerHarnessHelperTest.java | 32 +-
.../dataflow/worker/FakeWindmillServer.java | 55 +-
.../dataflow/worker/GroupingShuffleReaderTest.java | 2 +-
...ricsToPerStepNamespaceMetricsConverterTest.java | 293 ++++++++
.../dataflow/worker/PubsubDynamicSinkTest.java | 2 +-
.../runners/dataflow/worker/PubsubReaderTest.java | 2 +-
.../runners/dataflow/worker/PubsubSinkTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 2 +-
.../worker/RemoveSafeDeltaCounterCellTest.java | 102 +++
.../runners/dataflow/worker/ShuffleSinkTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 213 +++++-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
.../worker/StreamingModeExecutionContextTest.java | 8 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 2 +-
.../worker/StreamingStepMetricsContainerTest.java | 159 +++++
.../runners/dataflow/worker/TestShuffleReader.java | 2 +-
.../runners/dataflow/worker/TestShuffleWriter.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillReaderIteratorBaseTest.java | 2 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 84 ++-
.../worker/WorkerPipelineOptionsFactoryTest.java | 4 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 2 +-
.../worker/streaming/ActiveWorkStateTest.java | 52 +-
.../sideinput/SideInputStateFetcherTest.java | 16 +-
.../worker/BatchingShuffleEntryReaderTest.java | 2 +-
.../worker/GroupingShuffleEntryIteratorTest.java | 2 +-
.../util/common/worker/ShuffleEntryTest.java | 2 +-
.../grpc/GrpcGetWorkerMetadataStreamTest.java | 14 +-
.../client/grpc/GrpcWindmillServerTest.java | 207 +++++-
.../client/grpc/StreamingEngineClientTest.java | 16 +-
.../client/grpc/WindmillStreamSenderTest.java | 6 +-
.../windmill/state/WindmillStateCacheTest.java | 2 +-
.../windmill/state/WindmillStateInternalsTest.java | 110 ++-
.../windmill/state/WindmillStateReaderTest.java | 2 +-
.../budget/EvenGetWorkBudgetDistributorTest.java | 6 +-
.../worker/windmill/src/main/proto/windmill.proto | 67 +-
runners/java-fn-execution/build.gradle | 3 +-
.../artifact/ArtifactRetrievalService.java | 8 +-
.../artifact/ArtifactStagingService.java | 10 +-
.../control/BundleCheckpointHandlers.java | 2 +-
.../fnexecution/control/FnApiControlClient.java | 6 +-
.../control/FnApiControlClientPoolService.java | 6 +-
.../control/ProcessBundleDescriptors.java | 2 +-
.../runners/fnexecution/data/GrpcDataService.java | 2 +-
.../environment/ExternalEnvironmentFactory.java | 2 +-
.../fnexecution/logging/GrpcLoggingService.java | 2 +-
.../runners/fnexecution/provisioning/JobInfo.java | 2 +-
.../provisioning/StaticGrpcProvisionService.java | 2 +-
.../fnexecution/state/GrpcStateService.java | 4 +-
.../fnexecution/state/StateRequestHandlers.java | 2 +-
.../status/BeamWorkerStatusGrpcService.java | 2 +-
.../fnexecution/status/WorkerStatusClient.java | 2 +-
.../translation/PipelineTranslatorUtils.java | 2 +-
.../runners/fnexecution/wire/ByteStringCoder.java | 4 +-
.../runners/fnexecution/EmbeddedSdkHarness.java | 3 +
.../GrpcContextHeaderAccessorProviderTest.java | 22 +-
.../runners/fnexecution/ServerFactoryTest.java | 15 +-
.../artifact/ArtifactRetrievalServiceTest.java | 12 +-
.../artifact/ArtifactStagingServiceTest.java | 14 +-
.../control/BundleFinalizationHandlersTest.java | 4 +
.../control/DefaultJobBundleFactoryTest.java | 6 +-
.../control/FnApiControlClientPoolServiceTest.java | 11 +-
.../control/FnApiControlClientTest.java | 5 +-
.../control/ProcessBundleDescriptorsTest.java | 3 +
...eCountingExecutableStageContextFactoryTest.java | 2 +
.../fnexecution/control/RemoteExecutionTest.java | 5 +-
.../fnexecution/control/SdkHarnessClientTest.java | 5 +-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 3 +
.../control/TimerReceiverFactoryTest.java | 4 +
.../fnexecution/data/GrpcDataServiceTest.java | 11 +-
.../data/RemoteInputDestinationTest.java | 3 +
.../fnexecution/environment/DockerCommandTest.java | 3 +-
.../environment/DockerEnvironmentFactoryTest.java | 3 +-
.../environment/ProcessEnvironmentFactoryTest.java | 4 +-
.../environment/ProcessEnvironmentTest.java | 3 +
.../environment/ProcessManagerTest.java | 3 +
.../environment/RemoteEnvironmentTest.java | 3 +
.../logging/GrpcLoggingServiceTest.java | 9 +-
.../StaticGrpcProvisionServiceTest.java | 14 +-
.../fnexecution/state/GrpcStateServiceTest.java | 7 +-
.../state/StateRequestHandlersTest.java | 6 +-
.../status/BeamWorkerStatusGrpcServiceTest.java | 10 +-
.../fnexecution/status/WorkerStatusClientTest.java | 6 +-
.../BatchSideInputHandlerFactoryTest.java | 2 +
.../translation/PipelineTranslatorUtilsTest.java | 3 +
.../fnexecution/wire/ByteStringCoderTest.java | 5 +-
.../runners/fnexecution/wire/CommonCoderTest.java | 7 +-
.../wire/LengthPrefixUnknownCodersTest.java | 3 +
runners/java-job-service/build.gradle | 3 +-
.../runners/jobsubmission/InMemoryJobService.java | 10 +-
.../beam/runners/jobsubmission/JobInvocation.java | 2 +-
.../beam/runners/jobsubmission/JobInvoker.java | 2 +-
.../beam/runners/jobsubmission/JobPreparation.java | 2 +-
.../jobsubmission/PortablePipelineJarCreator.java | 4 +-
.../jobsubmission/PortablePipelineJarUtils.java | 6 +-
.../jobsubmission/InMemoryJobServiceTest.java | 6 +-
.../runners/jobsubmission/JobInvocationTest.java | 2 +-
.../PortablePipelineJarCreatorTest.java | 2 +-
runners/portability/java/build.gradle | 10 +-
.../portability/JobServicePipelineResult.java | 2 +-
.../beam/runners/portability/PortableRunner.java | 4 +-
.../portability/testing/TestJobService.java | 2 +-
.../runners/portability/PortableRunnerTest.java | 8 +-
runners/samza/build.gradle | 3 +-
runners/samza/job-server/build.gradle | 2 +
.../apache/beam/runners/samza/SamzaJobInvoker.java | 2 +-
.../samza/runtime/SamzaStateRequestHandlers.java | 2 +-
.../translation/SamzaTestStreamTranslator.java | 2 +-
.../samza/translation/WindowAssignTranslator.java | 2 +-
.../runners/samza/util/PipelineJsonRenderer.java | 6 +-
.../SamzaMetricsBundleProgressHandlerTest.java | 2 +-
runners/samza/src/test/resources/ExpectedDag.json | 68 +-
.../container/spark_job_server_container.gradle | 5 +-
runners/spark/job-server/spark_job_server.gradle | 4 +
runners/spark/spark_runner.gradle | 9 +-
.../apache/beam/runners/spark/SparkJobInvoker.java | 2 +-
.../beam/runners/spark/SparkPipelineRunner.java | 2 +-
runners/twister2/build.gradle | 2 +-
.../functions/AssignWindowsFunction.java | 2 +-
.../translators/functions/DoFnFunction.java | 2 +-
.../functions/GroupByWindowFunction.java | 2 +-
scripts/ci/pr-bot/updateReviewers.ts | 1 +
sdks/go.mod | 99 +--
sdks/go.sum | 229 ++++---
sdks/go/container/Dockerfile | 2 +-
sdks/go/container/build.gradle | 5 +-
sdks/go/pkg/beam/core/core.go | 2 +-
.../pkg/beam/core/runtime/contextreg/contextreg.go | 3 +-
sdks/go/pkg/beam/core/runtime/exec/datasampler.go | 28 +-
.../pkg/beam/core/runtime/exec/datasampler_test.go | 24 +-
sdks/go/pkg/beam/core/runtime/exec/datasink.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/pcollection.go | 14 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 6 +-
sdks/go/pkg/beam/core/runtime/exec/window.go | 18 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 18 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 22 +-
sdks/go/pkg/beam/core/runtime/harness/init/init.go | 4 +-
sdks/go/pkg/beam/core/timers/timers.go | 7 +-
sdks/go/pkg/beam/runners/prism/internal/coders.go | 2 +-
.../pkg/beam/runners/prism/internal/engine/data.go | 192 ++++++
.../prism/internal/engine/elementmanager.go | 572 ++++++++++++++--
.../prism/internal/engine/elementmanager_test.go | 4 +-
.../runners/prism/internal/engine/engine_test.go | 171 +++++
.../beam/runners/prism/internal/engine/timers.go | 174 +++++
sdks/go/pkg/beam/runners/prism/internal/execute.go | 30 +
.../beam/runners/prism/internal/execute_test.go | 8 +-
.../pkg/beam/runners/prism/internal/handlepardo.go | 10 +-
.../beam/runners/prism/internal/jobservices/job.go | 15 +-
.../prism/internal/jobservices/management.go | 52 +-
.../prism/internal/jobservices/management_test.go | 34 +
.../runners/prism/internal/jobservices/metrics.go | 18 +-
.../prism/internal/jobservices/server_test.go | 61 ++
.../pkg/beam/runners/prism/internal/preprocess.go | 14 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 126 +++-
.../runners/prism/internal/unimplemented_test.go | 70 +-
.../pkg/beam/runners/prism/internal/urns/urns.go | 5 +
.../beam/runners/prism/internal/worker/bundle.go | 75 ++-
.../runners/prism/internal/worker/bundle_test.go | 10 +-
.../beam/runners/prism/internal/worker/worker.go | 108 ++-
.../runners/prism/internal/worker/worker_test.go | 23 +-
sdks/go/test/integration/integration.go | 33 +-
sdks/go/test/integration/primitives/state.go | 53 +-
sdks/go/test/integration/primitives/timers.go | 152 +++++
sdks/go/test/integration/primitives/timers_test.go | 35 +
sdks/java/container/agent/build.gradle | 4 +
sdks/java/container/common.gradle | 5 +-
.../container/license_scripts/dep_urls_java.yaml | 2 +-
sdks/java/core/build.gradle | 5 +-
sdks/java/core/jmh/build.gradle | 2 +-
.../jmh/util/ByteStringOutputStreamBenchmark.java | 4 +-
.../apache/beam/sdk/jmh/util/VarIntBenchmark.java | 303 +++++++++
.../main/java/org/apache/beam/sdk/Pipeline.java | 10 +-
.../org/apache/beam/sdk/fn/CancellableQueue.java | 0
.../java/org/apache/beam/sdk/fn/IdGenerator.java | 0
.../java/org/apache/beam/sdk/fn/IdGenerators.java | 0
.../org/apache/beam/sdk/fn/JvmInitializers.java | 59 ++
.../sdk/fn/channel/AddHarnessIdInterceptor.java | 40 ++
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 143 ++++
.../beam/sdk/fn/channel/SocketAddressFactory.java | 63 ++
.../apache/beam/sdk/fn/channel/package-info.java | 0
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 265 ++++++++
.../sdk/fn/data/BeamFnDataInboundObserver.java | 0
.../sdk/fn/data/BeamFnDataOutboundAggregator.java | 413 ++++++++++++
.../beam/sdk/fn/data/CloseableFnDataReceiver.java | 0
.../org/apache/beam/sdk/fn/data/DataEndpoint.java | 0
.../beam/sdk/fn/data/DecodingFnDataReceiver.java | 46 ++
.../apache/beam/sdk/fn/data/FnDataReceiver.java | 0
.../apache/beam/sdk/fn/data/LogicalEndpoint.java | 0
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 68 ++
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 73 ++
.../org/apache/beam/sdk/fn/data/TimerEndpoint.java | 0
.../org/apache/beam/sdk/fn/data/WeightedList.java | 0
.../org/apache/beam/sdk/fn/data/package-info.java | 0
.../java/org/apache/beam/sdk/fn/package-info.java | 0
.../org/apache/beam/sdk/fn/server/FnService.java | 35 +
.../server/GrpcContextHeaderAccessorProvider.java | 75 +++
.../apache/beam/sdk/fn/server/GrpcFnServer.java | 163 +++++
.../apache/beam/sdk/fn/server/HeaderAccessor.java | 0
.../beam/sdk/fn/server/InProcessServerFactory.java | 69 ++
.../apache/beam/sdk/fn/server/ServerFactory.java | 283 ++++++++
.../apache/beam/sdk/fn/server/package-info.java | 0
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 0
.../sdk/fn/splittabledofn/WatermarkEstimators.java | 0
.../beam/sdk/fn/splittabledofn/package-info.java | 0
.../apache/beam/sdk/fn/stream/AdvancingPhaser.java | 0
.../sdk/fn/stream/BufferingStreamObserver.java | 140 ++++
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 353 ++++++++++
.../beam/sdk/fn/stream/DirectStreamObserver.java | 133 ++++
.../stream/ForwardingClientResponseObserver.java | 77 +++
.../sdk/fn/stream/OutboundObserverFactory.java | 149 +++++
.../beam/sdk/fn/stream/PrefetchableIterable.java | 0
.../beam/sdk/fn/stream/PrefetchableIterables.java | 0
.../beam/sdk/fn/stream/PrefetchableIterator.java | 0
.../beam/sdk/fn/stream/PrefetchableIterators.java | 0
.../sdk/fn/stream/SynchronizedStreamObserver.java | 63 ++
.../apache/beam/sdk/fn/stream/package-info.java | 0
.../org/apache/beam/sdk/fn/test/TestExecutors.java | 0
.../org/apache/beam/sdk/fn/test/TestStreams.java | 157 +++++
.../org/apache/beam/sdk/fn/test/package-info.java | 0
.../sdk/fn/windowing/EncodedBoundedWindow.java | 94 +++
.../apache/beam/sdk/fn/windowing/package-info.java | 0
.../main/java/org/apache/beam/sdk/io/FileIO.java | 57 ++
.../src/main/java/org/apache/beam/sdk/io/Read.java | 131 ++--
.../main/java/org/apache/beam/sdk/io/TextIO.java | 38 ++
.../java/org/apache/beam/sdk/io/TextSource.java | 7 +-
.../java/org/apache/beam/sdk/io/WriteFiles.java | 309 +++++++--
.../java/org/apache/beam/sdk/io/range/ByteKey.java | 4 +-
.../beam/sdk/options/PipelineOptionsFactory.java | 19 +-
.../GenerateSequenceSchemaTransformProvider.java | 201 ++++++
.../apache/beam/sdk/providers/package-info.java | 23 +
.../apache/beam/sdk/schemas/SchemaTranslation.java | 2 +-
.../schemas/logicaltypes/SchemaLogicalType.java | 2 +-
.../providers/FlattenTransformProvider.java | 100 +++
.../providers/LoggingTransformProvider.java | 188 ++++++
.../org/apache/beam/sdk/transforms/Combine.java | 24 +-
.../org/apache/beam/sdk/transforms/Create.java | 191 ++++++
.../java/org/apache/beam/sdk/transforms/DoFn.java | 61 ++
.../beam/sdk/transforms/DoFnOutputReceivers.java | 27 +
.../org/apache/beam/sdk/transforms/DoFnTester.java | 22 +
.../org/apache/beam/sdk/transforms/Reshuffle.java | 95 ++-
.../transforms/errorhandling/BadRecordRouter.java | 43 ++
.../sdk/transforms/errorhandling/ErrorHandler.java | 4 +
.../beam/sdk/transforms/join/CoGbkResult.java | 3 +-
.../transforms/resourcehints/ResourceHints.java | 2 +-
.../sdk/transforms/windowing/GlobalWindow.java | 5 +
.../sdk/transforms/windowing/ReshuffleTrigger.java | 3 -
.../beam/sdk/transforms/windowing/Window.java | 12 +-
.../beam/sdk/util/ByteStringOutputStream.java | 4 +-
.../java/org/apache/beam/sdk/util/CoderUtils.java | 2 +-
.../org/apache/beam/sdk/util/HistogramData.java | 44 +-
.../main/java/org/apache/beam/sdk/util/VarInt.java | 62 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 21 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 5 +-
.../apache/beam/sdk/SdkHarnessEnvironmentTest.java | 3 +-
.../apache/beam/sdk/fn/CancellableQueueTest.java | 0
.../org/apache/beam/sdk/fn/IdGeneratorsTest.java | 0
.../apache/beam/sdk/fn/JvmInitializersTest.java | 89 +++
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 73 ++
.../sdk/fn/channel/SocketAddressFactoryTest.java | 55 ++
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 363 ++++++++++
.../sdk/fn/data/BeamFnDataInboundObserverTest.java | 251 +++++++
.../fn/data/BeamFnDataOutboundAggregatorTest.java | 0
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 69 ++
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 69 ++
.../fn/splittabledofn/RestrictionTrackersTest.java | 0
.../fn/splittabledofn/WatermarkEstimatorsTest.java | 0
.../beam/sdk/fn/stream/AdvancingPhaserTest.java | 0
.../sdk/fn/stream/BufferingStreamObserverTest.java | 0
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 243 +++++++
.../sdk/fn/stream/DirectStreamObserverTest.java | 0
.../ForwardingClientResponseObserverTest.java | 92 +++
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 76 +++
.../sdk/fn/stream/PrefetchableIterablesTest.java | 0
.../sdk/fn/stream/PrefetchableIteratorsTest.java | 0
.../apache/beam/sdk/fn/test/TestExecutorsTest.java | 0
.../apache/beam/sdk/fn/test/TestStreamsTest.java | 0
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 48 ++
.../org/apache/beam/sdk/io/TextIOReadTest.java | 113 +++-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 143 ++++
...enerateSequenceSchemaTransformProviderTest.java | 61 ++
.../beam/sdk/schemas/SchemaTranslationTest.java | 2 +-
.../beam/sdk/schemas/utils/JsonUtilsTest.java | 4 +-
.../apache/beam/sdk/transforms/CombineTest.java | 24 +
.../org/apache/beam/sdk/transforms/CreateTest.java | 81 +++
.../apache/beam/sdk/transforms/ReshuffleTest.java | 139 ++++
.../errorhandling/ErrorHandlingTestUtils.java | 48 ++
.../sdk/transforms/reflect/DoFnInvokersTest.java | 22 +
.../beam/sdk/util/ByteStringOutputStreamTest.java | 2 +-
.../org/apache/beam/sdk/util/CoderUtilsTest.java | 2 +-
.../apache/beam/sdk/util/HistogramDataTest.java | 57 +-
sdks/java/expansion-service/build.gradle | 4 +-
sdks/java/expansion-service/container/build.gradle | 5 +-
.../sdk/expansion/service/ExpansionServer.java | 4 +-
.../sdk/expansion/service/ExpansionService.java | 22 +-
.../ExpansionServiceSchemaTransformProvider.java | 5 +-
.../service/JavaClassLookupTransformProvider.java | 7 +-
.../service/WindowIntoTransformProvider.java | 127 ++++
...xpansionServiceSchemaTransformProviderTest.java | 12 +-
.../expansion/service/ExpansionServiceTest.java | 2 +-
.../beam/sdk/expansion/service/ExternalTest.java | 34 +-
.../JavaClassLookupTransformProviderTest.java | 4 +-
.../apache/beam/sdk/extensions/avro/io/AvroIO.java | 20 +
.../sdk/extensions/gcp/options/GcpOptions.java | 14 +-
.../sdk/extensions/gcp/util/GceMetadataUtil.java | 1 +
sdks/java/extensions/protobuf/build.gradle | 3 +
.../sdk/extensions/protobuf/ProtoByteUtils.java | 162 ++++-
.../extensions/protobuf/ProtoSchemaTranslator.java | 9 +-
.../extensions/protobuf/ProtoByteUtilsTest.java | 121 ++++
sdks/java/extensions/python/build.gradle | 4 +-
.../extensions/python/PythonExternalTransform.java | 2 +-
.../python/ConsecutivePythonTransformsTest.java | 65 ++
.../schemaio-expansion-service/build.gradle | 2 +-
.../ExternalSchemaIOTransformRegistrar.java | 2 +-
.../extensions/sorter/BufferedExternalSorter.java | 6 +-
sdks/java/extensions/sql/build.gradle | 11 +
sdks/java/fn-execution/build.gradle | 43 --
.../org/apache/beam/sdk/fn/JvmInitializers.java | 57 --
.../sdk/fn/channel/AddHarnessIdInterceptor.java | 40 --
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 143 ----
.../beam/sdk/fn/channel/SocketAddressFactory.java | 63 --
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 265 --------
.../sdk/fn/data/BeamFnDataOutboundAggregator.java | 413 ------------
.../beam/sdk/fn/data/DecodingFnDataReceiver.java | 46 --
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 68 --
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 73 --
.../org/apache/beam/sdk/fn/server/FnService.java | 35 -
.../server/GrpcContextHeaderAccessorProvider.java | 75 ---
.../apache/beam/sdk/fn/server/GrpcFnServer.java | 163 -----
.../beam/sdk/fn/server/InProcessServerFactory.java | 69 --
.../apache/beam/sdk/fn/server/ServerFactory.java | 283 --------
.../sdk/fn/stream/BufferingStreamObserver.java | 140 ----
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 353 ----------
.../beam/sdk/fn/stream/DirectStreamObserver.java | 133 ----
.../stream/ForwardingClientResponseObserver.java | 77 ---
.../sdk/fn/stream/OutboundObserverFactory.java | 149 -----
.../sdk/fn/stream/SynchronizedStreamObserver.java | 63 --
.../org/apache/beam/sdk/fn/test/TestStreams.java | 157 -----
.../sdk/fn/windowing/EncodedBoundedWindow.java | 94 ---
.../apache/beam/sdk/fn/JvmInitializersTest.java | 88 ---
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 73 --
.../sdk/fn/channel/SocketAddressFactoryTest.java | 55 --
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 363 ----------
.../sdk/fn/data/BeamFnDataInboundObserverTest.java | 247 -------
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 69 --
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 69 --
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 243 -------
.../ForwardingClientResponseObserverTest.java | 92 ---
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 76 ---
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 48 --
sdks/java/harness/build.gradle | 8 +-
sdks/java/harness/jmh/build.gradle | 5 +-
.../fn/harness/jmh/ProcessBundleBenchmark.java | 2 +-
.../control/ExecutionStateSamplerBenchmark.java | 2 +-
.../fn/harness/jmh/control/MetricsBenchmark.java | 2 +-
.../jmh/logging/BeamFnLoggingClientBenchmark.java | 6 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 2 +-
.../beam/fn/harness/BeamFnDataWriteRunner.java | 1 -
.../java/org/apache/beam/fn/harness/Caches.java | 4 +-
.../beam/fn/harness/ExternalWorkerService.java | 2 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 311 ++++++++-
.../java/org/apache/beam/fn/harness/FnHarness.java | 4 +-
.../fn/harness/control/BeamFnControlClient.java | 4 +-
.../fn/harness/control/BundleProgressReporter.java | 2 +-
.../fn/harness/control/ExecutionStateSampler.java | 2 +-
.../apache/beam/fn/harness/control/Metrics.java | 2 +-
.../fn/harness/control/ProcessBundleHandler.java | 4 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 2 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 16 +-
.../harness/state/BeamFnStateGrpcClientCache.java | 4 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 2 +-
.../fn/harness/state/FnApiTimerBundleTracker.java | 2 +-
.../beam/fn/harness/state/MultimapSideInput.java | 2 +-
.../beam/fn/harness/state/MultimapUserState.java | 2 +-
.../beam/fn/harness/state/StateBackedIterable.java | 2 +-
.../fn/harness/state/StateFetchingIterators.java | 2 +-
.../beam/fn/harness/status/BeamFnStatusClient.java | 4 +-
.../stream/HarnessStreamObserverFactories.java | 2 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 2 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 2 +-
.../beam/fn/harness/ExternalWorkerServiceTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 38 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 8 +-
.../harness/control/BeamFnControlClientTest.java | 8 +-
.../control/BundleProgressReporterTest.java | 2 +-
.../harness/control/ExecutionStateSamplerTest.java | 2 +-
.../beam/fn/harness/control/MetricsTest.java | 2 +-
.../harness/control/ProcessBundleHandlerTest.java | 4 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 14 +-
.../data/PCollectionConsumerRegistryTest.java | 14 +-
.../beam/fn/harness/debug/DataSamplerTest.java | 2 +-
.../beam/fn/harness/debug/OutputSamplerTest.java | 2 +-
.../harness/logging/BeamFnLoggingClientTest.java | 32 +-
.../beam/fn/harness/state/BagUserStateTest.java | 2 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 12 +-
.../fn/harness/state/FakeBeamFnStateClient.java | 2 +-
.../fn/harness/state/IterableSideInputTest.java | 2 +-
.../fn/harness/state/MultimapSideInputTest.java | 2 +-
.../fn/harness/state/MultimapUserStateTest.java | 2 +-
.../fn/harness/state/StateBackedIterableTest.java | 2 +-
.../harness/state/StateFetchingIteratorsTest.java | 2 +-
.../fn/harness/status/BeamFnStatusClientTest.java | 6 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 4 +-
.../apache/beam/sdk/io/aws2/schemas/AwsTypes.java | 1 +
.../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java | 4 +-
sdks/java/io/cassandra/build.gradle | 8 +
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 19 +-
.../beam/sdk/io/clickhouse/ClickHouseWriter.java | 16 +
.../apache/beam/sdk/io/clickhouse/TableSchema.java | 26 +-
.../clickhouse/src/main/javacc/ColumnTypeParser.jj | 47 ++
.../beam/sdk/io/clickhouse/ClickHouseIOTest.java | 78 +++
.../beam/sdk/io/clickhouse/TableSchemaTest.java | 51 ++
sdks/java/io/components/build.gradle | 2 +-
sdks/java/io/contextualtextio/build.gradle | 2 +-
.../contextualtextio/ContextualTextIOSource.java | 2 +-
.../csv/providers/CsvWriteTransformProvider.java | 16 +-
sdks/java/io/debezium/build.gradle | 2 +-
.../apache/beam/io/debezium/SourceRecordJson.java | 4 +-
sdks/java/io/file-schema-transform/build.gradle | 2 +-
sdks/java/io/google-cloud-platform/build.gradle | 2 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 38 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 5 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 63 +-
.../sdk/io/gcp/bigquery/BigQueryIOMetadata.java | 38 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 735 +++++++++++----------
.../sdk/io/gcp/bigquery/BigQuerySinkMetrics.java | 49 +-
.../sdk/io/gcp/bigquery/CreateTableHelpers.java | 10 +-
.../gcp/bigquery/DynamicDestinationsHelpers.java | 42 +-
.../beam/sdk/io/gcp/bigquery/RetryManager.java | 2 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 12 +-
.../bigquery/StorageApiWritesShardedRecords.java | 11 +-
.../io/gcp/bigquery/UpdateSchemaDestination.java | 9 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 574 +++++++++-------
...ueryStorageWriteApiSchemaTransformProvider.java | 60 +-
.../io/gcp/bigtable/BigtableConfigTranslator.java | 177 +++--
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 173 ++++-
.../beam/sdk/io/gcp/bigtable/BigtableService.java | 7 +
.../io/gcp/bigtable/BigtableServiceFactory.java | 16 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 115 +++-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 349 ++++++++--
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 4 +-
.../PubsubLiteReadSchemaTransformProvider.java | 84 ++-
.../PubsubLiteWriteSchemaTransformProvider.java | 51 +-
.../beam/sdk/io/gcp/spanner/ReadSpannerSchema.java | 8 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 54 +-
.../io/gcp/spanner/SpannerTransformRegistrar.java | 2 +-
...erChangestreamsReadSchemaTransformProvider.java | 2 +-
.../changestreams/model/ValueCaptureType.java | 1 +
.../sdk/io/gcp/testing/FakeDatasetService.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIOJsonIT.java | 87 ++-
.../io/gcp/bigquery/BigQueryIOTranslationTest.java | 16 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 176 ++++-
.../io/gcp/bigquery/BigQuerySinkMetricsTest.java | 72 +-
...StorageWriteApiSchemaTransformProviderTest.java | 54 +-
.../gcp/bigtable/BigtableConfigTranslatorTest.java | 169 ++++-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 11 +
.../beam/sdk/io/gcp/bigtable/BigtableReadIT.java | 93 ++-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 207 ++++++
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 94 ++-
.../beam/sdk/io/gcp/datastore/SplitQueryFnIT.java | 17 +-
.../apache/beam/sdk/io/gcp/datastore/V1ReadIT.java | 26 +-
.../beam/sdk/io/gcp/datastore/V1TestUtil.java | 59 +-
.../beam/sdk/io/gcp/datastore/V1WriteIT.java | 11 +-
.../beam/sdk/io/gcp/healthcare/DicomIOReadIT.java | 40 +-
.../beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java | 9 +-
.../sdk/io/gcp/pubsub/PubsubIOExternalTest.java | 2 +-
.../gcp/pubsublite/internal/PubsubLiteDlqTest.java | 89 +++
.../internal/PubsubLiteWriteDlqTest.java | 31 +
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 3 +-
.../gcp/spanner/SpannerTransformRegistrarTest.java | 2 +-
.../mapper/ChangeStreamRecordMapperTest.java | 100 +++
sdks/java/io/hadoop-format/build.gradle | 15 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 27 +-
.../beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java | 5 +-
sdks/java/io/kafka/build.gradle | 1 +
.../KafkaReadSchemaTransformConfiguration.java | 23 +
.../io/kafka/KafkaReadSchemaTransformProvider.java | 157 ++---
.../kafka/KafkaWriteSchemaTransformProvider.java | 94 ++-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 20 +-
.../KafkaReadSchemaTransformProviderTest.java | 64 ++
.../KafkaWriteSchemaTransformProviderTest.java | 22 +-
sdks/java/io/kafka/upgrade/build.gradle | 4 +-
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 421 ++++++------
.../io/kafka/upgrade/KafkaIOTranslationTest.java | 16 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 10 +-
.../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 2 +-
.../java/org/apache/beam/sdk/io/neo4j/Neo4jIO.java | 28 +-
sdks/java/io/parquet/build.gradle | 2 +-
.../apache/beam/io/requestresponse/ApiIOError.java | 7 +-
.../org/apache/beam/io/requestresponse/Cache.java | 190 ++++--
.../org/apache/beam/io/requestresponse/Call.java | 377 ++++++++---
.../beam/io/requestresponse/CallShouldBackoff.java | 2 +-
...llShouldBackoffBasedOnRejectionProbability.java | 2 +-
.../DefaultSerializableBackoffSupplier.java | 35 +
.../apache/beam/io/requestresponse/Monitoring.java | 362 ++++++++++
.../org/apache/beam/io/requestresponse/Quota.java | 7 +-
.../beam/io/requestresponse/RedisClient.java | 4 +-
.../apache/beam/io/requestresponse/Repeater.java | 85 ++-
.../beam/io/requestresponse/RequestResponseIO.java | 552 ++++++++++++++--
.../org/apache/beam/io/requestresponse/Result.java | 90 +++
.../io/requestresponse/SerializableSupplier.java | 28 +
.../ThrottleWithExternalResource.java | 12 +-
.../UserCodeExecutionException.java | 8 +
.../io/requestresponse/UserCodeQuotaException.java | 6 +
.../UserCodeRemoteSystemException.java | 9 +
.../requestresponse/UserCodeTimeoutException.java | 9 +
.../requestresponse/WindowedCallShouldBackoff.java | 76 +++
.../apache/beam/io/requestresponse/CacheIT.java | 16 +-
...ouldBackoffBasedOnRejectionProbabilityTest.java | 2 +-
.../apache/beam/io/requestresponse/CallTest.java | 1 -
.../EchoGRPCCallerWithSetupTeardownIT.java | 2 +-
.../beam/io/requestresponse/EchoRequestCoder.java | 1 +
.../beam/io/requestresponse/EchoResponseCoder.java | 43 ++
.../beam/io/requestresponse/RedisClientIT.java | 2 +-
.../beam/io/requestresponse/RepeaterTest.java | 2 +-
.../io/requestresponse/RequestResponseIOIT.java | 95 +++
.../io/requestresponse/RequestResponseIOTest.java | 508 ++++++++++++++
.../ThrottleWithExternalResourceIT.java | 6 +-
.../WindowedCallShouldBackoffTest.java | 56 ++
.../singlestore/SingleStoreDefaultRowMapper.java | 5 +-
.../testing/expansion/TestExpansionService.java | 5 +-
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 14 +-
sdks/java/transform-service/app/build.gradle | 43 ++
sdks/java/transform-service/build.gradle | 5 +-
.../controller-container/build.gradle | 5 +-
sdks/java/transform-service/launcher/build.gradle | 28 +-
.../beam/sdk/transformservice/ArtifactService.java | 6 +-
.../beam/sdk/transformservice/Controller.java | 4 +-
.../sdk/transformservice/ExpansionService.java | 4 +-
.../sdk/transformservice/ArtifactServiceTest.java | 2 +-
.../sdk/transformservice/ExpansionServiceTest.java | 2 +-
sdks/python/apache_beam/dataframe/io.py | 3 +-
.../examples/dataframe/flight_delays.py | 2 +-
.../anomaly_detection_pipeline/setup.py | 2 +-
.../large_language_modeling/requirements.txt | 2 +-
.../inference/runinference_metrics/setup.py | 2 +-
.../kfp/components/preprocessing/requirements.txt | 2 +-
.../kfp/components/train/requirements.txt | 2 +-
sdks/python/apache_beam/io/avroio.py | 16 +-
.../io/external/xlang_bigqueryio_it_test.py | 77 ++-
.../io/external/xlang_kafkaio_it_test.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 304 +++++----
.../apache_beam/io/gcp/bigquery_file_loads.py | 25 +
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 65 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 201 +++++-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 2 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 62 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 85 ++-
.../apache_beam/io/gcp/healthcare/dicomio.py | 31 +-
.../apache_beam/io/gcp/healthcare/dicomio_test.py | 39 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 4 +-
sdks/python/apache_beam/io/parquetio.py | 7 +-
sdks/python/apache_beam/io/requestresponse.py | 413 ++++++++++++
.../apache_beam/io/requestresponse_it_test.py | 189 ++++++
sdks/python/apache_beam/io/requestresponse_test.py | 156 +++++
sdks/python/apache_beam/io/requestresponseio.py | 218 ------
.../apache_beam/io/requestresponseio_it_test.py | 184 ------
.../apache_beam/io/requestresponseio_test.py | 88 ---
sdks/python/apache_beam/ml/transforms/base.py | 94 ++-
sdks/python/apache_beam/ml/transforms/base_test.py | 142 +++-
.../ml/transforms/embeddings/huggingface.py | 14 +-
.../ml/transforms/embeddings/vertex_ai.py | 13 +-
sdks/python/apache_beam/ml/transforms/handlers.py | 9 +-
.../apache_beam/ml/transforms/handlers_test.py | 20 +-
.../python/apache_beam/options/pipeline_options.py | 14 +-
.../apache_beam/options/pipeline_options_test.py | 33 +
sdks/python/apache_beam/portability/common_urns.py | 1 +
.../runners/dataflow/dataflow_job_service.py | 9 +-
.../runners/dataflow/dataflow_job_service_test.py | 23 +
.../runners/dataflow/dataflow_runner.py | 3 +-
.../runners/dataflow/internal/apiclient.py | 6 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/runners/interactive/utils.py | 8 +-
.../runners/portability/fn_api_runner/execution.py | 11 +
.../portability/fn_api_runner/translations.py | 2 +-
.../portability/fn_api_runner/worker_handlers.py | 1 +
.../runners/portability/local_job_service_main.py | 4 +-
.../runners/portability/sdk_container_builder.py | 6 +-
sdks/python/apache_beam/runners/render.py | 3 +-
sdks/python/apache_beam/runners/trivial_runner.py | 2 +-
.../apache_beam/runners/worker/sdk_worker.py | 22 +-
.../apache_beam/testing/analyzers/perf_analysis.py | 6 +-
sdks/python/apache_beam/transforms/enrichment.py | 137 ++++
.../transforms/enrichment_handlers}/__init__.py | 0
.../transforms/enrichment_handlers/bigtable.py | 163 +++++
.../enrichment_handlers/bigtable_it_test.py | 341 ++++++++++
.../apache_beam/transforms/enrichment_it_test.py | 162 +++++
.../apache_beam/transforms/enrichment_test.py | 41 ++
sdks/python/apache_beam/transforms/external.py | 1 +
.../transforms/external_transform_provider.py | 277 ++++++++
.../transforms/external_transform_provider_test.py | 140 ++++
.../transforms/fully_qualified_named_transform.py | 13 +-
sdks/python/apache_beam/typehints/schemas.py | 4 +-
sdks/python/apache_beam/utils/retry.py | 33 +-
.../utils/transform_service_launcher.py | 2 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/README.md | 140 ++--
sdks/python/apache_beam/yaml/generate_yaml_docs.py | 52 +-
sdks/python/apache_beam/yaml/inline_python.md | 202 ++++++
sdks/python/apache_beam/yaml/json_utils.py | 49 ++
sdks/python/apache_beam/yaml/main.py | 15 +-
sdks/python/apache_beam/yaml/readme_test.py | 32 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 22 +-
.../apache_beam/yaml/standard_providers.yaml | 35 +-
sdks/python/apache_beam/yaml/yaml_io.py | 24 +
sdks/python/apache_beam/yaml/yaml_mapping.md | 8 +-
sdks/python/apache_beam/yaml/yaml_mapping.py | 65 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 304 +++++++--
.../apache_beam/yaml/yaml_provider_unit_test.py | 68 ++
sdks/python/apache_beam/yaml/yaml_transform.py | 61 +-
.../python/apache_beam/yaml/yaml_transform_test.py | 44 +-
.../apache_beam/yaml/yaml_transform_unit_test.py | 8 +-
sdks/python/build.gradle | 37 +-
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/boot.go | 9 +-
sdks/python/container/common.gradle | 5 +-
.../container/py310/base_image_requirements.txt | 114 ++--
.../container/py311/base_image_requirements.txt | 114 ++--
.../container/py38/base_image_requirements.txt | 110 +--
.../container/py39/base_image_requirements.txt | 116 ++--
sdks/python/expansion-service-container/Dockerfile | 2 +-
.../expansion-service-container/build.gradle | 5 +-
sdks/python/pyproject.toml | 4 +-
sdks/python/pytest.ini | 2 +-
sdks/python/setup.py | 13 +-
sdks/python/test-suites/dataflow/build.gradle | 4 +-
sdks/python/test-suites/dataflow/common.gradle | 2 +-
sdks/python/test-suites/direct/build.gradle | 8 +-
sdks/python/test-suites/direct/common.gradle | 2 +-
sdks/python/test-suites/gradle.properties | 4 +-
sdks/python/test-suites/tox/py38/build.gradle | 4 +
sdks/python/test-suites/xlang/build.gradle | 9 +-
sdks/python/tox.ini | 10 +-
sdks/typescript/build.sh | 3 -
sdks/typescript/container/Dockerfile | 2 +-
sdks/typescript/container/build.gradle | 5 +-
sdks/typescript/package-lock.json | 25 +-
sdks/typescript/package.json | 4 +-
sdks/typescript/src/apache_beam/coders/coders.ts | 6 +-
.../typescript/src/apache_beam/coders/js_coders.ts | 7 +-
.../src/apache_beam/coders/required_coders.ts | 34 +-
.../typescript/src/apache_beam/coders/row_coder.ts | 24 +-
.../src/apache_beam/coders/standard_coders.ts | 7 +-
.../src/apache_beam/examples/wordcount.ts | 2 +-
.../src/apache_beam/examples/wordcount_sql.ts | 4 +-
.../src/apache_beam/examples/wordcount_textio.ts | 2 +-
.../src/apache_beam/internal/environments.ts | 12 +-
.../src/apache_beam/internal/pipeline.ts | 58 +-
.../src/apache_beam/internal/serialize.ts | 13 +-
sdks/typescript/src/apache_beam/io/avroio.ts | 12 +-
sdks/typescript/src/apache_beam/io/bigqueryio.ts | 8 +-
sdks/typescript/src/apache_beam/io/index.ts | 3 +-
sdks/typescript/src/apache_beam/io/kafka.ts | 20 +-
sdks/typescript/src/apache_beam/io/parquetio.ts | 12 +-
sdks/typescript/src/apache_beam/io/pubsub.ts | 33 +-
sdks/typescript/src/apache_beam/io/pubsublite.ts | 12 +-
sdks/typescript/src/apache_beam/io/schemaio.ts | 10 +-
sdks/typescript/src/apache_beam/io/textio.ts | 32 +-
sdks/typescript/src/apache_beam/pvalue.ts | 57 +-
.../src/apache_beam/runners/artifacts.ts | 18 +-
.../typescript/src/apache_beam/runners/dataflow.ts | 19 +-
.../src/apache_beam/runners/direct_runner.ts | 48 +-
sdks/typescript/src/apache_beam/runners/flink.ts | 8 +-
.../apache_beam/runners/portable_runner/runner.ts | 36 +-
sdks/typescript/src/apache_beam/runners/runner.ts | 16 +-
.../src/apache_beam/runners/universal.ts | 6 +-
sdks/typescript/src/apache_beam/testing/assert.ts | 25 +-
.../apache_beam/testing/multi_pipeline_runner.ts | 22 +-
.../src/apache_beam/transforms/combiners.ts | 5 +-
.../src/apache_beam/transforms/create.ts | 2 +-
.../src/apache_beam/transforms/external.ts | 62 +-
.../src/apache_beam/transforms/flatten.ts | 4 +-
.../apache_beam/transforms/group_and_combine.ts | 59 +-
.../typescript/src/apache_beam/transforms/index.ts | 6 +-
.../src/apache_beam/transforms/internal.ts | 38 +-
.../typescript/src/apache_beam/transforms/pardo.ts | 57 +-
.../src/apache_beam/transforms/python.ts | 8 +-
sdks/typescript/src/apache_beam/transforms/sql.ts | 16 +-
.../src/apache_beam/transforms/transform.ts | 18 +-
.../typescript/src/apache_beam/transforms/utils.ts | 4 +-
.../src/apache_beam/transforms/window.ts | 22 +-
.../src/apache_beam/transforms/windowings.ts | 25 +-
.../src/apache_beam/utils/packageJson.ts | 27 +
sdks/typescript/src/apache_beam/utils/service.ts | 51 +-
sdks/typescript/src/apache_beam/utils/utils.ts | 2 +-
sdks/typescript/src/apache_beam/values.ts | 10 +-
sdks/typescript/src/apache_beam/worker/data.ts | 6 +-
.../apache_beam/worker/external_worker_service.ts | 10 +-
sdks/typescript/src/apache_beam/worker/logging.ts | 29 +-
sdks/typescript/src/apache_beam/worker/metrics.ts | 18 +-
.../typescript/src/apache_beam/worker/operators.ts | 121 ++--
.../src/apache_beam/worker/pardo_context.ts | 26 +-
sdks/typescript/src/apache_beam/worker/state.ts | 18 +-
sdks/typescript/src/apache_beam/worker/worker.ts | 50 +-
.../src/apache_beam/worker/worker_main.ts | 8 +-
settings.gradle.kts | 14 +-
vendor/README.md | 10 +-
vendor/grpc-1_54_0/build.gradle | 37 --
vendor/grpc-1_60_1/build.gradle | 38 ++
website/Dockerfile | 9 +-
website/www/site/config.toml | 2 +-
.../blog/apache-beam-flink-and-kubernetes-part2.md | 158 +++++
.../blog/apache-beam-flink-and-kubernetes-part3.md | 209 ++++++
website/www/site/content/en/blog/beam-2.53.0.md | 185 ++++++
.../content/en/blog/scaling-streaming-workload.md | 289 ++++++++
website/www/site/content/en/contribute/_index.md | 2 +
.../www/site/content/en/contribute/runner-guide.md | 12 +-
.../en/documentation/dsls/sql/zetasql/overview.md | 4 +-
.../documentation/io/built-in/google-bigquery.md | 2 +
.../site/content/en/documentation/io/connectors.md | 2 +-
.../content/en/documentation/ml/preprocess-data.md | 75 +--
.../content/en/documentation/programming-guide.md | 8 +-
.../en/documentation/runtime/environments.md | 12 +-
website/www/site/content/en/get-started/_index.md | 8 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
.../en/get-started/resources/learning-resources.md | 5 +
.../adaptive-timeout-kafka.png | Bin 0 -> 880831 bytes
.../checkpoint_monitoring-history-subtasks.png | Bin 0 -> 170772 bytes
.../flink-backlog-metrics.png | Bin 0 -> 580860 bytes
.../flink-partition-assignment.png | Bin 0 -> 66575 bytes
.../kafkaio-wait-reader.png | Bin 0 -> 399857 bytes
.../adaptive_scheduler_rescale.png | Bin 0 -> 859268 bytes
.../auto-tuned-worker.png | Bin 0 -> 114222 bytes
.../backlog_growth.png | Bin 0 -> 6568 bytes
.../backlog_time.png | Bin 0 -> 6127 bytes
.../cpurate_desired.png | Bin 0 -> 10111 bytes
.../flink-operator-chaining.png | Bin 0 -> 529398 bytes
.../operator-backlog.png | Bin 0 -> 350363 bytes
.../resource-allocation.png | Bin 0 -> 131782 bytes
.../worker_extra.png | Bin 0 -> 11497 bytes
.../worker_require.png | Bin 0 -> 11975 bytes
.../worker_scaleup.png | Bin 0 -> 7471 bytes
.../blog/scaling-streaming-workload/0-intro.png | Bin 0 -> 678815 bytes
.../blog/scaling-streaming-workload/0-pipeline.png | Bin 0 -> 41826 bytes
.../scaling-streaming-workload/1-default-ps.png | Bin 0 -> 80386 bytes
.../1-default-throughput.png | Bin 0 -> 177003 bytes
.../blog/scaling-streaming-workload/2-skeys-ps.png | Bin 0 -> 95852 bytes
.../2-skeys-throughput.png | Bin 0 -> 199363 bytes
.../3-autoshard-autoscale.png | Bin 0 -> 94179 bytes
.../3-autoshard-parallelism.png | Bin 0 -> 88744 bytes
.../3-autoshard-throughput-tr.png | Bin 0 -> 149305 bytes
.../3-autoshard-tr-input.png | Bin 0 -> 74404 bytes
.../3-autoshard-tr-output.png | Bin 0 -> 100177 bytes
.../3-autoshard-tr-overhead.png | Bin 0 -> 82261 bytes
.../4-format-transformation.png | Bin 0 -> 27127 bytes
.../4-ingestion-scale.png | Bin 0 -> 43609 bytes
.../scaling-streaming-workload/4-throughput.png | Bin 0 -> 27300 bytes
.../5-format-trasformation.png | Bin 0 -> 20761 bytes
.../5-ingestion-scale.png | Bin 0 -> 38156 bytes
.../scaling-streaming-workload/5-input-size.png | Bin 0 -> 31216 bytes
.../scaling-streaming-workload/5-output-size.png | Bin 0 -> 46655 bytes
.../6-format-transformation.png | Bin 0 -> 21764 bytes
.../6-ingestion-scale.png | Bin 0 -> 37699 bytes
.../scaling-streaming-workload/6-latencies.png | Bin 0 -> 38562 bytes
.../scaling-streaming-workload/7-current-input.png | Bin 0 -> 35557 bytes
.../7-current-shuffle-total.png | Bin 0 -> 5101 bytes
.../blog/scaling-streaming-workload/7-latency.png | Bin 0 -> 32395 bytes
.../7-previous-data-input.png | Bin 0 -> 34791 bytes
.../7-previous-shuffle-total.png | Bin 0 -> 5388 bytes
.../scaling-streaming-workload/7-resources.png | Bin 0 -> 34148 bytes
1204 files changed, 31193 insertions(+), 11702 deletions(-)
copy
learning/tour-of-beam/learning-content/io/text-io/text-io-local-write/go-example/myfile.txt
=> .github/trigger_files/beam_PostCommit_Java_DataflowV2.json (100%)
copy
learning/tour-of-beam/learning-content/io/text-io/text-io-local-write/go-example/myfile.txt
=> .github/trigger_files/beam_PostCommit_Python.json (100%)
create mode 100644
.github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Direct.json
delete mode 100644 .test-infra/jenkins/job_00_seed.groovy
delete mode 100644 .test-infra/jenkins/job_Inventory.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Direct.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_GoUsingJava_Dataflow.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_PythonUsingJava_Dataflow.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_DataflowV2.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_InfluxDBIO_IT.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Flink_Java11.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Python.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_IO_Dataflow.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Python_Examples_Direct.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Python_Examples_Flink.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Flink.groovy
delete mode 100644
.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy
delete mode 100644 .test-infra/jenkins/job_PostRelease_NightlySnapshot.groovy
delete mode 100644 .test-infra/jenkins/job_seed_standalone.groovy
delete mode 100644
.test-infra/terraform/google-cloud-platform/google-kubernetes-engine/apache-beam-testing.tfvars
create mode 100644
.test-infra/terraform/google-cloud-platform/google-kubernetes-engine/us-central1.apache-beam-testing.tfvars
create mode 100644
.test-infra/terraform/google-cloud-platform/google-kubernetes-engine/us-west1.apache-beam-testing.tfvars
delete mode 100644
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_54_0.groovy
create mode 100644
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_60_1.groovy
create mode 100644
examples/notebooks/beam-ml/data_preprocessing/compute_and_apply_vocab.ipynb
create mode 100644
examples/notebooks/beam-ml/data_preprocessing/huggingface_text_embeddings.ipynb
create mode 100644
examples/notebooks/beam-ml/data_preprocessing/scale_data.ipynb
create mode 100644
examples/notebooks/beam-ml/data_preprocessing/vertex_ai_text_embeddings.ipynb
create mode 100644 learning/prompts/README.md
create mode 100644 learning/prompts/code-explanation/01_io_kafka.md
create mode 100644 learning/prompts/code-explanation/02_io_pubsub.md
create mode 100644 learning/prompts/code-explanation/03_io_bigquery.md
create mode 100644 learning/prompts/code-explanation/04_io_bigtable.md
create mode 100644 learning/prompts/code-explanation/05_io_spanner.md
create mode 100644 learning/prompts/code-explanation/06_io_tfrecord.md
create mode 100644 learning/prompts/code-explanation/07_io_json.md
create mode 100644 learning/prompts/code-explanation/08_io_csv.md
create mode 100644 learning/prompts/code-explanation/09_io_avro.md
create mode 100644 learning/prompts/code-explanation/10_io_parquet.md
create mode 100644 learning/prompts/code-explanation/11_io_jdbc.md
create mode 100644 learning/prompts/code-generation/01_io_kafka.md
create mode 100644 learning/prompts/code-generation/02_io_pubsub.md
create mode 100644 learning/prompts/code-generation/03_io_bigquery.md
create mode 100644 learning/prompts/code-generation/04_io_bigtable.md
create mode 100644 learning/prompts/code-generation/05_io_spanner.md
create mode 100644 learning/prompts/code-generation/06_io_tfrecord.md
create mode 100644 learning/prompts/code-generation/07_io_json.md
create mode 100644 learning/prompts/code-generation/08_io_csv.md
create mode 100644 learning/prompts/code-generation/09_io_avro.md
create mode 100644 learning/prompts/code-generation/10_io_parquet.md
create mode 100644 learning/prompts/code-generation/11_io_jdbc.md
create mode 100644
learning/prompts/documentation-lookup/01_basic_learning_apache_beam.md
create mode 100644 learning/prompts/documentation-lookup/02_basic_pipelines.md
create mode 100644
learning/prompts/documentation-lookup/03_basic_configuring_pipelines.md
create mode 100644
learning/prompts/documentation-lookup/04_basic_pcollections.md
create mode 100644
learning/prompts/documentation-lookup/05_basic_ptransforms.md
create mode 100644 learning/prompts/documentation-lookup/06_basic_schema.md
create mode 100644 learning/prompts/documentation-lookup/07_basic_runners.md
create mode 100644 learning/prompts/documentation-lookup/08_basic_windowing.md
create mode 100644 learning/prompts/documentation-lookup/09_basic_triggers.md
create mode 100644 learning/prompts/documentation-lookup/10_basic_metrics.md
create mode 100644 learning/prompts/documentation-lookup/11_basic_state.md
create mode 100644 learning/prompts/documentation-lookup/12_basic_timers.md
create mode 100644
learning/prompts/documentation-lookup/13_advanced_splittable_dofn.md
create mode 100644
learning/prompts/documentation-lookup/14_advanced_pipeline_patterns.md
create mode 100644 learning/prompts/documentation-lookup/15_advanced_xlang.md
create mode 100644
learning/prompts/documentation-lookup/16_advanced_pipeline_lifecycle.md
create mode 100644 learning/prompts/documentation-lookup/17_advanced_ai_ml.md
create mode 100644 learning/prompts/documentation-lookup/18_io_kafka.md
create mode 100644 learning/prompts/documentation-lookup/19_io_pubsub.md
create mode 100644 learning/prompts/documentation-lookup/20_io_biguery.md
create mode 100644 learning/prompts/documentation-lookup/21_io_bigtable.md
create mode 100644 learning/prompts/documentation-lookup/22_io_spanner.md
create mode 100644 learning/prompts/documentation-lookup/23_io_gcs.md
create mode 100644 learning/prompts/documentation-lookup/24_io_json.md
create mode 100644 learning/prompts/documentation-lookup/25_io_csv.md
create mode 100644 learning/prompts/documentation-lookup/26_io_avro.md
create mode 100644 learning/prompts/documentation-lookup/27_io_parquet.md
create mode 100644 learning/prompts/documentation-lookup/28_io_jdbc.md
create mode 100644
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnknownCoderWrapper.java
create mode 100644
runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/EmptyUnboundedSource.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ActiveMessageMetadata.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateSampler.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/MetricsToPerStepNamespaceMetricsConverter.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/RemoveSafeDeltaCounterCell.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkItemCancelledException.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateSamplerTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/MetricsToPerStepNamespaceMetricsConverterTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/RemoveSafeDeltaCounterCellTest.java
create mode 100644
sdks/go/pkg/beam/runners/prism/internal/engine/engine_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/engine/timers.go
create mode 100644 sdks/go/test/integration/primitives/timers.go
create mode 100644 sdks/go/test/integration/primitives/timers_test.go
create mode 100644
sdks/java/core/jmh/src/main/java/org/apache/beam/sdk/jmh/util/VarIntBenchmark.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/CancellableQueue.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/IdGenerator.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/IdGenerators.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/channel/AddHarnessIdInterceptor.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/channel/package-info.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java
(100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/CloseableFnDataReceiver.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/DataEndpoint.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/DecodingFnDataReceiver.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/FnDataReceiver.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/TimerEndpoint.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/WeightedList.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/data/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/package-info.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/server/FnService.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/server/GrpcContextHeaderAccessorProvider.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/server/GrpcFnServer.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/HeaderAccessor.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/server/InProcessServerFactory.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/server/ServerFactory.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/server/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/splittabledofn/WatermarkEstimators.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/splittabledofn/package-info.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/AdvancingPhaser.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterables.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterator.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterators.java
(100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/stream/package-info.java (100%)
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/test/TestExecutors.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/test/package-info.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java
rename sdks/java/{fn-execution =>
core}/src/main/java/org/apache/beam/sdk/fn/windowing/package-info.java (100%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/providers/GenerateSequenceSchemaTransformProvider.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/providers/package-info.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/providers/FlattenTransformProvider.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/providers/LoggingTransformProvider.java
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/CancellableQueueTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/IdGeneratorsTest.java (100%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserverTest.java
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregatorTest.java
(100%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackersTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/splittabledofn/WatermarkEstimatorsTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/AdvancingPhaserTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserverTest.java
(100%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/DirectStreamObserverTest.java
(100%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/PrefetchableIterablesTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/stream/PrefetchableIteratorsTest.java
(100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/test/TestExecutorsTest.java (100%)
rename sdks/java/{fn-execution =>
core}/src/test/java/org/apache/beam/sdk/fn/test/TestStreamsTest.java (100%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/providers/GenerateSequenceSchemaTransformProviderTest.java
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/errorhandling/ErrorHandlingTestUtils.java
create mode 100644
sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/WindowIntoTransformProvider.java
create mode 100644
sdks/java/extensions/python/src/test/java/org/apache/beam/sdk/extensions/python/ConsecutivePythonTransformsTest.java
delete mode 100644 sdks/java/fn-execution/build.gradle
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/AddHarnessIdInterceptor.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactory.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/channel/SocketAddressFactory.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/DecodingFnDataReceiver.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/server/FnService.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/server/GrpcContextHeaderAccessorProvider.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/server/GrpcFnServer.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/server/InProcessServerFactory.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/server/ServerFactory.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/BufferingStreamObserver.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DataStreams.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserver.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactory.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/SynchronizedStreamObserver.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/test/TestStreams.java
delete mode 100644
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/ManagedChannelFactoryTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/channel/SocketAddressFactoryTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/BeamFnDataInboundObserverTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/DataStreamsTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/ForwardingClientResponseObserverTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/stream/OutboundObserverFactoryTest.java
delete mode 100644
sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/windowing/EncodedBoundedWindowTest.java
create mode 100644
sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/DefaultSerializableBackoffSupplier.java
create mode 100644
sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/Monitoring.java
create mode 100644
sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/Result.java
create mode 100644
sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/SerializableSupplier.java
create mode 100644
sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/WindowedCallShouldBackoff.java
create mode 100644
sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/EchoResponseCoder.java
create mode 100644
sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/RequestResponseIOIT.java
create mode 100644
sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/RequestResponseIOTest.java
create mode 100644
sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/WindowedCallShouldBackoffTest.java
create mode 100644 sdks/java/transform-service/app/build.gradle
create mode 100644 sdks/python/apache_beam/io/requestresponse.py
create mode 100644 sdks/python/apache_beam/io/requestresponse_it_test.py
create mode 100644 sdks/python/apache_beam/io/requestresponse_test.py
delete mode 100644 sdks/python/apache_beam/io/requestresponseio.py
delete mode 100644 sdks/python/apache_beam/io/requestresponseio_it_test.py
delete mode 100644 sdks/python/apache_beam/io/requestresponseio_test.py
create mode 100644 sdks/python/apache_beam/transforms/enrichment.py
copy {examples/notebooks/notebook_test_scripts =>
sdks/python/apache_beam/transforms/enrichment_handlers}/__init__.py (100%)
create mode 100644
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable.py
create mode 100644
sdks/python/apache_beam/transforms/enrichment_handlers/bigtable_it_test.py
create mode 100644 sdks/python/apache_beam/transforms/enrichment_it_test.py
create mode 100644 sdks/python/apache_beam/transforms/enrichment_test.py
create mode 100644
sdks/python/apache_beam/transforms/external_transform_provider.py
create mode 100644
sdks/python/apache_beam/transforms/external_transform_provider_test.py
create mode 100644 sdks/python/apache_beam/yaml/inline_python.md
create mode 100644 sdks/python/apache_beam/yaml/yaml_provider_unit_test.py
create mode 100644 sdks/typescript/src/apache_beam/utils/packageJson.ts
delete mode 100644 vendor/grpc-1_54_0/build.gradle
create mode 100644 vendor/grpc-1_60_1/build.gradle
create mode 100644
website/www/site/content/en/blog/apache-beam-flink-and-kubernetes-part2.md
create mode 100644
website/www/site/content/en/blog/apache-beam-flink-and-kubernetes-part3.md
create mode 100644 website/www/site/content/en/blog/beam-2.53.0.md
create mode 100644
website/www/site/content/en/blog/scaling-streaming-workload.md
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part2/adaptive-timeout-kafka.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part2/checkpoint_monitoring-history-subtasks.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part2/flink-backlog-metrics.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part2/flink-partition-assignment.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part2/kafkaio-wait-reader.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/adaptive_scheduler_rescale.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/auto-tuned-worker.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/backlog_growth.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/backlog_time.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/cpurate_desired.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/flink-operator-chaining.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/operator-backlog.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/resource-allocation.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/worker_extra.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/worker_require.png
create mode 100644
website/www/site/static/images/blog/apache-beam-flink-and-kubernetes-part3/worker_scaleup.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/0-intro.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/0-pipeline.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/1-default-ps.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/1-default-throughput.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/2-skeys-ps.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/2-skeys-throughput.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/3-autoshard-autoscale.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/3-autoshard-parallelism.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/3-autoshard-throughput-tr.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/3-autoshard-tr-input.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/3-autoshard-tr-output.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/3-autoshard-tr-overhead.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/4-format-transformation.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/4-ingestion-scale.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/4-throughput.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/5-format-trasformation.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/5-ingestion-scale.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/5-input-size.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/5-output-size.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/6-format-transformation.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/6-ingestion-scale.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/6-latencies.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/7-current-input.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/7-current-shuffle-total.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/7-latency.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/7-previous-data-input.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/7-previous-shuffle-total.png
create mode 100644
website/www/site/static/images/blog/scaling-streaming-workload/7-resources.png
