This is an automated email from the ASF dual-hosted git repository.
emilyye pushed a change to branch release-2.36.0
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard d5d0bc3 Merge pull request #16462 from ihji/cherry-pick-13599
omit 3fb0ff6 [BEAM-13599] Fix overflow error in Python Datastore
RampupThrottlingFn
discard bd46e7b [release-2.36.0][BEAM-13541] More intelligent caching of
CoGBK values. (#16354, #16407) (#16421)
omit d5e75c5 [release-2.36.0][BEAM-13575] Sickbay test that is flaky to
restore precommit test signal (#16410) (#16435)
omit 99e455e [release-2.36.0][BEAM-13459] Fix sdk_container_builder too
many values to unpack error (#16422)
omit f62cabf Set Dataflow container to release version.
add 28ea3f8 Moving to 2.37.0-SNAPSHOT on master branch.
add 12fbe03 Release website update for Beam 2.35.0 (#16115)
add 0af3689 [BEAM-13430] Re-enable checkerframework for the project
excluding the two classes it started to crash on. (#16389)
add e2189b3 [adhoc] Use predefined min/max combiners in SQS reader to
gather stats
add d8784ac Merge pull request #16390: [adhoc] Use predefined min/max
combiners in SQS reader to gather stats
add 52715c0 [BEAM-13009] Fix tests to ensure absence of duplicates only
per request to avoid issues when running with low targetParellelism
add fe456b7 Merge pull request #16367: [BEAM-13009] Fix tests to ensure
absence of duplicates only per request
add df26931 [BEAM-13430] Fix class not found error for example
integration tests and dataflow postcommit (#16388)
add eaacf70 [BEAM-13430] Re-enable dependency analysis for modules.
(#16395)
add 3a74728 Fix class not found in perf integration tests.
add 598f444 Update
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
add 2a9e8b6 Merge pull request #16398 from y1chi/perf-it
add e55f54e [BEAM-13484][Playground] Improve Terraform Scripts For Deploy
Infrastructure (#16378)
add 3ab26ce [BEAM-13541] More intelligent caching of CoGBK values.
(#16354)
add 0e69485 Merge pull request #15769 from [BEAM-13031] [Playground] Code
editor - Support Localization
add 503adc0 [BEAM-13555] playground - quick fix for safari
add 304680d Merge pull request #16362 from [BEAM-13555] [Bugfix]
[Playground] change html renderer for safari to fix freezes
add bbcfa4f [BEAM-13541] Fix spotbugs issue with DLS in tail (#16407)
add b9e8a78 [BEAM-13588] Add missing module for PVR tests and also
sickbay known broken test. (#16408)
add fdbb90b [BEAM-13575] Sickbay test that is flaky to restore precommit
test signal (#16410)
add 34118c6 [BEAM-12092] Bump jedis to version 3.8.0 (#16403)
add 7487bf0 [BEAM-13498, BEAM-13522] Sickbay broken test and test
category for Spark (#16411)
add df8b83b [BEAM-13587] Attempt to load AWS region from default provider
chain in AwsOptions
add 2cd5013 Merge pull request #16405: [BEAM-13587] Attempt to load AWS
region from default provider chain in AwsOptions
add 0fd6504 [BEAM-13509] Stop sharing SQS client between readers of same
source.
add c4fc398 Merge pull request #16368: [BEAM-13509] Stop sharing SQS
client between readers of same source.
add 17b62ad Merge pull request #16404: [BEAM-13586] Fix NPE in
DefaultS3ClientBuilderFactory
add 36ce02b [BEAM-13480] Sickbay
PubSubIntegrationTest.test_streaming_data_only on Dataflow (#16255)
add 3d272e7 Fix sdk_container_builder too many values to unpack error
add 1636a3a Merge pull request #16417 from y1chi/python_val
add 43a17a3 [BEAM-13430] Swap to use "mainClass" instead of "main" since
it was deprecated. (#16400)
add e6c73b4 [BEAM-13430] Replace deprecated "appendix" with
"archiveAppendix" (#16401)
add 061533c [BEAM-13591] Bump log4j2 version to 2.17.1
add 5a11778 Merge pull request #16415 from deadwind4/BEAM-13591
add 122ae34 [BEAM-13015] Add jamm as a java agent to the Java SDK harness
container (#16412)
add 2d4a17c [BEAM-13430] Partially revert
https://github.com/apache/beam/commit/eaacf709a861ea1c260e3856b720266348b1aecc
(#16419)
add 7b5c4cd [BEAM-12391] update avro sink to close the opened file
handle, instead of getting it by reference from the writer. This ensures that
when we load the python writer implementation we don't run into an attribute
error
add f63451e Merge pull request #16246: [BEAM-12391] update avro sink to
close the opened file handle directly
add 6045ee1 [BEAM-13482] Python fully qualified name external transforms.
add d13c21f Add Python expansion service entry point.
add 0c2f5a5 Merge pull request #16266 [BEAM-13482] Python fully qualified
name external transforms.
add 818428f Merge pull request #15863 from [BEAM-13184] Autosharding for
JdbcIO.write* transforms
add 9d2fcf3 Add Flink runner support for OrderedListState. This version
reads the list entirely into memory, so will not work well for large lists.
add b1341bc Merge pull request #15984: [BEAM-2791] OrderedListState for
Flink
add 3220631 [BEAM-11936] Enable FloatingPointAssertionWithinEpsilon
errorprone check (#16261)
add 9973c4f [BEAM-11936] Enable LockNotBeforeTry errorprone check (#16259)
add 7eeffe0 [BEAM-11936] Enable errorprone unused checks (#16262)
add 6b68536 Add Nexmark Query 14 (#16337)
add 83e9218 [BEAM-13569] Change Spark dependencies to implementation.
add f0289c0 remove redundant dependency
add e35eea8 remove redundant testImplementation dependencies
add 4c9d4f0 Merge pull request #16396 from ibzib/BEAM-13569
add ea65a05 [BEAM-13015] Migrate all user state and side implementations
to support caching. (#16263)
add eea07cf Fix remaining failing perf IT tests.
add 5ccb002 Merge pull request #16418 from y1chi/perf-it-2
add b82e407 [BEAM-13459] Update CHANGES.md, add note about artifact
caching for python jobs
add 56c2082 [BEAM-13459] Update CHANGES.md, add note about artifact
caching python jobs #16416
add e9e49ff [BEAM-13015] Use 20% of memory when the maximum has been
configured. (#16420)
add 38b833a [BEAM-13567] Consolidate runner flag definition. (#16426)
add 7d1d169 Upgrade python library versions in base_image_requirements.txt
add 91ce87d Merge pull request #16424 from y1chi/bump_pubsub
add 2a4a4cd [BEAM-13601] Don't cache Row types for a schema. (#16427)
add 3df986f [BEAM-13430] Re-enable checkerframework (#16429)
add 27908b4 [BEAM-13430] Ensure that testRuntimeMigration depends on
"default" configuration allowing us to get the main "jar" as a dependency.
(#16430)
add e2fadfb Merge pull request #16277 from [BEAM-13124][Playground]
Create readiness endpoint
add 84a2902 Merge pull request #16314 from
[BEAM-13260][Playground]Implement setup of CI test environment
add 5962f59 Merge pull request #16383 from [BEAM-13566][Playground] Add
logic of sending preparation's output on the backend side
add 6612983 [BEAM-13565][Playground] Add GetPreparationOutput method into
the .proto files; Regenerate files;
add 56a3c57 Merge pull request #16382 from [BEAM-13565][Playground] Add
GetPreparationOutput API method to .proto file
add 2c4f1d9 Merge pull request #16365 from [BEAM-13559][Playground]
Remove tag in examples CD
add d733e85 Merge pull request #16360 from [BEAM-13546][Playground]
Update nginx configuration to enable embedding iframes
add f51e1a6 Merge pull request #16192 from [BEAM-13395] [Playground] Tag
katas
add 5ea425b Merge pull request #16254 from [BEAM-13249][Playground]
Security – Mock Network
add b778f7d update bom
add 46b3ef5 Merge pull request #16377: [BEAM-13576] update bom
add 21d9f6d Merge pull request #16347: fix: move connector to use v1
BigQuery Storage Write API
add 9642de1 [BEAM-12879] Prevented missing permission from failing GCS I/O
add 70d9e2a [BEAM-12879] Prevented missing permission from failing GCS I/O
add c67bed4 [BEAM-13603] Fix bug in apache_beam.utils.Shared (#16437)
add 90f1675 [BEAM-10345] Add an import guard to support recent
google-cloud-spanner versions. (#16434)
add 3bbe6ff Fixed empty labels treated as wildcard when matching cache
files (#16440)
add 5c564d1 [BEAM-13570] Remove erroneous compileClasspath dependency.
(#16438)
add 17485ee [BEAM-13015] Plumb through process wide and bundle cache
through the FnApiStateAccessor. (#16423)
add 638687f [BEAM-13015] Cache the state backed iterable used for large
GBK results. (#16409)
add 50ef9e4 [BEAM-13604] NPE while getting null from BigDecimal column
add 19d60ea Merge pull request #16442: [BEAM-13604] NPE while getting
null from BigDecimal column
add ae5ea31 Fix formatting/alignment (#16443)
add b94d6b3 Merge pull request #16183 from [BEAM-13427] [Playground]
show logs for precompiled objects
add 52efbe4 [BEAM-10277] re-write encoding position tests to declare
schema protos explicitly (#16267)
add 3c67192 Update local_env_tests.yml (#16444)
add 3c53a78 [BEAM-13574] Filesystem abstraction Rename support (#16428)
add a1a530e [BEAM-13597] Setup Go in github actions (#16446)
add f656713 [BEAM-13091] Generate missing staged names from hash for
Dataflow runner
add a43eb11 add test
add cb652eb Merge pull request #15765 from ihji/BEAM-13091
add fffe4e9 Merge pull request #16161 from [BEAM-12164] Add Spanner
Partition Metadata DAOs
add 39c40f2 Merge pull request #16203 from [BEAM-12164] Add Spanner
Change Stream Mappers
add 55d16a6 [adhoc] Fix BigTableIO description
add ab20d7a Merge pull request #16453: [adhoc] Fix BigTableIO description
add a9e40c2 [BEAM-13015] Remove dead code now that all instances have
migrated to the state caching implementation. (#16447)
add fea9c4b [BEAM-13386] Add RLock support for cloudpickle (#16250)
add 3a7dcd7 Fix overflow
add e208b9d Merge pull request #16433 from danthev/py_overflow_fix
add b4ccfc6 [BEAM-8727] Bump software.amazon.awssdk to 2.17.106
add 7681258 Merge pull request #16456: [BEAM-8727] Bump
software.amazon.awssdk to 2.17.106
add 8d01b1e [BEAM-4868] Bump com.amazonaws to 1.12.135
add 2d3afbf Merge pull request #16454: [BEAM-4868] Bump com.amazonaws to
1.12.135
add 928da54 [BEAM-12092] Bump jedis to version 4.0.1 (#16287)
add 064764c don't close streams in finalize
add 6dabd6d Merge pull request #16464: [BEAM-13617] don't close streams
in finalize
add 1f766e3 [BEAM-13534] Add automated port polling to expansion service
runner if port isn't provided (#16470)
add bddbf92 Merge pull request #16344 from
[BEAM-13536][Playground][Bugfix] CI step doesn't log case with empty category
value
add 99d870d Merge pull request #16359 from [BEAM-13545][Playground] Add
GetValidationOutput API method to .proto file
add dc3961d Merge pull request #16384 from [BEAM-13308] [Playground]
Getting baseFileFolder from environment
add bcc46e6 Merge pull request #16306 from [BEAM-13447] [Playground] Add
filling of the chosen thumbs up/down button
add a7dae78 Merge pull request #16361 from [BEAM-13543][Playground] Add
logic of sending validation's output on the backend side
add 9bcfd1b [BEAM-12562] Dataframe pipe implementation (#16256)
add 0d583f4 Merge pull request #16338 from [BEAM-13528][Playground] Add
liveness check
add a3f617f [BEAM-13626] Remap expanded outputs after merging. (#16471)
add d082b05 Merge pull request #16147 from [BEAM-13359] [Playground] Tag
existing examples
add ee3b76a [BEAM-3221] Improve documentation in model pipeline protos
(#16474)
add 6390bcd [BEAM-13614] Add OnWindowExpiration support to the Java SDK
harness and proto translation. (#16458)
add 2ce793a Merge pull request #16156 from [BEAM-13391] Fix temporary
file format in WriteToBigQuery
add f8ecf24 Loosen typing extensions bound
add 5f3c17b Loosen typing extensions bound #16466
add a263858 [adhoc] Run spotlessApply on java examples to fix master
add 5201237 Merge pull request #16479: [adhoc] Run spotlessApply on java
examples to fix master
add 562346b [BEAM-13243][BEAM-8374] Add support for missing
PublishResponse fields in SnsIO.Write (AWS SDK v2)
add 5ad247b Merge pull request #16128: [BEAM-13243][BEAM-8374] Add
support for missing PublishResponse fields in SnsIO.Write (AWS SDK v2)
add ea5032c [BEAM-13628] Update SideInputCache to use full Transform and
SideInputIDs as token information (#16483)
add b2ed194 [BEAM-13602] Prevented metrics gathering from failing
bigtable io
add 6728737 make the code more pythonic
add e7bf967 [BEAM-13602] Prevented metrics gathering from failing
bigtable io
add 53d8ff7 Optional args and kwargs for named external transforms.
add e18e1b3 Merge pull request #16475 Optional args and kwargs for named
external transforms.
add fd1a7cd [BEAM-13432] Skip ExpansionService creation in Job Server
(#16222)
add 1688ac1 [BEAM-13616] Initial files for vendored gRPC 1.43.2 (#16460)
add a2db81d [BEAM-13638] Datatype of timestamp fields in SqsMessage for
AWS IOs for SDK v2 was changed from String to long, visibility of all fields
was fixed from package private to public
add a5d5b64 Merge pull request #16491: [BEAM-13638] Fix visibility of
fields in SqsMessage (AWS Sdk v2)
add fab75c9 Merge pull request #16469 from [BEAM-13623][Playground]
[Bugfix] During unit tests failing there is no any output
add 1c3266f Merge pull request #16149 from [BEAM-13113] [Playground]
playground frontend documentation
add e22444d Merge pull request #16363 from [BEAM-13557] [Playground] show
code execution time
add a3c3722 Merge pull request #16374 from [BEAM-13398][Playground] Split
LifeCycle to DTO and business logic
add f2468ba [BEAM-13616][BEAM-13646] Update vendored calcite 1.28.0 with
protobuf 3.19.2 (#16473)
add 1fa12d9 Merge pull request #16325 from [BEAM-13471] [Playground] Tag
existing unit-tests
add 08dd404 Adds several example multi-language Python pipelines
add b87d142 Merge pull request #16488: [BEAM-13637] Adds several example
multi-language Python pipelines
add 11b1a70 [BEAM-13399] Move service liveness polling to Runner type
(#16487)
add 7f5abe8 [BEAM-13480] Increase pipeline timeout for
PubSubIntegrationTest.test_streaming_data_only (#16496)
add 814a10b Stronger typing inference for CoGBK. (#16465)
add 5ab52a3 [BEAM-12464] Change ProtoSchemaTranslator beam schema
creation to match the order for protobufs containing Oneof fields (#14974)
add ad7a350 Introduce the notion of a JoinIndex for fewer shuffles.
(#16101)
add bd22c65 Merge pull request #16467 from [BEAM-12164]: SpannerIO
DetectNewPartitions SDF
add 7deb105 Merge pull request #16385 from [BEAM-13535] [Playground] add
cancel execution button
add 0a220a1 Merge pull request #16485 from [BEAM-13486] [Playground] For
unit tests (java) if one of tests fails the output goes to stdOutput
add cb08343 python sdk examples: Fixed typo in wordcount example.
add 8724555 Merge pull request #16413 from blais/master
add b42baf9 [BEAM-13455] Remove duplicated artifacts when using multiple
environments with Dataflow Java
add c0dc71e Merge pull request #16225 from ihji/BEAM-13455
add 3c8cd42 [BEAM-12572] Run java examples on multiple runners (#16450)
add cdd6f71 [BEAM-13400] JDBC IO does not support UUID and JSONB
PostgreSQL types and OTHER JDBC types in general
add a3bf36e Merge pull request #16284: [BEAM-13400] JdbcIO should support
UUID and JSONB PostgreSQL types and OTHER JDBC types in general
add e46408c Provide API to check whether a hint is known.
add 5b8ea6c [BEAM-12558] Fix doc typo.
add a221967 [BEAM-13656] Provide an API to check whether a hint is known.
#16508
add 5ad804b Split builder into several builder for each step of pipeline
execution
add c345fe1 Merge pull request #16477 from [BEAM-13560][Playground] Split
builder into several builders for each step of the pipeline execution
add 52ba860 Merge pull request #16482 from [BEAM-13429][Playground] Add
builder for preparers
add 9d40a4c [BEAM-13590] Fix abc imports from collections (#15850)
add a03093e Redirecting cross-language transforms content (#16504)
add f4e1e3a Remove tab from source.
add e38c6ac Merge pull request #16528 Remove tab from source.
add 205bafc [BEAM-10206] Resolve go vet errors in protox package
add 454ddba Merge pull request #16520: [BEAM-10206] Resolve go vet errors
in protox package
add 65a42b5 doc tweaks (#16498)
add a421b69 [BEAM-12621] - Update Jenkins VMs to modern Ubuntu version
(#16457)
add d88b492 Fix staticcheck errors in transforms directory
add 7ce4aa9 Remove unnecessary fmt.Sprintf() in partition.go
add 15c7d69 Replace bytes.Compare() with bytes.Equal() in test cases
add 56a0b55 Replace string(buf.Bytes()) with buf.String() in coder_test.go
add 235a9c3 Remove unnecessary blank identifier assignment in harness.go
add 413be9b fix capitalized error strings in expansionx
add 3a8fb56 Clean up string cast of bytes in vet.go and corresponding
tests
add f061f91 Remove unnecessary fmt call in universal.go
add 6d471c9 Merge pull request #16524: [BEAM-10206] Clean up some simple
staticcheck warnings in Go SDK
add 38b7e47 [BEAM-13664] Fix Primitives hashing benchmark (#16523)
add 8c2e1fd Merge pull request #16507: [BEAM-13137] Fixes ES utest size
flakiness with _flush api and index.store.stats_refresh_interval=0
add ed10912 [BEAM-8806] Integration test for SqsIO using Localstack
add 181403a Merge pull request #16468: [BEAM-8806] Integration test for
SqsIO using Localstack (AWS Sdk v2)
add 12330d8 Remove jcenter repositories from gradle configuration.
(#16532)
add ca46551 [BEAM-13641][Playground] Add SCIO SDK support on the CI/CD
step
add b45dc7e Merge pull request #16492 from [BEAM-13641][Playground] Add
SCIO SDK support on the CI/CD step
add 42c3b44 [BEAM-13430] Remove jcenter which will no longer contain any
updates. (#16536)
add 1838347 [BEAM-13616] Update com.google.cloud:libraries-bom to 24.2.0
(#16509)
add db28e19 [BEAM-13631] Add deterministic SQS message coder to fix
reading from SQS in batch mode
add 5187008 Merge pull request #16480: [BEAM-13631] Add deterministic SQS
message coder to fix reading from SQS in batch mode (AWS Sdk v1)
add 66be723 [BEAM-8806] Integration test for SqsIO
add b3e9c7d Merge pull request #16481: [BEAM-8806] Integration test for
SqsIO using Localstack (AWS Sdk v1)
add 7727dc9 [BEAM-13616][BEAM-13645] Switch to vendored grpc 1.43.2
(#16543)
add a5f789a Bump beam container version.
add ae75e47 Also bump FnAPI container.
add c54f53e Merge pull request #16529 Bump dataflow worker container
versions.
add 5e2e7ee [BEAM-13616][BEAM-13646] Upgrade vendored calcite to
1.28.0:0.2 (#16544)
add 745bee1 [BEAM-13680] Fixed code_repository (added pipelineUuid to
RunCodeResult when status is "Finished")
add ba97ea2 Merge pull request #16545 from [BEAM-13680] [Playground]
[Bugfix] Fix tests on frontend
add a241482 Merge pull request #16486 from [BEAM-13544][Playground] Add
logs to examples CI/CD to see the progress
add 424a462 [BEAM-13683] Correct SQL transform schema, fix expansion
address override bug (#16551)
add 91143f6 Update walkthrough.md (#16512)
add 77e924f [BEAM-11808][BEAM-9879] Support aggregate functions with two
arguments (#16200)
add bbaaa4b Merge pull request #16506 from [BEAM-13652][Playground] Send
examples' links to the frontend
add da359d9 Merge pull request #16322 from [BEAM-13407] [Playground]
Preload fonts for the web application
add 3759d44 [BEAM-13665] Make SpannerIO projectID optional again (#16547)
add 6b5a98f [BEAM-13015] Add state caching capability to be used as hint
for runners to not duplicate cached data if the SDK can do it for user state
and side inputs. (#16525)
add fd3f207 Merge pull request #16309: [BEAM-13503] Set a default value
to throwWriteErrors in BulkIO constructor
add b458f8f [BEAM-13015] Provide caching statistics in the status client.
(#16495)
add c392a00 [BEAM-13611] Skip test_xlang_jdbc_write (#16554)
add db769cb Merge pull request #16370 from [BEAM-13556] playground -
color and scroll tabs with new content
add 4fafa60 Merge pull request #16531 from [BEAM-13567] [playground]
Handle run code validation and preparation errors
add 2148a8e Merge pull request #16533 from [BEAM-13548] [Playground] Add
example description popover
add 33ae030 Merge pull request #16519 from [BEAM-13639] [Playground] Add
notification to Log/Output tabs about cached example
add 0cbd29f Merge pull request #16518 from [BEAM-13619] [Playground] Add
loading animation to the catalog
add 8ff406e Merge pull request #16243 from
darshan-sj/feature/support-priority-spannerio - Making rpcPriority a
ValueProvider in SpannerConfig
add d67e266 [BEAM-13683] Make cross-language SQL example pipeline (#16567)
add ef07638 [BEAM-13688] fixed type in BPG 4.5.3 window section (#16560)
add a5935c7 Remove obsolete commands from Inventory job. (#16564)
add 72a8c2c Disable logging for memoization test. (#16556)
add 2d1157e Merge pull request #16472: [BEAM-13697] Add SchemaFieldNumber
annotation
add f31cad0 Merge pull request #16373 from [BEAM-13515] [Playground]
Hiding lines in an example that are not necessary
add 421bc80 Revert "Merge pull request #15863 from [BEAM-13184]
Autosharding for JdbcIO.write* transforms"
add 4f29108 BEAM-13611 reactivating jdbcio xlang test
add a00634d Merge pull request #16569 from Revert "Merge pull request
#15863 from [BEAM-13184] Autosharding for …
add d43c755 Merge pull request #16371 from [BEAM-13518][Playground] Beam
Playground quickstart page on the Beam website
add 77b410c Update Java FnAPI beam master (#16572)
add 93f41ce [BEAM-13699] Replace fnv with maphash. (#16573)
add 2c22d8b [BEAM-13693] Bump
beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timeout to 12 hours
(#16576)
add 586739d [BEAM-13577] Beam Select's uniquifyNames function loses
nullability of Complex types while inferring schema
add ab6374e Merge pull request #16380 from
talatuyarer/BEAM-13577-select-uniquifyNames-nullablity
add bc39244 [BEAM-13689] Output token elements when BQ batch writes
complete.
add acd1d52 Merge pull request #16561 from [BEAM-13689] Output
TableDestination elements when BQ batch writes complete.
add 50e7235 [BEAM-10206] Remove Fatalf calls in non-test goroutines for
tests/benchmarks (#16575)
add e3c24f0 [BEAM-13430] Re-add provided configuration (#16552)
new 79120fd (release recut) Revert "Moving to 2.37.0-SNAPSHOT on master
branch."
new 5076eb1 Set Dataflow container to release version.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d5d0bc3)
\
N -- N -- N refs/heads/release-2.36.0 (5076eb1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/java_tests.yml | 9 +
.github/workflows/local_env_tests.yml | 3 +-
.github/workflows/playground_deploy_examples.yml | 129 ++
.github/workflows/python_tests.yml | 9 +
.test-infra/jenkins/job_Inventory.groovy | 2 -
.../job_PostCommit_Java_Examples_Direct.groovy | 46 +
.../job_PostCommit_Java_Examples_Flink.groovy | 46 +
.../job_PostCommit_Java_Examples_Spark.groovy | 47 +
...Java_PortableValidatesRunner_Spark_Batch.groovy | 4 +-
..._Java_ValidatesRunner_Dataflow_Streaming.groovy | 2 +-
...ava_PortableValidatesRunner_Flink_Docker.groovy | 34 +-
.test-infra/tools/README.md | 57 +
.test-infra/tools/python_installer.sh | 92 ++
.test-infra/validate-runner/build.gradle | 7 +-
CHANGES.md | 50 +-
build.gradle.kts | 7 +-
buildSrc/build.gradle.kts | 1 -
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 114 +-
...g_1_36_0.groovy => GrpcVendoring_1_43_2.groovy} | 47 +-
.../org/apache/beam/gradle/Repositories.groovy | 2 -
examples/java/build.gradle | 10 +-
.../apache/beam/examples/DebuggingWordCount.java | 11 +
.../org/apache/beam/examples/MinimalWordCount.java | 15 +-
.../java/org/apache/beam/examples/WordCount.java | 9 +
.../datatokenization/DataTokenization.java | 2 +-
.../examples/complete/datatokenization/README.md | 2 +-
.../transforms/DataProtectors.java | 6 +-
.../complete/kafkatopubsub/KafkaToPubsub.java | 2 +-
.../beam/examples/complete/kafkatopubsub/README.md | 2 +-
.../kafkatopubsub/kafka/consumer/Utils.java | 4 +-
.../beam/examples/cookbook/DistinctExample.java | 22 +-
.../apache/beam/examples/snippets/Snippets.java | 8 +
.../org/apache/beam/examples/complete/TfIdfIT.java | 2 +-
.../apache/beam/examples/complete/TfIdfTest.java | 9 +
.../beam/examples/complete/game/GameStatsTest.java | 9 +
.../examples/cookbook/BigQueryTornadoesIT.java | 2 +-
.../examples/cookbook/DistinctExampleTest.java | 9 +
examples/java/twitter/build.gradle | 7 +-
examples/kotlin/build.gradle | 6 +-
examples/multi-language/README.md | 41 +
.../app => examples/multi-language}/build.gradle | 31 +-
examples/multi-language/python/addprefix.py | 98 ++
examples/multi-language/python/javacount.py | 104 ++
.../multi-language/python/javadatagenerator.py | 89 ++
.../beam/examples/multilanguage/JavaCount.java | 14 +-
.../examples/multilanguage/JavaCountBuilder.java | 17 +-
.../multilanguage/JavaCountConfiguration.java | 9 +-
.../examples/multilanguage/JavaCountRegistrar.java | 37 +-
.../examples/multilanguage/JavaDataGenerator.java | 97 ++
.../beam/examples/multilanguage/JavaPrefix.java | 39 +-
.../examples/multilanguage/JavaPrefixBuilder.java | 16 +-
.../multilanguage/JavaPrefixConfiguration.java | 19 +-
.../multilanguage/JavaPrefixRegistrar.java | 37 +-
.../get-started/try-apache-beam-java.ipynb | 12 +-
.../commontransforms/aggregation/count/Task.java | 8 +
.../commontransforms/aggregation/max/Task.java | 8 +
.../commontransforms/aggregation/mean/Task.java | 8 +
.../commontransforms/aggregation/min/Task.java | 8 +
.../commontransforms/aggregation/sum/Task.java | 8 +
.../katas/commontransforms/filter/filter/Task.java | 8 +
.../katas/commontransforms/filter/pardo/Task.java | 8 +
.../katas/commontransforms/withkeys/Task.java | 8 +
.../katas/coretransforms/branching/Task.java | 10 +
.../combine/binarycombinefnlambda/Task.java | 9 +
.../combine/binarycombinefn/Task.java | 9 +
.../coretransforms/combine/combineperkey/Task.java | 9 +
.../coretransforms/combine/combinefn/Task.java | 9 +
.../katas/coretransforms/combine/simple/Task.java | 9 +
.../katas/coretransforms/composite/Task.java | 11 +
.../katas/coretransforms/flatten/Task.java | 10 +
.../katas/coretransforms/groupbykey/Task.java | 9 +
.../coretransforms/map/flatmapelements/Task.java | 9 +
.../katas/coretransforms/map/mapelements/Task.java | 8 +
.../coretransforms/map/pardoonetomany/Task.java | 9 +
.../katas/coretransforms/map/pardo/Task.java | 8 +
.../katas/coretransforms/partition/Task.java | 10 +
.../katas/coretransforms/sideoutput/Task.java | 10 +
.../learning/katas/examples/wordcount/Task.java | 8 +
.../beam/learning/katas/intro/hello/Task.java | 8 +
.../learning/katas/windowing/fixedwindow/Task.java | 9 +
learning/katas/java/build.gradle | 3 +-
learning/katas/kotlin/build.gradle | 3 +-
.../Common Transforms/Aggregation/Count/task.py | 8 +
.../Common Transforms/Aggregation/Largest/task.py | 8 +
.../Common Transforms/Aggregation/Mean/task.py | 8 +
.../Common Transforms/Aggregation/Smallest/task.py | 8 +
.../Common Transforms/Aggregation/Sum/task.py | 8 +
.../python/Common Transforms/Filter/Filter/task.py | 8 +
.../python/Common Transforms/Filter/ParDo/task.py | 8 +
.../Common Transforms/WithKeys/WithKeys/task.py | 8 +
.../Core Transforms/Branching/Branching/task.py | 10 +
.../CoGroupByKey/CoGroupByKey/task.py | 9 +
.../Core Transforms/Combine/Combine PerKey/task.py | 8 +
.../Core Transforms/Combine/CombineFn/task.py | 8 +
.../Combine/Simple Function/task.py | 8 +
.../Composite Transform/task.py | 9 +
.../python/Core Transforms/Flatten/Flatten/task.py | 8 +
.../Core Transforms/GroupByKey/GroupByKey/task.py | 8 +
.../python/Core Transforms/Map/FlatMap/task.py | 9 +
.../katas/python/Core Transforms/Map/Map/task.py | 8 +
.../Core Transforms/Map/ParDo OneToMany/task.py | 9 +
.../katas/python/Core Transforms/Map/ParDo/task.py | 8 +
.../Core Transforms/Partition/Partition/task.py | 9 +
.../Core Transforms/Side Input/Side Input/task.py | 8 +
.../Side Output/Side Output/task.py | 9 +
.../python/Examples/Word Count/Word Count/task.py | 8 +
.../Introduction/Hello Beam/Hello Beam/task.py | 8 +
.../Windowing/Adding Timestamp/ParDo/task.py | 8 +
.../Fixed Time Window/Fixed Time Window/task.py | 9 +
.../fn-execution/src/main/proto/beam_fn_api.proto | 18 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 25 +-
playground/api/v1/api.proto | 29 +-
playground/backend/cmd/server/controller.go | 34 +-
playground/backend/cmd/server/http.go | 14 +-
playground/backend/cmd/server/server.go | 2 +-
playground/backend/containers/go/Dockerfile | 19 +-
playground/backend/containers/go/build.gradle | 12 +
playground/backend/containers/go/entrypoint.sh | 25 +
playground/backend/containers/java/Dockerfile | 15 +-
playground/backend/containers/java/build.gradle | 12 +
playground/backend/containers/java/entrypoint.sh | 25 +
playground/backend/containers/python/Dockerfile | 15 +-
playground/backend/containers/python/build.gradle | 12 +
playground/backend/containers/python/entrypoint.sh | 26 +
playground/backend/internal/api/v1/api.pb.go | 957 ++++++++-----
playground/backend/internal/api/v1/api_grpc.pb.go | 76 +
playground/backend/internal/cache/cache.go | 6 +
.../backend/internal/cache/redis/redis_cache.go | 2 +-
.../internal/cloud_bucket/precompiled_objects.go | 1 +
.../internal/code_processing/code_processing.go | 273 ++--
.../code_processing/code_processing_test.go | 328 +++--
.../backend/internal/environment/application.go | 11 +-
.../internal/environment/application_test.go | 40 +
playground/backend/internal/environment/beam.go | 16 +-
.../internal/environment/environment_service.go | 27 +-
.../environment/environment_service_test.go | 17 +-
playground/backend/internal/executors/executor.go | 10 +-
.../backend/internal/executors/executor_builder.go | 85 +-
.../backend/internal/executors/executor_test.go | 145 +-
playground/backend/internal/fs_tool/fs.go | 101 +-
playground/backend/internal/fs_tool/fs_test.go | 531 +++----
playground/backend/internal/fs_tool/go_fs.go | 4 +-
playground/backend/internal/fs_tool/go_fs_test.go | 48 +-
playground/backend/internal/fs_tool/java_fs.go | 13 +-
.../backend/internal/fs_tool/java_fs_test.go | 68 +-
.../backend/internal/fs_tool/lc_constructor.go | 61 +-
playground/backend/internal/fs_tool/python_fs.go | 4 +-
.../backend/internal/fs_tool/python_fs_test.go | 44 +-
.../go_preparers.go} | 62 +-
.../go_preparers_test.go} | 18 +-
.../java_preparers.go} | 126 +-
.../java_preparers_test.go} | 42 +-
.../preparator.go => preparers/preparer.go} | 34 +-
.../python_preparers.go} | 30 +-
.../python_preparers_test.go} | 16 +-
.../internal/setup_tools/builder/setup_builder.go | 145 +-
.../setup_tools/builder/setup_builder_test.go | 289 +++-
.../setup_tools/life_cycle/life_cycle_setuper.go | 27 +-
.../life_cycle/life_cycle_setuper_test.go | 100 +-
.../internal/streaming/run_output_writer.go | 9 +-
.../internal/utils/precompiled_objects_utils.go | 1 +
.../backend/internal/utils/preparators_utils.go | 26 +-
playground/backend/internal/utils/system_utils.go | 51 +
.../backend/internal/utils/system_utils_test.go | 85 ++
playground/categories.yaml | 7 +
playground/frontend/CONTRIBUTE.md | 135 ++
playground/frontend/Dockerfile | 1 +
playground/frontend/README.md | 53 +-
playground/frontend/analysis_options.yaml | 2 +-
playground/frontend/assets/thumb_down_filled.svg | 24 +
playground/frontend/assets/thumb_up_filled.svg | 24 +
playground/frontend/l10n.yaml | 20 +
playground/frontend/lib/api/v1/api.pb.dart | 1462 ++++++++++++++------
playground/frontend/lib/api/v1/api.pbgrpc.dart | 381 ++---
playground/frontend/lib/api/v1/api.pbjson.dart | 283 +++-
.../loading_indicator/loading_indicator.dart} | 22 +-
.../toggle_theme_button/toggle_theme_button.dart | 9 +-
.../output_tabs.dart => config/locale.dart} | 28 +-
playground/frontend/lib/constants/assets.dart | 2 +
playground/frontend/lib/constants/sizes.dart | 2 +
playground/frontend/lib/l10n/app_en.arb | 158 +++
.../frontend/lib/{main.dart => l10n/l10n.dart} | 10 +-
playground/frontend/lib/main.dart | 4 +-
.../actions/components/new_example_action.dart | 5 +-
.../modules/actions/components/reset_action.dart | 5 +-
.../modules/editor/components/editor_textarea.dart | 62 +-
.../components/pipeline_options_text_field.dart | 11 +-
.../lib/modules/editor/components/run_button.dart | 61 +-
.../modules/editor/parsers/run_options_parser.dart | 24 +-
.../code_repository/code_client/code_client.dart | 12 +
.../code_client/grpc_code_client.dart | 32 +-
.../code_repository/code_repository.dart | 53 +-
.../code_repository/run_code_result.dart | 19 +-
.../description_popover.dart} | 41 +-
.../description_popover_button.dart | 80 ++
.../example_list/expansion_panel_item.dart | 31 +-
.../components/filter/category_bubble.dart | 9 +-
.../examples/components/filter/type_filter.dart | 25 +-
.../components/search_field/search_field.dart | 4 +-
.../lib/modules/examples/example_selector.dart | 40 +-
.../lib/modules/examples/models/example_model.dart | 6 +
.../example_client/example_client.dart | 2 +
.../example_client/grpc_example_client.dart | 26 +-
.../examples/repositories/example_repository.dart | 7 +
.../lib/modules/output/components/output.dart | 40 +-
.../lib/modules/output/components/output_area.dart | 21 +-
.../components/output_header/output_header.dart | 13 +-
.../output_header/output_placements.dart | 7 +-
.../components/output_header/output_tab.dart | 80 ++
.../components/output_header/output_tabs.dart | 44 +-
.../modules/output/components/output_result.dart | 29 +-
.../modules/output/models/output_placement.dart | 16 +-
.../lib/modules/sdk/components/sdk_selector.dart | 4 +-
.../shortcuts/components/shortcuts_modal.dart | 27 +-
.../components/embedded_actions.dart | 4 +-
.../embedded_playground_page.dart | 1 -
.../playground/components/close_listener.dart} | 41 +-
.../components/close_listener_nonweb.dart} | 14 +-
.../components/editor_textarea_wrapper.dart | 59 +-
.../feedback/feedback_dropdown_icon_button.dart | 16 +-
.../components/feedback/playground_feedback.dart | 33 +-
.../pages/playground/components/more_actions.dart | 27 +-
.../components/playground_page_footer.dart | 13 +-
.../components/playground_page_providers.dart | 4 +
.../lib/pages/playground/playground_page.dart | 88 +-
.../pages/playground/states/examples_state.dart | 24 +-
.../playground/states/feedback_state.dart} | 17 +-
.../pages/playground/states/playground_state.dart | 67 +-
playground/frontend/lib/playground_app.dart | 38 +-
playground/frontend/nginx_default.conf | 12 +-
playground/frontend/pubspec.lock | 12 +
playground/frontend/pubspec.yaml | 5 +
.../editor/parsers/run_options_parser_test.dart | 4 +
.../code_repository/code_repository_test.dart | 93 +-
.../code_repository_test.mocks.dart | 22 +
.../states/example_selector_state_test.mocks.dart | 7 +
playground/frontend/web/index.html | 19 +
playground/infrastructure/cd_helper.py | 19 +-
playground/infrastructure/ci_cd.py | 9 +
playground/infrastructure/ci_helper.py | 16 +-
playground/infrastructure/config.py | 12 +-
playground/infrastructure/grpc_client.py | 1 +
playground/infrastructure/helm/Chart.yaml | 20 +
.../infrastructure/helm/templates/_helpers.tpl | 76 +
.../helm/templates/deployment-backend-go.yaml | 71 +
.../helm/templates/deployment-backend-java.yaml | 71 +
.../helm/templates/deployment-backend-python.yaml | 71 +
.../infrastructure/helm/templates/service-go.yaml | 32 +
.../helm/templates/service-java.yaml | 32 +
.../helm/templates/service-python.yaml | 32 +
playground/infrastructure/helm/values.yaml | 65 +
playground/infrastructure/helper.py | 101 +-
playground/infrastructure/proxy/allow_list.py | 29 +
.../infrastructure/proxy/allow_list_proxy.py | 30 +
playground/infrastructure/requirements.txt | 3 +-
playground/infrastructure/test_cd_helper.py | 7 +-
playground/infrastructure/test_grpc_client.py | 1 +
playground/infrastructure/test_helper.py | 48 +-
.../terraform/applications/backend-go/main.tf | 4 +-
.../{backend-python => backend-java}/main.tf | 39 +-
.../{backend => backend-java}/variables.tf | 12 +
.../terraform/applications/backend-python/main.tf | 4 +-
.../{backend => backend-router}/main.tf | 30 +-
.../{backend-go => backend-router}/variables.tf | 0
playground/terraform/applications/frontend/main.tf | 5 +-
.../terraform/applications/frontend/variables.tf | 4 +
.../{applications/backend => modules/gke}/main.tf | 46 +-
.../modules/gke/output.tf} | 16 +-
.../frontend => modules/gke}/variables.tf | 19 +-
runners/core-construction-java/build.gradle | 2 +-
.../beam/runners/core/construction/BeamUrns.java | 2 +-
.../core/construction/CoderTranslation.java | 2 +-
.../core/construction/CoderTranslators.java | 2 +-
.../core/construction/CombineTranslation.java | 2 +-
.../CreatePCollectionViewTranslation.java | 2 +-
.../DefaultExpansionServiceClientFactory.java | 2 +-
.../core/construction/DisplayDataTranslation.java | 2 +-
.../runners/core/construction/Environments.java | 11 +-
.../beam/runners/core/construction/External.java | 6 +-
.../runners/core/construction/ModelCoders.java | 2 +-
.../construction/PCollectionViewTranslation.java | 2 +-
.../core/construction/ParDoTranslation.java | 90 +-
.../construction/PipelineOptionsTranslation.java | 6 +-
.../runners/core/construction/ReadTranslation.java | 4 +-
.../runners/core/construction/SdkComponents.java | 2 +-
.../runners/core/construction/SplittableParDo.java | 11 +-
.../core/construction/TestStreamTranslation.java | 2 +-
.../core/construction/WindowIntoTranslation.java | 2 +-
.../construction/WindowingStrategyTranslation.java | 8 +-
.../core/construction/WriteFilesTranslation.java | 2 +-
.../graph/GreedyPCollectionFusers.java | 2 +-
.../core/construction/graph/QueryablePipeline.java | 2 +-
.../runners/core/construction/CommonCoderTest.java | 2 +-
.../core/construction/EnvironmentsTest.java | 22 +
.../core/construction/PTransformMatchersTest.java | 21 -
.../core/construction/ParDoTranslationTest.java | 9 +
.../PipelineOptionsTranslationTest.java | 6 +-
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/construction/ValidateRunnerXlangTest.java | 8 +-
.../construction/WindowIntoTranslationTest.java | 2 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
runners/core-java/build.gradle | 2 +-
.../runners/core/metrics/MetricsContainerImpl.java | 2 +-
.../core/metrics/MetricsContainerStepMap.java | 6 +-
.../core/metrics/MonitoringInfoEncodings.java | 2 +-
.../runners/core/metrics/SimpleExecutionState.java | 2 +-
.../runners/core/metrics/SimpleStateRegistry.java | 2 +-
.../core/metrics/MonitoringInfoEncodingsTest.java | 2 +-
runners/direct-java/build.gradle | 45 +-
.../runners/direct/TransformEvaluatorRegistry.java | 16 -
.../runners/direct/CloningBundleFactoryTest.java | 67 -
runners/flink/flink_runner.gradle | 44 +-
runners/flink/job-server/flink_job_server.gradle | 49 +-
.../FlinkBatchPortablePipelineTranslator.java | 2 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 2 +-
.../beam/runners/flink/FlinkPipelineRunner.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 2 +-
.../flink/FlinkStreamingTransformTranslators.java | 17 -
.../utils/FlinkPortableRunnerUtils.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../streaming/ExecutableStageDoFnOperator.java | 4 +-
.../wrappers/streaming/FlinkKeyUtils.java | 2 +-
.../state/FlinkBroadcastStateInternals.java | 131 --
.../streaming/state/FlinkStateInternals.java | 153 +-
.../runners/flink/FlinkJobServerDriverTest.java | 47 +
.../FlinkExecutableStageFunctionTest.java | 2 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 5 +-
.../wrappers/streaming/FlinkKeyUtilsTest.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 19 +-
.../examples-streaming/build.gradle | 1 -
.../examples/build.gradle | 1 -
.../dataflow/DataflowPipelineTranslator.java | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 185 +--
.../dataflow/PrimitiveParDoSingleFactory.java | 51 +-
.../util/RowCoderCloudObjectTranslator.java | 2 +-
.../util/SchemaCoderCloudObjectTranslator.java | 2 +-
.../runners/dataflow/DataflowPipelineJobTest.java | 25 -
.../dataflow/DataflowPipelineTranslatorTest.java | 2 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 126 +-
.../google-cloud-dataflow-java/worker/build.gradle | 5 +-
.../worker/legacy-worker/build.gradle | 21 +-
.../runners/dataflow/worker/ByteStringCoder.java | 2 +-
.../dataflow/worker/DataflowRunnerHarness.java | 2 +-
.../worker/DataflowWorkerHarnessHelper.java | 4 +-
.../worker/GroupAlsoByWindowParDoFnFactory.java | 2 +-
.../beam/runners/dataflow/worker/PubsubSink.java | 2 +-
.../beam/runners/dataflow/worker/StateFetcher.java | 2 +-
.../dataflow/worker/StreamingDataflowWorker.java | 4 +-
.../worker/StreamingModeExecutionContext.java | 2 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 4 +-
.../dataflow/worker/WindmillComputationKey.java | 4 +-
.../dataflow/worker/WindmillNamespacePrefix.java | 2 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 2 +-
.../dataflow/worker/WindmillStateCache.java | 4 +-
.../dataflow/worker/WindmillStateInternals.java | 2 +-
.../dataflow/worker/WindmillStateReader.java | 4 +-
.../dataflow/worker/WindmillTimerInternals.java | 2 +-
.../dataflow/worker/WorkerCustomSources.java | 2 +-
.../dataflow/worker/fn/BeamFnControlService.java | 6 +-
.../control/RegisterAndProcessBundleOperation.java | 4 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 4 +-
.../fn/data/RemoteGrpcPortWriteOperation.java | 6 +-
.../worker/fn/logging/BeamFnLoggingService.java | 4 +-
.../fn/stream/ServerStreamObserverFactory.java | 6 +-
.../graph/CreateExecutableStageNodeFunction.java | 4 +-
...nsertFetchAndFilterStreamingSideInputNodes.java | 2 +-
.../worker/graph/RegisterNodeFunction.java | 4 +-
.../worker/status/SdkWorkerStatusServlet.java | 2 +-
.../worker/windmill/DirectStreamObserver.java | 4 +-
.../windmill/ForwardingClientResponseObserver.java | 6 +-
.../worker/windmill/GrpcWindmillServer.java | 30 +-
.../worker/windmill/StreamObserverFactory.java | 4 +-
.../runners/dataflow/harness/test/TestStreams.java | 4 +-
.../runners/dataflow/worker/DataflowMatchers.java | 2 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 2 +-
.../runners/dataflow/worker/PubsubReaderTest.java | 2 +-
.../runners/dataflow/worker/PubsubSinkTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 2 +-
.../runners/dataflow/worker/StateFetcherTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 6 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
.../worker/StreamingModeExecutionContextTest.java | 2 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillReaderIteratorBaseTest.java | 2 +-
.../dataflow/worker/WindmillStateCacheTest.java | 2 +-
.../worker/WindmillStateInternalsTest.java | 6 +-
.../dataflow/worker/WindmillStateReaderTest.java | 4 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 2 +-
.../worker/fn/BeamFnControlServiceTest.java | 4 +-
.../RegisterAndProcessBundleOperationTest.java | 2 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 8 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 6 +-
.../fn/stream/ServerStreamObserverFactoryTest.java | 4 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 2 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 2 +-
.../worker/windmill/GrpcWindmillServerTest.java | 14 +-
runners/java-fn-execution/build.gradle | 2 +-
.../artifact/ArtifactRetrievalService.java | 8 +-
.../artifact/ArtifactStagingService.java | 10 +-
.../control/BundleCheckpointHandlers.java | 2 +-
.../control/DefaultJobBundleFactory.java | 12 +-
.../fnexecution/control/FnApiControlClient.java | 6 +-
.../control/FnApiControlClientPoolService.java | 6 +-
.../control/ProcessBundleDescriptors.java | 2 +-
.../runners/fnexecution/data/GrpcDataService.java | 2 +-
.../environment/ExternalEnvironmentFactory.java | 2 +-
.../fnexecution/logging/GrpcLoggingService.java | 2 +-
.../runners/fnexecution/provisioning/JobInfo.java | 2 +-
.../provisioning/StaticGrpcProvisionService.java | 2 +-
.../fnexecution/state/GrpcStateService.java | 4 +-
.../fnexecution/state/StateRequestHandlers.java | 2 +-
.../status/BeamWorkerStatusGrpcService.java | 2 +-
.../fnexecution/status/WorkerStatusClient.java | 2 +-
.../translation/PipelineTranslatorUtils.java | 2 +-
.../runners/fnexecution/wire/ByteStringCoder.java | 2 +-
.../GrpcContextHeaderAccessorProviderTest.java | 20 +-
.../runners/fnexecution/ServerFactoryTest.java | 12 +-
.../artifact/ArtifactRetrievalServiceTest.java | 10 +-
.../artifact/ArtifactStagingServiceTest.java | 12 +-
.../control/DefaultJobBundleFactoryTest.java | 4 +-
.../control/FnApiControlClientPoolServiceTest.java | 8 +-
.../control/FnApiControlClientTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 216 ++-
.../fnexecution/control/SdkHarnessClientTest.java | 2 +-
.../fnexecution/data/GrpcDataServiceTest.java | 8 +-
.../logging/GrpcLoggingServiceTest.java | 6 +-
.../StaticGrpcProvisionServiceTest.java | 10 +-
.../fnexecution/state/GrpcStateServiceTest.java | 4 +-
.../state/StateRequestHandlersTest.java | 2 +-
.../status/BeamWorkerStatusGrpcServiceTest.java | 8 +-
.../fnexecution/status/WorkerStatusClientTest.java | 2 +-
.../fnexecution/wire/ByteStringCoderTest.java | 2 +-
runners/java-job-service/build.gradle | 3 +-
.../runners/jobsubmission/InMemoryJobService.java | 10 +-
.../beam/runners/jobsubmission/JobInvocation.java | 2 +-
.../beam/runners/jobsubmission/JobInvoker.java | 2 +-
.../beam/runners/jobsubmission/JobPreparation.java | 2 +-
.../runners/jobsubmission/JobServerDriver.java | 11 +-
.../jobsubmission/PortablePipelineJarCreator.java | 4 +-
.../jobsubmission/PortablePipelineJarUtils.java | 6 +-
.../jobsubmission/InMemoryJobServiceTest.java | 6 +-
.../runners/jobsubmission/JobInvocationTest.java | 2 +-
.../PortablePipelineJarCreatorTest.java | 2 +-
runners/jet/build.gradle | 2 +
runners/local-java/build.gradle | 1 +
runners/portability/java/build.gradle | 16 +-
.../portability/JobServicePipelineResult.java | 2 +-
.../beam/runners/portability/PortableRunner.java | 4 +-
.../portability/testing/TestJobService.java | 2 +-
.../runners/portability/PortableRunnerTest.java | 8 +-
runners/samza/build.gradle | 6 +-
runners/samza/job-server/build.gradle | 48 +-
.../apache/beam/runners/samza/SamzaJobInvoker.java | 2 +-
.../samza/runtime/SamzaStateRequestHandlers.java | 2 +-
.../translation/SamzaTestStreamTranslator.java | 2 +-
.../samza/translation/WindowAssignTranslator.java | 2 +-
runners/spark/job-server/spark_job_server.gradle | 190 +--
runners/spark/spark_runner.gradle | 78 +-
.../apache/beam/runners/spark/SparkJobInvoker.java | 2 +-
.../beam/runners/spark/SparkPipelineRunner.java | 2 +-
runners/twister2/build.gradle | 4 +-
.../functions/AssignWindowsFunction.java | 2 +-
.../translators/functions/DoFnFunction.java | 2 +-
.../functions/GroupByWindowFunction.java | 2 +-
sdks/go/examples/contains/contains.go | 10 +
.../debugging_wordcount/debugging_wordcount.go | 10 +
sdks/go/examples/forest/forest.go | 14 +
sdks/go/examples/grades/grades.go | 10 +
.../minimal_wordcount/minimal_wordcount.go | 20 +-
sdks/go/examples/multiout/multiout.go | 13 +
sdks/go/examples/stringsplit/stringsplit.go | 9 +
.../windowed_wordcount/windowed_wordcount.go | 10 +
sdks/go/examples/wordcount/wordcount.go | 9 +
sdks/go/examples/xlang/sql/sql.go | 130 ++
sdks/go/examples/yatzy/yatzy.go | 10 +
sdks/go/pkg/beam/coder_test.go | 2 +-
sdks/go/pkg/beam/combine_test.go | 8 +
sdks/go/pkg/beam/core/runtime/exec/hash.go | 7 +-
sdks/go/pkg/beam/core/runtime/exec/hash_test.go | 24 +-
sdks/go/pkg/beam/core/runtime/exec/pardo_test.go | 12 +-
sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 10 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 6 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 2 +-
.../core/runtime/harness/statecache/statecache.go | 47 +-
.../runtime/harness/statecache/statecache_test.go | 54 +-
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 4 +
.../core/runtime/xlangx/expansionx/download.go | 4 +-
.../beam/core/runtime/xlangx/expansionx/process.go | 58 +-
.../core/runtime/xlangx/expansionx/process_test.go | 95 +-
sdks/go/pkg/beam/core/util/protox/any_test.go | 4 +-
sdks/go/pkg/beam/external_test.go | 2 +-
sdks/go/pkg/beam/io/filesystem/filesystem.go | 31 +-
.../go/pkg/beam/io/filesystem/filesystem_test.go | 33 +-
sdks/go/pkg/beam/io/filesystem/gcs/gcs.go | 36 +
.../go/pkg/beam/io/filesystem/gcs/gcs_test.go | 34 +-
sdks/go/pkg/beam/io/filesystem/local/local.go | 16 +
sdks/go/pkg/beam/io/filesystem/local/local_test.go | 179 +++
sdks/go/pkg/beam/io/filesystem/memfs/memory.go | 70 +-
.../go/pkg/beam/io/filesystem/memfs/memory_test.go | 125 +-
sdks/go/pkg/beam/io/filesystem/util.go | 67 +
sdks/go/pkg/beam/io/filesystem/util_test.go | 399 ++++++
sdks/go/pkg/beam/pardo_test.go | 4 +-
sdks/go/pkg/beam/partition.go | 2 +-
sdks/go/pkg/beam/partition_test.go | 8 +
.../go/pkg/beam/runners/flag.go | 15 +-
sdks/go/pkg/beam/runners/universal/universal.go | 3 +-
sdks/go/pkg/beam/runners/vet/vet.go | 2 +-
sdks/go/pkg/beam/runners/vet/vet_test.go | 2 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 21 +-
.../go/pkg/beam/transforms/filter/distinct_test.go | 9 +
sdks/go/pkg/beam/transforms/filter/filter_test.go | 8 +
sdks/go/pkg/beam/transforms/sql/sql.go | 1 -
sdks/go/pkg/beam/transforms/sql/sqlx/sqlx.go | 2 +-
sdks/go/pkg/beam/transforms/top/top.go | 5 +-
sdks/go/pkg/beam/x/beamx/run.go | 21 +-
sdks/go/test/integration/xlang/expansion_test.go | 23 +-
sdks/java/container/Dockerfile | 3 +
sdks/java/container/boot.go | 9 +
sdks/java/container/build.gradle | 3 +-
sdks/java/container/common.gradle | 7 +-
.../container/license_scripts/dep_urls_java.yaml | 2 +-
sdks/java/core/build.gradle | 18 +-
.../java/org/apache/beam/sdk/io/TextSource.java | 2 +-
.../java/org/apache/beam/sdk/io/range/ByteKey.java | 4 +-
.../apache/beam/sdk/options/SdkHarnessOptions.java | 54 +-
.../apache/beam/sdk/schemas/AutoValueSchema.java | 54 +-
.../sdk/schemas/FieldValueTypeInformation.java | 25 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 45 +-
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 39 +-
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 20 -
.../apache/beam/sdk/schemas/SchemaTranslation.java | 2 +-
.../sdk/schemas/annotations/SchemaFieldNumber.java | 64 +
.../sdk/schemas/logicaltypes/UuidLogicalType.java | 80 ++
.../apache/beam/sdk/schemas/transforms/Select.java | 10 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 12 +-
.../sdk/testing/UsesSdkHarnessEnvironment.java} | 13 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 4 +-
.../transforms/resourcehints/ResourceHints.java | 2 +-
.../org/apache/beam/sdk/util/WeightedValue.java | 24 +
.../apache/beam/sdk/values/PCollectionViews.java | 59 -
.../apache/beam/sdk/SdkHarnessEnvironmentTest.java | 69 +
.../apache/beam/sdk/coders/CoderRegistryTest.java | 4 -
.../beam/sdk/options/SdkHarnessOptionsTest.java | 48 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 47 +
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 5 +-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 23 +-
.../beam/sdk/schemas/SchemaTranslationTest.java | 2 +-
.../sdk/schemas/logicaltypes/LogicalTypesTest.java | 16 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 30 +
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 39 +
.../apache/beam/sdk/testing/TestStreamTest.java | 8 +
.../apache/beam/sdk/transforms/CombineFnsTest.java | 9 +
.../org/apache/beam/sdk/transforms/FilterTest.java | 9 +
.../org/apache/beam/sdk/transforms/ParDoTest.java | 15 +-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 38 +-
.../GrowableOffsetRangeTrackerTest.java | 2 +-
sdks/java/expansion-service/app/build.gradle | 2 +-
sdks/java/expansion-service/build.gradle | 6 +-
.../sdk/expansion/service/ExpansionServer.java | 4 +-
.../sdk/expansion/service/ExpansionService.java | 8 +-
.../service/JavaClassLookupTransformProvider.java | 4 +-
.../expansion/service/ExpansionServiceTest.java | 2 +-
.../beam/sdk/expansion/service/ExternalTest.java | 4 +-
.../JavaClassLookupTransformProviderTest.java | 4 +-
sdks/java/extensions/arrow/build.gradle | 1 +
sdks/java/extensions/euphoria/build.gradle | 3 +-
.../core/translate/TimestampExtractTransform.java | 8 -
.../google-cloud-platform-core/build.gradle | 2 -
sdks/java/extensions/jackson/build.gradle | 1 +
sdks/java/extensions/ml/build.gradle | 2 -
.../extensions/protobuf/ProtoMessageSchema.java | 7 +-
.../extensions/protobuf/ProtoSchemaTranslator.java | 26 +-
.../protobuf/ProtoDynamicMessageSchemaTest.java | 86 ++
.../protobuf/ProtoMessageSchemaTest.java | 46 +
.../protobuf/ProtoSchemaTranslatorTest.java | 14 +
.../sdk/extensions/protobuf/TestProtoSchemas.java | 125 +-
.../src/test/proto/proto3_schema_messages.proto | 28 +
.../schemaio-expansion-service/build.gradle | 4 +-
.../ExternalSchemaIOTransformRegistrar.java | 2 +-
sdks/java/extensions/sorter/build.gradle | 7 +-
sdks/java/extensions/sql/build.gradle | 40 +-
sdks/java/extensions/sql/datacatalog/build.gradle | 11 +-
.../extensions/sql/expansion-service/build.gradle | 2 +-
sdks/java/extensions/sql/hcatalog/build.gradle | 15 +-
sdks/java/extensions/sql/jdbc/build.gradle | 2 +-
sdks/java/extensions/sql/perf-tests/build.gradle | 2 -
.../sdk/extensions/sql/impl/udaf/StringAgg.java | 59 +-
.../sdk/extensions/sql/BeamSqlDslArrayTest.java | 53 +-
.../rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java | 32 +-
sdks/java/extensions/sql/zetasql/build.gradle | 4 +-
.../zetasql/SupportedZetaSqlBuiltinFunctions.java | 6 +-
.../translation/AggregateScanConverter.java | 62 +-
.../zetasql/translation/ExpressionConverter.java | 2 +-
.../translation/SqlNullIfOperatorRewriter.java | 4 +-
.../translation/SqlOperatorMappingTable.java | 150 +-
.../sql/zetasql/translation/SqlOperators.java | 55 +-
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 62 +
sdks/java/extensions/zetasketch/build.gradle | 1 +
sdks/java/fn-execution/build.gradle | 4 +-
.../org/apache/beam/sdk/fn/CancellableQueue.java | 4 +-
.../sdk/fn/channel/AddHarnessIdInterceptor.java | 8 +-
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 18 +-
.../beam/sdk/fn/channel/SocketAddressFactory.java | 2 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 6 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer2.java | 4 +-
.../sdk/fn/data/BeamFnDataInboundObserver.java | 2 +-
...amFnDataSizeBasedBufferingOutboundObserver.java | 4 +-
...amFnDataTimeBasedBufferingOutboundObserver.java | 2 +-
.../beam/sdk/fn/data/DecodingFnDataReceiver.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 2 +-
.../org/apache/beam/sdk/fn/server/FnService.java | 6 +-
.../server/GrpcContextHeaderAccessorProvider.java | 16 +-
.../apache/beam/sdk/fn/server/GrpcFnServer.java | 2 +-
.../beam/sdk/fn/server/InProcessServerFactory.java | 8 +-
.../apache/beam/sdk/fn/server/ServerFactory.java | 20 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 4 +-
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 2 +-
.../beam/sdk/fn/stream/DirectStreamObserver.java | 4 +-
.../stream/ForwardingClientResponseObserver.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactory.java | 4 +-
.../beam/sdk/fn/stream/PrefetchableIterable.java | 3 +
.../beam/sdk/fn/stream/PrefetchableIterables.java | 53 +-
.../sdk/fn/stream/SynchronizedStreamObserver.java | 2 +-
.../fn/test/InProcessManagedChannelFactory.java | 4 +-
.../org/apache/beam/sdk/fn/test/TestStreams.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindow.java | 2 +-
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 6 +-
.../sdk/fn/channel/SocketAddressFactoryTest.java | 2 +-
.../fn/data/BeamFnDataGrpcMultiplexer2Test.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 2 +-
.../fn/data/BeamFnDataInboundObserver2Test.java | 2 +-
.../sdk/fn/data/BeamFnDataInboundObserverTest.java | 2 +-
...DataSizeBasedBufferingOutboundObserverTest.java | 2 +-
...DataTimeBasedBufferingOutboundObserverTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 2 +-
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 2 +-
.../ForwardingClientResponseObserverTest.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 4 +-
.../sdk/fn/stream/PrefetchableIterablesTest.java | 20 +
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 2 +-
sdks/java/harness/build.gradle | 10 +-
.../logging/BeamFnLoggingClientBenchmark.java | 6 +-
.../java/org/apache/beam/fn/harness/Cache.java | 3 +
.../java/org/apache/beam/fn/harness/Caches.java | 248 ++--
.../beam/fn/harness/ExternalWorkerService.java | 2 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 218 ++-
.../java/org/apache/beam/fn/harness/FnHarness.java | 7 +-
.../fn/harness/control/BeamFnControlClient.java | 4 +-
.../fn/harness/control/ProcessBundleHandler.java | 43 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 2 +-
.../harness/data/PCollectionConsumerRegistry.java | 2 +-
.../harness/data/PTransformFunctionRegistry.java | 2 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 12 +-
.../apache/beam/fn/harness/state/BagUserState.java | 59 +-
.../harness/state/BeamFnStateGrpcClientCache.java | 4 +-
.../fn/harness/state/CachingBeamFnStateClient.java | 177 ---
.../beam/fn/harness/state/FnApiStateAccessor.java | 197 ++-
.../fn/harness/state/FnApiTimerBundleTracker.java | 2 +-
.../beam/fn/harness/state/IterableSideInput.java | 47 +-
.../state/LazyCachingIteratorToIterable.java | 6 +-
.../beam/fn/harness/state/MultimapSideInput.java | 71 +-
.../beam/fn/harness/state/MultimapUserState.java | 186 ++-
.../beam/fn/harness/state/StateBackedIterable.java | 44 +-
.../fn/harness/state/StateFetchingIterators.java | 178 +--
.../beam/fn/harness/status/BeamFnStatusClient.java | 21 +-
.../stream/HarnessStreamObserverFactories.java | 2 +-
.../org/apache/beam/fn/harness/CachesTest.java | 82 +-
.../beam/fn/harness/ExternalWorkerServiceTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 38 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 8 +-
.../harness/control/BeamFnControlClientTest.java | 8 +-
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 14 +-
.../harness/logging/BeamFnLoggingClientTest.java | 16 +-
.../beam/fn/harness/state/BagUserStateTest.java | 172 ++-
.../state/BeamFnStateGrpcClientCacheTest.java | 16 +-
.../state/CachingBeamFnStateClientTest.java | 347 -----
.../fn/harness/state/FakeBeamFnStateClient.java | 2 +-
.../fn/harness/state/IterableSideInputTest.java | 96 ++
.../fn/harness/state/MultimapSideInputTest.java | 73 +-
.../fn/harness/state/MultimapUserStateTest.java | 526 +++++--
.../fn/harness/state/StateBackedIterableTest.java | 53 +-
.../harness/state/StateFetchingIteratorsTest.java | 88 +-
.../fn/harness/status/BeamFnStatusClientTest.java | 27 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 4 +-
sdks/java/io/amazon-web-services/build.gradle | 4 +-
.../apache/beam/sdk/io/aws/options/AwsOptions.java | 19 +-
.../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java | 24 +-
.../beam/sdk/io/aws/sqs/SqsMessageCoder.java | 82 ++
.../beam/sdk/io/aws/sqs/SqsUnboundedReader.java | 102 +-
.../beam/sdk/io/aws/sqs/SqsUnboundedSource.java | 29 +-
.../org/apache/beam/sdk/io/aws/ITEnvironment.java | 2 +-
.../sdk/io/aws/dynamodb/DynamoDBIOWriteTest.java | 56 +-
.../org/apache/beam/sdk/io/aws/sqs/SqsIOIT.java | 112 ++
.../beam/sdk/io/aws/sqs/SqsMessageCoderTest.java | 62 +
.../sdk/io/aws/sqs/SqsUnboundedReaderTest.java | 8 +-
.../sdk/io/aws/sqs/SqsUnboundedSourceTest.java | 3 +-
sdks/java/io/amazon-web-services2/build.gradle | 4 +-
.../apache/beam/sdk/io/aws2/coders/AwsCoders.java | 135 ++
.../beam/sdk/io/aws2/coders/package-info.java} | 10 +-
.../beam/sdk/io/aws2/options/AwsOptions.java | 19 +-
.../io/aws2/s3/DefaultS3ClientBuilderFactory.java | 27 +-
.../beam/sdk/io/aws2/sns/PublishResponseCoder.java | 48 -
.../sdk/io/aws2/sns/PublishResponseCoders.java | 129 ++
.../sdk/io/aws2/sns/SnsCoderProviderRegistrar.java | 3 +-
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 42 +-
.../apache/beam/sdk/io/aws2/sqs/SqsMessage.java | 36 +-
.../beam/sdk/io/aws2/sqs/SqsUnboundedReader.java | 103 +-
.../beam/sdk/io/aws2/sqs/SqsUnboundedSource.java | 12 -
.../org/apache/beam/sdk/io/aws2/ITEnvironment.java | 2 +-
.../beam/sdk/io/aws2/coders/AwsCodersTest.java | 69 +
.../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 54 +-
.../aws2/s3/DefaultS3ClientBuilderFactoryTest.java | 92 ++
.../sdk/io/aws2/sns/PublishResponseCodersTest.java | 92 ++
.../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 40 +
.../org/apache/beam/sdk/io/aws2/sqs/SqsIOIT.java | 121 ++
.../sdk/io/aws2/sqs/SqsUnboundedReaderTest.java | 2 +-
sdks/java/io/amqp/build.gradle | 1 +
sdks/java/io/azure/build.gradle | 2 -
sdks/java/io/bigquery-io-perf-tests/build.gradle | 2 -
sdks/java/io/cassandra/build.gradle | 3 +-
sdks/java/io/clickhouse/build.gradle | 3 +-
sdks/java/io/common/build.gradle | 1 +
sdks/java/io/contextualtextio/build.gradle | 4 +-
.../contextualtextio/ContextualTextIOSource.java | 2 +-
sdks/java/io/debezium/build.gradle | 5 +-
.../apache/beam/io/debezium/SourceRecordJson.java | 4 +-
.../elasticsearch-tests-5/build.gradle | 5 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 9 +-
.../elasticsearch-tests-6/build.gradle | 6 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 9 +-
.../elasticsearch-tests-7/build.gradle | 5 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 9 +-
.../elasticsearch-tests-common/build.gradle | 5 +-
.../elasticsearch/ElasticsearchIOTestCommon.java | 23 +-
.../io/elasticsearch/ElasticsearchIOTestUtils.java | 20 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 1 +
sdks/java/io/expansion-service/build.gradle | 2 +-
sdks/java/io/file-based-io-tests/build.gradle | 3 -
sdks/java/io/google-cloud-platform/build.gradle | 8 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 165 ++-
.../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 12 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 35 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 2 +-
.../gcp/bigquery/StorageApiFinalizeWritesDoFn.java | 8 +-
.../bigquery/StorageApiFlushAndFinalizeDoFn.java | 4 +-
.../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 2 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 20 +-
.../bigquery/StorageApiWritesShardedRecords.java | 6 +-
.../sdk/io/gcp/bigquery/StreamingWriteTables.java | 4 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 22 +-
.../beam/sdk/io/gcp/bigquery/WriteResult.java | 71 +-
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 45 +-
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 32 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 24 -
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 10 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 24 +-
.../io/gcp/spanner/SpannerTransformRegistrar.java | 2 +-
.../spanner/changestreams/ChangeStreamMetrics.java | 186 +++
.../gcp/spanner/changestreams/dao/DaoFactory.java | 136 ++
.../dao/PartitionMetadataAdminDao.java | 193 +++
.../changestreams/dao/PartitionMetadataDao.java | 470 +++++++
.../dofn/DetectNewPartitionsDoFn.java | 264 ++++
.../spanner/changestreams/dofn/package-info.java} | 10 +-
.../mapper/ChangeStreamRecordMapper.java | 324 +++++
.../changestreams/mapper/MapperFactory.java | 67 +
.../mapper/PartitionMetadataMapper.java | 98 ++
.../changestreams/mapper/package-info.java} | 13 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 14 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 1 -
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 94 +-
.../sdk/io/gcp/pubsub/PubsubIOExternalTest.java | 4 +-
.../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 21 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 19 +-
.../mapper/ChangeStreamRecordMapperTest.java | 223 +++
.../mapper/PartitionMetadataMapperTest.java | 142 ++
.../changestreams/util/TestStructMapper.java | 232 ++++
sdks/java/io/hadoop-common/build.gradle | 11 +-
sdks/java/io/hadoop-file-system/build.gradle | 2 +
sdks/java/io/hadoop-format/build.gradle | 28 +-
.../sdk/io/hadoop/format/TestRowDBWritable.java | 10 -
sdks/java/io/hbase/build.gradle | 5 +-
sdks/java/io/hcatalog/build.gradle | 25 +-
sdks/java/io/jdbc/build.gradle | 1 -
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 13 +
.../org/apache/beam/sdk/io/jdbc/LogicalTypes.java | 13 +-
.../org/apache/beam/sdk/io/jdbc/SchemaUtil.java | 34 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 37 +-
.../apache/beam/sdk/io/jdbc/OtherJdbcTypesIT.java | 147 ++
sdks/java/io/jms/build.gradle | 1 +
sdks/java/io/kafka/build.gradle | 12 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 4 +-
.../beam/sdk/io/kinesis/AmazonKinesisMock.java | 11 +-
sdks/java/io/mongodb/build.gradle | 1 -
sdks/java/io/parquet/build.gradle | 4 +-
sdks/java/io/rabbitmq/build.gradle | 3 +-
sdks/java/io/redis/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/redis/RedisIO.java | 74 +-
.../org/apache/beam/sdk/io/redis/RedisIOTest.java | 10 +-
sdks/java/io/synthetic/build.gradle | 1 +
.../apache/beam/sdk/io/thrift/ThriftSchema.java | 2 +-
.../org/apache/beam/sdk/io/xml/XmlSourceTest.java | 4 +-
sdks/java/maven-archetypes/examples/build.gradle | 2 +-
.../maven-archetypes/gcp-bom-examples/build.gradle | 2 +-
sdks/java/testing/expansion-service/build.gradle | 4 +-
sdks/java/testing/jpms-tests/build.gradle | 2 +-
sdks/java/testing/kafka-service/build.gradle | 4 +-
sdks/java/testing/load-tests/build.gradle | 3 +-
sdks/java/testing/nexmark/build.gradle | 4 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 3 +
.../apache/beam/sdk/nexmark/NexmarkQueryName.java | 1 +
.../apache/beam/sdk/nexmark/queries/Query14.java} | 37 +-
sdks/java/testing/tpcds/build.gradle | 7 +-
sdks/java/testing/watermarks/build.gradle | 2 +-
sdks/python/apache_beam/coders/row_coder_test.py | 121 +-
sdks/python/apache_beam/dataframe/convert_test.py | 8 +-
sdks/python/apache_beam/dataframe/expressions.py | 1 +
sdks/python/apache_beam/dataframe/frame_base.py | 2 +-
sdks/python/apache_beam/dataframe/frames.py | 20 +-
sdks/python/apache_beam/dataframe/frames_test.py | 36 +
.../apache_beam/dataframe/pandas_doctests_test.py | 3 -
sdks/python/apache_beam/dataframe/partitionings.py | 57 +-
.../apache_beam/dataframe/partitionings_test.py | 11 +-
sdks/python/apache_beam/dataframe/transforms.py | 79 +-
.../apache_beam/dataframe/transforms_test.py | 40 +
.../examples/cookbook/combiners_test.py | 8 +
.../apache_beam/examples/cookbook/filters_test.py | 8 +
.../examples/cookbook/multiple_output_pardo.py | 13 +
.../examples/streaming_wordcount_it_test.py | 9 +
sdks/python/apache_beam/examples/wordcount.py | 11 +-
.../apache_beam/examples/wordcount_debugging.py | 13 +
.../apache_beam/examples/wordcount_minimal.py | 12 +
.../apache_beam/examples/wordcount_minimal_test.py | 9 +
sdks/python/apache_beam/examples/wordcount_test.py | 9 +
.../apache_beam/examples/wordcount_with_metrics.py | 10 +
.../apache_beam/internal/cloudpickle_pickler.py | 9 +
.../internal/cloudpickle_pickler_test.py | 7 +
sdks/python/apache_beam/internal/metrics/metric.py | 12 +-
sdks/python/apache_beam/internal/pickler_test.py | 7 +
sdks/python/apache_beam/io/avroio.py | 25 +-
sdks/python/apache_beam/io/avroio_test.py | 15 +
.../apache_beam/io/gcp/bigquery_file_loads.py | 11 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 95 +-
sdks/python/apache_beam/io/gcp/bigtableio.py | 14 +-
.../apache_beam/io/gcp/experimental/spannerio.py | 12 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 26 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 47 +
.../apache_beam/io/gcp/pubsub_integration_test.py | 2 +-
sdks/python/apache_beam/metrics/metric.py | 5 +-
sdks/python/apache_beam/metrics/metric_test.py | 8 +
sdks/python/apache_beam/runners/common.py | 3 +-
...low_exercise_streaming_metrics_pipeline_test.py | 9 +
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/direct/direct_runner_test.py | 3 +-
.../runners/direct/transform_evaluator.py | 4 +-
.../runners/interactive/cache_manager.py | 4 +-
.../runners/interactive/cache_manager_test.py | 12 +
.../runners/interactive/caching/streaming_cache.py | 6 +-
.../interactive/caching/streaming_cache_test.py | 3 +
.../runners/interactive/interactive_runner.py | 2 +-
.../runners/interactive/pipeline_fragment_test.py | 13 +-
.../runners/portability/expansion_service_main.py | 72 +
.../runners/portability/fn_api_runner/fn_runner.py | 13 +-
.../portability/fn_api_runner/fn_runner_test.py | 10 +-
.../runners/portability/portable_metrics.py | 4 -
.../apache_beam/runners/worker/sideinputs.py | 4 +-
sdks/python/apache_beam/transforms/external.py | 10 +-
.../transforms/fully_qualified_named_transform.py | 110 ++
.../fully_qualified_named_transform_test.py | 134 ++
sdks/python/apache_beam/transforms/ptransform.py | 2 +-
sdks/python/apache_beam/transforms/resources.py | 4 +
.../apache_beam/transforms/resources_test.py | 2 +
sdks/python/apache_beam/transforms/sql_test.py | 9 +
sdks/python/apache_beam/transforms/trigger.py | 3 +-
sdks/python/apache_beam/transforms/util.py | 48 +-
sdks/python/apache_beam/typehints/schemas.py | 21 +-
sdks/python/apache_beam/typehints/typecheck.py | 4 +-
sdks/python/apache_beam/typehints/typehints.py | 16 +-
.../python/apache_beam/typehints/typehints_test.py | 1 +
sdks/python/apache_beam/utils/shared.py | 1 +
sdks/python/apache_beam/utils/shared_test.py | 11 +
.../container/py36/base_image_requirements.txt | 18 +-
.../container/py37/base_image_requirements.txt | 28 +-
.../container/py38/base_image_requirements.txt | 28 +-
sdks/python/setup.py | 2 +-
settings.gradle.kts | 3 +-
vendor/README.md | 43 +-
vendor/calcite-1_28_0/build.gradle | 22 +-
vendor/{grpc-1_36_0 => grpc-1_43_2}/build.gradle | 18 +-
website/www/site/assets/scss/_global.sass | 3 +
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.35.0.md | 160 +++
.../en/documentation/dsls/sql/walkthrough.md | 2 +-
.../content/en/documentation/programming-guide.md | 2 +-
.../java-multi-language-pipelines.md} | 11 +-
.../www/site/content/en/documentation/sdks/java.md | 4 +
.../www/site/content/en/get-started/downloads.md | 8 +
.../site/content/en/get-started/quickstart-java.md | 2 +-
.../content/en/get-started/try-beam-playground.md | 102 ++
website/www/site/data/capability_matrix.yaml | 4 +-
.../www/site/data/capability_matrix_snapshot.yaml | 4 +-
website/www/site/data/io_matrix.yaml | 2 +-
.../partials/section-menu/en/documentation.html | 1 -
.../partials/section-menu/en/get-started.html | 1 +
.../layouts/partials/section-menu/en/sdks.html | 1 +
.../www/site/layouts/shortcodes/playground.html | 19 +
website/www/site/static/.htaccess | 2 +-
913 files changed, 20510 insertions(+), 6998 deletions(-)
create mode 100644 .github/workflows/playground_deploy_examples.yml
create mode 100644
.test-infra/jenkins/job_PostCommit_Java_Examples_Direct.groovy
create mode 100644
.test-infra/jenkins/job_PostCommit_Java_Examples_Flink.groovy
create mode 100644
.test-infra/jenkins/job_PostCommit_Java_Examples_Spark.groovy
copy playground/frontend/lib/modules/output/components/output.dart =>
.test-infra/jenkins/job_PreCommit_Java_PortableValidatesRunner_Flink_Docker.groovy
(51%)
create mode 100644 .test-infra/tools/README.md
create mode 100644 .test-infra/tools/python_installer.sh
rename
buildSrc/src/main/groovy/org/apache/beam/gradle/{GrpcVendoring_1_36_0.groovy =>
GrpcVendoring_1_43_2.groovy} (83%)
create mode 100644 examples/multi-language/README.md
copy {sdks/java/expansion-service/app => examples/multi-language}/build.gradle
(57%)
create mode 100644 examples/multi-language/python/addprefix.py
create mode 100644 examples/multi-language/python/javacount.py
create mode 100644 examples/multi-language/python/javadatagenerator.py
copy
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaCount.java
(64%)
copy
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaCountBuilder.java
(59%)
copy
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaCountConfiguration.java
(78%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/util/WeightedValue.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaCountRegistrar.java
(55%)
create mode 100644
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaDataGenerator.java
copy sdks/java/core/src/main/java/org/apache/beam/sdk/util/WeightedValue.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefix.java
(53%)
copy
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefixBuilder.java
(60%)
copy playground/frontend/lib/main.dart =>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefixConfiguration.java
(72%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/util/WeightedValue.java
=>
examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefixRegistrar.java
(55%)
create mode 100755 playground/backend/containers/go/entrypoint.sh
create mode 100755 playground/backend/containers/java/entrypoint.sh
create mode 100755 playground/backend/containers/python/entrypoint.sh
rename playground/backend/internal/{preparators/go_preparators.go =>
preparers/go_preparers.go} (50%)
rename playground/backend/internal/{preparators/go_preparators_test.go =>
preparers/go_preparers_test.go} (86%)
rename playground/backend/internal/{preparators/java_preparators.go =>
preparers/java_preparers.go} (67%)
rename playground/backend/internal/{preparators/java_preparators_test.go =>
preparers/java_preparers_test.go} (79%)
copy playground/backend/internal/{preparators/preparator.go =>
preparers/preparer.go} (50%)
rename playground/backend/internal/{preparators/python_preparators.go =>
preparers/python_preparers.go} (78%)
rename playground/backend/internal/{preparators/python_preparators_test.go =>
preparers/python_preparers_test.go} (86%)
create mode 100644 playground/frontend/CONTRIBUTE.md
create mode 100644 playground/frontend/assets/thumb_down_filled.svg
create mode 100644 playground/frontend/assets/thumb_up_filled.svg
create mode 100644 playground/frontend/l10n.yaml
copy
playground/frontend/lib/{modules/output/components/output_header/output_tabs.dart
=> components/loading_indicator/loading_indicator.dart} (71%)
copy
playground/frontend/lib/{modules/output/components/output_header/output_tabs.dart
=> config/locale.dart} (68%)
create mode 100644 playground/frontend/lib/l10n/app_en.arb
copy playground/frontend/lib/{main.dart => l10n/l10n.dart} (78%)
copy
playground/frontend/lib/modules/examples/components/{filter/type_filter.dart =>
description_popover/description_popover.dart} (57%)
create mode 100644
playground/frontend/lib/modules/examples/components/description_popover/description_popover_button.dart
create mode 100644
playground/frontend/lib/modules/output/components/output_header/output_tab.dart
copy playground/frontend/lib/{modules/output/components/output_area.dart =>
pages/playground/components/close_listener.dart} (58%)
copy playground/frontend/lib/{main.dart =>
pages/playground/components/close_listener_nonweb.dart} (78%)
copy playground/frontend/lib/{main.dart =>
pages/playground/states/feedback_state.dart} (78%)
create mode 100644 playground/infrastructure/helm/Chart.yaml
create mode 100644 playground/infrastructure/helm/templates/_helpers.tpl
create mode 100644
playground/infrastructure/helm/templates/deployment-backend-go.yaml
create mode 100644
playground/infrastructure/helm/templates/deployment-backend-java.yaml
create mode 100644
playground/infrastructure/helm/templates/deployment-backend-python.yaml
create mode 100644 playground/infrastructure/helm/templates/service-go.yaml
create mode 100644 playground/infrastructure/helm/templates/service-java.yaml
create mode 100644 playground/infrastructure/helm/templates/service-python.yaml
create mode 100644 playground/infrastructure/helm/values.yaml
create mode 100644 playground/infrastructure/proxy/allow_list.py
create mode 100644 playground/infrastructure/proxy/allow_list_proxy.py
copy playground/terraform/applications/{backend-python =>
backend-java}/main.tf (70%)
rename playground/terraform/applications/{backend =>
backend-java}/variables.tf (90%)
copy playground/terraform/applications/{backend => backend-router}/main.tf
(69%)
copy playground/terraform/applications/{backend-go =>
backend-router}/variables.tf (100%)
rename playground/terraform/{applications/backend => modules/gke}/main.tf (57%)
copy playground/{infrastructure/requirements.txt =>
terraform/modules/gke/output.tf} (80%)
copy playground/terraform/{applications/frontend => modules/gke}/variables.tf
(71%)
create mode 100644 sdks/go/examples/xlang/sql/sql.go
copy playground/backend/internal/utils/system_utils_test.go =>
sdks/go/pkg/beam/io/filesystem/filesystem_test.go (65%)
copy playground/backend/internal/utils/system_utils_test.go =>
sdks/go/pkg/beam/io/filesystem/gcs/gcs_test.go (63%)
create mode 100644 sdks/go/pkg/beam/io/filesystem/local/local_test.go
create mode 100644 sdks/go/pkg/beam/io/filesystem/util_test.go
rename playground/backend/internal/preparators/preparator.go =>
sdks/go/pkg/beam/runners/flag.go (68%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaFieldNumber.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/UuidLogicalType.java
copy
sdks/java/{fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
core/src/main/java/org/apache/beam/sdk/testing/UsesSdkHarnessEnvironment.java}
(76%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/SdkHarnessEnvironmentTest.java
delete mode 100644
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/CachingBeamFnStateClient.java
delete mode 100644
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/CachingBeamFnStateClientTest.java
create mode 100644
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/state/IterableSideInputTest.java
create mode 100644
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsMessageCoder.java
create mode 100644
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsIOIT.java
create mode 100644
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsMessageCoderTest.java
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/coders/AwsCoders.java
copy
sdks/java/{fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/coders/package-info.java}
(78%)
delete mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
create mode 100644
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoders.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/coders/AwsCodersTest.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/s3/DefaultS3ClientBuilderFactoryTest.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCodersTest.java
create mode 100644
sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/SqsIOIT.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/ChangeStreamMetrics.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/DaoFactory.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataAdminDao.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataDao.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/DetectNewPartitionsDoFn.java
copy
sdks/java/{fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/package-info.java}
(78%)
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/mapper/ChangeStreamRecordMapper.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/mapper/MapperFactory.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/mapper/PartitionMetadataMapper.java
copy
sdks/java/{fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/PrefetchableIterable.java
=>
io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/mapper/package-info.java}
(75%)
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/mapper/ChangeStreamRecordMapperTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/mapper/PartitionMetadataMapperTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/util/TestStructMapper.java
create mode 100644
sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/OtherJdbcTypesIT.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/util/WeightedValue.java
=>
testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/queries/Query14.java}
(52%)
create mode 100644
sdks/python/apache_beam/runners/portability/expansion_service_main.py
create mode 100644
sdks/python/apache_beam/transforms/fully_qualified_named_transform.py
create mode 100644
sdks/python/apache_beam/transforms/fully_qualified_named_transform_test.py
rename vendor/{grpc-1_36_0 => grpc-1_43_2}/build.gradle (71%)
create mode 100644 website/www/site/content/en/blog/beam-2.35.0.md
rename website/www/site/content/en/documentation/{patterns/cross-language.md
=> sdks/java-multi-language-pipelines.md} (93%)
create mode 100644
website/www/site/content/en/get-started/try-beam-playground.md
create mode 100644 website/www/site/layouts/shortcodes/playground.html
