This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a change to branch direct_runner_bug
in repository https://gitbox.apache.org/repos/asf/beam.git
from aeb7661f272 Create empty ListBuffer when buffer is none
add 22c4cc216e7 Replace empty buffer with a List/GroupBuffer
add 946324ae137 set _isPlaygroundShown after _currentUnitContent (#27242)
add bac98cf2075 add gcloud auth and push to registry on merge PR (#27341)
add e8d8043f441 Upgrade obsolete Avro plugin to 0.16.0 (#27232)
add 2c30acfe75b Consider beam_services option when determining the
expansion service type
add c47438802d9 Merge pull request #27345: Consider beam_services option
when determining the expansion service type
add 4d38d953788 updated go precommit with new condition, re-run action,
pull_req_targ… (#27321)
add e00e19c0887 adds GitHub CLI to runner image (#27342)
add 619e41c7c0a Bump github.com/proullon/ramsql in /sdks (#27337)
add 54adbcb0b9c Bump actions/setup-python from 3 to 4 (#27336)
add 8b8fdcaf71c Verify script improvements (#27309)
add 3457434de74 fix repo and add paths to push trigger (#27357)
add 4a7a2579c70 Update Storage Write API Autosharding documentation
(#27316)
add dc5bdb0d1e0 Bump golang.org/x/text from 0.10.0 to 0.11.0 in /sdks
(#27351)
add a4141189b83 Updates PythonExternalTransform to support IPv6 addresses
add aa6f0a364db Merge pull request #27332: Updates PythonExternalTransform
to support IPv6 addresses
add 35bdebc4c1b wrong path fix (#27359)
add 1842dd2f07a Respect BigQuery insert byte size limits (#24979)
add 0ce0d9acead Keep roles for gcp SA updated (#27306)
add cb8adf123e7 adding glcoud cli and kubectl (#27358)
add 2a1276544f6 Add Windmill support for MultimapState (#23492)
add 24fa67f958b Fix windmill proto (#27361)
add 9a11e28ce79 Use DOCKERHUB_USER instead of username (#27362)
add 15a54a00ac0 [Go SDK][Prism] Improve failed pipeline handling (#27297)
add 79fcfc6fcb5 Fix flakiness for BeamFnLoggingClientTest (#27308)
add 5bc80aa2e6d Inherit Generic for TimestampedValue (#26290)
add c28fa0639c2 Update 2.50 release notes to include new Kafka
topicPattern feature (#27367)
add a6760153543 Bump golang.org/x/oauth2 from 0.9.0 to 0.10.0 in /sdks
(#27376)
add cafb3da0523 Bump @grpc/grpc-js from 1.6.7 to 1.8.8 in /sdks/typescript
(#27372)
add 556d93365a1 Update dataflow container versions (#27381)
add 44f165fedf6 Update python options in runners/spark.md. (#27334)
add a5873792a36 Fix mismatched section title in Release Guide (#27383)
add b529efecc9d MLTransform (#26795)
add 9e6420d9f90 Arc additional pools (#27369)
add 7eaef18bcbf Address failing BT read tests in #27319 (#27322)
add ccf6544a636 Automatically use Docker Compose based transform service
for PythonExternalTransform when needed.
add f19c384af1f Merge pull request #27365: Automatically use Docker
Compose based transform service for PythonExternalTransform when needed.
add a12577ae68a python precommit jobs and readme (#27379)
add a0c9aa2e4f7 Bump google.golang.org/grpc from 1.56.1 to 1.56.2 in /sdks
(#27387)
add 71cb3ca2403 Bump google.golang.org/api from 0.129.0 to 0.130.0 in
/sdks (#27377)
add 7ce2d00464f Revert "Replace StorageV1 client with GCS client (#25965)"
(#27388)
add 4ba5e43a789 Fix CombineGlobally with GlobalWindows (#26922)
add f643872f0b7 Restore trailing slash accidentally removed in
https://github.com/apache/beam/pull/25665
add dbf9ecce8f5 Merge pull request #27386 from Restore trailing slash in
Dockerfile.
add cbfeb2e5ec7 fixes rerun call in matrix jobs (#27392)
add 79d210c7f01 added beam_PreCommit_Java_Examples_Dataflow job (#27233)
add 4026d9e3482 Migrate "beam_PreCommit_Typescript" Jenkins job to Github
Actions (#27141)
add c06247bf79e Fix staged RC source name in GA (#27394)
add e6cccdef7d2 Ensure CancellableQueue doesn't keep references to
logically removed objects. (#27403)
add ed7cbcaa0f9 temporarily disable PreCommit_Java_Examples_Dataflow.yml
(#27420)
add e809912fa30 Fix a typo in the GenAI notebook (#27418)
add 88e635a2b0a [Python] Make BigQuery load job names deterministic for
retry resilience (#27384)
add a576094632f Merge remote-tracking branch 'upstream/master' into
direct_runner_bug
No new revisions were added by this update.
Summary of changes:
.github/actions/rerun-job-action/action.yml | 7 +
.../gh-actions-self-hosted-runners/arc/README.md | 44 +-
.../arc/config/arc_autoscaler.tpl | 4 +-
.../arc/config/arc_deployment.tpl | 35 +-
.../arc/environments/beam.env | 42 +-
.github/gh-actions-self-hosted-runners/arc/gke.tf | 53 +-
.github/gh-actions-self-hosted-runners/arc/helm.tf | 2 +-
.../arc/images/Dockerfile | 14 +-
.../arc/kubernetes.tf | 30 +-
.../arc/variables.tf | 87 +-
.github/workflows/README.md | 30 +
.github/workflows/beam_PreCommit_Go.yml | 78 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 109 +
.github/workflows/beam_PreCommit_Typescript.yml | 81 +
.github/workflows/build_release_candidate.yml | 6 +-
.github/workflows/build_runner_image.yml | 38 +-
.github/workflows/job_PreCommit_Python.yml | 94 +
.../job_PreCommit_PythonAutoformatter.yml | 78 +
.../workflows/job_PreCommit_Python_Dataframes.yml | 92 +
.../workflows/job_PreCommit_Python_DockerBuild.yml | 94 +
.../workflows/job_PreCommit_Python_Examples.yml | 94 +
.github/workflows/job_PreCommit_Python_Runners.yml | 95 +
.../workflows/job_PreCommit_Python_Transforms.yml | 95 +
.github/workflows/job_Precommit_PythonLint.yml | 81 +
.../workflows/job_Precommit_Python_Coverage.yml | 84 +
CHANGES.md | 1 +
CI.md | 3 +
buildSrc/build.gradle.kts | 7 +-
.../beam-ml/run_inference_generative_ai.ipynb | 2 +-
.../frontend/lib/pages/tour/state.dart | 2 +-
ownership/JAVA_DEPENDENCY_OWNERS.yaml | 10 +-
.../assets/symbols/python.g.yaml | 8 +
release/src/main/scripts/mass_comment.py | 2 +-
release/src/main/scripts/verify_release_build.sh | 12 +-
.../beam/runners/dataflow/DataflowRunner.java | 11 +-
.../dataflow/worker/WindmillStateInternals.java | 674 ++++-
.../dataflow/worker/WindmillStateReader.java | 319 ++-
.../worker/WindmillStateInternalsTest.java | 1225 +++++++++
.../dataflow/worker/WindmillStateReaderTest.java | 723 +++++-
.../worker/windmill/src/main/proto/windmill.proto | 82 +-
sdks/go.mod | 20 +-
sdks/go.sum | 40 +-
.../beam/runners/prism/internal/execute_test.go | 14 +
.../beam/runners/prism/internal/jobservices/job.go | 7 +-
.../prism/internal/jobservices/management.go | 17 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 12 +-
.../pkg/beam/runners/prism/internal/testdofns.go | 5 +
.../beam/runners/prism/internal/testdofns_test.go | 1 +
.../beam/runners/prism/internal/worker/bundle.go | 7 +
.../beam/runners/prism/internal/worker/worker.go | 26 +-
.../src/main/resources/beam/suppressions.xml | 1 +
.../extensions/python/PythonExternalTransform.java | 69 +-
.../org/apache/beam/sdk/fn/CancellableQueue.java | 26 +-
.../apache/beam/sdk/fn/CancellableQueueTest.java | 28 +
.../harness/logging/BeamFnLoggingClientTest.java | 50 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 2 +-
.../BigtableReadSchemaTransformProviderIT.java | 42 +-
.../launcher/TransformServiceLauncher.java | 8 +-
.../examples/complete/game/user_score.py | 1 -
.../examples/ml_transform/ml_transform_basic.py | 118 +
sdks/python/apache_beam/internal/gcp/auth.py | 7 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 74 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 21 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 56 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 36 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 2 +-
sdks/python/apache_beam/io/gcp/gcsfilesystem.py | 33 +-
.../apache_beam/io/gcp/gcsfilesystem_test.py | 17 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 625 +++--
.../apache_beam/io/gcp/gcsio_integration_test.py | 149 +-
sdks/python/apache_beam/io/gcp/gcsio_overrides.py | 55 +
sdks/python/apache_beam/io/gcp/gcsio_test.py | 886 +++++--
.../io/gcp/internal/clients/storage/__init__.py | 33 +
.../internal/clients/storage/storage_v1_client.py | 1517 +++++++++++
.../clients/storage/storage_v1_messages.py | 2714 ++++++++++++++++++++
.../python/apache_beam/ml/transforms}/__init__.py | 0
sdks/python/apache_beam/ml/transforms/base.py | 165 ++
sdks/python/apache_beam/ml/transforms/base_test.py | 246 ++
sdks/python/apache_beam/ml/transforms/handlers.py | 410 +++
.../apache_beam/ml/transforms/handlers_test.py | 355 +++
sdks/python/apache_beam/ml/transforms/tft.py | 440 ++++
sdks/python/apache_beam/ml/transforms/tft_test.py | 395 +++
sdks/python/apache_beam/ml/transforms/utils.py | 56 +
.../options/pipeline_options_validator_test.py | 1 -
.../runners/dataflow/internal/apiclient.py | 52 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../apache_beam/runners/interactive/utils.py | 26 +-
.../apache_beam/runners/interactive/utils_test.py | 41 +-
.../runners/portability/fn_api_runner/fn_runner.py | 15 +-
.../runners/portability/sdk_container_builder.py | 41 +-
.../apache_beam/transforms/combiners_test.py | 44 +
sdks/python/apache_beam/transforms/core.py | 17 +-
sdks/python/apache_beam/transforms/external.py | 8 +
.../transforms/timestamped_value_type_test.py | 139 +
sdks/python/apache_beam/transforms/window.py | 9 +-
sdks/python/apache_beam/typehints/typecheck.py | 18 +-
sdks/python/apache_beam/utils/subprocess_server.py | 7 +-
sdks/python/mypy.ini | 3 +
sdks/python/scripts/generate_pydoc.sh | 2 +-
sdks/python/setup.py | 1 -
sdks/python/test-suites/tox/py38/build.gradle | 4 +
sdks/python/tox.ini | 6 +
sdks/typescript/container/Dockerfile | 2 +-
sdks/typescript/package-lock.json | 349 ++-
sdks/typescript/package.json | 2 +-
.../site/content/en/contribute/release-guide.md | 2 +-
.../documentation/io/built-in/google-bigquery.md | 11 +-
.../site/content/en/documentation/runners/spark.md | 2 +-
108 files changed, 13341 insertions(+), 799 deletions(-)
create mode 100644 .github/workflows/README.md
create mode 100644 .github/workflows/beam_PreCommit_Java_Examples_Dataflow.yml
create mode 100644 .github/workflows/beam_PreCommit_Typescript.yml
create mode 100644 .github/workflows/job_PreCommit_Python.yml
create mode 100644 .github/workflows/job_PreCommit_PythonAutoformatter.yml
create mode 100644 .github/workflows/job_PreCommit_Python_Dataframes.yml
create mode 100644 .github/workflows/job_PreCommit_Python_DockerBuild.yml
create mode 100644 .github/workflows/job_PreCommit_Python_Examples.yml
create mode 100644 .github/workflows/job_PreCommit_Python_Runners.yml
create mode 100644 .github/workflows/job_PreCommit_Python_Transforms.yml
create mode 100644 .github/workflows/job_Precommit_PythonLint.yml
create mode 100644 .github/workflows/job_Precommit_Python_Coverage.yml
create mode 100644
sdks/python/apache_beam/examples/ml_transform/ml_transform_basic.py
create mode 100644 sdks/python/apache_beam/io/gcp/gcsio_overrides.py
create mode 100644
sdks/python/apache_beam/io/gcp/internal/clients/storage/__init__.py
create mode 100644
sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py
create mode 100644
sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py
copy {examples/notebooks/notebook_test_scripts =>
sdks/python/apache_beam/ml/transforms}/__init__.py (100%)
create mode 100644 sdks/python/apache_beam/ml/transforms/base.py
create mode 100644 sdks/python/apache_beam/ml/transforms/base_test.py
create mode 100644 sdks/python/apache_beam/ml/transforms/handlers.py
create mode 100644 sdks/python/apache_beam/ml/transforms/handlers_test.py
create mode 100644 sdks/python/apache_beam/ml/transforms/tft.py
create mode 100644 sdks/python/apache_beam/ml/transforms/tft_test.py
create mode 100644 sdks/python/apache_beam/ml/transforms/utils.py
create mode 100644
sdks/python/apache_beam/transforms/timestamped_value_type_test.py
