This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a change to branch users/damccorm/test-revert
in repository https://gitbox.apache.org/repos/asf/beam.git
from 2f8f498b52f Test #35816
add 111c83f9fc6 fix(bigquery): use correct project for temp dataset
operations (#35817)
add 4dfdb57bc12 Re-enable parallel for KafkaIO precommit (#35814)
add 68c09915799 Use transformers 4.55.0
add 2af502b5aad Update transformers version
add 18ce59be33e Fix PostCommit Python Dependency job (#35819)
add f30aa9655d2 Fix JdbcIO.read log spam when autocommit disabled (#35825)
add 55febcbf6fc Add javadoc to JdbcIO.PreparedStatementSetter (#35796)
add 86931222e6f contributor-docs: add milvus enrichment handler doc
(#35829)
add 64625a753ee Added NewFeatures Line to Changes.Md (#35798)
add 7299b2525bf Adds support for updating Schema state fields. (#35229)
add a0abcbe76b0 Install yarn via APT (#35840)
add bdf7912cd89 docs: update CHANGES.md with BigQueryIO bugfix (#35827)
add 14f887037d8 fix commons-lang3 vulnerability (#35824)
add 932f078e340 [Java Dataflow Streaming] Add support to
AbstractWindmillStream to transition between physical streams within the same
logical stream (#35523)
add ea8e08ba561 Secret management service (#35524)
add ddaae131603 Fix API doc index page broken format and link (#35845)
add 112791da919 feature added for the duration/timeout in test_pipeline
(#35831)
add 4509ecd3ef3 Adding release-2.67.0-postrelease to protected branches in
.asf.yaml
add 0d87a9cc177 use gemma2-2b (#32603) (#35842)
add 55f80c8bdbb Update Beam website to release 2.67.0 (#35759)
add df2c5fcb65c vLLM Gemma Benchmarks (GPU) (#35809)
add 0d320f9c68c Fix race condition in FnAPIDoFnRunner.java (#35757)
add 962bc6ebb3a Cloudsql embeddings colab. (#35695)
add 9ce7ff831c1 Bump DF Java Python container tags (#35846)
add 5b8ddd421ca Update Republish Released Docker Containers (#35853)
add c371236d0a8 JUnit5 support (#35688)
add ad2371e51ed Update code-change-guide add back build SDK container
(#35855)
add 2c2211edb33 Add Action to generate ManagedIO doc PR (#35828)
add 576c508245e Fix #[35617] Windows compatibility for YAML
JavaJarProvider (#35803)
add 059d00b1291 Fix ml transform (#35774)
add 5184b949dca fix(test): improve redis container startup reliability in
tests (#35857)
add cb35ff2eb10 Fix exception escaping construction is :runners:local-java
(caught by spotbugs)
add b63fb6fd662 Merge pull request #35335: Adjust runners/local-java so
constructor does not throw exception
add 0e47f80865e Deprecate Python native SpannerIO, with exposed bug fix
(#35859)
add 4da46b0ae5b Fix duplicated prefix when handing an absolute jar path
for provider. (#35849)
add 02842e46cd5 Bump golang.org/x/net from 0.42.0 to 0.43.0 in /sdks
(#35818)
add 901058be0d8 Bump cloud.google.com/go/pubsub from 1.49.0 to 1.50.0 in
/sdks (#35807)
add 726c96e8a8b Add mask for PyPI API token
add 1a000dae094 Add mask for PyPI API token (#35863)
add bb9ab00de97 Fix a bug in cogbk for not using registered coder. (#35862)
add 3f491411ea8 Fix website license formatting (#35866)
add a0be7bfb635 Update Release Guide to add managed-io doc info (#35861)
add 9a15261c00d Adding Timesfm anomaly detection colab example (#35844)
add afe94ad8c6f Add helper functions for computing and using code object
identifiers to facilitate more efficient pickling of lambdas. (#35656)
add b2a3ea41120 Fix installing certificates
add b53f31f8625 Fix Deploy-Update-Playground-environment-stg Cloud Build
trigger (#35871)
add 14bb49c50aa Bump google.golang.org/api from 0.243.0 to 0.247.0 in
/sdks (#35879)
add 7ecbb2a7037 Pin a mocking test to DirectRunner (#35883)
add d2e1129ff13 [2/3] sdks/python: enrich data with Milvus Search [Vector,
Keyword, Hybrid] (#35577)
new 07d7dc7bd6f Merge branch 'master' of https://github.com/apache/beam
into users/damccorm/test-revert
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 1 +
.../actions/build-push-docker-action/action.yml | 45 +
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
.../beam_PostCommit_Yaml_Xlang_Direct.json | 2 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 26 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 2 +-
.github/workflows/build_release_candidate.yml | 71 +-
.github/workflows/finalize_release.yml | 6 +-
...Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt | 36 +
.../republish_released_docker_containers.yml | 2 +-
.test-infra/tools/refresh_looker_metrics.py | 1 +
CHANGES.md | 33 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
contributor-docs/code-change-guide.md | 23 +-
contributor-docs/discussion-docs/2025.md | 17 +-
contributor-docs/release-guide.md | 15 +-
.../anomaly_detection_timesfm.ipynb | 2712 ++++++++++++++++++++
...loudsql_mysql_product_catalog_embeddings.ipynb} | 557 ++--
infra/keys/README.md | 93 +
.../kustomization.yaml => infra/keys/config.yaml | 34 +-
infra/keys/gcp_logger.py | 153 ++
infra/keys/keys.py | 390 +++
.../redis-values.yaml => infra/keys/keys.yaml | 24 +-
infra/{iam => keys}/requirements.txt | 7 +-
infra/keys/secret_manager.py | 787 ++++++
infra/keys/service_account.py | 425 +++
infra/keys/test_secret_manager.py | 839 ++++++
infra/keys/test_service_account.py | 601 +++++
playground/backend/containers/router/Dockerfile | 19 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../util/RowCoderCloudObjectTranslator.java | 1 -
.../util/SchemaCoderCloudObjectTranslator.java | 42 -
.../windmill/client/AbstractWindmillStream.java | 314 ++-
.../worker/windmill/client/WindmillStream.java | 4 -
.../windmill/client/grpc/GrpcCommitWorkStream.java | 40 +-
.../client/grpc/GrpcDirectGetWorkStream.java | 25 +-
.../windmill/client/grpc/GrpcGetDataStream.java | 51 +-
.../windmill/client/grpc/GrpcGetWorkStream.java | 25 +-
.../client/grpc/GrpcGetWorkerMetadataStream.java | 27 +-
.../client/grpc/GrpcWindmillStreamFactory.java | 108 +-
.../dataflow/worker/FakeWindmillServer.java | 6 +-
.../worker/StreamingDataflowWorkerTest.java | 10 +-
.../client/AbstractWindmillStreamTest.java | 26 +-
.../client/TriggeredScheduledExecutorService.java | 140 +
.../client/grpc/FakeWindmillGrpcService.java | 47 +-
.../client/grpc/GrpcCommitWorkStreamTest.java | 674 +++++
.../client/grpc/GrpcGetDataStreamTest.java | 831 +++++-
.../apache/beam/runners/local/StructuralKey.java | 32 +-
sdks/go.mod | 27 +-
sdks/go.sum | 58 +-
sdks/java/core/build.gradle | 8 +
.../apache/beam/sdk/schemas/SchemaTranslation.java | 46 +
.../org/apache/beam/sdk/testing/TestPipeline.java | 35 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 11 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 22 +-
.../harness/control/ExecutionStateSamplerTest.java | 8 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 37 +-
sdks/java/javadoc/overview.html | 4 -
.../sketching => testing/junit}/build.gradle | 36 +-
.../beam/sdk/testing/TestPipelineExtension.java | 213 ++
.../org/apache/beam/sdk/testing}/package-info.java | 5 +-
.../testing/TestPipelineExtensionAdvancedTest.java | 88 +
.../sdk/testing/TestPipelineExtensionTest.java | 56 +
.../examples/inference/vllm_gemma_batch.py | 130 +
.../apache_beam/internal/code_object_pickler.py | 461 ++++
.../internal/code_object_pickler_test.py | 574 +++++
.../internal/{metrics => test_data}/__init__.py | 4 +-
.../__init__.py => internal/test_data/module_1.py} | 12 +-
.../test_data/module_1_class_added.py} | 23 +-
.../test_data/module_1_function_added.py} | 22 +-
.../test_data/module_1_global_variable_added.py} | 15 +-
.../test_data/module_1_lambda_variable_added.py} | 14 +-
.../test_data/module_1_local_variable_added.py} | 14 +-
.../test_data/module_1_local_variable_removed.py} | 11 +-
.../test_data/module_1_nested_function_2_added.py} | 21 +-
.../test_data/module_1_nested_function_added.py} | 20 +-
.../apache_beam/internal/test_data/module_2.py | 82 +
.../internal/test_data/module_2_modified.py | 92 +
.../__init__.py => internal/test_data/module_3.py} | 11 +-
.../test_data/module_3_modified.py} | 11 +-
.../test_data/module_with_default_argument.py} | 9 +-
.../apache_beam/io/gcp/bigquery_read_internal.py | 20 +-
.../io/gcp/bigquery_read_internal_test.py | 170 ++
sdks/python/apache_beam/io/gcp/bigquery_test.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 14 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 21 +
.../apache_beam/io/gcp/experimental/spannerio.py | 12 +-
.../apache_beam/io/requestresponse_it_test.py | 5 +-
.../python/apache_beam/metrics/monitoring_infos.py | 4 +-
sdks/python/apache_beam/ml/anomaly/transforms.py | 12 +-
.../ml/inference/test_resources/vllm.dockerfile | 63 +-
.../{vllm.dockerfile => vllm.dockerfile.old} | 2 +-
...equirements.txt => vllm_tests_requirements.txt} | 4 +-
.../apache_beam/ml/rag/enrichment/milvus_search.py | 8 +-
.../ml/rag/enrichment/milvus_search_it_test.py | 54 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
...ment_benchmarks.py => vllm_gemma_benchmarks.py} | 22 +-
.../apache_beam/testing/load_tests/build.gradle | 11 +-
sdks/python/apache_beam/testing/test_pipeline.py | 9 +-
.../enrichment_handlers/bigquery_it_test.py | 4 +-
.../enrichment_handlers/bigtable_it_test.py | 5 +-
.../vertex_ai_feature_store_it_test.py | 7 +-
sdks/python/apache_beam/transforms/util.py | 5 +-
sdks/python/apache_beam/transforms/util_test.py | 65 +-
.../apache_beam/yaml/tests/runinference.yaml | 2 +-
sdks/python/apache_beam/yaml/yaml_mapping.py | 1 +
sdks/python/apache_beam/yaml/yaml_ml.py | 57 +-
sdks/python/apache_beam/yaml/yaml_ml_test.py | 87 +
sdks/python/apache_beam/yaml/yaml_provider.py | 36 +-
.../apache_beam/yaml/yaml_provider_unit_test.py | 66 +
.../container/py310/base_image_requirements.txt | 52 +-
.../container/py310/ml_image_requirements.txt | 62 +-
.../container/py311/base_image_requirements.txt | 54 +-
.../container/py311/ml_image_requirements.txt | 64 +-
.../container/py312/base_image_requirements.txt | 54 +-
.../container/py312/ml_image_requirements.txt | 64 +-
.../container/py313/base_image_requirements.txt | 129 +-
.../container/py39/base_image_requirements.txt | 52 +-
.../container/py39/ml_image_requirements.txt | 60 +-
sdks/python/gen_managed_doc.py | 3 +
sdks/python/setup.py | 9 +-
sdks/python/test-suites/dataflow/common.gradle | 2 +-
sdks/python/tox.ini | 6 +-
settings.gradle.kts | 1 +
website/Dockerfile | 19 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.67.0.md | 73 +
.../site/content/en/documentation/io/managed-io.md | 3 +
.../www/site/content/en/get-started/downloads.md | 19 +-
website/www/site/content/en/performance/_index.md | 1 +
.../en/performance/vllmgemmabatchtesla/_index.md | 43 +
website/www/site/data/performance.yaml | 16 +
132 files changed, 11827 insertions(+), 1245 deletions(-)
create mode 100644 .github/actions/build-push-docker-action/action.yml
create mode 100644
.github/workflows/load-tests-pipeline-options/beam_Inference_Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt
create mode 100644
examples/notebooks/beam-ml/anomaly_detection/anomaly_detection_timesfm.ipynb
copy
examples/notebooks/beam-ml/{cloudsql_postgres_product_catalog_embeddings.ipynb
=> cloudsql_mysql_product_catalog_embeddings.ipynb} (83%)
create mode 100644 infra/keys/README.md
copy
.test-infra/mock-apis/infrastructure/kubernetes/refresher/overlays/echo-10-per-1s-quota/kustomization.yaml
=> infra/keys/config.yaml (50%)
create mode 100644 infra/keys/gcp_logger.py
create mode 100644 infra/keys/keys.py
copy .test-infra/mock-apis/infrastructure/kubernetes/redis/redis-values.yaml
=> infra/keys/keys.yaml (52%)
copy infra/{iam => keys}/requirements.txt (88%)
create mode 100644 infra/keys/secret_manager.py
create mode 100644 infra/keys/service_account.py
create mode 100644 infra/keys/test_secret_manager.py
create mode 100644 infra/keys/test_service_account.py
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/TriggeredScheduledExecutorService.java
copy sdks/java/{extensions/sketching => testing/junit}/build.gradle (56%)
create mode 100644
sdks/java/testing/junit/src/main/java/org/apache/beam/sdk/testing/TestPipelineExtension.java
copy
{examples/java/cdap/hubspot/src/main/java/org/apache/beam/examples/complete/cdap/hubspot/utils
=>
sdks/java/testing/junit/src/main/java/org/apache/beam/sdk/testing}/package-info.java
(89%)
create mode 100644
sdks/java/testing/junit/src/test/java/org/apache/beam/sdk/testing/TestPipelineExtensionAdvancedTest.java
create mode 100644
sdks/java/testing/junit/src/test/java/org/apache/beam/sdk/testing/TestPipelineExtensionTest.java
create mode 100644
sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py
create mode 100644 sdks/python/apache_beam/internal/code_object_pickler_test.py
copy sdks/python/apache_beam/internal/{metrics => test_data}/__init__.py (89%)
copy sdks/python/apache_beam/{tools/__init__.py =>
internal/test_data/module_1.py} (76%)
copy
sdks/python/apache_beam/{testing/benchmarks/cloudml/cloudml_benchmark_constants_lib.py
=> internal/test_data/module_1_class_added.py} (68%)
copy
sdks/python/apache_beam/{testing/benchmarks/cloudml/cloudml_benchmark_constants_lib.py
=> internal/test_data/module_1_function_added.py} (70%)
copy sdks/python/apache_beam/{runners/direct/__init__.py =>
internal/test_data/module_1_global_variable_added.py} (73%)
copy sdks/python/apache_beam/{runners/direct/__init__.py =>
internal/test_data/module_1_lambda_variable_added.py} (71%)
copy sdks/python/apache_beam/{runners/direct/__init__.py =>
internal/test_data/module_1_local_variable_added.py} (72%)
copy sdks/python/apache_beam/{tools/__init__.py =>
internal/test_data/module_1_local_variable_removed.py} (79%)
copy
sdks/python/apache_beam/{testing/benchmarks/cloudml/cloudml_benchmark_constants_lib.py
=> internal/test_data/module_1_nested_function_2_added.py} (70%)
copy
sdks/python/apache_beam/{testing/benchmarks/cloudml/cloudml_benchmark_constants_lib.py
=> internal/test_data/module_1_nested_function_added.py} (70%)
create mode 100644 sdks/python/apache_beam/internal/test_data/module_2.py
create mode 100644
sdks/python/apache_beam/internal/test_data/module_2_modified.py
copy sdks/python/apache_beam/{tools/__init__.py =>
internal/test_data/module_3.py} (79%)
copy sdks/python/apache_beam/{tools/__init__.py =>
internal/test_data/module_3_modified.py} (79%)
copy sdks/python/apache_beam/{tools/__init__.py =>
internal/test_data/module_with_default_argument.py} (81%)
create mode 100644
sdks/python/apache_beam/io/gcp/bigquery_read_internal_test.py
copy sdks/python/apache_beam/ml/inference/test_resources/{vllm.dockerfile =>
vllm.dockerfile.old} (97%)
copy sdks/python/apache_beam/ml/inference/{torch_tests_requirements.txt =>
vllm_tests_requirements.txt} (94%)
copy
sdks/python/apache_beam/testing/benchmarks/inference/{pytorch_sentiment_benchmarks.py
=> vllm_gemma_benchmarks.py} (71%)
create mode 100644 website/www/site/content/en/blog/beam-2.67.0.md
create mode 100644
website/www/site/content/en/performance/vllmgemmabatchtesla/_index.md
