This is an automated email from the ASF dual-hosted git repository.
derrickaw pushed a change to branch 20260624_zizmorInitial
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 2a75b952672 add comment
discard e6b5c40ba5f add rule about action hashes
discard 755ea5c2f64 Initial zizmor findings
add 8d252c4659b update changes log for yaml improvements etc (#39093)
add 3fba9950feb Update beam-master container
add e56ed7718a5 Trigger failed postcommit tests.
add 7454aa6a0bf Merge pull request #39100 from
shunping/update-python-beam-master
add d71c4aca4f4 Fix race condition in DirectRunner executor shutdown
add 0af28db1142 Apply suggested fix
add 23521069470 Merge pull request #39096 from shunping/fix-flaky-rrio-test
add 96733599e6c Fix JsonToRow swallowing downstream errors when runners
fuse transforms. (#39098)
add 8092349ca28 Enable playground tests triggered on cron schedule (#39069)
add 8ecdab8da94 [Iceberg CDC] Import BaseIncrementalChangelogScan (#38823)
add 8c311e45672 [Iceberg CDC] Add utils base, delete reader, and
serializable files (#38821)
add c1898ff68b8 Fix pip ResolutionTooDeep (#39017)
add 9d3694165e1 [Python] Add UnboundedSource SDF wrapper (#19137) (#38724)
add 5eefc931b5d remove infra change (#39104)
add eef26ca8b71 Revert "[Python] Optimize BigQuery copy jobs in file loads
using multi-source copy" (#39106)
add 0c51b0d288b Write append tables can be asynchronous (#39120)
add b642c5cbd0e Fix typo (#39123)
add 56b5a52294d Bump google-cloud-aiplatform (#39113)
add d77d12a47b2 Update cython requirement from <4,>=3.2.5 to >=3.2.6,<4 in
/sdks/python (#39099)
add 5cb2f09298c [Website] Update Python version compatibility table
(#38969)
add 698420027d5 Docs: update Python version support in Python quickstart
(#39122)
add 8ccd1d70c04 including GitHub Actions to schedule daily audits and Add
documentati… (#39116)
add 5a2e98bb712 Fix a flaky test in ApproximateQuantilesTest (#39132)
add e3b4d8c5d0e Fix flaky ML RunInference tests by disabling reshuffle on
beam.Create (#39118)
add 3cca8fa3dac [Dataflow Streaming] Access state/timerinternals via
StepContext (#38991)
add 2e8f75aaf4b Bump actions/checkout from 6 to 7 (#39147)
add b63ea65d6a3 Bump github.com/aws/smithy-go from 1.27.2 to 1.27.3 in
/sdks (#39145)
add 31cd92f5f55 Bump cloud.google.com/go/storage from 1.62.3 to 1.63.0 in
/sdks (#39144)
add edd3bfca48e Fix flaky GcsIOIntegrationTest and handle GCS bucket
creation race conditions (#39130)
add af005ef679c fix post commit yaml test (#39142)
add 31175f9ab82 fix race condition in statesampler with lock for reset
(#39129)
add ccac91ad9b8 add zizmor to existing precommit GHA workflow (#39108)
add f17b370e74a Fix flaky PubSub integration test by deduplicating
messages (#39134)
add 402b41c726d Fix ThreadSanitizer data race and failed RPC calls (#39076)
add 961ed7c31ad [IcebergIO] Fix stale openWriters count in
RecordWriterManager (#39157)
add f909e8a7af9 Use Flink 2.2 container for load tests
add 984df275bf4 debug only change -- preserve cluster to check error
add 397d9c61a27 Set configs for Flink 2
add a0fd9502818 Refactor Flink Load Tests to use native Dataproc
components and Auto Zone
add 3e76ff518fa Remove unnecessary gcloud volume mount from Job Server
add 5d28c414f62 Change memory allocation, use resize feature
add 30a5a00d584 Merge pull request #39154 from apache/flink-load-test-ver
add f9171da59bc Bump github.com/nats-io/nats-server/v2 from 2.14.2 to
2.14.3 in /sdks (#39168)
add aa6942a9853 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#39169)
add fcd105de015 Bump js-yaml from 3.14.2 to 3.15.0 in /website/www (#39172)
add 24a2ccc830a Bump com.gradle.common-custom-user-data-gradle-plugin
(#39166)
add 1c4b2047016 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#39170)
add aec5b5da813 Add embedded NVIDIA Dynamo support to vLLM ModelHandler
(#38701)
add 8f60e69177b Fixed DirectRunner PubSub subscriber client lifecycle
(#39079)
add ed0a5832eb1 Add open telemetry dependency to dataflow worker (#38941)
add a86e6115270 Retry starting SubprocessServer on failure (#39159)
add e34cc96df4d [yaml] - mongodb read normalization (#38772)
add e23ef261fe9 Initial zizmor findings
add 1df117d2b4f add rule about action hashes
add 668a9e5927a add comment
add 6960d4b737d fix another workflow
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (2a75b952672)
\
N -- N -- N refs/heads/20260624_zizmorInitial (6960d4b737d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../beam_CloudML_Benchmarks_Dataflow.json | 2 +-
..._ValidatesRunner_Dataflow_Streaming_Engine.json | 2 +-
.github/trigger_files/beam_PostCommit_Python.json | 6 +-
.../beam_PostCommit_Python_Examples_Dataflow.json | 4 +-
.../beam_PostCommit_Yaml_Xlang_Direct.json | 2 +-
.github/workflows/README.md | 1 +
.../beam_Infrastructure_AuditUnmanagedKeys.yml | 71 +
.../beam_LoadTests_Go_CoGBK_Flink_batch.yml | 8 +-
.../beam_LoadTests_Go_Combine_Flink_Batch.yml | 14 +-
.../beam_LoadTests_Go_GBK_Flink_Batch.yml | 14 +-
.../beam_LoadTests_Go_ParDo_Flink_Batch.yml | 6 +-
.../beam_LoadTests_Go_SideInput_Flink_Batch.yml | 8 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 6 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 14 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 6 +-
.../beam_LoadTests_Python_GBK_Flink_Batch.yml | 12 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 6 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 6 +-
.github/workflows/beam_Playground_Precommit.yml | 1 +
.../workflows/beam_PreCommit_Flink_Container.yml | 6 +-
.github/workflows/beam_PreCommit_GHA.yml | 6 +-
.test-infra/dataproc/flink_cluster.sh | 78 +-
.test-infra/dataproc/init-actions/docker.sh | 103 --
.test-infra/dataproc/init-actions/flink.sh | 217 ---
CHANGES.md | 5 +
infra/enforcement/README.md | 12 +-
.../org/apache/beam/runners/core/DoFnRunners.java | 4 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 17 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 4 +-
.../apache/beam/runners/core/ReduceFnTester.java | 20 +-
.../direct/ExecutorServiceParallelExecutor.java | 28 +-
.../wrappers/streaming/WindowDoFnOperator.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 3 +
.../dataflow/worker/GroupAlsoByWindowsParDoFn.java | 2 +-
.../worker/StreamingModeExecutionContext.java | 20 +
.../dataflow/worker/WindmillTimerInternals.java | 23 +
.../windmill/state/WindmillStateInternals.java | 17 +
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 3 +-
.../worker/StreamingModeExecutionContextTest.java | 60 +
sdks/go.mod | 28 +-
sdks/go.sum | 60 +-
.../apache/beam/sdk/transforms/AsyncWrapper.java | 10 +-
.../org/apache/beam/sdk/transforms/JsonToRow.java | 8 +-
.../beam/sdk/transforms/AsyncWrapperTest.java | 66 +-
.../apache/beam/sdk/transforms/JsonToRowTest.java | 26 +
.../beam/sdk/io/iceberg/IcebergScanConfig.java | 78 +-
.../apache/beam/sdk/io/iceberg/PartitionUtils.java | 100 +-
.../apache/beam/sdk/io/iceberg/ReadFromTasks.java | 4 +-
.../org/apache/beam/sdk/io/iceberg/ReadUtils.java | 137 +-
.../beam/sdk/io/iceberg/RecordWriterManager.java | 5 +-
.../apache/beam/sdk/io/iceberg/ScanTaskReader.java | 4 +-
.../beam/sdk/io/iceberg/SerializableDataFile.java | 100 +-
.../sdk/io/iceberg/SerializableDeleteFile.java | 335 +++++
.../beam/sdk/io/iceberg/cdc/CdcOutputUtils.java | 178 +++
.../beam/sdk/io/iceberg/cdc/CdcReadUtils.java | 698 ++++++++++
.../beam/sdk/io/iceberg/cdc/DeleteReader.java | 309 +++++
.../io/iceberg/cdc/IcebergCdcMetadataColumns.java | 93 ++
.../io/iceberg/cdc/SerializableChangelogTask.java | 282 ++++
.../beam/sdk/io/iceberg/cdc}/package-info.java | 4 +-
.../iceberg/BaseIncrementalChangelogScan.java | 1014 ++++++++++++++
.../java/org/apache/iceberg}/package-info.java | 4 +-
.../IcebergCdcReadSchemaTransformProviderTest.java | 12 +-
.../beam/sdk/io/iceberg/IcebergIOReadTest.java | 83 +-
.../IcebergSchemaTransformTranslationTest.java | 33 +-
.../beam/sdk/io/iceberg/PartitionUtilsTest.java | 68 +
.../apache/beam/sdk/io/iceberg/ReadUtilsTest.java | 59 +-
.../sdk/io/iceberg/SerializableDataFileTest.java | 3 +
.../sdk/io/iceberg/SerializableDeleteFileTest.java | 222 ++++
.../beam/sdk/io/iceberg/TestDataWarehouse.java | 2 +-
.../io/iceberg/catalog/IcebergCatalogBaseIT.java | 7 +-
.../beam/sdk/io/iceberg/cdc/CdcReadUtilsTest.java | 381 ++++++
.../beam/sdk/io/iceberg/cdc/DeleteReaderTest.java | 419 ++++++
.../iceberg/cdc/SerializableChangelogTaskTest.java | 256 ++++
sdks/java/io/mongodb/build.gradle | 2 +
.../beam/sdk/io/mongodb/AggregationQuery.java | 9 +-
...> MongoDbReadSchemaTransformConfiguration.java} | 34 +-
.../MongoDbReadSchemaTransformProvider.java | 151 +++
.../apache/beam/sdk/io/mongodb/MongoDbUtils.java | 131 ++
.../MongoDbReadSchemaTransformProviderTest.java | 221 ++++
.../beam/sdk/io/mongodb/MongoDbUtilsTest.java | 123 ++
.../examples/inference/vllm_text_completion.py | 24 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 162 +--
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 205 +--
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 18 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 13 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 21 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 40 +
.../apache_beam/io/gcp/pubsub_integration_test.py | 22 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 79 +-
sdks/python/apache_beam/io/iobase.py | 14 +-
sdks/python/apache_beam/io/iobase_test.py | 49 +-
sdks/python/apache_beam/io/unbounded_source.py | 996 ++++++++++++++
.../python/apache_beam/io/unbounded_source_test.py | 1380 ++++++++++++++++++++
sdks/python/apache_beam/ml/inference/base_test.py | 2 +-
.../ml/inference/pytorch_inference_test.py | 5 +-
.../ml/inference/sklearn_inference_test.py | 4 +-
.../ml/inference/tensorflow_inference_test.py | 4 +-
.../ml/inference/tensorrt_inference_test.py | 11 +-
.../inference/test_resources/vllm.dockerfile.old | 14 +-
.../apache_beam/ml/inference/vllm_inference.py | 331 ++++-
.../ml/inference/vllm_inference_test.py | 157 +++
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../runners/direct/transform_evaluator.py | 113 +-
.../runners/worker/statesampler_fast.pyx | 11 +-
.../testing/benchmarks/cloudml/constraints.txt} | 46 +-
.../testing/benchmarks/cloudml/requirements.txt | 17 +-
sdks/python/apache_beam/transforms/async_dofn.py | 4 +-
.../apache_beam/transforms/async_dofn_test.py | 47 +
sdks/python/apache_beam/transforms/stats_test.py | 20 +-
sdks/python/apache_beam/transforms/util_test.py | 49 +
sdks/python/apache_beam/utils/subprocess_server.py | 89 +-
.../apache_beam/utils/subprocess_server_test.py | 22 +-
.../yaml/extended_tests/databases/iceberg.yaml | 54 +-
sdks/python/apache_beam/yaml/integration_tests.py | 74 ++
sdks/python/apache_beam/yaml/standard_io.yaml | 13 +-
sdks/python/apache_beam/yaml/tests/mongodb.yaml | 40 +-
sdks/python/apache_beam/yaml/yaml_io.py | 59 +-
sdks/python/pyproject.toml | 2 +-
sdks/python/test-suites/dataflow/common.gradle | 50 +-
sdks/standard_external_transforms.yaml | 34 +
settings.gradle.kts | 2 +-
.../site/content/en/documentation/sdks/python.md | 6 +-
.../site/content/en/get-started/quickstart-py.md | 2 +-
website/www/yarn.lock | 6 +-
125 files changed, 9673 insertions(+), 1220 deletions(-)
create mode 100644 .github/workflows/beam_Infrastructure_AuditUnmanagedKeys.yml
delete mode 100644 .test-infra/dataproc/init-actions/docker.sh
delete mode 100644 .test-infra/dataproc/init-actions/flink.sh
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableDeleteFile.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcOutputUtils.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtils.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/DeleteReader.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/IcebergCdcMetadataColumns.java
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/SerializableChangelogTask.java
copy sdks/java/{core/jmh/src/main/java/org/apache/beam/sdk/jmh/io =>
io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc}/package-info.java
(91%)
create mode 100644
sdks/java/io/iceberg/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java
copy
{runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state
=> sdks/java/io/iceberg/src/main/java/org/apache/iceberg}/package-info.java
(90%)
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/SerializableDeleteFileTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/CdcReadUtilsTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/DeleteReaderTest.java
create mode 100644
sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/SerializableChangelogTaskTest.java
copy
sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/{MongoDbWriteSchemaTransformConfiguration.java
=> MongoDbReadSchemaTransformConfiguration.java} (69%)
create mode 100644
sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbReadSchemaTransformProvider.java
create mode 100644
sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbReadSchemaTransformProviderTest.java
create mode 100644 sdks/python/apache_beam/io/unbounded_source.py
create mode 100644 sdks/python/apache_beam/io/unbounded_source_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/vllm_inference_test.py
copy sdks/python/{scripts/run_eslint.sh =>
apache_beam/testing/benchmarks/cloudml/constraints.txt} (53%)
mode change 100755 => 100644
