This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/pip/sdks/python/dill-gte-0.3.1.1-and-lt-0.4.1
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 00500ef68fb Update dill requirement in /sdks/python
add febe6affadf Allow nullable subfields for OneOfType (#36297)
add 53bba283ce0 Fix typevar reduce for PYPI. (#36299)
add bbf3613ea40 Set latest tag based on branch in sdk and docker snapshots
(#36298)
add bacd18b3731 CONTRIBUTING.md: include workflows README doc (#36303)
add d9209d9c876 Fix BigQuery Integration tests
add b9ba54d20b8 Fix location
add 8bde4429a3c fix location
add 08e76a0ca12 fix formatting
add 9cd994eb086 Merge pull request #36302 from
tarun-google/fix_bigquery_tests
add 18d1d9fbb22 increase timeout (#36317)
add 50e14ace7f6 feat(bigquery): add BigLake configuration support for
Storage Write API (#36225)
add c84f28f84aa Add option to pickle relative filepaths in cloudpickle.
(#36300)
add 8cdd2f595ff Fix PerformanceTests jobs (#36315)
add abdec1bcecd Bump github.com/nats-io/nats.go from 1.45.0 to 1.46.0 in
/sdks (#36312)
add 34aa17de6cd Allow users to pass service name for profiler for Java And
Go SDK (#35903)
add 970f6b26c5d [GSoC 25][Blog Post]: Beam ML Vector DB/Feature Store
Project (#36301)
add 292484f96a2 fix license script (#36328)
add bb340c2f66a further increase timeout (#36329)
add 4d87e040ac2 Add deterministic redistribute sharding for KafkaIO read.
(#36112)
add 449ab577d32 Update python SDK container image (#36341)
add de8a6375242 Add Python 3.13
add 7a815d458f4 Add Python 3.13 to supported versions
add 881de9ba98a Update tensorflow version for Python 3.13
add 5f216696371 Merge pull request #36319 from
apache/fix-update-dependencies
add da00474b4ea Add temp_location to all BigQuery Write examples and
integration tests
add 75866588752 Merge pull request #36336 from
tarun-google/fix_temp_location
add 09b9c471c8d Add @SchemaFieldDescription annotations and description to
Bigtable config (#36344)
add 15b8560975a Fix Managed I/O link in the blog (#36268)
add 2f9a910e1f7 Implement a hierarchy for Beam SQL Calcite Schemas (#35787)
add 69b70263e72 Add ability for coders to set version tags for update
compat checks. (#36316)
add e4c891fffe7 Update coders.py
add 7de704e9e9a Add dill test workflow (#36348)
add 340d420c2d3 Remove workflow_run. (#36351)
add a0831e0d4b1 Add AftersynchronizedProcessing Time as continuation
trigger (#36285)
add 97b35eb5d7f Bump docker/login-action from 2 to 3 (#36232)
add d78d004d0a5 Skip SqlTransformExample (#36354)
add 6b9d534137c Set user agent when downloading jars with local_jar
(#36342)
add e61b4f39b09 Add Java25 container (#36349)
add 666c75adc07 Add google-cloud-secret-managager to beam python container
deps (#36353)
add a3fd151f8f9 Fix sql example in playground (#36358)
add b2954eef3b5 Fix playground example dataset format value
add 3f6d8210845 Merge pull request #36360 from apache/fix-dataset-path
add 3903623850d Exit gradle process after build complete (#36361)
add 7c88161b18b Consolidate beam Publish Bem SDK Snapshot matrix (#36363)
add 3534960b9c8 [Prism] Support AfterProcessingTime triggers - part 2
(#36333)
add df3384f96a2 Also check the existence of _fields in addition
to__annotations__ for NamedTuple. (#36337)
add 1c8bcf9bae1 Fix race condition and nil pointer dereferencing (#36370)
add 45c36901859 Bump google.golang.org/protobuf from 1.36.8 to 1.36.10 in
/sdks (#36371)
add 24d3a7bac85 Fix errorprone move serialization to private method
(#36352)
add a03e96ad981 Update GitHub Actions to run Python 3.13 Tests (#35056)
add 1058c020edf Update dill requirement in /sdks/python
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (00500ef68fb)
\
N -- N -- N
refs/heads/dependabot/pip/sdks/python/dill-gte-0.3.1.1-and-lt-0.4.1
(1058c020edf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../gradle-command-self-hosted-action/action.yml | 2 +-
.../test-properties.json | 8 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.json | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.json | 2 +-
...PostCommit_XVR_PythonUsingJavaSQL_Dataflow.json | 2 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Dependency.yml | 2 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 2 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 2 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 2 +-
.github/workflows/beam_PreCommit_Prism_Python.yml | 2 +-
.github/workflows/beam_PreCommit_Python.yml | 2 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
...ython_ML.yml => beam_PreCommit_Python_Dill.yml} | 50 ++--
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 6 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 38 +--
.../workflows/beam_Publish_Docker_Snapshots.yml | 7 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/build_release_candidate.yml | 2 +-
.github/workflows/build_wheels.yml | 6 +-
.github/workflows/finalize_release.yml | 2 +-
.github/workflows/python_dependency_tests.yml | 1 +
.github/workflows/python_tests.yml | 3 +-
.github/workflows/update_python_dependencies.yml | 3 +-
.../postgres/postgres-service-for-local-dev.yml | 15 +-
CHANGES.md | 1 +
CONTRIBUTING.md | 4 +
build.gradle.kts | 3 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 16 +-
examples/java/sql/build.gradle | 2 +-
local-env-setup.sh | 4 +-
playground/backend/containers/java/Dockerfile | 14 +-
playground/infrastructure/ci_cd.py | 2 +
playground/infrastructure/models.py | 2 +-
runners/prism/java/build.gradle | 10 +
sdks/go.mod | 4 +-
sdks/go.sum | 8 +-
sdks/go/container/boot.go | 42 ++-
sdks/go/container/boot_test.go | 127 +++++---
.../prism/internal/engine/elementmanager.go | 176 +++++++++--
.../beam/runners/prism/internal/engine/strategy.go | 43 +++
.../prism/internal/jobservices/management.go | 2 +-
.../runners/prism/internal/unimplemented_test.go | 3 +-
sdks/go/test/integration/integration.go | 1 +
sdks/go/test/integration/primitives/windowinto.go | 24 +-
.../test/integration/primitives/windowinto_test.go | 6 +
sdks/java/container/boot.go | 88 ++++--
sdks/java/container/boot_test.go | 45 +++
sdks/java/container/build.gradle | 1 +
sdks/java/container/distroless/build.gradle | 7 +-
.../java/container/{java21 => java25}/build.gradle | 4 +-
.../java25-security.properties} | 0
.../container/{java21 => java25}/option-jamm.json | 0
.../option-java25-security.json} | 2 +-
.../license_scripts/pull_licenses_java.py | 3 +-
.../beam/sdk/schemas/logicaltypes/OneOfType.java | 3 -
.../apache/beam/sdk/transforms/Redistribute.java | 2 +-
.../beam/sdk/util/construction/Environments.java | 5 +-
.../sdk/schemas/logicaltypes/LogicalTypesTest.java | 17 --
.../sdk/util/construction/EnvironmentsTest.java | 10 +-
.../extensions/protobuf/ProtoBeamConverter.java | 9 +-
.../beam/sdk/extensions/protobuf/ProtobufUtil.java | 18 --
.../sql/meta/provider/iceberg/IcebergCatalog.java | 30 +-
.../meta/provider/iceberg/IcebergMetastore.java | 154 ++++++++++
.../sql/meta/provider/iceberg/IcebergTable.java | 5 +-
.../provider/iceberg/IcebergTableProvider.java | 96 ------
.../provider/iceberg/BeamSqlCliIcebergTest.java | 116 ++++++-
.../provider/iceberg/IcebergMetastoreTest.java | 97 ++++++
.../meta/provider/iceberg/IcebergReadWriteIT.java | 46 ++-
.../provider/iceberg/IcebergTableProviderTest.java | 83 -----
.../meta/provider/iceberg/PubsubToIcebergIT.java | 18 +-
.../sql/src/main/codegen/includes/parserImpls.ftl | 30 +-
.../beam/sdk/extensions/sql/SqlTransform.java | 5 +-
.../apache/beam/sdk/extensions/sql/TableUtils.java | 4 +
.../sdk/extensions/sql/impl/BeamCalciteSchema.java | 58 ++--
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 18 +-
.../extensions/sql/impl/CatalogManagerSchema.java | 278 +++++++++++++++++
.../sdk/extensions/sql/impl/CatalogSchema.java | 229 ++++++++++++++
.../sdk/extensions/sql/impl/JdbcConnection.java | 6 +-
.../beam/sdk/extensions/sql/impl/TableName.java | 30 ++
.../sql/impl/parser/SqlCreateCatalog.java | 43 +--
.../sql/impl/parser/SqlCreateDatabase.java | 74 +++--
.../sql/impl/parser/SqlCreateExternalTable.java | 46 ++-
.../extensions/sql/impl/parser/SqlDdlNodes.java | 29 +-
.../extensions/sql/impl/parser/SqlDropCatalog.java | 46 +--
.../sql/impl/parser/SqlDropDatabase.java | 59 +---
.../extensions/sql/impl/parser/SqlDropTable.java | 45 +++
.../sql/impl/parser/SqlSetOptionBeam.java | 35 ++-
.../extensions/sql/impl/parser/SqlUseCatalog.java | 38 +--
.../extensions/sql/impl/parser/SqlUseDatabase.java | 61 ++--
.../apache/beam/sdk/extensions/sql/meta/Table.java | 3 +-
.../sdk/extensions/sql/meta/catalog/Catalog.java | 25 +-
.../sql/meta/catalog/CatalogManager.java | 13 +-
.../sql/meta/catalog/EmptyCatalogManager.java | 14 +-
.../sql/meta/catalog/InMemoryCatalog.java | 46 ++-
.../sql/meta/catalog/InMemoryCatalogManager.java | 29 +-
.../sql/meta/store/InMemoryMetaStore.java | 60 ++--
.../sdk/extensions/sql/meta/store/MetaStore.java | 6 +
.../sdk/extensions/sql/BeamSqlCliCatalogTest.java | 333 +++++++++++++++++++++
.../sdk/extensions/sql/BeamSqlCliDatabaseTest.java | 133 +++++++-
.../beam/sdk/extensions/sql/BeamSqlCliTest.java | 182 +----------
.../sdk/extensions/sql/impl/JdbcDriverTest.java | 22 +-
.../sql/impl/parser/BeamDDLNestedTypesTest.java | 2 +-
.../extensions/sql/impl/parser/BeamDDLTest.java | 28 +-
.../sdk/extensions/sql/impl/rel/BaseRelTest.java | 2 +
.../sql/impl/rule/JoinReorderingTest.java | 6 +-
.../sql/meta/store/InMemoryMetaStoreTest.java | 9 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 3 +
.../providers/BigQueryWriteConfiguration.java | 11 +
.../BigtableReadSchemaTransformProvider.java | 19 ++
.../BigtableWriteSchemaTransformProvider.java | 18 ++
.../beam/sdk/io/iceberg/IcebergCatalogConfig.java | 40 ++-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 53 +++-
.../KafkaIOReadImplementationCompatibility.java | 6 +
.../beam/sdk/io/kafka/KafkaReadRedistribute.java | 124 ++++++++
...KafkaIOReadImplementationCompatibilityTest.java | 6 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 87 +++++-
.../sdk/io/kafka/KafkaReadRedistributeTest.java | 231 ++++++++++++++
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 10 +
.../io/kafka/upgrade/KafkaIOTranslationTest.java | 1 +
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java | 5 +-
sdks/python/apache_beam/coders/coders.py | 9 +
.../internal/cloudpickle/cloudpickle.py | 119 ++++++--
.../io/external/xlang_bigqueryio_it_test.py | 96 ++++++
sdks/python/apache_beam/io/gcp/bigquery.py | 8 +-
.../apache_beam/io/gcp/bigquery_biglake_test.py | 117 ++++++++
.../python/apache_beam/options/pipeline_options.py | 6 +
sdks/python/apache_beam/pipeline.py | 2 +
sdks/python/apache_beam/pipeline_test.py | 33 ++
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../testing/integration/tests/screen_diff_test.py | 2 -
.../portability/flink_uber_jar_job_server.py | 4 +-
.../apache_beam/runners/portability/job_server.py | 5 +-
.../portability/spark_uber_jar_job_server.py | 4 +-
sdks/python/apache_beam/transforms/core.py | 12 +
sdks/python/apache_beam/transforms/external.py | 43 ++-
.../python/apache_beam/transforms/external_test.py | 2 +-
.../apache_beam/transforms/ptransform_test.py | 19 ++
sdks/python/apache_beam/transforms/trigger.py | 119 +++++++-
sdks/python/apache_beam/transforms/trigger_test.py | 50 ++++
.../typehints/native_type_compatibility.py | 2 +-
sdks/python/apache_beam/utils/subprocess_server.py | 22 +-
.../apache_beam/utils/subprocess_server_test.py | 7 +-
.../utils/transform_service_launcher.py | 7 +-
.../transforms/blueprint/gcs_text_to_bigquery.yaml | 3 +
.../transforms/blueprint/jdbc_to_bigquery.yaml | 3 +-
.../transforms/blueprint/mysql_to_bigquery.yaml | 3 +
.../transforms/blueprint/oracle_to_bigquery.yaml | 3 +
.../transforms/blueprint/postgres_to_bigquery.yaml | 3 +
.../transforms/blueprint/spanner_to_bigquery.yaml | 3 +
.../blueprint/sqlserver_to_bigquery.yaml | 3 +
.../ml/log_analysis/ml_preprocessing.yaml | 1 +
.../yaml/extended_tests/data/enrichment.yaml | 3 +
.../yaml/extended_tests/databases/bigquery.yaml | 4 +-
sdks/python/apache_beam/yaml/integration_tests.py | 5 +-
sdks/python/build.gradle | 2 +-
.../ml_image_requirements.txt} | 126 ++++----
.../container/py310/base_image_requirements.txt | 92 +++---
.../container/py311/base_image_requirements.txt | 94 +++---
.../container/py312/base_image_requirements.txt | 94 +++---
.../container/py313/base_image_requirements.txt | 101 +++----
.../container/py39/base_image_requirements.txt | 82 ++---
sdks/python/pyproject.toml | 2 +-
sdks/python/setup.py | 76 +++--
sdks/python/test-suites/gradle.properties | 28 +-
sdks/python/tox.ini | 12 +
settings.gradle.kts | 1 +
.../site/content/en/blog/gsoc-25-ml-connectors.md | 254 ++++++++++++++++
.../en/blog/gsoc-25-yaml-user-accessibility.md | 2 +-
website/www/site/data/authors.yml | 4 +
197 files changed, 4376 insertions(+), 1553 deletions(-)
copy .github/workflows/{beam_PreCommit_Python_ML.yml =>
beam_PreCommit_Python_Dill.yml} (73%)
copy sdks/java/container/{java21 => java25}/build.gradle (97%)
copy sdks/java/container/{java21/java21-security.properties =>
java25/java25-security.properties} (100%)
copy sdks/java/container/{java21 => java25}/option-jamm.json (100%)
copy sdks/java/container/{java11/option-java11-security.json =>
java25/option-java25-security.json} (86%)
create mode 100644
sdks/java/extensions/sql/iceberg/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/IcebergMetastore.java
delete mode 100644
sdks/java/extensions/sql/iceberg/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/IcebergTableProvider.java
create mode 100644
sdks/java/extensions/sql/iceberg/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/IcebergMetastoreTest.java
delete mode 100644
sdks/java/extensions/sql/iceberg/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/IcebergTableProviderTest.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CatalogManagerSchema.java
create mode 100644
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CatalogSchema.java
create mode 100644
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlCliCatalogTest.java
create mode 100644
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadRedistribute.java
create mode 100644
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadRedistributeTest.java
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_biglake_test.py
copy sdks/python/container/ml/{py312/base_image_requirements.txt =>
py313/ml_image_requirements.txt} (75%)
create mode 100644 website/www/site/content/en/blog/gsoc-25-ml-connectors.md
