This is an automated email from the ASF dual-hosted git repository.
Abacn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ffcbd7e464f Update IO support matrix after Beam 2.76.0 release (#39982)
ffcbd7e464f is described below
commit ffcbd7e464f8244bd623d23e7fa9fcc750e81bbb
Author: Yi Hu <[email protected]>
AuthorDate: Thu Sep 3 12:01:32 2026 -0400
Update IO support matrix after Beam 2.76.0 release (#39982)
* Update IO support matrix after Beam 2.76.0 release
* Fix CHANGES.md wrong version
* Fix Yamldoc empty for certain transforms due to expansion service is not
present
* Remove item that not affect any release
---
CHANGES.md | 10 +-
sdks/python/build.gradle | 8 +
website/www/site/content/en/blog/beam-2.76.0.md | 4 -
.../site/content/en/documentation/io/connectors.md | 181 +++++++--------------
website/www/site/data/capability_matrix.yaml | 1 -
5 files changed, 71 insertions(+), 133 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 365605c7065..f0d5d06b9d9 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -76,6 +76,9 @@
* (Java/Python) `Watch` can bound its deduplication state by event time,
retiring an output key once the greatest emitted timestamp has moved more than
the allowed lateness past it. Java adds
`Watch.growthOf(...).withTimestampCursor()`. Python adds `allowed_lateness` for
the existing `timestamp_cursor` option
([#18459](https://github.com/apache/beam/issues/18459)).
* (Java) Spark Structured Streaming runner: stateful ParDo with state, timers,
`@RequiresTimeSortedInput` and tagged outputs is now supported in batch mode
([#39779](https://github.com/apache/beam/issues/39779)).
* (Python) Added support for Vertex AI Model Monitoring V2 in RunInference
([#39738](https://github.com/apache/beam/issues/39738)).
+* [Flink Runner] Added opt-in static round-robin split assignment for small
bounded sources via the new `sourceStaticSplitThresholdMb` pipeline option. The
default of 0 keeps the existing lazy pull-based assignment
([#39873](https://github.com/apache/beam/issues/39873)).
+* Added automatic caching of bounded, single-pane side-input views for classic
Java Flink DataStream execution
([#39866](https://github.com/apache/beam/issues/39866)).
+* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`,
which returns up to n arbitrary elements from a PCollection
([#18552](https://github.com/apache/beam/issues/18552)).
## Breaking Changes
@@ -89,7 +92,6 @@
## Bugfixes
-* (Java) Restored binary compatibility for `CoderTranslatorRegistrar`
implementations compiled against Beam 2.76 and earlier
([#38714](https://github.com/apache/beam/issues/38714)).
* (Java) Fixed the Spark runner firing processing-time timers in reverse
timestamp order ([#39824](https://github.com/apache/beam/issues/39824)).
* (Python) Fixed incorrect profiler options handling on portable runners
([#39613](https://github.com/apache/beam/issues/39613)).
* (Java) KafkaIO dynamic reads no longer require the obsolete `beam_fn_api`
experiment ([#29998](https://github.com/apache/beam/issues/29998)).
@@ -124,8 +126,6 @@
## New Features / Improvements
-* [Flink Runner] Added opt-in static round-robin split assignment for small
bounded sources via the new `sourceStaticSplitThresholdMb` pipeline option. The
default of 0 keeps the existing lazy pull-based assignment
([#39873](https://github.com/apache/beam/issues/39873)).
-* Added automatic caching of bounded, single-pane side-input views for classic
Java Flink DataStream execution
([#39866](https://github.com/apache/beam/issues/39866)).
* Added `GroupIntoBatches` transform and the standard
`beam:coder:sharded_key:v1` coder to the Go SDK, along with
`beam.Coder.IsDeterministic`, `beam.PCollection.WindowingStrategy`,
@@ -146,7 +146,6 @@
* (Python) Added `Watch`, a transform that polls a growing set of outputs for
each input element, deduplicates outputs across poll rounds, and stops per a
user-supplied termination condition
([#21521](https://github.com/apache/beam/issues/21521)).
* (Python) Added support to analyze core dumps created after python worker
segmentation faults with `pystack` (or `gdb` if installed) using the
`--profiler_agent=coredump` pipeline option.
([#39484](https://github.com/apache/beam/issues/39484)).
-* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`,
which returns up to n arbitrary elements from a PCollection
([#18552](https://github.com/apache/beam/issues/18552)).
## Breaking Changes
@@ -161,9 +160,6 @@
Use pipeline option `--updateCompatibilityVersion=2.75.0` (or any older
version) to keep the old behavior
([#39344](https://github.com/apache/beam/issues/39344)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable
wrapping one) now raises a `TypeError` rather than silently iterating
per-character/byte/key (Python)
([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including
runner state mappings and Dataflow update handling
([#39020](https://github.com/apache/beam/issues/39020)).
-* (Python) Typehints of dataclass fields are honored during type inferences.
To restore the behavior of fallback-to-any,
- use pipeline option `--exclude_infer_dataclass_field_type`
([#38797](https://github.com/apache/beam/issues/38797)).
- However fixing forward is recommended.
* (Java) IcebergIO and projects that use it must now be built with Java 17 or
later as a result of Iceberg 1.11.0 upgrade
([#38925](https://github.com/apache/beam/issues/38925)).
## Bugfixes
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 75afa5d7c96..339d87c76cb 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -107,16 +107,24 @@ tasks.register("prepareExpansionServicesForYamlDocs") {
dependsOn ":sdks:java:extensions:schemaio-expansion-service:shadowJar"
dependsOn ":sdks:java:extensions:sql:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:amazon-web-services2:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:debezium:expansion-service:shadowJar"
dependsOn ":sdks:java:io:expansion-service:shadowJar"
dependsOn ":sdks:java:io:google-cloud-platform:expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:messaging-expansion-service:shadowJar"
+ dependsOn ":sdks:java:io:snowflake:expansion-service:shadowJar"
doLast {
// Copy expansion service jar into cache path (.apache_beam/cache/jars/).
copy {
from
project.tasks.findByPath(":sdks:java:extensions:schemaio-expansion-service:shadowJar")
from
project.tasks.findByPath(":sdks:java:extensions:sql:expansion-service:shadowJar")
+ from
project.tasks.findByPath(":sdks:java:io:amazon-web-services2:expansion-service:shadowJar")
+ from
project.tasks.findByPath(":sdks:java:io:debezium:expansion-service:shadowJar")
from
project.tasks.findByPath(":sdks:java:io:expansion-service:shadowJar")
from
project.tasks.findByPath(":sdks:java:io:google-cloud-platform:expansion-service:shadowJar")
+ from
project.tasks.findByPath(":sdks:java:io:messaging-expansion-service:shadowJar")
+ from
project.tasks.findByPath(":sdks:java:io:snowflake:expansion-service:shadowJar")
into "${System.getProperty('user.home')}/.apache_beam/cache/jars/"
}
}
diff --git a/website/www/site/content/en/blog/beam-2.76.0.md
b/website/www/site/content/en/blog/beam-2.76.0.md
index cea653eaade..240c961cbd9 100644
--- a/website/www/site/content/en/blog/beam-2.76.0.md
+++ b/website/www/site/content/en/blog/beam-2.76.0.md
@@ -63,7 +63,6 @@ For more information on changes in 2.76.0, check out the
[detailed release notes
* (Python) Added `Watch`, a transform that polls a growing set of outputs for
each input element, deduplicates outputs across poll rounds, and stops per a
user-supplied termination condition
([#21521](https://github.com/apache/beam/issues/21521)).
* (Python) Added support to analyze core dumps created after python worker
segmentation faults with `pystack` (or `gdb` if installed) using the
`--profiler_agent=coredump` pipeline option.
([#39484](https://github.com/apache/beam/issues/39484)).
-* (Python) Added `Sample.Any`, the Python equivalent of Java's `Sample.any`,
which returns up to n arbitrary elements from a PCollection
([#18552](https://github.com/apache/beam/issues/18552)).
### Breaking Changes
@@ -78,9 +77,6 @@ For more information on changes in 2.76.0, check out the
[detailed release notes
Use pipeline option `--updateCompatibilityVersion=2.75.0` (or any older
version) to keep the old behavior
([#39344](https://github.com/apache/beam/issues/39344)).
* `DoFn.process` returning a `str`, `bytes`, or `dict` (instead of an iterable
wrapping one) now raises a `TypeError` rather than silently iterating
per-character/byte/key (Python)
([#18712](https://github.com/apache/beam/issues/18712)).
* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including
runner state mappings and Dataflow update handling
([#39020](https://github.com/apache/beam/issues/39020)).
-* (Python) Typehints of dataclass fields are honored during type inferences.
To restore the behavior of fallback-to-any,
- use pipeline option `--exclude_infer_dataclass_field_type`
([#38797](https://github.com/apache/beam/issues/38797)).
- However fixing forward is recommended.
* (Java) IcebergIO and projects that use it must now be built with Java 17 or
later as a result of Iceberg 1.11.0 upgrade
([#38925](https://github.com/apache/beam/issues/38925)).
### Bugfixes
diff --git a/website/www/site/content/en/documentation/io/connectors.md
b/website/www/site/content/en/documentation/io/connectors.md
index 679b6bf1e0e..f3078c6b378 100644
--- a/website/www/site/content/en/documentation/io/connectors.md
+++ b/website/www/site/content/en/documentation/io/connectors.md
@@ -429,7 +429,10 @@ This table provides a consolidated, at-a-glance overview
of the available built-
✔
<a
href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html";>native</a>
</td>
- <td>Not available</td>
+ <td class="present">
+ ✔
+ <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.xlang.io.html#apache_beam.transforms.xlang.io.ReadFromJms";>via
X-language</a>
+ </td>
<td>Not available</td>
<td>Not available</td>
<td>Not available</td>
@@ -790,7 +793,11 @@ This table provides a consolidated, at-a-glance overview
of the available built-
<a
href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/debeziumio";>via
X-language</a>
</td>
<td>Not available</td>
- <td>Not available</td>
+ <td class="present">
+ ✔
+ <br>
+ <a
href="https://beam.apache.org/releases/yamldoc/current/#readfromdebezium";>read</a>
+ </td>
<td class="present">✔</td>
<td class="present">✔</td>
<td class="absent">✘</td>
@@ -1322,18 +1329,25 @@ This table provides a consolidated, at-a-glance
overview of the available built-
<tr>
<td>Delta Lake</td>
<td class="present">✔</td>
- <td class="present">✘</td>
+ <td class="absent">✘</td>
<td class="present">
✔
<a
href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/managed/Managed.html";>native</a>
</td>
+ <td class="present">
+ ✔
+ <a
href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.managed.html#module-apache_beam.transforms.managed";>via
X-language</a>
+ </td>
<td>Not available</td>
<td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td class="present">
+ ✔
+ <br>
+ <a
href="https://beam.apache.org/releases/yamldoc/current/#readfromdelta";>read</a>
+ </td>
<td class="present">✔</td>
<td class="absent">✘</td>
- <td class="absent">✔</td>
+ <td class="present">✔</td>
</tr>
</table>
</div>
@@ -1346,205 +1360,130 @@ This table provides a consolidated, at-a-glance
overview of the available built-
<th>Connector Name</th>
<th>Source Supported</th>
<th>Sink Supported</th>
- <th>Java</th>
- <th>Python</th>
- <th>Go</th>
- <th>Typescript</th>
- <th>Yaml</th>
+ <th>Language</th>
<th>Batch Supported</th>
<th>Streaming Supported</th>
</tr>
<tr>
<td>
- <a href="https://github.com/google/hana-bq-beam-connector";>SAP Hana to
Google BigQuery</a>
+ <a
href="https://cloud.google.com/bigtable/docs/hbase-dataflow-java";>Cloud
Bigtable (HBase based)</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td>Java</td>
<td class="present">✔</td>
<td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/esakik/beam-mysql-connector";>MySQL</a>
+ <a href="https://github.com/cognitedata/cdf-beam-connector-java";>Cognite
Data Fusion</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td class="present">
- ✔
- <br>
- <a
href="https://beam.apache.org/releases/yamldoc/current/#readfrommysql";>read</a>
/ <a
href="https://beam.apache.org/releases/yamldoc/current/#writetomysql";>write</a>
- </td>
+ <td>Java</td>
+ <td class="present">✔</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a
href="https://github.com/Refinitiv/trep-websockets-beam-io";>TrepWsIO</a>
+ <a
href="https://github.com/kineticadb/kinetica-connector-beam";>KineticaDB</a>
</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
+ <td>Java</td>
<td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a
href="https://github.com/kineticadb/kinetica-connector-beam";>KineticaDB</a>
+ <a href="https://github.com/google/hana-bq-beam-connector";>SAP Hana to
Google BigQuery</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td>Java</td>
<td class="present">✔</td>
<td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/cognitedata/cdf-beam-connector-java";>Cognite
Data Fusion</a>
+ <a
href="https://github.com/Refinitiv/trep-websockets-beam-io";>TrepWsIO</a>
</td>
<td class="present">✔</td>
- <td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td class="absent">✘</td>
+ <td>Java</td>
<td class="present">✔</td>
<td class="present">✔</td>
</tr>
<tr>
<td>
- <a href="https://github.com/314e/beam-pyodbc-connector";>Pyodbc</a>
+ <a href="https://github.com/beam-pyio";>Beam PyIO (Collection of Python
IO connectors)</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td>Python</td>
+ <td class="present">✔</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/amient/goconnect";>Go Connect</a>
+ <a href="https://github.com/jccatrinck/dataflow-cloud-sql-python";>Cloud
SQL</a>
</td>
<td class="present">✔</td>
- <td class="present">✔</td>
<td class="absent">✘</td>
- <td class="absent">✘</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/tinybirdco/tinybird-beam";>Tinybird</a>
+ <a href="https://github.com/esakik/beam-mysql-connector";>MySQL</a>
</td>
- <td class="absent">✘</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/jccatrinck/dataflow-cloud-sql-python";>Cloud
SQL</a>
+ <a href="https://github.com/314e/beam-pyodbc-connector";>Pyodbc</a>
</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
+ <td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
<td class="absent">✘</td>
</tr>
<tr>
<td>
- <a
href="https://cloud.google.com/bigtable/docs/hbase-dataflow-java";>Cloud
Bigtable (HBase based)</a>
+ <a href="https://github.com/tinybirdco/tinybird-beam";>Tinybird</a>
</td>
+ <td class="absent">✘</td>
<td class="present">✔</td>
+ <td>Python</td>
<td class="present">✔</td>
- <td class="present">
- ✔
- native
- </td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
- <td class="absent">✘</td>
</tr>
<tr>
<td>
- <a href="https://github.com/beam-pyio";>Beam PyIO (Collection of Python
IO connectors)</a>
+ <a href="https://github.com/amient/goconnect";>Go Connect</a>
</td>
<td class="present">✔</td>
<td class="present">✔</td>
- <td>Not available</td>
- <td class="present">
- ✔
- native
+ <td>Go</td>
+ <td class="present">✔</td>
+ <td class="present">✔</td>
+ </tr>
+ <tr>
+ <td>
+ <a
href="https://beam.apache.org/releases/yamldoc/current/#readfrommysql";>MySQL,
PostgreSQL, Oracle, SQL Server</a>
</td>
- <td>Not available</td>
- <td>Not available</td>
- <td>Not available</td>
<td class="present">✔</td>
<td class="present">✔</td>
+ <td>Yaml</td>
+ <td class="present">✔</td>
+ <td class="absent">✘</td>
</tr>
</table>
</div>
diff --git a/website/www/site/data/capability_matrix.yaml
b/website/www/site/data/capability_matrix.yaml
index e4db68c0cbe..3f3bf39594c 100644
--- a/website/www/site/data/capability_matrix.yaml
+++ b/website/www/site/data/capability_matrix.yaml
@@ -1736,7 +1736,6 @@ capability-matrix:
l1: "Unverified"
l2:
l3:
-
- class: kafka-streams
l1: "No"
l2: not implemented
