gemini-code-assist[bot] commented on code in PR #39178:
URL: https://github.com/apache/beam/pull/39178#discussion_r3501468918
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -736,294 +840,279 @@ and Beam SQL is invoked via the Managed API under the
hood.
</tr>
<tr>
<td>
- keep
+ from_snapshot
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A subset of column names to read exclusively. If null or empty, all
columns will be read.
+ Starts reading from this snapshot ID (inclusive).
</td>
</tr>
- </table>
-</div>
-
-### `ICEBERG` Write
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>table</strong>
+ from_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A fully-qualified table identifier. You may also provide a template to
write to multiple dynamic destinations, for example:
`dataset.my_{col1}_{col2.nested}_table`.
+ Starts reading from the first snapshot (inclusive) that was created
after this timestamp (in milliseconds).
</td>
</tr>
<tr>
<td>
- autosharding
+ keep
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- Enables dynamic sharding to automatically adjust the number of
parallel writers based on data volume. It handles data skew by further
sub-dividing partitions into multiple shards to prevent bottlenecks during
high-throughput writes. Only available with 'hash' distribution mode.
+ A subset of column names to read exclusively. If null or empty, all
columns will be read.
</td>
</tr>
<tr>
<td>
- catalog_name
+ poll_interval_seconds
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Name of the catalog containing the table.
+ The interval at which to poll for new snapshots. Defaults to 60
seconds.
</td>
</tr>
<tr>
<td>
- catalog_properties
+ starting_strategy
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- Properties used to set up the Iceberg catalog.
+ The source's starting strategy. Valid options are: "earliest" or
"latest". Can be overriden by setting a starting snapshot or timestamp.
Defaults to earliest for batch, and latest for streaming.
Review Comment:
There is a typo in the description: "overriden" should be spelled
"overridden".
```suggestion
The source's starting strategy. Valid options are: "earliest" or
"latest". Can be overridden by setting a starting snapshot or timestamp.
Defaults to earliest for batch, and latest for streaming.
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -736,294 +840,279 @@ and Beam SQL is invoked via the Managed API under the
hood.
</tr>
<tr>
<td>
- keep
+ from_snapshot
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A subset of column names to read exclusively. If null or empty, all
columns will be read.
+ Starts reading from this snapshot ID (inclusive).
</td>
</tr>
- </table>
-</div>
-
-### `ICEBERG` Write
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>table</strong>
+ from_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A fully-qualified table identifier. You may also provide a template to
write to multiple dynamic destinations, for example:
`dataset.my_{col1}_{col2.nested}_table`.
+ Starts reading from the first snapshot (inclusive) that was created
after this timestamp (in milliseconds).
</td>
</tr>
<tr>
<td>
- autosharding
+ keep
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- Enables dynamic sharding to automatically adjust the number of
parallel writers based on data volume. It handles data skew by further
sub-dividing partitions into multiple shards to prevent bottlenecks during
high-throughput writes. Only available with 'hash' distribution mode.
+ A subset of column names to read exclusively. If null or empty, all
columns will be read.
</td>
</tr>
<tr>
<td>
- catalog_name
+ poll_interval_seconds
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Name of the catalog containing the table.
+ The interval at which to poll for new snapshots. Defaults to 60
seconds.
</td>
</tr>
<tr>
<td>
- catalog_properties
+ starting_strategy
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- Properties used to set up the Iceberg catalog.
+ The source's starting strategy. Valid options are: "earliest" or
"latest". Can be overriden by setting a starting snapshot or timestamp.
Defaults to earliest for batch, and latest for streaming.
</td>
</tr>
<tr>
<td>
- config_properties
+ streaming
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Properties passed to the Hadoop Configuration.
+ Enables streaming reads, where source continuously polls for snapshots
forever.
</td>
</tr>
<tr>
<td>
- direct_write_byte_limit
+ to_snapshot
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- For a streaming pipeline, sets the limit for lifting bundles into the
direct write path.
+ Reads up to this snapshot ID (inclusive).
</td>
</tr>
<tr>
<td>
- distribution_mode
+ to_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- Defines distribution of write data. Supported distributions:
-- none: don't shuffle rows (default)
-- hash: shuffle rows by partition key before writing data
+ Reads up to the latest snapshot (inclusive) created before this
timestamp (in milliseconds).
</td>
</tr>
+ </table>
+</div>
+
+### `POSTGRES` Read
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- drop
+ <strong>jdbc_url</strong>
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- A list of field names to drop from the input record before writing. Is
mutually exclusive with 'keep' and 'only'.
+ Connection URL for the JDBC source.
</td>
</tr>
<tr>
<td>
- keep
+ connection_properties
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- A list of field names to keep in the input record. All other fields
are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
</td>
Review Comment:
The phrase "as standalone parameter" is grammatically incorrect. It should
be "as a standalone parameter" or "as standalone parameters".
```suggestion
Used to set connection properties passed to the JDBC driver not
already defined as a standalone parameter (e.g. username and password can be
set using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -736,294 +840,279 @@ and Beam SQL is invoked via the Managed API under the
hood.
</tr>
<tr>
<td>
- keep
+ from_snapshot
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A subset of column names to read exclusively. If null or empty, all
columns will be read.
+ Starts reading from this snapshot ID (inclusive).
</td>
</tr>
- </table>
-</div>
-
-### `ICEBERG` Write
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>table</strong>
+ from_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A fully-qualified table identifier. You may also provide a template to
write to multiple dynamic destinations, for example:
`dataset.my_{col1}_{col2.nested}_table`.
+ Starts reading from the first snapshot (inclusive) that was created
after this timestamp (in milliseconds).
</td>
</tr>
<tr>
<td>
- autosharding
+ keep
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- Enables dynamic sharding to automatically adjust the number of
parallel writers based on data volume. It handles data skew by further
sub-dividing partitions into multiple shards to prevent bottlenecks during
high-throughput writes. Only available with 'hash' distribution mode.
+ A subset of column names to read exclusively. If null or empty, all
columns will be read.
</td>
</tr>
<tr>
<td>
- catalog_name
+ poll_interval_seconds
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Name of the catalog containing the table.
+ The interval at which to poll for new snapshots. Defaults to 60
seconds.
</td>
</tr>
<tr>
<td>
- catalog_properties
+ starting_strategy
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- Properties used to set up the Iceberg catalog.
+ The source's starting strategy. Valid options are: "earliest" or
"latest". Can be overriden by setting a starting snapshot or timestamp.
Defaults to earliest for batch, and latest for streaming.
</td>
</tr>
<tr>
<td>
- config_properties
+ streaming
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Properties passed to the Hadoop Configuration.
+ Enables streaming reads, where source continuously polls for snapshots
forever.
</td>
</tr>
<tr>
<td>
- direct_write_byte_limit
+ to_snapshot
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- For a streaming pipeline, sets the limit for lifting bundles into the
direct write path.
+ Reads up to this snapshot ID (inclusive).
</td>
</tr>
<tr>
<td>
- distribution_mode
+ to_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- Defines distribution of write data. Supported distributions:
-- none: don't shuffle rows (default)
-- hash: shuffle rows by partition key before writing data
+ Reads up to the latest snapshot (inclusive) created before this
timestamp (in milliseconds).
</td>
</tr>
+ </table>
+</div>
+
+### `POSTGRES` Read
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- drop
+ <strong>jdbc_url</strong>
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- A list of field names to drop from the input record before writing. Is
mutually exclusive with 'keep' and 'only'.
+ Connection URL for the JDBC source.
</td>
</tr>
<tr>
<td>
- keep
+ connection_properties
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- A list of field names to keep in the input record. All other fields
are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
</td>
</tr>
<tr>
<td>
- only
+ fetch_size
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- The name of a single record field that should be written. Is mutually
exclusive with 'keep' and 'drop'.
+ This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
</td>
Review Comment:
"per every database call" is slightly redundant. It should be "per database
call" or "for each database call".
```suggestion
This method is used to override the size of the data that is going
to be fetched and loaded in memory per database call. It should ONLY be used if
the default value throws memory errors.
```
##########
website/www/site/content/en/documentation/io/managed-io.md:
##########
@@ -736,294 +840,279 @@ and Beam SQL is invoked via the Managed API under the
hood.
</tr>
<tr>
<td>
- keep
+ from_snapshot
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A subset of column names to read exclusively. If null or empty, all
columns will be read.
+ Starts reading from this snapshot ID (inclusive).
</td>
</tr>
- </table>
-</div>
-
-### `ICEBERG` Write
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>table</strong>
+ from_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- A fully-qualified table identifier. You may also provide a template to
write to multiple dynamic destinations, for example:
`dataset.my_{col1}_{col2.nested}_table`.
+ Starts reading from the first snapshot (inclusive) that was created
after this timestamp (in milliseconds).
</td>
</tr>
<tr>
<td>
- autosharding
+ keep
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code>list[<span style="color: green;">str</span>]</code>
</td>
<td>
- Enables dynamic sharding to automatically adjust the number of
parallel writers based on data volume. It handles data skew by further
sub-dividing partitions into multiple shards to prevent bottlenecks during
high-throughput writes. Only available with 'hash' distribution mode.
+ A subset of column names to read exclusively. If null or empty, all
columns will be read.
</td>
</tr>
<tr>
<td>
- catalog_name
+ poll_interval_seconds
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Name of the catalog containing the table.
+ The interval at which to poll for new snapshots. Defaults to 60
seconds.
</td>
</tr>
<tr>
<td>
- catalog_properties
+ starting_strategy
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- Properties used to set up the Iceberg catalog.
+ The source's starting strategy. Valid options are: "earliest" or
"latest". Can be overriden by setting a starting snapshot or timestamp.
Defaults to earliest for batch, and latest for streaming.
</td>
</tr>
<tr>
<td>
- config_properties
+ streaming
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Properties passed to the Hadoop Configuration.
+ Enables streaming reads, where source continuously polls for snapshots
forever.
</td>
</tr>
<tr>
<td>
- direct_write_byte_limit
+ to_snapshot
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- For a streaming pipeline, sets the limit for lifting bundles into the
direct write path.
+ Reads up to this snapshot ID (inclusive).
</td>
</tr>
<tr>
<td>
- distribution_mode
+ to_timestamp
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- Defines distribution of write data. Supported distributions:
-- none: don't shuffle rows (default)
-- hash: shuffle rows by partition key before writing data
+ Reads up to the latest snapshot (inclusive) created before this
timestamp (in milliseconds).
</td>
</tr>
+ </table>
+</div>
+
+### `POSTGRES` Read
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- drop
+ <strong>jdbc_url</strong>
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- A list of field names to drop from the input record before writing. Is
mutually exclusive with 'keep' and 'only'.
+ Connection URL for the JDBC source.
</td>
</tr>
<tr>
<td>
- keep
+ connection_properties
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- A list of field names to keep in the input record. All other fields
are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
</td>
</tr>
<tr>
<td>
- only
+ fetch_size
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- The name of a single record field that should be written. Is mutually
exclusive with 'keep' and 'drop'.
+ This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
</td>
</tr>
<tr>
<td>
- partition_fields
+ location
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: green">str</code>
</td>
<td>
- Fields used to create a partition spec that is applied when tables are
created. For a field 'foo', the available partition transforms are:
-
-- `foo`
-- `truncate(foo, N)`
-- `bucket(foo, N)`
-- `hour(foo)`
-- `day(foo)`
-- `month(foo)`
-- `year(foo)`
-- `void(foo)`
-
-For more information on partition transforms, please visit
https://iceberg.apache.org/spec/#partition-transforms.
+ Name of the table to read from.
</td>
</tr>
<tr>
<td>
- sort_fields
+ num_partitions
</td>
<td>
- <code>list[<span style="color: green;">str</span>]</code>
+ <code style="color: #f54251">int32</code>
</td>
<td>
- Fields used to set the table's sort order, applied when the table is
created. Each entry has the form `<term> [asc|desc] [nulls first|nulls last]`,
where `<term>` is a field name or one of the partition transforms (e.g.
`bucket(col, 4)`, `day(ts)`). Direction defaults to ascending; null order
defaults to nulls-first for ascending and nulls-last for descending. Note: this
sets the table's declared sort order as metadata; it does not cause Beam to
physically sort records before writing.
-For more information on sort orders, please visit
https://iceberg.apache.org/spec/#sort-orders.
+ The number of partitions
</td>
</tr>
<tr>
<td>
- table_properties
+ output_parallelization
</td>
<td>
- <code>map[<span style="color: green;">str</span>, <span style="color:
green;">str</span>]</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Iceberg table properties to be set on the table when it is created.
-For more information on table properties, please visit
https://iceberg.apache.org/docs/latest/configuration/#table-properties.
+ Whether to reshuffle the resulting PCollection so results are
distributed to all workers.
</td>
</tr>
<tr>
<td>
- triggering_frequency_seconds
+ partition_column
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: green">str</code>
</td>
<td>
- For a streaming pipeline, sets the frequency at which snapshots are
produced.
+ Name of a column of numeric type that will be used for partitioning.
</td>
</tr>
- </table>
-</div>
-
-### `SQLSERVER` Read
-
-<div class="table-container-wrapper">
- <table class="table table-bordered">
- <tr>
- <th>Configuration</th>
- <th>Type</th>
- <th>Description</th>
- </tr>
<tr>
<td>
- <strong>jdbc_url</strong>
+ password
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Connection URL for the JDBC source.
+ Password for the JDBC source.
</td>
</tr>
<tr>
<td>
- connection_properties
+ read_query
</td>
<td>
<code style="color: green">str</code>
</td>
<td>
- Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
+ SQL query used to query the JDBC source.
</td>
</tr>
<tr>
<td>
- disable_auto_commit
+ username
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- Whether to disable auto commit on read. Defaults to true if not
provided. The need for this config varies depending on the database platform.
Informix requires this to be set to false while Postgres requires this to be
set to true.
+ Username for the JDBC source.
</td>
</tr>
+ </table>
+</div>
+
+### `POSTGRES` Write
+
+<div class="table-container-wrapper">
+ <table class="table table-bordered">
+ <tr>
+ <th>Configuration</th>
+ <th>Type</th>
+ <th>Description</th>
+ </tr>
<tr>
<td>
- fetch_size
+ <strong>jdbc_url</strong>
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: green">str</code>
</td>
<td>
- This method is used to override the size of the data that is going to
be fetched and loaded in memory per every database call. It should ONLY be used
if the default value throws memory errors.
+ Connection URL for the JDBC sink.
</td>
</tr>
<tr>
<td>
- location
+ autosharding
</td>
<td>
- <code style="color: green">str</code>
+ <code style="color: orange">boolean</code>
</td>
<td>
- Name of the table to read from.
+ If true, enables using a dynamically determined number of shards to
write.
</td>
</tr>
<tr>
<td>
- num_partitions
+ batch_size
</td>
<td>
- <code style="color: #f54251">int32</code>
+ <code style="color: #f54251">int64</code>
</td>
<td>
- The number of partitions
+ n/a
</td>
</tr>
<tr>
<td>
- output_parallelization
+ connection_properties
</td>
<td>
- <code style="color: orange">boolean</code>
+ <code style="color: green">str</code>
</td>
<td>
- Whether to reshuffle the resulting PCollection so results are
distributed to all workers.
+ Used to set connection properties passed to the JDBC driver not
already defined as standalone parameter (e.g. username and password can be set
using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
Review Comment:
The phrase "as standalone parameter" is grammatically incorrect. It should
be "as a standalone parameter" or "as standalone parameters".
```suggestion
Used to set connection properties passed to the JDBC driver not
already defined as a standalone parameter (e.g. username and password can be
set using parameters above accordingly). Format of the string must be
"key1=value1;key2=value2;".
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
