YannByron commented on code in PR #6256:
URL: https://github.com/apache/hudi/pull/6256#discussion_r945547502
##########
rfc/rfc-51/rfc-51.md:
##########
@@ -64,69 +65,72 @@ We follow the debezium output format: four columns as shown
below
Note: the illustration here ignores all the Hudi metadata columns like
`_hoodie_commit_time` in `before` and `after` columns.
-## Goals
+## Design Goals
1. Support row-level CDC records generation and persistence;
2. Support both MOR and COW tables;
3. Support all the write operations;
4. Support Spark DataFrame/SQL/Streaming Query;
-## Implementation
+## Configurations
-### CDC Architecture
+| key | default | description
|
+|-----------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.table.cdc.enabled | `false` | The master
switch of the CDC features. If `true`, writers and readers will respect CDC
configurations and behave accordingly. |
+| hoodie.table.cdc.supplemental.logging | `false` | If `true`,
persist the required information about the changed data, including `before`. If
`false`, only `op` and record keys will be persisted. |
+| hoodie.table.cdc.supplemental.logging.include_after | `false` | If `true`,
persist `after` as well.
|
-
+To perform CDC queries, users need to set `hoodie.table.cdc.enable=true` and
`hoodie.datasource.query.type=incremental`.
-Note: Table operations like `Compact`, `Clean`, `Index` do not write/change
any data. So we don't need to consider them in CDC scenario.
-
-### Modifiying code paths
+| key | default | description
|
+|----------------------------------------|------------|--------------------------------------|
+| hoodie.table.cdc.enabled | `false` | set to `true` for CDC
queries |
+| hoodie.datasource.query.type | `snapshot` | set to `incremental`
for CDC queries |
+| hoodie.datasource.read.start.timestamp | - | requried.
|
+| hoodie.datasource.read.end.timestamp | - | optional.
|
-
+### Logical File Types
-### Config Definitions
+We define 4 logical file types for the CDC scenario.
Review Comment:
I think keeping them can help the developers to understand this design
details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
