Andreas Neumann created SPARK-56651:
---------------------------------------
Summary: Add AutoCDC Python API
Key: SPARK-56651
URL: https://issues.apache.org/jira/browse/SPARK-56651
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines, PySpark
Affects Versions: 4.1.1
Reporter: Andreas Neumann
Add an API to the exiting pipelines package to configure an AutoCDC flow.
Example:
{code:java}
dp.create_auto_cdc_flow(
target="customers",
source="customers_cdc_clean",
stored_as_scd_type="1",
keys=["id"],
sequence_by=col("operation_date"),
ignore_null_updates_column_list=[]
apply_as_deletes=expr("operation = 'DELETE'"),
except_column_list=["operation", "operation_date", "_rescued_data"],
) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
