Hi Airflow community,
I would like to start a discussion regarding a new provider:
apache-airflow-providers-dq.
While Airflow already includes SQL check operators that many users rely on
for data quality, this new provider builds on that foundation by
introducing DQRule and RuleSet objects, stable rule identity, persisted
history, and direct connections to Airflow assets. This approach makes
quality results easier to inspect over time, allows downstream consumers to
gate tasks based on recent quality results, and provides a unified schema
for LLM-assisted workflows. Execution will continue to utilize existing
DbApiHook connections.
The initial version of the provider is intentionally focused:
- Declarative DQRule and RuleSet objects.
- DQCheckOperator and @task.dq_check.
- DbApiHook-based SQL checks, including built-in checks and custom_sql.
- Persisted results for tasks, runs, and rules.
- A minimal Airflow UI plugin for viewing results and rule history.
- Experimental asset helpers such as asset_quality() and
require_quality().
Regarding scope, this first iteration uses object storage only to persist
DQ results and history; checks are executed via database connections.
Future iterations may include file or object-store based checks (e.g., S3,
GCS) where Airflow runs quality rules against data directly.
This proposal does not require changes to Airflow core. Asset support is
currently provider-owned metadata, with static configuration stored on the
asset and runtime summaries stored on asset events. If the provider gains
traction, we can discuss making Data Quality a first-class component of
Airflow assets.
This work also serves as a practical follow-up to the data quality
direction mentioned in AIP-99. Persisted history is valuable for users and
future LLM-assisted workflows, such as those from Anthropic or common.ai,
to understand rule performance and generate candidate rules based on schema
context.
A rough pseudo-flow is provided below:
seed_rules = RuleSet(
name="orders_quality",
rules=[
DQRule(name="order_id_not_null", check="null_count",
column="order_id", condition={"equal_to": 0}),
DQRule(name="amount_valid", check="min", column="amount",
condition={"geq_to": 0}),
],
)
orders_asset = asset_quality(
Asset("orders"),
conn_id="warehouse",
table="orders",
ruleset=seed_rules,
)
# Optional: common.ai / Anthropic provider can generate a RuleSet from
schema context.
generated_rules = generate_rules_from_schema(...)
@task.dq_check(asset=orders_asset)
def check_orders(ruleset):
return ruleset
checked_orders = check_orders(generated_rules)
with DAG("orders_consumer", schedule=orders_asset):
require_quality(orders_asset, min_score=0.95) >> consume_orders()
The UI remains deliberately minimal for this initial release, focusing on
result and history inspection.
You can view examples [1] of how it's integrated with assets/llms.
currently i named it providers `apache-airflow-providers-dq`. if any other
preference likely with `dataquality`. Please let me know if you have a
preference. naming is hard :)
[1]:
https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
Thanks,
Pavan
