Sorry, I forgot to add: the draft PR is here https://github.com/apache/airflow/pull/69413; it's still a WIP.
some screenshots https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 :) Pavan On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <[email protected]> wrote: > Hi Airflow community, > > I would like to start a discussion regarding a new provider: > apache-airflow-providers-dq. > > While Airflow already includes SQL check operators that many users rely on > for data quality, this new provider builds on that foundation by > introducing DQRule and RuleSet objects, stable rule identity, persisted > history, and direct connections to Airflow assets. This approach makes > quality results easier to inspect over time, allows downstream consumers to > gate tasks based on recent quality results, and provides a unified schema > for LLM-assisted workflows. Execution will continue to utilize existing > DbApiHook connections. > > The initial version of the provider is intentionally focused: > > - Declarative DQRule and RuleSet objects. > - DQCheckOperator and @task.dq_check. > - DbApiHook-based SQL checks, including built-in checks and custom_sql. > - Persisted results for tasks, runs, and rules. > - A minimal Airflow UI plugin for viewing results and rule history. > - Experimental asset helpers such as asset_quality() and > require_quality(). > > Regarding scope, this first iteration uses object storage only to persist > DQ results and history; checks are executed via database connections. > Future iterations may include file or object-store based checks (e.g., S3, > GCS) where Airflow runs quality rules against data directly. > > This proposal does not require changes to Airflow core. Asset support is > currently provider-owned metadata, with static configuration stored on the > asset and runtime summaries stored on asset events. If the provider gains > traction, we can discuss making Data Quality a first-class component of > Airflow assets. > > This work also serves as a practical follow-up to the data quality > direction mentioned in AIP-99. Persisted history is valuable for users and > future LLM-assisted workflows, such as those from Anthropic or common.ai, > to understand rule performance and generate candidate rules based on schema > context. > > A rough pseudo-flow is provided below: > > seed_rules = RuleSet( > name="orders_quality", > rules=[ > DQRule(name="order_id_not_null", check="null_count", > column="order_id", condition={"equal_to": 0}), > DQRule(name="amount_valid", check="min", column="amount", > condition={"geq_to": 0}), > ], > ) > > orders_asset = asset_quality( > Asset("orders"), > conn_id="warehouse", > table="orders", > ruleset=seed_rules, > ) > > # Optional: common.ai / Anthropic provider can generate a RuleSet from > schema context. > generated_rules = generate_rules_from_schema(...) > > @task.dq_check(asset=orders_asset) > def check_orders(ruleset): > return ruleset > > checked_orders = check_orders(generated_rules) > > with DAG("orders_consumer", schedule=orders_asset): > require_quality(orders_asset, min_score=0.95) >> consume_orders() > > The UI remains deliberately minimal for this initial release, focusing on > result and history inspection. > > You can view examples [1] of how it's integrated with assets/llms. > > currently i named it providers `apache-airflow-providers-dq`. if any > other preference likely with `dataquality`. Please let me know if you have > a preference. naming is hard :) > > [1]: > https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/ > > Thanks, > Pavan > >
