Hi Pavan, First of all, +1 to this.
Now, few things: * On naming: dataquality over dq for me honestly. Our existing provider names spell things out (common.sql, openlineage, not abbreviated forms) and dq is genuinely ambiguous outside context. * On scope: I also agree with Niko that #69413 is too large for one pass & I am glad to see the backend/UI split already happening in #69575. Would also suggest keeping the LLM assisted rule generation pieces (*schema-based generate_rules_from_schema*) out of the initial provider PR entirely cos as I see it, its a separable capability and bundling it will slow review of the core DQRule or RuleSet or operator surface, which is the part that actually needs the most detailed review. In short: go for it! Thanks & Regards, Amogh Desai On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <[email protected]> wrote: > In the meantime, the PR is ready for review. Feel free to review and > provide any feedback. > > Regards, > Pavan > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu < > [email protected]> > wrote: > > > Sorry, I forgot to add: the draft PR is here > > https://github.com/apache/airflow/pull/69413; it's still a WIP. > > > > some screenshots > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 :) > > > > Pavan > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu < > > [email protected]> wrote: > > > >> Hi Airflow community, > >> > >> I would like to start a discussion regarding a new provider: > >> apache-airflow-providers-dq. > >> > >> While Airflow already includes SQL check operators that many users rely > >> on for data quality, this new provider builds on that foundation by > >> introducing DQRule and RuleSet objects, stable rule identity, persisted > >> history, and direct connections to Airflow assets. This approach makes > >> quality results easier to inspect over time, allows downstream > consumers to > >> gate tasks based on recent quality results, and provides a unified > schema > >> for LLM-assisted workflows. Execution will continue to utilize existing > >> DbApiHook connections. > >> > >> The initial version of the provider is intentionally focused: > >> > >> - Declarative DQRule and RuleSet objects. > >> - DQCheckOperator and @task.dq_check. > >> - DbApiHook-based SQL checks, including built-in checks and > custom_sql. > >> - Persisted results for tasks, runs, and rules. > >> - A minimal Airflow UI plugin for viewing results and rule history. > >> - Experimental asset helpers such as asset_quality() and > >> require_quality(). > >> > >> Regarding scope, this first iteration uses object storage only to > persist > >> DQ results and history; checks are executed via database connections. > >> Future iterations may include file or object-store based checks (e.g., > S3, > >> GCS) where Airflow runs quality rules against data directly. > >> > >> This proposal does not require changes to Airflow core. Asset support is > >> currently provider-owned metadata, with static configuration stored on > the > >> asset and runtime summaries stored on asset events. If the provider > gains > >> traction, we can discuss making Data Quality a first-class component of > >> Airflow assets. > >> > >> This work also serves as a practical follow-up to the data quality > >> direction mentioned in AIP-99. Persisted history is valuable for users > and > >> future LLM-assisted workflows, such as those from Anthropic or > common.ai, > >> to understand rule performance and generate candidate rules based on > schema > >> context. > >> > >> A rough pseudo-flow is provided below: > >> > >> seed_rules = RuleSet( > >> name="orders_quality", > >> rules=[ > >> DQRule(name="order_id_not_null", check="null_count", > >> column="order_id", condition={"equal_to": 0}), > >> DQRule(name="amount_valid", check="min", column="amount", > >> condition={"geq_to": 0}), > >> ], > >> ) > >> > >> orders_asset = asset_quality( > >> Asset("orders"), > >> conn_id="warehouse", > >> table="orders", > >> ruleset=seed_rules, > >> ) > >> > >> # Optional: common.ai / Anthropic provider can generate a RuleSet from > >> schema context. > >> generated_rules = generate_rules_from_schema(...) > >> > >> @task.dq_check(asset=orders_asset) > >> def check_orders(ruleset): > >> return ruleset > >> > >> checked_orders = check_orders(generated_rules) > >> > >> with DAG("orders_consumer", schedule=orders_asset): > >> require_quality(orders_asset, min_score=0.95) >> consume_orders() > >> > >> The UI remains deliberately minimal for this initial release, focusing > on > >> result and history inspection. > >> > >> You can view examples [1] of how it's integrated with assets/llms. > >> > >> currently i named it providers `apache-airflow-providers-dq`. if any > >> other preference likely with `dataquality`. Please let me know if you > have > >> a preference. naming is hard :) > >> > >> [1]: > >> > https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/ > >> > >> Thanks, > >> Pavan > >> > >> >
