The GitHub Actions job "Tests (AMD)" on airflow.git/main has failed. Run started by GitHub user potiuk (triggered by potiuk).
Head commit for run: 40311ffe4d7fb19da91115959233dc218b3dbb15 / Hussein Awala <[email protected]> Add partition_key and partition_key_regexp_pattern filters for asset events (#64610) * Add partition_key and partition_key_pattern filters for asset events Add two new query parameters to the asset events API endpoints: - partition_key: exact-match filter leveraging the new composite B-tree index on (asset_id, partition_key) - partition_key_pattern: regex-based filter using database-native regexp_match for flexible pattern matching Includes Core API, Execution API, Task SDK (InletEventsAccessor), client, documentation, migration for the composite index, and tests. Made-with: Cursor * Regenerate supervisor schema snapshot for partition_key filters * Add ReDoS safeguards for partition_key_pattern regex filter The pattern is evaluated by the database's regex engine, so a pathological pattern could consume DB CPU (ReDoS). Mitigations: - Cap the pattern length (MAX_REGEX_PATTERN_LENGTH) in both the Core and Execution API validators, returning 400 for over-long patterns. - Bound the regex query with a transaction-local statement_timeout on PostgreSQL (apply_regex_query_timeout); no-op elsewhere. MySQL bounds regex evaluation via its built-in regexp_time_limit; SQLite has no server-side regex and MariaDB is unsupported. - Document the safeguards in assets.rst. * Gate regex query filters behind a config flag with configurable timeout Add two [api] configs (Airflow 3.4.0): - enable_regexp_query_filters (bool, default False): the partition_key_pattern filter passes a user-supplied regex to the database engine, which is a ReDoS vector, so it is disabled by default and must be explicitly enabled. Exact-match partition_key filtering is unaffected. - regexp_query_timeout (int seconds, default 30): the primary runtime mitigation, enforced as a PostgreSQL transaction-local statement_timeout; 0 disables it. When the flag is off, partition_key_pattern requests return HTTP 400 on both the Core and Execution APIs. Config descriptions document the security rationale. Adds tests for the disabled path and the timeout helper. * Revert pattern length cap for regex filters Drop MAX_REGEX_PATTERN_LENGTH; the opt-in flag and the PostgreSQL statement_timeout are the meaningful ReDoS safeguards, and a length cap does not stop short catastrophic patterns while risking false rejections of legitimate ones. * Fix CI: enable regex flag for by-alias tests, regenerate TS SDK schema - Add the enable_regexp_query_filters conf_vars fixture to TestGetAssetEventByAssetAliasPartitionKey (was missed), fixing the 4 by-alias regex tests that now hit the default-off flag; add a disabled->400 test for the alias endpoint too. - Regenerate ts-sdk/src/generated/supervisor.ts for the partition_key / partition_key_pattern comms fields (check-ts-sdk-supervisor-schema). * Address review feedback on partition key regexp filter - Rename the public/SDK parameter partition_key_pattern -> partition_key_regexp_pattern to distinguish regexp filtering from the substring "*_pattern" filters on other APIs. - Move the enable_regexp_query_filters gate into _RegexParam construction and the regex compile check into depends_regex, so a regexp filter can never be instantiated without the setting being enabled. - Drop the mutually-exclusive 400 between partition_key and partition_key_regexp_pattern (both now combine with AND) on the Core and Execution APIs and in the Task SDK client/accessor. - Reuse the shared QueryAssetEventPartitionKeyFilter/Regex params in the Execution API asset-event routes instead of duplicating the query params and validation. - Keep backend implementation details out of the public parameter description; simplify the config docs; revert an unrelated limit check. - Regenerate OpenAPI specs, UI TS clients, and the supervisor schema snapshots. * Collapse regexp config into one setting and scope the query timeout Address further review feedback: - Replace the enable_regexp_query_filters + regexp_query_timeout pair with a single [api] regexp_query_timeout: 0 (default) disables regexp filtering entirely, any positive value enables it and bounds the query runtime. This removes the "enabled but unbounded" combination. - Make apply_regex_query_timeout a context manager that sets a transaction-local statement_timeout on enter and resets it on exit, so the bound is scoped to the regexp query and does not leak to other statements in the request's transaction. Both asset-event routes apply it automatically around query execution instead of relying on a manual call in the view. * Regenerate UI query client for partition_key_regexp_pattern rename The hand-applied rename missed the type-block declarations and JSDoc in the generated query hooks (they were not on lines matching the AssetEvents hook name). Regenerate with `pnpm codegen` so the UI client matches the OpenAPI spec, fixing the "Compile / format / lint UI" static check. * Fix provide_session positional static check in partition-key tests The check-no-new-provide-session-positional hook flagged the partition-key regexp tests. The nine test methods never used the injected session, so drop their @provide_session decorator and unused session parameter; make session keyword-only in the _create_partition_key_test_data helper that does use it. * Address review: MySQL query timeout, float config, doc link, test tidy-up - Enforce regexp_query_timeout on MySQL too (session max_execution_time), not just PostgreSQL statement_timeout; apply_regex_query_timeout now branches by dialect and is a no-op on SQLite. - Make [api] regexp_query_timeout a float so fractional seconds (e.g. 0.5) are allowed; read it via conf.getfloat. - Link the config option in assets.rst via :ref: and note the MySQL bound. - Tests: create partition-key rows via an autouse fixture instead of a per-test helper call, fold the exact-match cases into parametrized tests (core + execution API), and cover the MySQL/float timeout paths. * Restore previous db timeout instead of clearing it Address review nit: apply_regex_query_timeout now captures the current statement_timeout (PostgreSQL) / max_execution_time (MySQL) before overriding it and restores that value on exit, instead of resetting to 0. This preserves a server- or role-level global timeout rather than clobbering it. * Apply regexp query timeout automatically via the filter dependency Address review feedback: instead of relying on each view to wrap its query in apply_regex_query_timeout (easy to forget -> ReDoS), the regexp filter's dependency (regex_param_factory) now applies the timeout to the request's session itself. Any endpoint using QueryAssetEventPartitionKeyRegex is bounded automatically. The dependency is function-scoped so it can depend on the session it bounds; the previous timeout value is restored on teardown. Removes the manual wrappers from the core and execution asset-event routes and adds tests for the dependency behavior. Report URL: https://github.com/apache/airflow/actions/runs/29751645499 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
