mennanov opened a new pull request, #75570:
URL: https://github.com/apache/airflow/pull/75570
Hi! I'd like to introduce the tool i'm working on which replaces some of the
manual CI and pre-commit checks this repo has with intuitive declarative rules
written directly in the source code (and `blockwatch.toml` is some cases).
In this PR: 4 existing `prek` hooks are replaced with just one hook that
performs similar and even stricter checks. Same checks, much less code.
In the follow-up PR: add checks for parts that are supposed to stay in sync
but have no checks today.
## Replaced checks
| Removed hook | Script + test lines | Replaced by |
|---|---|---|
| `check-go-version-in-sync` | 197 + 130 | 5 entries in `blockwatch.toml` |
| `check-java-sdk-version-in-sync` | 152 | 5 entries in `blockwatch.toml` |
| `check-secrets-search-path-sync` | 92 | a `<block>` tag on each of the two
lists |
| `check-applies-to-roots-in-sync` | 117 + 84 | a `<block>` tag on each of
the two tables |
## How it works
Rules are declared in the comments next to the code:
```python
DEFAULT_SECRETS_SEARCH_PATH = [
# <block name="secrets-search-path"
same-as="task-sdk/src/airflow/sdk/execution_time/secrets/__init__.py:secrets-search-path">
"airflow.secrets.environment_variables.EnvironmentVariablesBackend",
"airflow.secrets.metastore.MetastoreBackend",
# </block>
]
```
Rules for the files that can't (or should not) have comments are declared in
`blockwatch.toml`
Running `blockwatch` from the repo root validates all the rules.
See [blockwatch](https://github.com/mennanov/blockwatch) for details about
the tool itself.
#### Found along the way
- the Go rule now also covers `k8s-tests.yml`, which is supposed to stay in
sync with `go-sdk/go.mod` but had no checks for that
- The common/ai plugin's `package.json` pins `[email protected]`. Everything else,
including `PNPM_VERSION` in the images, uses `10.28.1`. A follow-up PR fixes it
and adds a rule.
- Go is pinned at 1.25 (`go.mod`), 1.24.4 (Dockerfile defaults) and 1.27.2
(`Dockerfile.ci:1982`). `JAVA_SDK_VERSION` in breeze is 17, while Gradle
targets 11. These need a maintainer to say which value is right.
#### The hook
It is pinned to blockwatch v0.16.0 like the other external hooks, and
installs a prebuilt binary from PyPI (`blockwatch-cli`, MIT), so contributors
don't need Rust. A full run takes under a second. CI runs it in the Static
checks job.
#### Testing
`prek run blockwatch --all-files` passes. For each removed check, I made a
one-sided edit (bumped `go.mod`, swapped the secrets entries, changed an
`applies_to` value in the TS table only, and so on). Each time, blockwatch
failed and named exactly the files out of sync. The fast prek checks pass. I
left mypy to CI, since the Python changes are comments only.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 5.5) mostly for finding the existing checks
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
