1fanwang opened a new issue, #73768:
URL: https://github.com/apache/airflow/issues/73768
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.2
### What happened and how to reproduce it?
A client can't update just the slots of a pool. `PATCH
/api/v2/pools/{pool_name}?update_mask=slots` with `{"slots": 8}` returns 422
because the pool name and `include_deferred` are still required, even though
the mask leaves them out. A bulk update with the same mask fails the same way.
The same request works on `default_pool`.
As a result, every client has to send `include_deferred` on each update.
`airflowctl pools update --pool etl --slots 16` sends its flag default, false,
which turns deferred slots off on a pool that had them on.
To reproduce, run the script below against a local API server (SQLite,
simple auth manager). It takes the `airflow` of a venv with
`apache-airflow==3.3.2` from PyPI and an `airflowctl` built from main at
28cab90ccbb51d9dcf5daf8af178a959b8c9bbea:
```
$ bash pool-patch-e2e.sh /tmp/gt-af332/bin/airflow
/tmp/gt-ctl-main/bin/airflowctl
$ airflow version # the server
3.3.2
$ curl -X POST http://127.0.0.1:61485/api/v2/pools -d '{"name": "etl",
"slots": 4, "include_deferred": true}'
{"name":"etl","slots":4,"description":null,"include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":4,"deferred_slots":0,"team_name":null}
HTTP 201
$ curl -X PATCH 'http://127.0.0.1:61485/api/v2/pools/etl?update_mask=slots'
-d '{"slots": 8}'
{"detail":[{"type":"missing","loc":["pool"],"msg":"Field
required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"},{"type":"missing","loc":["include_deferred"],"msg":"Field
required","input":{"slots":8},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
$ curl -X PATCH
'http://127.0.0.1:61485/api/v2/pools/default_pool?update_mask=slots' -d
'{"slots": 150}'
{"name":"default_pool","slots":150,"description":"Default
pool","include_deferred":false,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":150,"deferred_slots":0,"team_name":null}
HTTP 200
$ curl -X PATCH http://127.0.0.1:61485/api/v2/pools -d '{"actions":
[{"action": "update", "entities": [{"name": "etl", "slots": 8}], "update_mask":
["slots"]}]}'
{"detail":[{"type":"missing","loc":["include_deferred"],"msg":"Field
required","input":{"slots":8,"pool":"etl"},"url":"https://errors.pydantic.dev/2.13/v/missing"}]}
HTTP 422
$ curl http://127.0.0.1:61485/api/v2/pools/etl
{"name":"etl","slots":4,"description":null,"include_deferred":true,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":4,"deferred_slots":0,"team_name":null}
HTTP 200
$ airflowctl pools update --pool etl --slots 16 # airflowctl has to send
include_deferred
[{"name": "etl", "slots": "16", "description": null, "include_deferred":
"False", "occupied_slots": "0", "running_slots": "0", "queued_slots": "0",
"scheduled_slots": "0", "open_slots": "16", "deferred_slots": "0", "team_name":
null}]
$ curl http://127.0.0.1:61485/api/v2/pools/etl
{"name":"etl","slots":16,"description":null,"include_deferred":false,"occupied_slots":0,"running_slots":0,"queued_slots":0,"scheduled_slots":0,"open_slots":16,"deferred_slots":0,"team_name":null}
HTTP 200
```
<details>
<summary>Reproducer source: pool-patch-e2e.sh</summary>
```bash
#!/usr/bin/env bash
# PATCH /api/v2/pools/{pool_name} with partial bodies against a real Airflow
API server (SQLite, simple auth manager).
# Usage: pool-patch-e2e.sh <airflow executable for the server> [airflowctl
executable]
set -u
AIRFLOW=${1:?usage: pool-patch-e2e.sh <airflow> [airflowctl]}
CTL=${2:-}
PORT=$(python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1",
0)); print(s.getsockname()[1])')
URL="http://127.0.0.1:$PORT";
SERVER_HOME="$(mktemp -d)"
CLIENT_HOME="$(mktemp -d)"
SERVER_ENV=(
AIRFLOW_HOME="$SERVER_HOME"
AIRFLOW__CORE__LOAD_EXAMPLES=False
AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_USERS=admin:admin
)
say() { printf '\n$ %s\n' "$1"; }
env "${SERVER_ENV[@]}" "$AIRFLOW" db migrate > "$SERVER_HOME/setup.log" 2>&1
|| { cat "$SERVER_HOME/setup.log"; exit 1; }
env "${SERVER_ENV[@]}" "$AIRFLOW" api-server --port "$PORT" --workers 1 >
"$SERVER_HOME/api-server.log" 2>&1 &
SERVER_PID=$!
trap 'kill "$SERVER_PID" 2>/dev/null; wait "$SERVER_PID" 2>/dev/null; rm -rf
"$SERVER_HOME" "$CLIENT_HOME"' EXIT
PASSWORDS="$SERVER_HOME/simple_auth_manager_passwords.json.generated"
for _ in $(seq 90); do
curl -sf "$URL/api/v2/monitor/health" > /dev/null && [ -s "$PASSWORDS" ]
&& break
sleep 1
done
PASSWORD=$(python3 -c 'import json, sys;
print(json.load(open(sys.argv[1]))["admin"])' "$PASSWORDS")
TOKEN=$(curl -s -X POST "$URL/auth/token" -H 'Content-Type:
application/json' \
-d "{\"username\": \"admin\", \"password\": \"$PASSWORD\"}" | python3 -c
'import json, sys; print(json.load(sys.stdin)["access_token"])')
api() { curl -s -w '\nHTTP %{http_code}\n' -H "Authorization: Bearer $TOKEN"
-H 'Content-Type: application/json' "$@"; }
say 'airflow version # the server'
env "${SERVER_ENV[@]}" "$AIRFLOW" version 2> /dev/null
say "curl -X POST $URL/api/v2/pools -d '{\"name\": \"etl\", \"slots\": 4,
\"include_deferred\": true}'"
api -X POST "$URL/api/v2/pools" -d '{"name": "etl", "slots": 4,
"include_deferred": true}'
say "curl -X PATCH '$URL/api/v2/pools/etl?update_mask=slots' -d '{\"slots\":
8}'"
api -X PATCH "$URL/api/v2/pools/etl?update_mask=slots" -d '{"slots": 8}'
say "curl -X PATCH '$URL/api/v2/pools/default_pool?update_mask=slots' -d
'{\"slots\": 150}'"
api -X PATCH "$URL/api/v2/pools/default_pool?update_mask=slots" -d
'{"slots": 150}'
BULK='{"actions": [{"action": "update", "entities": [{"name": "etl",
"slots": 8}], "update_mask": ["slots"]}]}'
say "curl -X PATCH $URL/api/v2/pools -d '$BULK'"
api -X PATCH "$URL/api/v2/pools" -d "$BULK"
say "curl $URL/api/v2/pools/etl"
api "$URL/api/v2/pools/etl"
if [ -n "$CTL" ]; then
export AIRFLOW_HOME="$CLIENT_HOME" AIRFLOW_CLI_TOKEN="$TOKEN"
"$CTL" auth login --api-url "$URL" --skip-keyring > /dev/null
say 'airflowctl pools update --pool etl --slots 16 # airflowctl has to
send include_deferred'
"$CTL" pools update --pool etl --slots 16
say "curl $URL/api/v2/pools/etl"
api "$URL/api/v2/pools/etl"
fi
```
</details>
### What you think should happen instead?
With `update_mask`, a PATCH should only need the fields named in the mask
and should leave the others as they are, for every pool and in bulk updates.
`default_pool` already works that way, because its branch in
`update_orm_from_pydantic` validates only the masked fields. Other pools are
validated against `BasePool`, which requires `pool`, `slots` and
`include_deferred`:
- The validation:
https://github.com/apache/airflow/blob/18fe1c6098605e089dcb7571b3839183f7b32752/airflow-core/src/airflow/api_fastapi/core_api/services/public/pools.py#L77-L89
- `BasePool`:
https://github.com/apache/airflow/blob/18fe1c6098605e089dcb7571b3839183f7b32752/airflow-core/src/airflow/api_fastapi/core_api/datamodels/pools.py#L44-L50
- `PoolPatchBody`, where every field is optional:
https://github.com/apache/airflow/blob/18fe1c6098605e089dcb7571b3839183f7b32752/airflow-core/src/airflow/api_fastapi/core_api/datamodels/pools.py#L78-L84
Once the server accepts that, airflowctl can send `update_mask` with the
fields a user passes and leave `include_deferred` out unless
`--include-deferred` or `--no-include-deferred` is given.
### Operating System
macOS 26.6.2
### Deployment
Virtualenv installation
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
Not Applicable
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Not Applicable
### Anything else?
The pool API code is the same on main at 18fe1c6. The existing tests cover a
full body, a name-only body without a mask (422), masks on `default_pool` and a
bulk mask with a complete entity, but not a partial body with a mask on another
pool.
https://github.com/apache/airflow/pull/71220 meets the same requirement from
the other side: for its cluster-wide setting it fills in the configured
`include_deferred` so that `BasePool` validation passes. A fix here would touch
those same lines.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
