LeonardoSanBenitez commented on issue #58310: URL: https://github.com/apache/airflow/issues/58310#issuecomment-5225906585
On the api-server memory, there is a FAQ entry for that exact symptom that I should have pointed at the first time rather than guessing: https://airflow.apache.org/docs/apache-airflow/stable/faq.html#how-to-prevent-api-server-memory-growth. The api-server holds deserialized Dag objects in memory keyed by Dag version id, and the entry ties the growth directly to version inflation. The two levers are in different states by default on 3.3, which is worth checking before anything else: `[api] dag_cache_size` is new in 3.3.0 and already bounded at 64 (`dag_cache_ttl` 3600), so unless your deployment sets it to 0 the cache is not holding 54k versions — but rolling worker recycling is off, since `[api] server_type` defaults to `uvicorn` and `worker_refresh_interval` to `0`, and the recycling only takes effect under `gunicorn`. The docs note that worker recycling releases memory from any source, not only that cache. I have not measured either against a Dag your size, so that is the docu mented lever rather than a diagnosis of your crash. I also need to correct something in my earlier comment. I said that an unchanged `dag_code.source_code_hash` together with changed serialized data points at the parsing environment rather than at the Dag. That is wrong as a way of telling the two apart, and it matters for the monitoring you are describing. `source_code_hash` is a hash of the Dag file's text as read off disk (`DagCode.write_code` → `get_code_from_file`), and a file containing a random value does not change its text between parses either. On 3.3.0, five versions of a Dag with `uuid.uuid4()` in it share one `source_code_hash` and have five different `dag_hash` values — the same signature I attributed to #66103; as a control, editing the file between parses does give five different `source_code_hash` values. So that comparison separates "the file was edited" from "something else changed", and does not tell you which cause you have. Which makes the query you are planning a better instrument than the check for your case rather than a worse one: counting versions that share a `source_code_hash` looks at the result instead of the syntax, so it catches the helper-method pattern the static check misses. To go from the alarm to the cause, diffing the `serialized_dag` payload between two consecutive versions and seeing which field moved is the step that does separate them. And you are right about the check, with the limit a bit wider than "used directly": it cannot see through a function call at all, whether or not the helper lives in another file. It parses the single Dag file and only resolves a name it has itself seen assigned a recognised varying value, so `run_id = uuid.uuid4()` used in a task argument warns while `run_id = build_run_id()` does not — same file or imported, function or class method, and not even when the helper is what builds the task. Closing that would be a change to Airflow rather than something you can configure, so it is not worth waiting on; your own query is the faster route to the same alarm. --- Drafted-by: Claude Code (Opus 5); reviewed by @LeonardoSanBenitez before posting. If you believe this contribution was not useful to the airflow community, please let me know and I will stop posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
