fahadabdul commented on PR #73433:
URL: https://github.com/apache/airflow/pull/73433#issuecomment-6339492432
A related symptom from the same versioned callback entry, which this PR
doesn't seem to cover: `dag_processing.import_errors` stays at 1 when there are
no import errors.
**What seems to happen**
1. A callback runs under a `DagFileInfo` with `bundle_version` set, so it
gets its own `_file_stats` entry.
2. If that callback run exceeds `dag_file_processor_timeout`,
`_kill_timed_out_processors` writes `DagFileStat(import_errors=1, ...)` to the
versioned entry. No `ParseImportError` row is written, so the UI and API show 0.
3. `remove_orphaned_file_stats` compares `presence_key`, so the versioned
entry survives while the file exists (as intended since #66484).
4. `emit_metrics` sums `import_errors` across all `_file_stats` values, so
the gauge keeps reporting 1 until the DAG processor restarts.
In the production case, the stats table showed 0 errors for the same file at
the same time, since `_log_file_processing_stats` reads the scan entry by exact
key.
**Repro** (Astronomer Runtime build of 3.3.2, `3.3.2+astro.2`, versioned
bundle, `dag_file_processor_timeout = 30`):
```python
import time
from datetime import datetime
from airflow.sdk import dag, task
def slow_callback(context):
time.sleep(120)
@dag(start_date=datetime(2026, 1, 1), schedule=None, catchup=False,
on_failure_callback=slow_callback)
def ghost_import_error_repro():
@task
def fail():
raise ValueError("fail on purpose")
fail()
ghost_import_error_repro()
```
After one triggered run, the callback was killed after about 30s, and
`dag_processing.import_errors` went from 0 to 1. The `import_error` table was
empty while the gauge was at 1. The gauge stayed at 1 after a new bundle
version was picked up (about 27 minutes on the same DAG processor), and went
back to 0 once the DAG processor was replaced.
We first saw this in production on a 3.2.2 build with #66484 backported. The
relevant code (`presence_key` cleanup, the timeout path, and `emit_metrics`) is
the same in OSS 3.3.2 and on main.
**Possible directions** (not sure which fits best):
- Don't record `import_errors=1` when the timed-out processor was a
versioned callback run, since a slow callback isn't an import error.
- Or have `emit_metrics` resolve per `presence_key` the same way this PR now
does for the queue checks (preferring the scan entry).
Happy to open a separate issue if you'd prefer to keep this PR focused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
