fat-catTW commented on PR #71214: URL: https://github.com/apache/airflow/pull/71214#issuecomment-6397898881
I benchmarked the current backfill approach, and I would prefer to keep it without a read-side fallback to `callback.data`. Environment: - Breeze - PostgreSQL 14 - Synthetic legacy executor callbacks - `migration_batch_size=1,000` - Each scenario was run once | Rows | Legacy-data distribution | Upgrade: deadline | Upgrade: payload | Upgrade: total | Downgrade | | ---: | --- | ---: | ---: | ---: | ---: | | 10,000 | 100% deadline | 2.91s | 0.00s | 2.91s | 3.36s | | 10,000 | 50% deadline / 50% payload | 0.97s | 2.05s | 3.02s | 2.13s | | 10,000 | 100% payload | 0.01s | 3.68s | 3.69s | 1.80s | | 100,000 | 100% deadline | 38.18s | 0.01s | 38.19s | 16.61s | | 100,000 | 50% deadline / 50% payload | 13.95s | 33.55s | 47.50s | 16.59s | | 100,000 | 100% payload | 0.04s | 55.73s | 55.77s | 17.14s | | 1,000,000 | 100% deadline | 3m 03s | 0.32s | 3m 03s | 2m 58s | | 1,000,000 | 50% deadline / 50% payload | 1m 33s | 2m 58s | 4m 31s | 2m 55s | | 1,000,000 | 100% payload | 0.00s | 6m 05s | 6m 05s | 2m 58s | Note: During upgrade, the migration first backfills from `deadline.dagrun_id`, then falls back to legacy `callback.data["dag_run_id"]` for remaining callbacks. All scenarios backfilled every callback successfully. This is a local synthetic benchmark, so production hardware, concurrent workload, replication, and real data distribution may differ. However, it suggests the payload-only worst case is a bounded, one-time upgrade cost of about six minutes at one million rows. Based on this test, I think keeping the one-time backfill and avoiding a permanent read-side fallback is a reasonable trade-off. It lets the Scheduler rely consistently on the database-managed `callback.dagrun_id` relationship instead of continuing to support the legacy payload representation. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
