GitHub user bkarankar added a comment to the discussion: Error observed: AirflowRuntimeError in airflow version 3.1.2
This looks like a race condition around inserts into rendered_task_instance_fields, where multiple processes attempt to write the same (dag_id, task_id, run_id, map_index) record concurrently. We've seen similar behavior in highly parallel environments (e.g., CeleryExecutor with high worker concurrency). As a workaround, try disabling rendered task field storage: AIRFLOW__CORE__STORE_RENDERED_TI_FIELDS=False Also check your Airflow version, scheduler/API server replica count, and worker concurrency settings. It would be helpful to know whether this is reproducible on the latest Airflow patch release, as there may already be a fix or ongoing work related to this race condition. GitHub link: https://github.com/apache/airflow/discussions/61442#discussioncomment-17219171 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
