GitHub user rahrajlat added a comment to the discussion: [Idea] LLM-powered smart retry operator for Airflow 3.x
This is a really interesting idea — the limitations of static retries in Airflow are definitely real, and your examples make a lot of sense. I’ve been exploring something similar from a broader AIOps perspective here: https://github.com/rahrajlat/airflow-ai-ops In my case, I’ve also been trying a self-learning approach, where the system improves its error classification over time based on past logs and outcomes, instead of relying only on fixed rules. The idea is to combine historical patterns with LLM-based reasoning, so decisions become more accurate as the system sees more failures. <img width="864" height="576" alt="image" src="https://github.com/user-attachments/assets/a1b52502-9baa-4b35-927d-d5943a2b1967"; /> There’s definitely overlap in using logs + LLMs to guide decisions like retries or alerts. I’ve been looking at it not just for retries, but also for things like smarter alerting, failure categorisation, and suggesting next actions. Personally, this feels like a great fit as a provider package rather than core Airflow, since it keeps things optional and flexible, and avoids adding complexity to the core retry mechanism. GitHub link: https://github.com/apache/airflow/discussions/64334#discussioncomment-16634651 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
