GitHub user kdickinson87 edited a discussion: Add a new task status that would exist between Success and Failure, such as Warn, Needs Attention, or Degraded
We have several tasks across our airflow environment that are full load tasks, that process all records for a given API or dataset(some don't have a way to be made incremental/don't support it on vendor end), and failed records get put into a failed output/reprocess queue, and successful records are processed. Right now, when, say for example, >99% of the rows are successful, we either have to set the status of the run as successful, when there are failures, or failed, when >99% of the rows were successfully processed. Either way we currently continue processing of the successful rows, and the failures are investigated by our product team and get updated/picked up on the next run. It would be super helpful if there were another task status between success and failure to give to these tasks that aren't full Failures, but also not fully successful. Something like "Needs Attention" with an orange color, or "Warning" with an orange color. This would help parse out critical errors that mean "dag should stop processing completely and return error" and data errors, especially when it's a small subset of data, that would allow continued processing of the downstream tasks, while giving notice that there are items(be it data, etc) that need investigated. Similar to error severity levels. GitHub link: https://github.com/apache/airflow/discussions/54093 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
