jedcunningham commented on code in PR #50715:
URL: https://github.com/apache/airflow/pull/50715#discussion_r2103350513
##########
airflow-core/src/airflow/utils/log/log_reader.py:
##########
@@ -96,10 +99,15 @@ def read_log_stream(self, ti: TaskInstance, try_number: int
| None, metadata: di
TaskInstanceState.RUNNING,
TaskInstanceState.DEFERRED,
):
- if not logs[0]:
+ if not logs:
# we did not receive any logs in this loop
# sleeping to conserve resources / limit requests on
external services
time.sleep(self.STREAM_LOOP_SLEEP_SECONDS)
+ empty_iterations += 1
Review Comment:
We will never hit this block if the task running or deferred. Honestly,
bailing after multiple empty iterations was just me being a bit defensive -
bailing after the first instance would probably be fine too.
This is hit when we will never get the end of log mark - imagine that line
is simply missing for a finished task from ES. You can look and look and look,
but you will never get another line. In theory, each log handler could handle
this scenario gracefully, but having something generic here as a fail safe felt
like a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
