TRReeve commented on issue #8212: URL: https://github.com/apache/airflow/issues/8212#issuecomment-665915862
@Siddharthk If you're using any kind of distributed execution (including kubernetes) you wont be able to read logs from the airflow UI while the executor is running using airflow "out of the box". Airflow pushes the log to remote storage when the task is completed and then the UI reads them from there. I would guess if you do need to get real time logs from batch tasks while they are running maybe you can set up some sort of worker image with fluentd installed or something similar to scrape stdout and push it somewhere else while airflow is running on it? Either way there'll have to be some sort of external integration ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
