Fang-Pen Lin created AIRFLOW-2325: ------------------------------------- Summary: Task logging with AWS Cloud watch Key: AIRFLOW-2325 URL: https://issues.apache.org/jira/browse/AIRFLOW-2325 Project: Apache Airflow Issue Type: New Feature Components: logging Reporter: Fang-Pen Lin
In many cases, it's ideal to use remote logging while running Airflow in production, as the worker could be easily scale down or scale up. Or the worker is running in containers, where the local storage is not meant to be there forever. In that case, the S3 task logging handler could be used [https://github.com/apache/incubator-airflow/blob/master/airflow/utils/log/s3_task_handler.py] However, it comes with drawback. S3 logging handler only uploads the log when the task completed or failed. For long running tasks, it's hard to know what's going on with the process until it finishes. To make more real-time logging, I built a logging handler based on AWS CloudWatch. It uses a third party python package `watchtower` [https://github.com/kislyuk/watchtower/tree/master/watchtower] -- This message was sent by Atlassian JIRA (v7.6.3#76005)