[ https://issues.apache.org/jira/browse/AIRFLOW-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allison Wang updated AIRFLOW-1325: ---------------------------------- Summary: Airflow Streaming Log Backed By ElasticSearch (was: Make Airflow Logging Backed By Elasticsearch) > Airflow Streaming Log Backed By ElasticSearch > --------------------------------------------- > > Key: AIRFLOW-1325 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1325 > Project: Apache Airflow > Issue Type: Improvement > Reporter: Allison Wang > Assignee: Allison Wang > > Currently, Airflow uses S3/GCS as the log storage backend. Workers, when > executing the task, flushes logs into local files. When tasks are completed, > those log files will be uploaded to the remote storage system like S3 or GCS. > This approach makes log streaming and analysis difficult. Also when worker > servers are down while executing the task, the entire task log will be lost > until worker servers are recovered. It's also considered a bad practice for > airflow webserver to communicate directly with worker servers. > This change adds functionality to use customized logging backend. Users are > able to configure logging backend that supports streaming logs and more > advanced queries. Currently, Elasticsearch logging backend is implemented. > This feature will also be backward compatible. It will direct users to the > old logging flow if logging_backend_url is not set. A new UI will be created > to support above features and old page won't be modified. -- This message was sent by Atlassian JIRA (v6.4.14#64029)