XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify 
the logging configuration template
URL: https://github.com/apache/airflow/pull/6644#discussion_r349981433
 
 

 ##########
 File path: airflow/config_templates/airflow_local_settings.py
 ##########
 @@ -148,26 +128,61 @@
     }
 }
 
-REMOTE_HANDLERS = {
-    's3': {
+# Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is 
set.
+# This is to avoid exceptions when initializing RotatingFileHandler multiple 
times
+# in multiple processes.
+if os.environ.get('CONFIG_PROCESSOR_MANAGER_LOGGER') == 'True':
+    DEFAULT_LOGGING_CONFIG['handlers'] \
+        .update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'])
+    DEFAULT_LOGGING_CONFIG['loggers'] \
+        .update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['loggers'])
+
+    # Manually create log directory for processor_manager handler as 
RotatingFileHandler
+    # will only create file but not the directory.
+    processor_manager_handler_config = 
DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'][
+        'processor_manager']
+    directory = os.path.dirname(processor_manager_handler_config['filename'])
+    mkdirs(directory, 0o755)
+
+# Remote logging configuration
+
+# Storage bucket URL for remote logging
+# S3 buckets should start with "s3://"
+# GCS buckets should start with "gs://"
+# WASB buckets should start with "wasb"
+# just to help Airflow select correct handler
+REMOTE_BASE_LOG_FOLDER = conf.get('core', 'REMOTE_BASE_LOG_FOLDER')
+
+ELASTICSEARCH_HOST = conf.get('elasticsearch', 'HOST')
+
+REMOTE_LOGGING = conf.getboolean('core', 'remote_logging')
+
+if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
+    S3_REMOTE_HANDLERS = {
         'task': {
             'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
             'formatter': 'airflow',
 
 Review comment:
   `formatter`, `base_log_folder`, and `filename_template` are shared and are 
always the same in different handlers.
   Wonder if it's a good idea to have a `task template` (inside `formatter`, 
`base_log_folder`, and `filename_template` are defined), and create your 
`*_REMOTE_HANDLERS` using this template.
   
   This can avoid duplicated lines, and make potential future updating on 
`formatter`/`base_log_folder`/`filename_template` easier.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to