MaicoTimmerman commented on code in PR #28626:
URL: https://github.com/apache/airflow/pull/28626#discussion_r1060149945
##########
airflow/config_templates/airflow_local_settings.py:
##########
@@ -286,6 +287,17 @@
},
}
DEFAULT_LOGGING_CONFIG["handlers"].update(OSS_REMOTE_HANDLERS)
+ elif REMOTE_BASE_LOG_FOLDER.startswith("webhdfs://"):
+ WEBHDFS_REMOTE_HANDLERS = {
+ "task": {
+ "class":
"airflow.providers.apache.hdfs.log.webhdfs_task_handler.WebHDFSTaskHandler",
+ "formatter": "airflow",
+ "base_log_folder": os.path.expanduser(BASE_LOG_FOLDER),
+ "webhdfs_log_folder": REMOTE_BASE_LOG_FOLDER,
+ "filename_template": FILENAME_TEMPLATE,
+ }
+ }
+ DEFAULT_LOGGING_CONFIG["handlers"].update(WEBHDFS_REMOTE_HANDLERS)
Review Comment:
I agree with you, the current solution scales very poorly.
Before I try and implement an improvement, I'm not sure what direction we
should take on this. I could create a class/function that does the same switch
logic and centralizes the common parameters, `base_log_folder`,
`REMOTE_BASE_LOG_FOLDER`, `FILENAME_TEMPLATE` and `formatter`. Is that what you
mean?
Adding to that, I feel the issue here is more regarding the non-uniformity
of the remote loggers, as I described in my other comment. If we are looking
for a better uniform solution, I propose we discuss the remote logging
architecture in an issue, as it would be a huge scope creep for this MR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
