gtxu commented on issue #17678: URL: https://github.com/apache/airflow/issues/17678#issuecomment-5233948515
Hi @ArshiAAkhavan, After reviewing this issue, I understand the motivation. However, from my understanding there is a concern with the idea of having the Scheduler directly fetch logs from all worker pods via the Kubernetes API. Firstly, Log aggregation is an I/O-intensive operation. Having the Scheduler maintain continuous API connections to stream logs from all active Pods would introduce massive network and memory overhead. Secondly, the current DaemonSet approach is exactly the official [Kubernetes best practice for cluster-level logging](https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures). Kubernetes encourages decoupling the core application logic from observability. Having a node-level logging agent (DaemonSet) read from `/var/log/containers/*.log` ensures that log processing doesn't consume the resources allocated to your application pods, and it handles log rotation and node failures gracefully. If the primary pain point is the heavy resource consumption of your current logging stack (like Logstash), a highly recommended approach is to switch to a more lightweight log shipper. Using **Fluent Bit** (instead of Filebeat/Logstash) as a DaemonSet typically consumes only a few megabytes of RAM per node while natively forwarding to Elasticsearch. If the Airflow want to implement API-based log fetching natively in it, it could not run within the main Scheduler loop. I think this would require a completely isolated, asynchronous logging daemon/process (to prevent blocking the Scheduler) — which essentially brings us back to managing a separate logging architecture anyway. Just want to comment my thought. Hope it helps in your workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
