I don't think Allison's PR fixes logging, but it's a step in the right
direction. The current approach creates an abstraction around reading logs,
whereas the final solution should define an interface for writings to logs
in addition to reading logs (which could indeed use something like
Hi Bolke,
I agree that we should make logging configurable lbut I wouldn't think
using handlers like python-elasticsearch-logger is a good idea over
flushing logs into files. Here are some reasons:
1. Such handlers do not have the built-in backpressure-sensitive
protocol that can prevent
In the light of fixing logging, I would definitely appreciate written design.
Especially, as there have been multiple attempts to fix some issues but these
have been more like stop gap fixes.
In my opinion Airflow should not stipulate in a hard coded fashion where and
how logging takes place.
Responding to some of Bolke's concerns in the github PR for this change:
> Mmm still not convinced. Especially on elastic search it is just easier
to use the start_date to shard on.
sharding on start_date isn't great because there is still some risk of
collisions and it means that we are coupling
