potiuk edited a comment on pull request #17545: URL: https://github.com/apache/airflow/pull/17545#issuecomment-896678364
> One thing I observed from some recent discussions on operators (including this one) is, the operator actually support a lot of hooking mechanism if you are willing to subclass them, but maybe users are either unaware of the possibility, or (rationally or not) resisting doing so. If it’s the former, maybe we should push the idea more (but that’s not easy since inheritance hurts discoverability by its very nature). I think it is a bit irrational (because everything we do in DAG is Python, so subclassing the operator in DAG is pretty much the same as passing a function to constructor). But not entirely. I agree there is indeed a mental barrier there to a) discover that you can do it b) actually do sub-classing. A lot of people who interact with DAGs know python's functional side much more than the Object-Oriented one, and there is big group of people (this group has a large intersection with data-science profession) that sees the need of creating new classes as "alien". And I think we should embrace the feelings of those people rather than change their approach - especially that we can do it easily. Eventually you could also do it by monkey-patching your operator and yet no-one in the right mind thinks that is a good idea (I hope :D) Taking that in the consideration - I'd be for adding an optional `pre_execute` parameter actually (and `post_execute` as well). I am not too afraid of having more parameters in BaseOperator `__init__`, to be honest. When you have auto-complete, and when they are all optional, there is very little drawback to it, and we already force the kwarg-way of passing those operators. With the right documentation we could explicitly tell what is the difference and be even very explicit that "callbacks" might happen asynchronously, and "pre_", "post_" are always synchronous (and it actually makes perfect sense because nowhere in the callback it is actually strictly specified that it will be executed "BEFORE" execution. This is what "pre_" guarantee is all about. BTW. I thought a bit maybe a "Builder" pattern could be better in this case, but I think there is very little difference in python of whether you use constructor parameters, or builder-kind methods (and it would be far too much hassle to have Builders around). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
