andrewgodwin commented on issue #17834: URL: https://github.com/apache/airflow/issues/17834#issuecomment-906001148
It's not _quite_ that flexible sadly - Operators must have an async version written along with an accompanying trigger, and since the trigger runs on a different process it still suffers the same serialisation problem. I doubt we'll ever be able to make any generic operator (PythonOperator counting as the "most generic" of all operators) run well at scale unless we change the Operator contract to bake-in multi-tenancy and serializability from the get-go, which seems... unlikely. Deferrable Operators (AIP-40), and Smart Sensors to a similar extent, require that the operator that's being made "more efficient" have a central "delay portion" that can be done from anywhere - for example, waiting on an external system. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
