mobuchowski opened a new pull request, #40078: URL: https://github.com/apache/airflow/pull/40078
This PR builds on https://github.com/apache/airflow/pull/39890 After this change, OpenLineage will execute metadata extraction in separate, forked process. It's a technique modeled to what interaction between `LocalTaskJobRunner` and `StandardTaskRunner` looks like - a process, in this case process of `StandardTaskRunner` watches over OpenLineage listener process during metadata extraction. This adds a layer of isolation between task execution and OpenLineage, adding a level of assurance that OpenLineage execution does not interfere with task execution in a way other than taking time. Additionally, this allows us to add configurable timeout for OL execute methods. The reason for that is, beyond configurability, that sometimes metadata extraction code can hang - for example, when dealing with Snowflake connection issue https://github.com/snowflakedb/snowflake-connector-python/pull/1898 - and we want to give as much guarantees that OL will not cause task to fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
