merobi-hub opened a new issue, #40309: URL: https://github.com/apache/airflow/issues/40309
### Apache Airflow version 2.9.2 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When running Airflow locally with the latest version of the OpenLineage provider, the scheduler shuts down after the attempt to pickle OpenLineageListener initializer fails. Fixed issue by downgrading the OpenLineage provider to 1.5.0. ### What you think should happen instead? ``` scheduler | [2024-06-18T13:57:05.429-0400] {scheduler_job_runner.py:860} ERROR - Exception when executing SchedulerJob._run_scheduler_loop scheduler | Traceback (most recent call last): scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/airflow/jobs/scheduler_job_runner.py", line 843, in _execute scheduler | self._run_scheduler_loop() scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/airflow/jobs/scheduler_job_runner.py", line 975, in _run_scheduler_loop scheduler | num_queued_tis = self._do_scheduling(session) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/airflow/jobs/scheduler_job_runner.py", line 1051, in _do_scheduling scheduler | self._start_queued_dagruns(session) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/airflow/jobs/scheduler_job_runner.py", line 1391, in _start_queued_dagruns scheduler | dag_run.notify_dagrun_state_changed() scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/airflow/models/dagrun.py", line 980, in notify_dagrun_state_changed scheduler | get_listener_manager().hook.on_dag_run_running(dag_run=self, msg=msg) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/pluggy/_hooks.py", line 513, in __call__ scheduler | return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/pluggy/_manager.py", line 120, in _hookexec scheduler | return self._inner_hookexec(hook_name, methods, kwargs, firstresult) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 139, in _multicall scheduler | raise exception.with_traceback(exception.__traceback__) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/pluggy/_callers.py", line 103, in _multicall scheduler | res = hook_impl.function(*args) scheduler | File "/Users/michael/Library/Python/3.9/lib/python/site-packages/airflow/providers/openlineage/plugins/listener.py", line 323, in on_dag_run_running scheduler | self.executor.submit( scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/process.py", line 697, in submit scheduler | self._adjust_process_count() scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/process.py", line 675, in _adjust_process_count scheduler | p.start() scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start scheduler | self._popen = self._Popen(self) scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen scheduler | return Popen(process_obj) scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__ scheduler | super().__init__(process_obj) scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__ scheduler | self._launch(process_obj) scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch scheduler | reduction.dump(process_obj, fp) scheduler | File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump scheduler | ForkingPickler(file, protocol).dump(obj) scheduler | AttributeError: Can't pickle local object 'OpenLineageListener.executor.<locals>.initializer' ``` ### How to reproduce Run `airflow standalone` using the latest versions of Airflow and the OpenLineage provider. ### Operating System MacOS Sonoma 14.5 ### Versions of Apache Airflow Providers apache-airflow-providers-celery==3.7.2 apache-airflow-providers-common-io==1.3.2 apache-airflow-providers-common-sql==1.14.0 apache-airflow-providers-fab==1.1.1 apache-airflow-providers-ftp==3.9.1 apache-airflow-providers-google==10.19.0 apache-airflow-providers-http==4.11.1 apache-airflow-providers-imap==3.6.1 apache-airflow-providers-openlineage==1.8.0 apache-airflow-providers-smtp==1.7.1 apache-airflow-providers-sqlite==3.8.1 ### Deployment Virtualenv installation ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org