ltshb commented on issue #50188:
URL: https://github.com/apache/arrow/issues/50188#issuecomment-6981019921

   I'm seeing the same hang with pyarrow 26.0.1.
   
   Setup: Airflow 3.4, Python 3.14, Airflow on EKS. Each task runs in its own 
EKS pod and gets AWS credentials through IRSA (web identity), which matches the 
STS/web-identity trigger mentioned above.
   
   The task finishes its work, but the process never exits. Downgrading to 
pyarrow 23.0.1 fixes it.
   
   Workaround for Airflow if you want to stay on a recent pyarrow: hard-exit 
the task process in an `on_success_callback`. In Airflow 3 the task has already 
reported SUCCESS when this callback runs, and exit code 0 keeps that state, so 
the deadlocking atexit handler is skipped:
   
   ```python
   import os
   import sys
   
   
   def hard_exit_on_success(context):
       # Workaround for https://github.com/apache/arrow/issues/50188
       sys.stdout.flush()
       sys.stderr.flush()
       os._exit(0)  # must be 0, otherwise Airflow marks the task as failed
   
   
   @dag(default_args={"on_success_callback": hard_exit_on_success}, ...)
   def my_dag(): ...
   ```
   
   Caveat: it must be the last success callback, because anything after it 
(other callbacks, success listeners) won't run.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to