hussein-awala commented on issue #30062:
URL: https://github.com/apache/airflow/issues/30062#issuecomment-1467175342

   When using `LocalExecutor`, the 
[parallelism](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism)
 conf is considered as worker concurrency, in your case the scheduler may 
create 1800 tasks running in sub-processes (and there are already other 
processes created to process dag files, handle SLA, process callbacks, ...).
   
   Your system has a limit for the maximum number of open file descriptors, so 
when you reach this limit, any python script will try to open a file, will fail 
with this error `ValueError: filedescriptor out of range in select()`. (same 
for all the processes and not only python)
   
   You can check the limit in your system using the command:
   ```terminal
   $ ulimit -n
   1024
   ```
   You can try to increase the limit and check if Airflow tasks will still fail 
or not:
   ```terminal
   $ ulimit -n 8192
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to