hussein-awala commented on issue #30062: URL: https://github.com/apache/airflow/issues/30062#issuecomment-1467175342
When using `LocalExecutor`, the [parallelism](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism) conf is considered as worker concurrency, in your case the scheduler may create 1800 tasks running in sub-processes (and there are already other processes created to process dag files, handle SLA, process callbacks, ...). Your system has a limit for the maximum number of open file descriptors, so when you reach this limit, any python script will try to open a file, will fail with this error `ValueError: filedescriptor out of range in select()`. (same for all the processes and not only python) You can check the limit in your system using the command: ```terminal $ ulimit -n 1024 ``` You can try to increase the limit and check if Airflow tasks will still fail or not: ```terminal $ ulimit -n 8192 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
