I seriously doubt it's the problem. There should be dag/tasks logs in your logs folder as well and they should tell you what happened. What database are you using ? Can you please dig deeper and provide more logs?
J, On Thu, Dec 19, 2019 at 1:28 AM Reed Villanueva <[email protected]> wrote: > Looking again at my lscpu specs, I noticed... > > [airflow@airflowetl airflow]$ lscpuArchitecture: x86_64 > CPU op-mode(s): 32-bit, 64-bitByte Order: Little Endian > CPU(s): 8On-line CPU(s) list: 0-7Thread(s) per core: > 1Core(s) per socket: 4Socket(s): 2 > > Notice Thread(s) per core: 1 > > Looking at my airflow.cfg settings I see max_threads = 2. Setting max_threads > = 1 and restarting both the scheduler > <https://www.astronomer.io/guides/airflow-scaling-workers/> seems to have > fixed the problem. > > If anyone knows more about what exactly is going wrong under the hood (eg. > why the task fails rather than just waiting for another thread to become > available), would be interested to hear about it. > > On Wed, Dec 18, 2019 at 11:45 AM Reed Villanueva <[email protected]> > wrote: > >> Running airflow dag that ran fine with SequentialExecutor now has many >> (though not all) simple tasks that fail without any log information when >> running with LocalExecutor and minimal parallelism, eg. >> >> <airflow.cfg> >> # overall task concurrency limit for airflow >> parallelism = 8 # which is same as number of cores shown by lscpu# max tasks >> per dag >> dag_concurrency = 2# max instances of a given dag that can run on airflow >> max_active_runs_per_dag = 1# max threads used per worker / core >> max_threads = 2 >> >> see https://www.astronomer.io/guides/airflow-scaling-workers/ >> >> Looking at the airflow-webserver.* logs nothing looks out of the >> ordinary, but looking at airflow-scheduler.out I see... >> >> [airflow@airflowetl airflow]$ tail -n 20 >> airflow-scheduler.out....[2019-12-18 11:29:17,773] {scheduler_job.py:1283} >> INFO - Executor reports execution of mydag.task_level1_table1 >> execution_date=2019-12-18 21:21:48.424900+00:00 exited with status failed >> for try_number 1[2019-12-18 11:29:17,779] {scheduler_job.py:1283} INFO - >> Executor reports execution of mydag.task_level1_table2 >> execution_date=2019-12-18 21:21:48.424900+00:00 exited with status failed >> for try_number 1[2019-12-18 11:29:17,782] {scheduler_job.py:1283} INFO - >> Executor reports execution of mydag.task_level1_table3 >> execution_date=2019-12-18 21:21:48.424900+00:00 exited with status failed >> for try_number 1[2019-12-18 11:29:18,833] {scheduler_job.py:832} WARNING - >> Set 1 task instances to state=None as their associated DagRun was not in >> RUNNING state[2019-12-18 11:29:18,844] {scheduler_job.py:1283} INFO - >> Executor reports execution of mydag.task_level1_table4 >> execution_date=2019-12-18 21:21:48.424900+00:00 exited with status success >> for try_number 1.... >> >> but not really sure what to take away from this. >> >> Anyone know what could be going on here or how to get more helpful >> debugging info? >> > > This electronic message is intended only for the named > recipient, and may contain information that is confidential or > privileged. If you are not the intended recipient, you are > hereby notified that any disclosure, copying, distribution or > use of the contents of this message is strictly prohibited. If > you have received this message in error or are not the named > recipient, please notify us immediately by contacting the > sender at the electronic mail address noted above, and delete > and destroy all copies of this message. Thank you. > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
