We are getting the logs like {local_executor.py:43} INFO - LocalWorker running airflow run <TaskCommand> {models.py:1595} ERROR - Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally? {models.py:1616} INFO - Marking task as UP_FOR_RETRY
It seems that in LocalExecutor.py file "subprocess_checkcall" command is returning successfully without throwing an error but Task state did not get updated from queued to running in mysql store. One possibility is that "_check_and_change_state_before_execution" function in models.py is returning false due to which task state is not getting updated to "running" state and task get stuck in up_for_retry in handle_failure function. Other possibility could be unavailability of system resources to create a new process inside "subprocess_checkcall" function call but I think in that case it would be throwing an error. Is there a way to enable the "verbose" logging flag in _check_and_change_state_before_execution call. It would help us in debugging it further. Thanks, Raman Gupta On 2018/08/17 15:22:27, Matthias Huschle <matthias.husc...@paymill.de> wrote: > Hi Raman, > > Does it happen only occasionally, or can it be easily reproduced? > What happens if you start it with "airflow run" or " airflow test"? > What is in the logs about it? > What is your user process limit ("ulimit -u") on that machine? > > > 2018-08-17 15:39 GMT+02:00 ramandu...@gmail.com <ramandu...@gmail.com>: > > > Thanks Taylor, > > We are getting this issue even after restart. We are observing that task > > instance state is transitioned from > > scheduled->queued->up_for_retry and dag gets stuck in up_for_retry state. > > Behind the scenes executor keep on retrying the dag's task exceeding the > > max retry limit. > > In normal scenario task state should have following transition > > scheduled->queued->running->up_for_retry > > but we are seeing task is not entered in to running state rather it moves > > directly to up_for_retry from queued state. > > Any pointer on this would be helpful. > > > > Thanks, > > raman Gupta > > > > > > On 2018/08/16 16:05:31, Taylor Edmiston <tedmis...@gmail.com> wrote: > > > Does a scheduler restart make a difference? > > > > > > *Taylor Edmiston* > > > Blog <https://blog.tedmiston.com/> | CV > > > <https://stackoverflow.com/cv/taylor> | LinkedIn > > > <https://www.linkedin.com/in/tedmiston/> | AngelList > > > <https://angel.co/taylor> | Stack Overflow > > > <https://stackoverflow.com/users/149428/taylor-edmiston> > > > > > > > > > On Thu, Aug 16, 2018 at 4:25 AM, ramandu...@gmail.com < > > ramandu...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > We are using airflow 1.9 with Local Executor more. Intermittently we > > are > > > > observing that tasks are getting stuck in "up_for_retry" mode and are > > > > getting retried again and again exceeding their configured max retries > > > > count. like we have configured max retries as 2 but task is retried 15 > > > > times and got stuck in up_for_retry state. > > > > Any pointer on this would be helpful. > > > > > > > > Thanks, > > > > Raman Gupta > > > > > > > > > > > > > -- > > *< Dr. Matthias Huschle />* > > Data and Analytics Manager > > > > > > E: matthias.husc...@paymill.de > > Connect with me LinkedIn > <https://de.linkedin.com/pub/matthias-huschle/b0/912/91>| Xing > <https://www.xing.com/profile/Matthias_Huschle> > > > Credit & debit cards or PayPal? Now you can have both through the PAYMILL > API. Find out more » <https://www.paymill.com/payment-methods/paypal> > > > <https://paymill.atlassian.net/wiki/www.paymill.com/payment-methods/paypal>PAYMILL > GmbH | St.-Martin-Straße 63, 81669 München <https://www.paymill.com/> > <https://paymill.atlassian.net/wiki/www.paymill.com/payment-methods/paypal> > > Follow us @PAYMILL <https://twitter.com/Paymill> | www.PAYMILL.com > <http://www.paymill.com/> > > > The information in this e-mail is confidential and may be protected by > professional secrecy. It is intended solely for the addressee. Any access > to this e-mail is prohibited by persons other than the addressee. If you > are not the named addressee, you are prohibited to make any attempt to > publicise, reproduce or distribute this e-mail, this includes the taking or > refraining of action in regards to the information obtained. Please notify > the sender by e-mail immediately if you have received this e-mail by > mistake and delete this e-mail from your system. > PAYMILL GmbH | Sitz München | Amtsgericht München | HRB 226526 | > Steuernummer: 143/169/70894 | USt-ID: DE308345749 | Geschäftsführer: Daniel > Georges. >