Subhashini2610 commented on issue #29031:
URL: https://github.com/apache/airflow/issues/29031#issuecomment-1399294156
@potiuk
I have been facing this issue for quite a while now and have been clueless
about the fix. Your insights into what might have caused the issue is
interesting. Excuse me for the lack of information provided.
My deployment: I have my deployment running on an AKS cluster in Azure with
2 user pools with 3 nodes each with sufficient memory and CPU (current
utilization stands at a max of 8% memory and 6% CPU). Also, I connect to the
PostgresDB hosted again in Azure and protected by a firewall. I understand this
could also be the cause. I am currently using the Kubernetes Executor.
Since the issue occurs randomly, after keeping a watch today, I was able to
get some debug logs of the executor when the scheduler stopped heartbeating for
sometime. Please find the logs captured during the occurrence of issue (between
2 heartbeats) below:
```
[2023-01-21T17:02:32.801+0000] {base_job.py:229} DEBUG - [heartbeat]
[2023-01-21T17:02:32.805+0000] {scheduler_job.py:898} DEBUG - Next timed
event is in 1.923924
[2023-01-21T17:02:32.805+0000] {scheduler_job.py:900} DEBUG - Ran scheduling
loop in 0.06 seconds
[2023-01-21T17:02:33.805+0000] {retries.py:84} DEBUG - Running
SchedulerJob._create_dagruns_for_dags with retries. Try 1 of 3
[2023-01-21T17:02:33.816+0000] {retries.py:84} DEBUG - Running
SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2023-01-21T17:02:33.826+0000] {retries.py:84} DEBUG - Running
SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2023-01-21T17:02:33.827+0000] {retries.py:84} DEBUG - Running
SchedulerJob._schedule_all_dag_runs with retries. Try 1 of 3
[2023-01-21T17:02:33.840+0000] {scheduler_job.py:355} DEBUG - No tasks to
consider for execution.
[2023-01-21T17:02:33.844+0000] {base_executor.py:163} DEBUG - 0 running task
instances
[2023-01-21T17:02:33.844+0000] {base_executor.py:164} DEBUG - 0 in queue
[2023-01-21T17:02:33.844+0000] {base_executor.py:165} DEBUG - 32 open slots
[2023-01-21T17:02:33.844+0000] {base_executor.py:174} DEBUG - Calling the
<class 'airflow.executors.kubernetes_executor.KubernetesExecutor'> sync method
[2023-01-21T17:02:33.845+0000] {kubernetes_executor.py:371} DEBUG - Syncing
KubernetesExecutor
[2023-01-21T17:02:33.845+0000] {kubernetes_executor.py:293} DEBUG -
KubeJobWatcher alive, continuing
[2023-01-21T17:02:33.845+0000] {kubernetes_executor.py:670} DEBUG - Next
timed event is in 0.805033
[2023-01-21T17:02:33.845+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:02:33.846+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:02:33.856+0000] {scheduler_job.py:898} DEBUG - Next timed
event is in 0.872712
[2023-01-21T17:02:33.856+0000] {scheduler_job.py:900} DEBUG - Ran scheduling
loop in 0.05 seconds
[2023-01-21T17:02:34.730+0000] {retries.py:84} DEBUG - Running
SchedulerJob._create_dagruns_for_dags with retries. Try 1 of 3
[2023-01-21T17:02:34.740+0000] {retries.py:84} DEBUG - Running
SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2023-01-21T17:02:34.749+0000] {retries.py:84} DEBUG - Running
SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2023-01-21T17:02:34.750+0000] {retries.py:84} DEBUG - Running
SchedulerJob._schedule_all_dag_runs with retries. Try 1 of 3
[2023-01-21T17:02:34.764+0000] {scheduler_job.py:355} DEBUG - No tasks to
consider for execution.
[2023-01-21T17:02:34.768+0000] {base_executor.py:163} DEBUG - 0 running task
instances
[2023-01-21T17:02:34.768+0000] {base_executor.py:164} DEBUG - 0 in queue
[2023-01-21T17:02:34.768+0000] {base_executor.py:165} DEBUG - 32 open slots
[2023-01-21T17:02:34.768+0000] {base_executor.py:174} DEBUG - Calling the
<class 'airflow.executors.kubernetes_executor.KubernetesExecutor'> sync method
[2023-01-21T17:02:34.768+0000] {kubernetes_executor.py:371} DEBUG - Syncing
KubernetesExecutor
[2023-01-21T17:02:34.768+0000] {kubernetes_executor.py:293} DEBUG -
KubeJobWatcher alive, continuing
[2023-01-21T17:02:34.769+0000] {kubernetes_executor.py:677} DEBUG - Looking
for pending worker pods older than 300 seconds
[2023-01-21T17:02:55.234+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6026)
[2023-01-21T17:02:55.245+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6030)
[2023-01-21T17:02:55.437+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor846-Process, stopped)>
[2023-01-21T17:02:55.532+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor847-Process, stopped)>
[2023-01-21T17:02:56.876+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6040)
[2023-01-21T17:02:57.131+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor848-Process, stopped)>
[2023-01-21T17:03:26.210+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6051)
[2023-01-21T17:03:26.212+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6047)
[2023-01-21T17:03:26.415+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor850-Process, stopped)>
[2023-01-21T17:03:26.510+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor849-Process, stopped)>
[2023-01-21T17:03:27.859+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6061)
[2023-01-21T17:03:28.034+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor851-Process, stopped)>
[2023-01-21T17:03:57.173+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6072)
[2023-01-21T17:03:57.186+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6068)
[2023-01-21T17:03:57.359+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor853-Process, stopped)>
[2023-01-21T17:03:57.467+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor852-Process, stopped)>
[2023-01-21T17:03:58.796+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6082)
[2023-01-21T17:03:59.017+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor854-Process, stopped)>
[2023-01-21T17:04:28.185+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6093)
[2023-01-21T17:04:28.231+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6089)
[2023-01-21T17:04:28.412+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor856-Process, stopped)>
[2023-01-21T17:04:28.516+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor855-Process, stopped)>
[2023-01-21T17:04:29.846+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6103)
[2023-01-21T17:04:30.021+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor857-Process, stopped)>
[2023-01-21T17:04:59.198+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6110)
[2023-01-21T17:04:59.208+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6114)
[2023-01-21T17:04:59.421+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor858-Process, stopped)>
[2023-01-21T17:04:59.512+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor859-Process, stopped)>
[2023-01-21T17:05:00.888+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6124)
[2023-01-21T17:05:01.056+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor860-Process, stopped)>
[2023-01-21T17:05:30.227+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6131)
[2023-01-21T17:05:30.229+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6135)
[2023-01-21T17:05:30.437+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor861-Process, stopped)>
[2023-01-21T17:05:30.542+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor862-Process, stopped)>
[2023-01-21T17:05:31.902+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6145)
[2023-01-21T17:05:32.082+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor863-Process, stopped)>
[2023-01-21T17:06:01.197+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6162)
[2023-01-21T17:06:01.199+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6166)
[2023-01-21T17:06:01.394+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor864-Process, stopped)>
[2023-01-21T17:06:01.513+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor865-Process, stopped)>
[2023-01-21T17:06:02.854+0000] {settings.py:407} DEBUG - Disposing DB
connection pool (PID 6183)
[2023-01-21T17:06:03.033+0000] {processor.py:296} DEBUG - Waiting for
<ForkProcess(DagFileProcessor866-Process, stopped)>
[2023-01-21T17:06:08.626+0000] {connectionpool.py:813} WARNING - Retrying
(Retry(total=2, connect=None, read=None, redirect=None, status=None)) after
connection broken by 'ProtocolError('Connection aborted.',
RemoteDisconnected('Remote end closed connection without response'))':
/api/v1/namespaces/airflow/pods?fieldSelector=status.phase%3DPending&labelSelector=airflow-worker%3D130&limit=100
[2023-01-21T17:06:08.652+0000] {rest.py:231} DEBUG - response body:
{"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"282652"},"items":[]}
[2023-01-21T17:06:08.653+0000] {kubernetes_executor.py:466} DEBUG - Clearing
tasks that have not been launched
[2023-01-21T17:06:08.657+0000] {kubernetes_executor.py:476} INFO - Found 0
queued task instances
[2023-01-21T17:06:08.657+0000] {kubernetes_executor.py:670} DEBUG - Next
timed event is in 59.999990
[2023-01-21T17:06:08.657+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.658+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.659+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.660+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.661+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.662+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.663+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.664+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.665+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.666+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.667+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.668+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.669+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.670+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.671+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.672+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.673+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.674+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.675+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.676+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.677+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.678+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.679+0000] {manager.py:275} DEBUG - Received message of
type DagParsingStat
[2023-01-21T17:06:08.702+0000] {base_job.py:229} DEBUG - [heartbeat]
```
Could you please help me localize the issue with the above logs? I hope the
above would help. Kindly help me proceed in the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
