Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
jason810496 commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3846041722 It seems like the frontend problem? I found out that the Logs are stuck in Frontend sometime (are stuck at specific line number but the Frontend is still calling `get_log` API periodically) and the Frontend still showing the same amount of line numbers even I refresh the page. However, when I call the `get_log` API directly with `curl` and pipe to `wc` several times, the total line numbers increase. ```bash # first try curl .../logs/1?map_index=-1 | wc % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 413320 413320 0 249k 0 --:--:-- --:--:-- --:--:-- 250k 219 898 41332 # second try after several seconds curl .../logs/1?map_index=-1 | wc % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 418870 418870 0 542k 0 --:--:-- --:--:-- --:--:-- 545k 222 910 41887 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
jason810496 commented on issue #58676:
URL: https://github.com/apache/airflow/issues/58676#issuecomment-3845092376
> Any insights or examples you can share? Thanks!
I use the same Dag and same setup in Breeze today, but I'm unable to to
reproduce right now 😥
**Breeze setup**:
- Add the dag to `${AIRFLOW_REPO}/files/dags`
- `breeze start-airflow --use-airflow-version 3.1.6 --backend postgres`
- Add `/files/dags` to `PYTHONPATH` ( or `cp /files/dags/.py
/opt/airflow` for quick workaround )
**The Dag I use**:
```python
from __future__ import annotations
import asyncio
import os
import signal
import subprocess
from typing import Any
from airflow.sdk import dag, task, teardown
from airflow.sdk.bases.operator import BaseOperator
from airflow.triggers.base import BaseTrigger, TriggerEvent
class ProcessWaitTrigger(BaseTrigger):
"""Trigger that waits for a process to finish."""
def __init__(self, pid: int):
super().__init__()
self.pid = pid
def serialize(self) -> tuple[str, dict[str, Any]]:
return (
"no_logs_during_deferrable.ProcessWaitTrigger",
{"pid": self.pid},
)
async def run(self):
while True:
try:
# Check if process exists (pid 0 signal does not kill)
# os.kill is synchronous, but fast.
os.kill(self.pid, 0)
self.log.info("Process %d still running", self.pid)
await asyncio.sleep(1)
except OSError:
# Process dead
yield TriggerEvent({"status": "success"})
return
class ProcessWaitOperator(BaseOperator):
"""Operator that defers until a process finishes."""
template_fields = ("pid",)
def __init__(self, pid: int, **kwargs):
super().__init__(**kwargs)
self.pid = pid
def execute(self, context):
print(f">> Deferring process {self.pid} wait")
self.defer(
trigger=ProcessWaitTrigger(pid=int(self.pid)),
method_name="execute_complete"
)
def execute_complete(self, context, event=None):
print(f"Process {self.pid} finished")
return
@dag(
dag_id="no_logs_during_deferrable",
schedule=None,
catchup=False,
)
def no_logs_during_deferrable():
@task(task_id="submit_job")
def submit_job(**context):
# Run sleep 600 in background
process = subprocess.Popen(["sleep", "600"], close_fds=True)
print(f"Submitted process {process.pid}")
return process.pid
@teardown
@task(task_id="cancel_job")
def cancel_job(pid: int):
print(f"Killing process {pid}")
try:
os.kill(pid, signal.SIGTERM)
except OSError:
print(f"Process {pid} already gone")
pid = submit_job()
wait_job_finish = ProcessWaitOperator(task_id="wait_job_finish", pid=pid)
wait_job_finish >> cancel_job(pid)
no_logs_during_deferrable()
```
**Todays Logs**:
```
Log message source details
sources=["/root/airflow/logs/dag_id=no_logs_during_deferrable/run_id=manual__2026-02-04T03:37:09+00:00/task_id=wait_job_finish/attempt=1.log","/root/airflow/logs/dag_id=no_logs_during_deferrable/run_id=manual__2026-02-04T03:37:09+00:00/task_id=wait_job_finish/attempt=1.log.trigger.3.log","http://630c65abc04d:8794/log/dag_id=no_logs_during_deferrable/run_id=manual__2026-02-04T03:37:09+00:00/task_id=wait_job_finish/attempt=1.log.trigger.3.log";]
[2026-02-04 11:37:13] INFO - DAG bundles loaded: dags-folder
source=airflow.dag_processing.bundles.manager.DagBundlesManager
loc=manager.py:179
[2026-02-04 11:37:13] INFO - Filling up the DagBag from
/files/dags/no_logs_during_deferrable.py source=airflow.models.dagbag.DagBag
loc=dagbag.py:593
[2026-02-04 11:37:13] INFO - >> Deferring process 942 wait source=task.stdout
[2026-02-04 11:37:13] INFO - Pausing task as DEFERRED.
dag_id=no_logs_during_deferrable task_id=wait_job_finish
run_id=manual__2026-02-04T03:37:09+00:00 source=task loc=task_runner.py:928
[2026-02-04 11:37:14] INFO - trigger
no_logs_during_deferrable/manual__2026-02-04T03:37:09+00:00/wait_job_finish/-1/1
(ID 3) starting loc=triggerer_job_runner.py:1117
[2026-02-04 11:37:14] INFO - Process 942 still running
source=no_logs_during_deferrable.ProcessWaitTrigger
loc=no_logs_during_deferrable.py:33
[2026-02-04 11:37:15] INFO - Process 942 still running
source=no_logs_during_deferrable.ProcessWaitTrigger
loc=no_logs_during_deferrable.py:33
[2026-02-04 11:37:16] INFO - Process 942 still running
source=no_logs_during_deferrable.ProcessWaitTrigger
loc=no_logs_during_deferrable.py:33
[2026-02-0
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
Lee-W commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3844726680 > > then it disappears for the time of deferral, and then it re-appears after the task has finished. > > It seems like some of the triggers have the above behavior, so maybe the root cause will be in Triggerer. I just wrote a few DeferrableOperator and tested it out with `3.1.6` and all resulted as above described. Any insights or examples you can share? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
jason810496 commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3842886065 > then it disappears for the time of deferral, and then it re-appears after the task has finished. It seems like **all** the triggers have the above behavior, so maybe the root cause will be in Triggerer. I just wrote a few DeferrableOperator and tested it out with `3.1.6` and all resulted as above described. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
MarcusCramer91 commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3833619014 Hey, thanks for the answer! It is deployed via the `apache-airflow` (`apache-airflow/airflow`) helm chart v. 1.18.0 (which should be latest). I am overriding a some of the default values, but those overrides mostly concern resources, PVC and secrets. I don't think any of those should matter. I could send you sections of the compiled helm chart YAML with overrides, which sections would be particularly interesting? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
github-actions[bot] commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3802428554 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
Lee-W commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3707597521 > Hey [@Lee-W](https://github.com/Lee-W), sorry for the late answer (christmas days and all), > > I tried out the DAG you sent me. I can briefly see `pre_execute called by self.log`, then it disappears for the time of deferral, and then it re-appears after the task has finished. > > Initial: https://private-user-images.githubusercontent.com/7646847/531440862-963084fa-aff2-48dc-93d5-d3a63798be6f.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Njc0OTkwOTMsIm5iZiI6MTc2NzQ5ODc5MywicGF0aCI6Ii83NjQ2ODQ3LzUzMTQ0MDg2Mi05NjMwODRmYS1hZmYyLTQ4ZGMtOTNkNS1kM2E2Mzc5OGJlNmYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDEwNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjAxMDRUMDM1MzEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmI3YmNjNGJkOGNiZjk0YjI1MWYzODQ1YzNjZWM3MzNiMmJhN2E4NGE1NzMzY2RiNDdlMzE5MDBmZTNmZDA1MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.AGcgnw2OQQ9owpEu4AUB4XLETbdcHPjXPORzxgfe9Oc";> > > During wait: https://private-user-images.githubusercontent.com/7646847/531440931-7400175f-377b-475f-9e40-28bb45468685.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Njc0OTkwOTMsIm5iZiI6MTc2NzQ5ODc5MywicGF0aCI6Ii83NjQ2ODQ3LzUzMTQ0MDkzMS03NDAwMTc1Zi0zNzdiLTQ3NWYtOWU0MC0yOGJiNDU0Njg2ODUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDEwNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjAxMDRUMDM1MzEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NDAwMDNlMDQ2ZjYxNDQwZmUyMDk1MjBjN2ZlNWE0NzEyNjc0ZWRhNjliMjEwNjkyN2UxMDJkZjI3ZWNiYmQ3MSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.XJsiihjRpPJvZpIwBW2FpNImn58s9z8oMzpVyDZPGnE";> > > After completion: https://private-user-images.githubusercontent.com/7646847/531440955-383fd41d-73cd-4b94-8de7-9e0097779d4a.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Njc0OTkwOTMsIm5iZiI6MTc2NzQ5ODc5MywicGF0aCI6Ii83NjQ2ODQ3LzUzMTQ0MDk1NS0zODNmZDQxZC03M2NkLTRiOTQtOGRlNy05ZTAwOTc3NzlkNGEucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDEwNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjAxMDRUMDM1MzEzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NTMwZTNlNDJlMWEwOTRjYWM5MTE5ZGFhMmJlODcwMGQ0OTgxMDY2MzMwMDkwM2U4MWI2YjkyMTA4MTBmYjI1YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.PWMFoOH4oMpsry5sCxDFxhEOloOyyrqCB1VvRvLOxWQ";> > > You can see the source in the header change from `airflow-worker-0` to `airflow-triggerer-0`, so that must be the issue? No worries 🙂 Yep, I think this is a valid issue. Would you mind sharing the airflow configuration or how you deployed Airflow? Ideally, a minimum step-by-step setup could be `uv pip install airflow` or a Helm chart. I've tried a few ways but failed to reproduce it. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
MarcusCramer91 commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3704887120 Hey @Lee-W, sorry for the late answer (christmas days and all), I tried out the DAG you sent me. I can briefly see `pre_execute called by self.log`, then it disappears for the time of deferral, and then it re-appears after the task has finished. Initial: https://github.com/user-attachments/assets/963084fa-aff2-48dc-93d5-d3a63798be6f"; /> During wait: https://github.com/user-attachments/assets/7400175f-377b-475f-9e40-28bb45468685"; /> After completion: https://github.com/user-attachments/assets/383fd41d-73cd-4b94-8de7-9e0097779d4a"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
Lee-W commented on issue #58676:
URL: https://github.com/apache/airflow/issues/58676#issuecomment-3631572442
@MarcusCramer91, is there any airflow config you feel is suspicious? Or
maybe let's check whether the error happens in core or google-provider. Could
you please test with a Dag like the following one and see whether the log is
still disappearing?
```python
from __future__ import annotations
from datetime import datetime, timedelta
from airflow.providers.standard.triggers.temporal import TimeDeltaTrigger
from airflow.sdk import DAG, BaseOperator, Context
class SimpleDeferrableOperator(BaseOperator):
def __init__(self, wait_seconds: int = 5, **kwargs):
super().__init__(**kwargs)
self.wait_seconds = wait_seconds
def pre_execute(self, context: Context):
self.log.info(">>> pre_execute called by self.log")
super().pre_execute(context)
def execute(self, context: Context):
print(">>> Deferring task...")
self.defer(
method_name="execute_complete",
trigger=TimeDeltaTrigger(timedelta(seconds=self.wait_seconds)),
)
def execute_complete(self, context: Context, event=None):
print(">>> execute_complete called, event:", event)
return "completed"
with DAG(
dag_id="minimal_deferrable_dag",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
):
SimpleDeferrableOperator(
task_id="wait_then_complete",
wait_seconds=10,
)
```
If so, then we can almost confirm this is an issue from the core. And the
next step could be turn on and turn off your airflow config one by one and see
what makes this happened
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
Lee-W commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3622294831 If we could have a minimum Dag (ideally with a minimum custom deferrable operator that does almost nothing but logging), that would help us investigate it faster. But I'll try to create one operator that stays in deferred state longer and see whether I can reproduce it on my end as well. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Logs disappearing for deferred CloudRunExecuteJobOperator while state is deferred [airflow]
MarcusCramer91 commented on issue #58676: URL: https://github.com/apache/airflow/issues/58676#issuecomment-3616295617 @Lee-W Thanks for investigating. Happens 100% for me. Logs briefly appear (for like 10s), then they disappear while the task is deferred, and they re-appear once the task has finished. I suspect it has to do with deferred tasks being passed over to the triggerer. Any particular information regarding my setup that would help debugging? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
