nailo2c opened a new pull request, #67140: URL: https://github.com/apache/airflow/pull/67140
closes: #36963 # How + Use the [Cloud Logging](https://docs.cloud.google.com/logging/docs/reference/libraries#listing_log_entries) library to fetch container logs. + Support both normal and deferrable mode. ### How to reproduce 1. Use `gcloud` to create a job template. ```bash export PROJECT_ID="<put-your-project-id>" export REGION="us-central1" export JOB_NAME="issue-36963-hello-job" gcloud run jobs create "${JOB_NAME}" \ --image="us-docker.pkg.dev/cloudrun/container/job:latest" \ --region="${REGION}" \ --project="${PROJECT_ID}" \ --labels="created-by=airflow-36963-repro" ``` 2. Set up the GCP connection. 3. Run the reproduction Dag. ```python with DAG( dag_id="issue_36963_cloud_run_logs_repro", start_date=datetime(2026, 1, 1), schedule=None, catchup=False, tags=["repro", "issue-36963", "cloud-run"], ) as dag: execute_job = CloudRunExecuteJobOperator( task_id="execute_cloud_run_job", project_id=PROJECT_ID, region=REGION, job_name=JOB_NAME, deferrable=False, ) execute_job_verbose = CloudRunExecuteJobOperator( task_id="execute_cloud_run_job_verbose", project_id=PROJECT_ID, region=REGION, job_name=JOB_NAME, deferrable=False, verbose=True, ) execute_job_deferrable = CloudRunExecuteJobOperator( task_id="execute_cloud_run_job_deferrable", project_id=PROJECT_ID, region=REGION, job_name=JOB_NAME, deferrable=True, ) execute_job_deferrable_verbose = CloudRunExecuteJobOperator( task_id="execute_cloud_run_job_deferrable_verbose", project_id=PROJECT_ID, region=REGION, job_name=JOB_NAME, deferrable=True, verbose=True, ) execute_job_failing_verbose = CloudRunExecuteJobOperator( task_id="execute_cloud_run_job_failing_verbose", project_id=PROJECT_ID, region=REGION, job_name=FAILING_JOB_NAME, verbose=True, ) ( execute_job >> execute_job_verbose >> execute_job_deferrable >> execute_job_deferrable_verbose >> execute_job_failing_verbose ) ``` # What ### Before the fix + Cloud Run has container logs such as `Starting Task #0, Attempt #0 ...`. <img width="1434" height="500" alt="36963_cloud_run_log" src="https://github.com/user-attachments/assets/603b1f9a-7788-4350-80d8-84dc238f7dbd"; /> + However, these logs are not shown in the Airflow UI. <img width="1910" height="571" alt="36963_before_the_fix" src="https://github.com/user-attachments/assets/99f1ea3a-8435-4c73-a7da-65b593e31214"; /> ### After the fix + Cloud Run container logs are shown in the Airflow UI as expected. <img width="1920" height="629" alt="36963_after_the_fix" src="https://github.com/user-attachments/assets/9caa8cae-e1be-4277-9588-b924f5d0ff40"; /> + Failed jobs also show their container logs before the task failure is raised. <img width="2556" height="553" alt="36963_cloud_run_fail_job_log_2" src="https://github.com/user-attachments/assets/3f7ad72b-2881-439e-839a-7924fd80af6e"; /> <img width="2549" height="759" alt="36963_fail_log_show_on_ui" src="https://github.com/user-attachments/assets/381c45fc-7cd4-48e1-bcb8-f39966d7efd3"; /> <br><br> --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [x] Yes (please specify the tool below) Generated-by: Codex 5.5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
