nikoki opened a new pull request, #72322: URL: https://github.com/apache/airflow/pull/72322
When a task attempt's log is uploaded to GCS more than once, the same lines get stored again and again — a log that should be a few MB can end up hundreds of MB or GB of repeated content. The common trigger is a reschedule-mode sensor. `UP_FOR_RESCHEDULE` does not increment `try_number`, so every poke writes to the same `attempt=N.log` key, and every poke is a separate worker process. With N pokes, poke 1's lines are stored N times, poke 2's N-1 times, and so on (duplication grows like N(N+1)/2). It only happens when a later lifecycle reuses the same local log path (long-lived Celery/Local workers, or a shared logs volume). This is the same bug already fixed for S3 (#67144) and WASB (#70860), both by truncating the local log after a successful upload. GCS never received the equivalent fix. ## Fix Truncate the local log file after a successful upload when `delete_local_copy` is `False`. ## Tests Updated `test_upload` to assert the local file is truncated after a successful upload (when not deleting) and left untouched after a failed upload (so a retry can still send it). Verified: reverting the fix makes exactly the 2 duplication-relevant `test_upload` parametrizations fail; with the fix, all 49 tests in `test_gcs_task_handler.py` pass. `ruff check` and `mypy` clean on the changed files. related: #67144, #70860 <!-- Read the Pull Request Guidelines for more information. https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
