hkc-8010 commented on issue #72759:
URL: https://github.com/apache/airflow/issues/72759#issuecomment-5630507280
We hit the same root cause in production with a different leftover state,
which matters for your
"does #71535 cover this" question.
Airflow 3.3.0, providers-git 0.4.x, long-lived worker pods with pod-local
bundle storage. SIGTERM
landed 315 ms into the working clone, and `git clone` had got far enough to
write files into the
working tree but not far enough to record them in the index. So instead of
`pathspec ... did not match any file(s) known to git`, the checkout aborted
on untracked files:
```
git checkout master
error: The following untracked working tree files would be overwritten by
checkout:
.agents/instructions/airflow-patterns.md
<...>
Aborting
```
Every one of the 30 tasks that landed on that pod afterwards failed at
`task_runner.py parse` -> `git.py:215`, and it stayed broken until the
worker was replaced. Your
four reasons it does not self-heal apply unchanged.
The relevant part: #71535 would not recover this one. `_has_version(repo,
tracking_ref)` fails and
the fetch runs, the refs come back from the local bare mirror, and then the
checkout still aborts
because the untracked files are still sitting in the working tree. #72787's
broader move of the
checkout into the retry and `rmtree` block does recover it. That seems like
a concrete argument
for the broader fix rather than the narrow fetch, at least for this failure
family.
Your point about the residual case, where the clone dies before the remote
configuration is
written and `self.repo.remotes.origin` raises `AttributeError`, is the one
that convinced me the
recovery path is not the whole answer. I have written that up in #72922 with
a PR that stages the
clone in a sibling directory and renames it into place, so nothing lands at
`repo_path` until the
clone is finished. That removes the residual case too, and composes with
#72787 rather than
competing with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
