jhgoebbert opened a new issue, #54829: URL: https://github.com/apache/airflow/issues/54829
### Description Hi everyone, I recently came across the problem, that the git-fetch-requests of the dag-bundle hit the rate-limit of the git-server. I found out that Airflow cannot **take advantage of a higher rate-limit for authorized git-requests** if the repo is public as it never sends the credentials to the server. But Airflow could support to **force authentication** for a git connection type via a custom http.extraHeader as additional argument to the git command. This would allow authenticated Git operations even for public repositories. ### Use case/motivation Git does not authenticate when syncing public repositories, even if login and password are set. This is due to Git’s HTTPS behavior: **it always tries anonymous access first** and only sends credentials if the server responds with a 401 Unauthorized. => Public repos typically do not issue a 401, so credentials are never used. This is problematic because GitLab/GitHub apply different rate limits for authenticated vs anonymous requests. As a result, Airflow tasks can hit anonymous rate limits even when valid credentials are provided. **A known workaround** is to use Git’s http.extraHeader option to force authentication: ``` B64=$(printf 'x:%s' "<PROJECT_ACCESS_TOKEN>" | base64) git -c http.extraHeader="Authorization: Basic $B64" clone https://gitlab.com/<group>/<repo>.git ``` This ensures that all requests are authenticated from the first request, avoiding anonymous rate limits. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
