Hi all, I was able to reproduce this issue, but at the same time I looked closer at the logs and I found the cause of it: it was a misconfigured nginx server block. All the files that were reported 404 by pacman are *.sig, and I had a location block config that turns off access logging for *.sig, and this config shouldn't have been on the reverse proxy at all which causes the reverse proxy to respond 404 to *.sig file requests instead of passing the requests to upstreams, that's why the 404s didn't appear in the access log, which had me confused for a few days.
Sorry everyone... stupid mistakes here. <-- That's why getting enough sleep is important, otherwise things like this happen :( On Wed, Sep 6, 2023 at 6:16 AM Tyler Dence <[email protected]> wrote: > > As a potential cause/possible test condition - I'd be curious if there was > some sort of time desync between the machine and mirror. Theoretically, if > the pacman client were "in the future" compared to the mirror, an > if-modified-since header could result in a 304. > > libalpm/dload.c:362 potentially supports this line of thinking, since the > timestamp used to query is the mtime of the file on the local filesystem as > returned by the stat call on 359 (looking at v6.0.1-133-gf9d8beef). > > On Tue, Sep 5, 2023, 14:30 pitastrudl <[email protected]> wrote: >> >> On 8/29/23 12:12, Jing Luo wrote: >> > Hi all, >> > >> > I don't know if this is an intended behavior for pacman. Today I ran >> > `pacman -Syyu` against the mirror hosted by me, and pacman output 404 >> > for all the packages: >> > >> > ``` >> > Total Download Size: 395.48 MiB >> > Total Installed Size: 2273.64 MiB >> > Net Upgrade Size: 133.43 MiB >> > >> > :: Proceed with installation? [Y/n] >> > :: Retrieving packages... >> > noto-fonts-cjk-20230817-1-any is up to date >> > linux-6.4.12.arch1-1-x86_64 is up to date >> > ktuberling-23.08.0-1-x86_64 is up to date >> > marble-common-23.08.0-1-x86_64 is up to date >> > klettres-23.08.0-1-x86_64 is up to date >> > linux-headers-6.4.12.arch1-1-x86_64 is up to date >> > minuet-23.08.0-1-x86_64 is up to date >> > kalzium-23.08.0-1-x86_64 is up to date >> > linux-docs-6.4.12.arch1-1-x86_64 is up to date >> > archlinux-appstream-data-20230827-2-any is up to date >> > Total ( 10/217) 395.5 MiB >> > 22.7 GiB/s 00:00 >> > [#######################################################################################] >> > 100% >> > error: failed retrieving file >> > 'noto-fonts-cjk-20230817-1-any.pkg.tar.zst.sig' from repo.jing.rocks : >> > The requested URL returned error: 404 >> > error: failed retrieving file >> > 'linux-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : >> > The requested URL returned error: 404 >> > error: failed retrieving file >> > 'archlinux-appstream-data-20230827-2-any.pkg.tar.zst.sig' from >> > repo.jing.rocks : The requested URL returned error: 404 >> > warning: too many errors from repo.jing.rocks, skipping for the >> > remainder of this transaction >> > error: failed retrieving file >> > 'ktuberling-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : >> > The requested URL returned error: 404 >> > error: failed retrieving file >> > 'linux-headers-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from >> > repo.jing.rocks : The requested URL returned error: 404 >> > error: failed retrieving file >> > 'marble-common-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks >> > : The requested URL returned error: 404 >> > error: failed retrieving file >> > 'kalzium-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The >> > requested URL returned error: 404 >> > error: failed retrieving file >> > 'klettres-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The >> > requested URL returned error: 404 >> > error: failed retrieving file >> > 'minuet-23.08.0-1-x86_64.pkg.tar.zst.sig' from repo.jing.rocks : The >> > requested URL returned error: 404 >> > error: failed retrieving file >> > 'linux-docs-6.4.12.arch1-1-x86_64.pkg.tar.zst.sig' from >> > repo.jing.rocks : The requested URL returned error: 404 >> > warning: failed to retrieve some files >> > error: failed to commit transaction (failed to retrieve some files) >> > Errors occurred, no packages were upgraded. >> > ``` >> > >> > I thought my mirror was out of date at first, so I ran the syncrepo >> > script manually, but it looks like only the file lists were >> > transferred. I searched on the arch forum, but in their cases, they >> > have out-of-date package index or they change the mirrorlist to solve >> > this problem. Not exactly my case. So I looked at the nginx access.log: >> > >> > ``` >> > - - - [29/Aug/2023:08:06:42 +0900] "GET >> > /archlinux/core/os/x86_64/core.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 >> > (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:42 +0900] "GET >> > /archlinux/multilib/os/x86_64/multilib.db HTTP/2.0" 304 0 "-" >> > "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:42 +0900] "GET >> > /archlinux/extra/os/x86_64/extra.db HTTP/2.0" 304 0 "-" "pacman/6.0.2 >> > (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/noto-fonts-cjk-20230817-1-any.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/core/os/x86_64/linux-6.4.12.arch1-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/ktuberling-23.08.0-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/marble-common-23.08.0-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/klettres-23.08.0-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/core/os/x86_64/linux-headers-6.4.12.arch1-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/minuet-23.08.0-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/kalzium-23.08.0-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/core/os/x86_64/linux-docs-6.4.12.arch1-1-x86_64.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > - - - [29/Aug/2023:08:06:43 +0900] "GET >> > /archlinux/extra/os/x86_64/archlinux-appstream-data-20230827-2-any.pkg.tar.zst >> > HTTP/2.0" 304 0 "-" "pacman/6.0.2 (Linux x86_64) libalpm/13.0.2" >> > ``` >> > >> > After a little bit of research, I found that nginx seems to conform to >> > the standard according to my setup: gzip off, telling the browser (in >> > this case, pacman) to use the client side local cache, returning a 304. >> > Note that my mirror site is set up like this: a load-balancer/reverse >> > proxy (repo.jing.rocks) in front of two web servers (repo1 and repo2) >> > connected to the same NFS storage backend. Caching to disk is not >> > enabled in nginx. >> > >> > After that I decided to put these three lines into my location block >> > so that nginx always returns 200: >> > >> > ``` >> > add_header Cache-Control "no-cache"; >> > etag off; >> > if_modified_since off; >> > ``` >> > >> > It looks all good now, except I can't reproduce this pacman bug (?) >> > anymore. Now that arch's gitlab closed new account registration, I >> > thought I should probably look for experienced mirror operators' input >> > before submitting a bug report. What do y'all think? Is this a bug? >> > >> > (Also my mirror hasn't been accepted after 2 months >> > https://bugs.archlinux.org/task/78991 >> > looks like the mirror admin team is taking their sweet sweet time?) >> > >> > BR, >> > >> >> Hey, >> >> I myself have not seen this error in my time being here and it seems it >> would be a good idea to open an issue on gitlab if you'd want someone to >> possibly look into it but if you cannot reliably reproduce it ,you can >> still ask for any ideas. Were there any updates between the problem and >> any possible upgrades that might fix it? The registration is disabled >> due to spam issues that affect generally most public gitlab instances >> and you can email [email protected] to open an account and >> open a bugreport there and someone who has more indepth knowledge of >> pacman might be able to help. >> >> Regards, >> Arun >> -- Jing Luo About me: https://jing.rocks/about/ PGP Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC
