On 06-01-26, Thibaut wrote: > > > > Le 6 janv. 2026 à 15:27, Baptiste Jonglez <[email protected]> a > > écrit : > > > > I like provocative thinking ;) > > Then here’s another thought :) > > > One thing that is maybe not clear: we don't manage the mirrors ourselves, > > and the mirrors are not used by downloads.openwrt.org. All "official" > > traffic is already handled by the CDN and our single download server, with > > old releases handled by the archive server. So, mirrors don't cost us > > much in term of time and infrastructure (except this traffic problem). > > So far, the machinery is just "the rsync server is open to all". > > > > There are two kind of mirrors as far as I know: "far-flung institutions > > showing their support" as you say, and people running private mirrors > > (which I guess they have good reasons for, maybe they have large fleets of > > OpenWrt devices) > > About that. I can give you an excellent reason to run a local mirror when you > depend on packages repositories being available and « working », e.g. when > frequently building lots of images using IB: when the CDN becomes out of sync > (say Packages doesn’t match the actual files available for download - and it > happens quite often), you’re screwed.
This is a separate issue we should fix. I believe we can just disable CDN caching for packages and Packages* files, and keep caching only for images and SDK/IB. I will try this. > As for the crazy amount of traffic, as mentioned on IRC I believe it’s partly > due to our (honestly broken) way of updating the packages repositories: > updating the entire archive whenever a single commit that affects a single > package is pushed cannot scale. > > I believe your proposal simply delays the inevitable: it will push down this > problem to tier-1 mirrors but eventually we will be caught back again as the > number of packages and platforms grow. > > I believe it would make more sense to work on having a saner build system > that can (build - that would also save CPU cycles - and) update only actually > modified packages. There’s been preliminary work in phase1 to address this > (at least for the ‘update’ part). Then all forms of mirror traffic should > naturally decrease as untouched bits remain valid in cache. Agreed but this is a much bigger undertaking. As suggested by Paul and Hauke, we will exclude snapshots from rsync mirroring as a first step, this should already help quite a lot (well, "delay the inevitable" as you say ;) ). The only thing that will be constantly updated by mirrors are packages for the active stable releases, typically the last two major releases. Baptiste _______________________________________________ openwrt-devel mailing list [email protected] https://lists.openwrt.org/mailman/listinfo/openwrt-devel
