On 06-01-26, Rich Brown wrote: > Thank you Baptiste, for this analysis. > > I re-sorted your list to by data consumption. it’s clear that the mirror > traffic is 4x the second place consumer, and 3x ALL the traffic. In fact, the > mirror traffic on its own seems to exceed the traffic cap. > > - rsync downloads from mirrors : 4 TB / day > - HTTP downloads.openwrt.org : 1 TB / day (mostly origin traffic for Fastly > CDN) > - HTTP sources.cdn.openwrt.org : 100 GB / day (mostly origin traffic for > Fastly CDN) > - rsync downloads from buildbot workers : 30 GB /day > - rsync sources : 5 GB /day > > I understand (and would support) your multi-tier suggestion below, but I am > going to suggest a heretical solution, not because I advocate for it, but > simply to start a discussion. > > Proposal: We should eliminate all mirrors. > > Background: 20 years ago (more or less) mirror download sites for popular > projects were essential. Primary hosts didn’t have the horsepower to handle > all the download requests and traffic. In addition, it allowed “far flung” > institutions to show their camaraderie and support for the project. They were > a good thing. Today, in the presence of CDNs and much higher speed hosts, > it’s not clear that all this “mirror machinery” is justified by the > complexity. > > Analysis: We should determine whether these mirrors actually help our central > traffic problem. How many mirror sites do we support? Do we know the outbound > traffic for In the mirror sites? What kind of data do they provide (sources? > binary images?) > > It is thinkable to me (although I have no information either way), that our > central servers + Fastly CDN could easily handle the requests from > individuals who otherwise would go to the mirrors. > > If that’s the case, we could eliminate all these mirrors, eliminate the > (human) communication and whatever infrastructure we support, save us some > bandwidth, and some brainpower. > > Of course, I may totally misunderstand the problem, or be proven to be > totally wrong - that’s OK. But still I felt I need to ask the question.
I like provocative thinking ;) One thing that is maybe not clear: we don't manage the mirrors ourselves, and the mirrors are not used by downloads.openwrt.org. All "official" traffic is already handled by the CDN and our single download server, with old releases handled by the archive server. So, mirrors don't cost us much in term of time and infrastructure (except this traffic problem). So far, the machinery is just "the rsync server is open to all". There are two kind of mirrors as far as I know: "far-flung institutions showing their support" as you say, and people running private mirrors (which I guess they have good reasons for, maybe they have large fleets of OpenWrt devices) There is an incomplete list here: https://openwrt.org/downloads#mirrors I think public mirrors can still be useful, it leaves alternative choices if we have a big outage of the download server and the archive server (unlikely, but still possible). Another reason is better performance: the CDN helps but still needs to fetch content from the download server in Germany. That being said, I agree the current number of mirrors is kind of unreasonable for this purpose. The idea of the 2-tiers system is precisely to delegate mirror distribution to a few trusted people/org that have the knowledge and infrastructure for this. Baptiste > > On Jan 6, 2026, at 06:13, Baptiste Jonglez <[email protected]> > > wrote: > > > > I think we should move to a 2-tier mirror architecture: have only 2-3 > > tier-1 mirrors that are allowed to pull from us, and then ask all other > > mirrors to become tier-2 and to pull from a tier-1 mirror. Tier 1 mirrors > > should mirror the entire archive and should commit to a long-term > > relationship. I can contact mirror operators I know, and if they agree, > > set things up this way. > > > > Of course buildbot workers and the archive server would still be allowed > > to rsync from the download server. > > > > > _______________________________________________ > openwrt-devel mailing list > [email protected] > https://lists.openwrt.org/mailman/listinfo/openwrt-devel _______________________________________________ openwrt-devel mailing list [email protected] https://lists.openwrt.org/mailman/listinfo/openwrt-devel
