Hi all
First thing I would suggest is ask Netcup to increase the traffic limit.
Does it cost significantly ? If so how much ? Traffic nowadays is
normally pretty cheap everywhere in Datacenters.
If that is not an option reducing traffic generated (for example by
eliminating the SNAPSHOT mirror, or reduce it to fewer places) could be
an option.
Even finding another provider could be a preferred option I think.
Regarding the suggestion to focus on CDN usage I kind of understand the
idea, and most of it make sense, however in the other hand mirrors don't
cost anything (except for this current limitation) and have their value
in different ways.
Couldn't a sync process for mirrors to fetch data from the CDN instead
of the server in Germany be something to be considered ? I know it
wouldn't be done using rsync but perhaps some similar method can be
developed to achieve that securely.
Another idea in the line of what suggested would be to have one main
mirror in each country/region and ask other mirrors to sync from them,
but I would rather explore the above options first.
Regards
Fernando
On 1/6/2026 10:54 AM, Rich Brown wrote:
Thank you Baptiste, for this analysis.
I re-sorted your list to by data consumption. it’s clear that the mirror
traffic is 4x the second place consumer, and 3x ALL the traffic. In fact, the
mirror traffic on its own seems to exceed the traffic cap.
- rsync downloads from mirrors : 4 TB / day
- HTTP downloads.openwrt.org : 1 TB / day (mostly origin traffic for Fastly CDN)
- HTTP sources.cdn.openwrt.org : 100 GB / day (mostly origin traffic for Fastly
CDN)
- rsync downloads from buildbot workers : 30 GB /day
- rsync sources : 5 GB /day
I understand (and would support) your multi-tier suggestion below, but I am
going to suggest a heretical solution, not because I advocate for it, but
simply to start a discussion.
Proposal: We should eliminate all mirrors.
Background: 20 years ago (more or less) mirror download sites for popular
projects were essential. Primary hosts didn’t have the horsepower to handle all
the download requests and traffic. In addition, it allowed “far flung”
institutions to show their camaraderie and support for the project. They were a
good thing. Today, in the presence of CDNs and much higher speed hosts, it’s
not clear that all this “mirror machinery” is justified by the complexity.
Analysis: We should determine whether these mirrors actually help our central
traffic problem. How many mirror sites do we support? Do we know the outbound
traffic for In the mirror sites? What kind of data do they provide (sources?
binary images?)
It is thinkable to me (although I have no information either way), that our
central servers + Fastly CDN could easily handle the requests from individuals
who otherwise would go to the mirrors.
If that’s the case, we could eliminate all these mirrors, eliminate the (human)
communication and whatever infrastructure we support, save us some bandwidth,
and some brainpower.
Of course, I may totally misunderstand the problem, or be proven to be totally
wrong - that’s OK. But still I felt I need to ask the question.
Best regards,
Rich
On Jan 6, 2026, at 06:13, Baptiste Jonglez <[email protected]> wrote:
I think we should move to a 2-tier mirror architecture: have only 2-3
tier-1 mirrors that are allowed to pull from us, and then ask all other
mirrors to become tier-2 and to pull from a tier-1 mirror. Tier 1 mirrors
should mirror the entire archive and should commit to a long-term
relationship. I can contact mirror operators I know, and if they agree,
set things up this way.
Of course buildbot workers and the archive server would still be allowed
to rsync from the download server.
_______________________________________________
openwrt-adm mailing list
[email protected]
https://lists.openwrt.org/mailman/listinfo/openwrt-adm
_______________________________________________
openwrt-devel mailing list
[email protected]
https://lists.openwrt.org/mailman/listinfo/openwrt-devel
