Re: Routed optical networks
"I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen" Often the type of people making these kinds of predictions that a tire pressure sensor generates 20 gigabytes of traffic a day. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Vasilenko Eduard via NANOG" To: "Dave Taht" Cc: "NANOG" Sent: Thursday, May 11, 2023 2:33:58 AM Subject: RE: Routed optical networks But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.t...@gmail.com] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard Cc: Phil Bedard ; Etienne-Victor Depasquale ; NANOG Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG < nanog@nanog.org > wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto: nanog-bounces+vasilenko.eduard = huawei@nanog.org ] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale < ed...@ieee.org >; NANOG < nanog@nanog.org > Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds
Weekly Global IPv4 Routing Table Report
This is an automated weekly mailing describing the state of the Global IPv4 Routing Table as seen from APNIC's router in Japan. The posting is sent to APOPS, NANOG, AfNOG, SANOG, PacNOG, SAFNOG TZNOG, MENOG, BJNOG, SDNOG, CMNOG, LACNOG and the RIPE Routing WG. Daily listings are sent to bgp-st...@lists.apnic.net. For historical data, please see https://thyme.apnic.net. If you have any comments please contact Philip Smith . IPv4 Routing Table Report 04:00 +10GMT Sat 13 May, 2023 BGP Table (Global) as seen in Japan. Report Website: https://thyme.apnic.net Detailed Analysis: https://thyme.apnic.net/current/ Analysis Summary BGP routing table entries examined: 922283 Prefixes after maximum aggregation (per Origin AS): 349580 Deaggregation factor: 2.64 Unique aggregates announced (without unneeded subnets): 449017 Total ASes present in the Internet Routing Table: 74436 Prefixes per ASN: 12.39 Origin-only ASes present in the Internet Routing Table: 63869 Origin ASes announcing only one prefix: 26242 Transit ASes present in the Internet Routing Table: 10567 Transit-only ASes present in the Internet Routing Table:431 Average AS path length visible in the Internet Routing Table: 4.3 Max AS path length visible: 55 Max AS path prepend of ASN (265020) 50 Prefixes from unregistered ASNs in the Routing Table: 1095 Number of instances of unregistered ASNs: 1095 Number of 32-bit ASNs allocated by the RIRs: 41848 Number of 32-bit ASNs visible in the Routing Table: 34589 Prefixes from 32-bit ASNs in the Routing Table: 171578 Number of bogon 32-bit ASNs visible in the Routing Table:13 Special use prefixes present in the Routing Table:1 Prefixes being announced from unallocated address space:633 Number of addresses announced to Internet: 3057963648 Equivalent to 182 /8s, 68 /16s and 210 /24s Percentage of available address space announced: 82.6 Percentage of allocated address space announced: 82.6 Percentage of available address space allocated: 100.0 Percentage of address space in use by end-sites: 99.6 Total number of prefixes smaller than registry allocations: 308163 APNIC Region Analysis Summary - Prefixes being announced by APNIC Region ASes: 244341 Total APNIC prefixes after maximum aggregation: 69435 APNIC Deaggregation factor:3.52 Prefixes being announced from the APNIC address blocks: 238480 Unique aggregates announced from the APNIC address blocks:98015 APNIC Region origin ASes present in the Internet Routing Table: 13438 APNIC Prefixes per ASN: 17.75 APNIC Region origin ASes announcing only one prefix: 3959 APNIC Region transit ASes present in the Internet Routing Table: 1801 Average APNIC Region AS path length visible:4.5 Max APNIC Region AS path length visible: 26 Number of APNIC region 32-bit ASNs visible in the Routing Table: 8741 Number of APNIC addresses announced to Internet: 773569024 Equivalent to 46 /8s, 27 /16s and 186 /24s APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431 (pre-ERX allocations) 23552-24575, 37888-38911, 45056-46079, 55296-56319, 58368-59391, 63488-64098, 64297-64395, 131072-151865 APNIC Address Blocks 1/8, 14/8, 27/8, 36/8, 39/8, 42/8, 43/8, 49/8, 58/8, 59/8, 60/8, 61/8, 101/8, 103/8, 106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8, 116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8, 123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8, 163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8, 203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8, 222/8, 223/8, ARIN Region Analysis Summary Prefixes being announced by ARIN Region ASes:269926 Total ARIN prefixes after maximum aggregation: 122606 ARIN Deaggregation factor: 2.20 Prefixes being announced from the ARIN address blocks: 272521 Unique aggregates announced from the ARIN address blocks:130293 ARIN Region origin ASes present in the Internet Routing Table:19109 ARIN Prefixes per ASN:
LibreQos
Changing the topic... On Fri, May 12, 2023 at 7:11 AM Mark Tinka wrote: > > > > On 5/12/23 15:03, Dave Taht wrote: > > > Libreqos is free software, working as a bridge, you can plug it in > > between any two points on your network, and on cheap (350 bucks off of > > ebay) xeon gold hardware easily cracks 25Gbits while shaping with a > > goal of cracking 100Gbits one day soon. > > This is fantastic! :blush: We have done a couple podcasts about it, like this one: https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/ and have perhaps made a mistake by using matrix chat, rather than a web forum, to too-invisibly, do development and support in, but it has been a highly entertaining way to get a better picture of the real problems caring ISPs have. I see you are in Africa? We have a few ISPs playing with this in kenya... > > I also found your post about it here: > > https://www.reddit.com/r/HomeNetworking/comments/11pmc9a/a_latency_on_the_internet_update_bufferbloat_sqm/ > > If you could throw more hardware at it, could it do several 100's of Gbps? We do not know. Presently our work is supported by equinix´s open source program, with four servers in their Dallas DC, and they are 25Gbit ports. Putting together enough dough to get to 100Gbit or finding someone willing to send traffic through more bare metal at that data center or elsewhere is on my mind. In other words, we can easily spin up the ability to L2 route some traffic through a box in their DCs, if only we knew where to find it. :) If you assume linearity to cores (which is a lousy assumption, ok?), 64 Xeon cores could do about 200Gbit, running flat out. I am certain it will not scale linearly and we will hit multiple bottlenecks on a way to that goal. Limits we know about: A) Trying to drive 10s of gbits of realistic traffic through this requires more test clients and servers than we have, or someone with daring and that kind of real traffic in the first place. For example one of our most gung-ho clients has 100Gbit ports, but not anywhere near that amount of inbound traffic. (they are crazy enough to pull git head, try it for a few minutes in production, and then roll back or leave it up) B) A brief test of a 64 core AMD + Nvidia ethernet was severely outperformed by our current choice of a 20 core xeon gold + intel 710 or 810 card. It is far more the ethernet card that is the dominating factor. I would kill if I could find one that did a LPM -> CPU mapping... (e.g. instead of a LPM->route mapping, LPM to what cpu to interrupt). We also tried an 80 core arm to inconclusive results early on. Tests of the latest ubuntu release are ongoing. I am not prepared to bless that or release any results yet. C) A single cake instance on one of the more high end Xeons can *almost* push 10Gbit/sec while eating a core. D) Our model is one cake instance per subscriber + the ability to establish trees emulating links further down the chain. One ISP is modeling 10 mmwave hops. Another is just putting in multiple boxes closer to the towers. So in other words, 100s of gbits is achievable today if you throw boxes at it, and more cost effective to do that way. We will of course, keep striving to crack 100gbit native on a single box with multiple cards. It is a nice goal to have. E) In our present, target markets, 10k typical residential subscribers only eat 11Gbit/sec at peak. That is a LOT of the smaller ISPs and networks that fit into that space, so of late we have been focusing more on analytics and polish than pushing more traffic. Some of our new R/T analytics break down at 10k cake instances (that is 40 million fq_codel queues, ok?), and we cannot sample at 10ms rates, falling back to (presently) 1s conservatively. We are nearing putting out a v1.4-rc7 which is just features and polish, you can get a .deb of v1.4-rc6 here: https://github.com/LibreQoE/LibreQoS/releases/tag/v1.4-rc6 There is an optional, and anonymized reporting facility built into that. In the last two months, 44404 cake shaped devices shaping .19Tbits that we know of have come online. Aside from that we have no idea how many ISPs have picked it up! a best guess would be well over 100k subs at this point. Putting in libreqos is massively cheaper than upgrading all the cpe to good queue management, (it takes about 8 minutes to get it going in monitor mode, but exporting shaping data into it requires glue, and time) but better cpe remains desirable - especially that the uplink component of the cpe also do sane shaping natively. "And dang, it, ISPs of the world, please ship decent wifi!?", because we can see the wifi going south in many cases from this vantage point now. In the past year mikrotik in particular has done a nice update to fq_codel and cake in RouterOS, eero 6s have got quite good, much of openwifi/openwrt, evenroute is good... It feels good, after 14 years of trying to fix the internet, to be seeing such progress, on
Re: Routed optical networks
On 5/12/23 15:03, Dave Taht wrote: Libreqos is free software, working as a bridge, you can plug it in between any two points on your network, and on cheap (350 bucks off of ebay) xeon gold hardware easily cracks 25Gbits while shaping with a goal of cracking 100Gbits one day soon. This is fantastic! I also found your post about it here: https://www.reddit.com/r/HomeNetworking/comments/11pmc9a/a_latency_on_the_internet_update_bufferbloat_sqm/ If you could throw more hardware at it, could it do several 100's of Gbps? Also, when you say "bridge", if the server dies, does it become a wire, or would that require specialized hardware builds? Mark.
Re: Routed optical networks
On Thu, May 11, 2023 at 7:32 AM Mark Tinka wrote: > > > > On 5/11/23 13:25, Vasilenko Eduard via NANOG wrote: > > Sandvine is not representative of global traffic because DPI is installed > mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – > it is not the biggest source. Moreover, Mobiles would look better growing > because the limiting factor was on technology (5G proposed more than 4G, 4G > proposed much more than 3G) – it would probably would less disruptive in the > future. > > Fixed Carriers do not pay DPI premiums. And rarely share their traffic > publicly. Sandvine could not see it. > > > And QUIC is making the the DPI model so loved by MNO's quickly irrelevant. What we have been doing with libreqos is add in FQ + AQM (sch_cake), while pioneering some nifty new eBPF traffic monitoring abilities like being able to do "mice and elephants" plots, and more importantly sample TCP RTT statistics using a variant of kathie nichols' "passive ping" (pping) utility. This latter feature is proving very useful in diagnosing problems deeper in the network. Quic does respond to good ole queuing delays, packet drop, and in some cases ecn, but I am going to miss pping as it deploys more. It is the FQ that helps the most on videoconferencing and voip traffic. Libreqos is free software, working as a bridge, you can plug it in between any two points on your network, and on cheap (350 bucks off of ebay) xeon gold hardware easily cracks 25Gbits while shaping with a goal of cracking 100Gbits one day soon. demo here: https://payne.taht.net , running a variety of flent based stress tests, tcp up and downloads, the rrul test, etc. The interesting part to me, is that *real* traffic actually looks nothing like that on a per subscriber basis. Here´s a screen-movie example of real-world netflix queuing depth and delay: https://www.youtube.com/watch?v=C-2oSBr2200 Our new "piano roll" mice and elephants plotter is pretty neat, if I can say so myself. ... What we see from those ISPs that have shared their data from their Libreqos deployment is that the principal usage of major bandwidth is movie streaming, and there is essentially no difference in average usage between a 50Mbit residential plan and a gbit plan and only barely discernible at 25 vs 50. Given how new this codebase is we do not have any long term statistics for traffic growth or decline, or other patterns. But I kind of hope more folk here fire it up in their labs, at least. Please drop in on our chat channel #libreqos:matrix.org if you need help getting it setup. Sampling as we can at 10ms is a very different view of the net from 5 minute averages. Specifically plugging because I would like to understand CDN traffic patterns better, and do not have much data as yet on that side, collected this way. > Mark. -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos
Re: Routed optical networks
On Thu, May 11, 2023 at 4:25 AM Vasilenko Eduard wrote: > > I did investigate traffic for every Carrier while dealing with it as a > consultant (repeated many dozens of times). > > I have seen over a decade how traffic growth dropped year-over-year (from 60% > to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). There was a covid spike, but the trendline appeared to be down to 5% projected this year in a british study for fixed residential that I cannot find right now. > Sometimes I talk to old connections and they confirm that it is even less now. > > In rear cases, It is typically possible to find this information on the > public Internet (I remember the case when Google disclosed traffic for > Pakistan at the conference with the explanation that 30% is attributed to new > subscribers, and an additional +30% is to more heavy content per subscriber). > > But mostly, it was confidential information from a discussion with Carriers – > they all know very well their traffic growth. > > In general, traffic stat is pretty confidential. I did not have the > motivation to aggregate it. > > > > Sandvine is not representative of global traffic because DPI is installed > mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – > it is not the biggest source. Moreover, Mobiles would look better growing > because the limiting factor was on technology (5G proposed more than 4G, 4G > proposed much more than 3G) – it would probably would less disruptive in the > future. > > Fixed Carriers do not pay DPI premiums. And rarely share their traffic > publicly. Sandvine could not see it. > > > > VNI is claiming so many things. Please show where exactly they show traffic > growth (I am not interested in prediction speculations). Is it possible to > understand CAGR for the 5 last years? Is it declining or growing? (traffic > itself is for sure still growing) > > > > Of course, the disruption could come at any year and add a new S-curve > (Metaverse?). But disruption is by definition not predictable. > > > > PS: Everything above and below in this thread is just my personal opinion. > > > > Eduard > > From: Etienne-Victor Depasquale [mailto:ed...@ieee.org] > Sent: Thursday, May 11, 2023 12:48 PM > To: Vasilenko Eduard > Cc: Dave Taht ; Phil Bedard ; > NANOG > Subject: Re: Routed optical networks > > > > Eduard, academics cite the VNI (and the Sandvine Global reports). > > > > Do you know of alternative sources that show traffic growth data you're more > comfortable with? > > > > Cheers, > > > > Etienne > > > > On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard > wrote: > > But it is speculation, not a trend yet. > > I remember 10y ago every presentation started from the claim that 100B of IoT > would drive XXX traffic. It did not happen. > > Now we see presentations that AI would be talking to AI that generates > traffic. > > Maybe some technology would push traffic next S-curve, maybe not. It is still > speculation. > > > > The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, > 2) video quality for subscribers. Nothing else yet. > > It is almost finished for both trends. We are close to the plateau of these > S-curves. > > For some years (2013-2020) I was carefully looking at numbers for many > countries: it was always possible to split CAGR for these 2 components. The > video part was extremely consistent between countries. The subscriber part > was 100% proportional to subscriber CAGR. > > Everything else up to now was “marketing” to say it mildly. > > > > Reminder: nothing in nature could grow indefinitely. The limit always exists. > It is only a question of when. > > > > PS: Of course, marketing people could draw you any traffic growth – it > depends just on the marketing budget. > > > > Eduard > > From: Dave Taht [mailto:dave.t...@gmail.com] > Sent: Tuesday, May 9, 2023 11:41 PM > To: Vasilenko Eduard > Cc: Phil Bedard ; Etienne-Victor Depasquale > ; NANOG > Subject: Re: Routed optical networks > > > > Up until this moment I was feeling that my take on the decline of traffic > growth was somewhat isolated, in that I have long felt that we are nearing > the top of the S curve of the data we humans can create and consume. About > the only source of future traffic growth I can think of comes from getting > more humans online, and that is a mere another doubling. > > > > On the other hand, predictions such as 640k should be enough for everyone did > not pan out. > > > > On the gripping hand, there has been an explosion of LLM stuff of late, with > enormous models being widely distributed in just the past month: > > > > https://lwn.net/Articles/930939/ > > > > Could the AIs takeoff lead to a resumption of traffic growth? I still don´t > think so... > > > > > > On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG > wrote: > > Disclaimer: Metaverse has not changed Metro traffic yet. Then … > > > > I am
