Routed optical networks
Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
Maybe some clarification as to what you're asking for would help. You're mixing fiber, networks, and a MAN. Fiber is just the medium. It could be for IP switching or projecting a light show. Are you asking if there are diverse paths throughout a metro area? On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG < nanog@nanog.org> wrote: > Hello folks, > > Simple question: does "routed optical networks" have a clear meaning in > the metro area context, or not? > > Put differently: does it call to mind a well-defined stack of technologies > in the control and data planes of metro-area networks? > > I'm asking because I'm having some thoughts about the clarity of this > term, in the process of carrying out a qualitative survey of the results of > the metro-area networks survey. > > Cheers, > > Etienne > > -- > Ing. Etienne-Victor Depasquale > Assistant Lecturer > Department of Communications & Computer Engineering > Faculty of Information & Communication Technology > University of Malta > Web. https://www.um.edu.mt/profile/etiennedepasquale >
Re: Routed optical networks
Hi Etienne In short, the idea is that optical networks are wasteful and routers do a better job making more use of a network's capacity than ROADMs. Take the extra router hop (or 3 or 8) versus short-cutting it with an optical network because the silicon is so low-latency anyway that it hardly makes a difference now. Putting more GBs per second on fewer strands means saving a lot of money on infrastructure costs. 400G ZR comes to mind as a foundational technology since it basically made active optical muxponder equipment obsolete in the metro. The savings here means telcos/enterprises can afford more router ports, which we've already established can utilize paths more efficiently anyway. Otherwise, this is more of a concept and can be executed with a variety of pre-existing technologies, or someone's new secret sauce that bakes everything together like SD-WAN did to its constituent technologies. -Matt On Mon, May 1, 2023 at 12:30 PM Etienne-Victor Depasquale via NANOG < nanog@nanog.org> wrote: > Hello folks, > > Simple question: does "routed optical networks" have a clear meaning in > the metro area context, or not? > > Put differently: does it call to mind a well-defined stack of technologies > in the control and data planes of metro-area networks? > > I'm asking because I'm having some thoughts about the clarity of this > term, in the process of carrying out a qualitative survey of the results of > the metro-area networks survey. > > Cheers, > > Etienne > > -- > Ing. Etienne-Victor Depasquale > Assistant Lecturer > Department of Communications & Computer Engineering > Faculty of Information & Communication Technology > University of Malta > Web. https://www.um.edu.mt/profile/etiennedepasquale > -- Matt Erculiani ERCUL-ARIN
RE: Routed optical networks
Hi Etienne, It depends on who is the owner of the fiber. The incumbent carrier typically has enough fiber strands to avoid any colored interfaces (that are 3x expensive compare to gray) in the Metro. Metro ring typically has 8-10 nodes (or similar). 16-20 strands of fiber were not possible to construct anyway – any cable is bigger. It is the same cost to lay down fiber on 16 strands or 32. Hence, PTT just does not need DWDM in Metro, not at all. Hence, the DWDM optimization that you are talking about below is not needed too. If you rent a single pair of fiber then you need colored interfaces to multiplex 8-10 nodes into 1 pair on the ring. Then the movement of transponders from DWDM into the router would eliminate 2 gray interfaces on every node (4 per link): one on the router side, and another on the DWDM side. Overall, it is about a 25% cost cut of the whole “router+DWDM”. It is still 2x more expensive compare to using additional fiber strands on YOUR fiber. By the way, about “well-defined stack of technologies”: NMS (polished by SDN our days) should be cross-layer: it should manage at the same time: ROADM/OADM in DWDM and colored laser in Router. It is a vendor lock up to now (no multi-vendor). Hence, 25% cost savings would go to the vendor that has such NMS, not to the carrier. Technology still does not make sense because no multivendor support between the NMS of one vendor and the router or DWDM of another. Looking at the NMS history, it would probably never be multi-vendor. For that reason, I am pessimistic about the future of the colored interfaces in routers (and alien lambdas in DWDM). Despite a potential 25% cost advantage in eliminating gray interfaces. PS: "routed optical networks" is proprietary marketing. Nobody understands what you mean. I did google to understand. Eduard From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On Behalf Of Etienne-Victor Depasquale via NANOG Sent: Monday, May 1, 2023 9:29 PM To: NANOG mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
Josh, thank you, your remarks (and those of Matt and Eduard) are helping me to understand better. For some context, please look at this graphic <https://drive.google.com/open?id=1HkXAbd2PIQF_ACOeQYDtpeGxTEaTT1Y7&authuser=etienne.depasquale%40um.edu.mt&usp=drive_fs> that shows the results of the question "Which of the following best describes your current dominant form of metro-aggregation?" The left bar chart shows *NOG reponses; the right bar chart shows responses obtained through market research among Tier 1 and regional operators. Operator group respondents overwhelmingly selected "routed optical networks over Ethernet without ROADMs". Now, during an interview I held to assess the answers, the interviewee (an experienced network engineer) questioned the meaning. I realized that I may have been conditioned by Cisco marketing (I attend a few webinars), and I wanted to understand what respondents understood. Summarizing an answer to your observation, I was conditioned by Cisco marketing. Cheers, Etienne On Mon, May 1, 2023 at 10:30 PM Josh Luthman wrote: > Maybe some clarification as to what you're asking for would help. You're > mixing fiber, networks, and a MAN. Fiber is just the medium. It could be > for IP switching or projecting a light show. Are you asking if there are > diverse paths throughout a metro area? > > On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG < > nanog@nanog.org> wrote: > >> Hello folks, >> >> Simple question: does "routed optical networks" have a clear meaning in >> the metro area context, or not? >> >> Put differently: does it call to mind a well-defined stack of >> technologies in the control and data planes of metro-area networks? >> >> I'm asking because I'm having some thoughts about the clarity of this >> term, in the process of carrying out a qualitative survey of the results of >> the metro-area networks survey. >> >> Cheers, >> >> Etienne >> >> -- >> Ing. Etienne-Victor Depasquale >> Assistant Lecturer >> Department of Communications & Computer Engineering >> Faculty of Information & Communication Technology >> University of Malta >> Web. https://www.um.edu.mt/profile/etiennedepasquale >> > -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
Hi Etienne, Below is our (Cisco) definition of the Routed Optical Network. The goal, metro or long haul or subsea, is to reduce the number of control planes. By migration TDM traffic using CEM or PLE to the IP layer, you eliminate the OTN control plane and management. Eventually, when standards are settled the ultimate goal is to have a single control plane for the network. I'm not trying to be a commercial here, but you can read more in the resources section on this page: https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html HTH, Eve Routed optical networking, is an architecture that delivers improved operational efficiencies and simplicity. The solution works by merging IP and private line services onto a single layer where all the switching is done at Layer 3. Routers are connected with standardized 400G ZR/ZR+ coherent pluggable optics. With a single service layer based upon IP, flexible management tools can leverage telemetry and model-driven programmability to streamline lifecycle operations. This simplified architecture integrates open data models and standard APIs, enabling a provider to focus on automation initiatives for a simpler topology. On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG < nanog@nanog.org> wrote: > Hello folks, > > Simple question: does "routed optical networks" have a clear meaning in > the metro area context, or not? > > Put differently: does it call to mind a well-defined stack of technologies > in the control and data planes of metro-area networks? > > I'm asking because I'm having some thoughts about the clarity of this > term, in the process of carrying out a qualitative survey of the results of > the metro-area networks survey. > > Cheers, > > Etienne > > -- > Ing. Etienne-Victor Depasquale > Assistant Lecturer > Department of Communications & Computer Engineering > Faculty of Information & Communication Technology > University of Malta > Web. https://www.um.edu.mt/profile/etiennedepasquale >
Re: Routed optical networks
On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote: > In short, the idea is that optical networks are wasteful and routers do a > better job making more use of a network's capacity than ROADMs. Take the > extra router hop (or 3 or 8) versus short-cutting it with an optical > network because the silicon is so low-latency anyway that it hardly makes a > difference now. Putting more GBs per second on fewer strands means saving a > lot of money on infrastructure costs. This is a very convoluted way of backing into the ole packet-switched vs. circuit switched decision. -- . ___ ___ . . ___ . \/ |\ |\ \ . _\_ /__ |-\ |-\ \__
Re: Routed optical networks
Hello Eve, Thank you for weighing in; I'm eager for feedback from the field. This eagerness stems from my work, over the past two years, to form my understanding of where current- and next-gen metro area networks are heading. I need this understanding to help academics in my field of specialization to better understand energy consumption in metro-area networks. Your observation about elimination of OTN resonates well with what I've heard from webinars, and what I've read in studies. It also matches what I've shown in the graphic I linked to in an earlier post in this thread (this graphic <https://drive.google.com/open?id=1HkXAbd2PIQF_ACOeQYDtpeGxTEaTT1Y7&authuser=etienne.depasquale%40um.edu.mt&usp=drive_fs> ). However, the larger operators are less inclined to drop OTN as a server layer network (layer network used as defined in G.805). Indeed, part of the scope of the question leading to the results shown, actually was to try to understand the prevalence of OTN in operators' current networks. As regards greenfield, the *NOG results are a bit more nuanced <https://drive.google.com/file/d/1Hl-UKJNMTguGCH3_g9cTiSDBdQAK4lya/view?usp=sharing> . IP/MPLS over Ethernet over DWDM with ROADMs for node bypass gets 34% of the vote, up from about 13% of what is currently in their networks. Cheers, Etienne On Tue, May 2, 2023 at 4:01 PM Eve Griliches wrote: > Hi Etienne, > Below is our (Cisco) definition of the Routed Optical Network. The goal, > metro or long haul or subsea, is to reduce the number of control planes. By > migration TDM traffic using CEM or PLE to the IP layer, you eliminate the > OTN control plane and management. Eventually, when standards are settled > the ultimate goal is to have a single control plane for the network. I'm > not trying to be a commercial here, but you can read more in the resources > section on this page: > https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html > HTH, > Eve > > Routed optical networking, is an architecture that delivers improved > operational efficiencies and simplicity. The solution works by merging IP > and private line services onto a single layer where all the switching is > done at Layer 3. Routers are connected with standardized 400G ZR/ZR+ > coherent pluggable optics. > > With a single service layer based upon IP, flexible management tools can > leverage telemetry and model-driven programmability to streamline lifecycle > operations. This simplified architecture integrates open data models and > standard APIs, enabling a provider to focus on automation initiatives for a > simpler topology. > > On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG < > nanog@nanog.org> wrote: > >> Hello folks, >> >> Simple question: does "routed optical networks" have a clear meaning in >> the metro area context, or not? >> >> Put differently: does it call to mind a well-defined stack of >> technologies in the control and data planes of metro-area networks? >> >> I'm asking because I'm having some thoughts about the clarity of this >> term, in the process of carrying out a qualitative survey of the results of >> the metro-area networks survey. >> >> Cheers, >> >> Etienne >> >> -- >> Ing. Etienne-Victor Depasquale >> Assistant Lecturer >> Department of Communications & Computer Engineering >> Faculty of Information & Communication Technology >> University of Malta >> Web. https://www.um.edu.mt/profile/etiennedepasquale >> > -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
Very helpful observations, Matt, thank you. How comfortably does the phrase "routed optical networks over Ethernet without ROADMs" sit with you? I mean: would you accept a limitation of "optical network" to the case of a network without optical layer switching (of the type done by add-drop multiplexers)? Cheers, Etienne On Mon, May 1, 2023 at 10:57 PM Matt Erculiani wrote: > Hi Etienne > > In short, the idea is that optical networks are wasteful and routers do a > better job making more use of a network's capacity than ROADMs. Take the > extra router hop (or 3 or 8) versus short-cutting it with an optical > network because the silicon is so low-latency anyway that it hardly makes a > difference now. Putting more GBs per second on fewer strands means saving a > lot of money on infrastructure costs. > > 400G ZR comes to mind as a foundational technology since it basically made > active optical muxponder equipment obsolete in the metro. The savings here > means telcos/enterprises can afford more router ports, which we've already > established can utilize paths more efficiently anyway. Otherwise, this is > more of a concept and can be executed with a variety of pre-existing > technologies, or someone's new secret sauce that bakes everything together > like SD-WAN did to its constituent technologies. > > -Matt > > > On Mon, May 1, 2023 at 12:30 PM Etienne-Victor Depasquale via NANOG < > nanog@nanog.org> wrote: > >> Hello folks, >> >> Simple question: does "routed optical networks" have a clear meaning in >> the metro area context, or not? >> >> Put differently: does it call to mind a well-defined stack of >> technologies in the control and data planes of metro-area networks? >> >> I'm asking because I'm having some thoughts about the clarity of this >> term, in the process of carrying out a qualitative survey of the results of >> the metro-area networks survey. >> >> Cheers, >> >> Etienne >> >> -- >> Ing. Etienne-Victor Depasquale >> Assistant Lecturer >> Department of Communications & Computer Engineering >> Faculty of Information & Communication Technology >> University of Malta >> Web. https://www.um.edu.mt/profile/etiennedepasquale >> > > > -- > Matt Erculiani > ERCUL-ARIN > -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
> On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG > wrote: > > On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote: > > In short, the idea is that optical networks are wasteful and routers do a > > better job making more use of a network's capacity than ROADMs. Take the > > extra router hop (or 3 or 8) versus short-cutting it with an optical > > network because the silicon is so low-latency anyway that it hardly makes a > > difference now. Putting more GBs per second on fewer strands means saving a > > lot of money on infrastructure costs. > > This is a very convoluted way of backing into the ole packet-switched > vs. circuit switched decision. > > I don't follow. > While ROADMs can be thought of as circuit-switchers, > the number of concurrent clients and switching latency put ROADMs on a > different operational level than packet switchers, right? I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that. I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing. I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical. - Jared
Re: Routed optical networks
So right Jaredmagic has been in the NPU capacity increase that's driven the cost per 100G down on 1RU routers; and the integration of DSPs and more into QSFP-DD form factors at much lower power than expected. The standards for optical links are maturing as well, but we still have work to do on the management side for the electrical interfaces. Eve On Tue, May 2, 2023 at 3:33 PM Jared Mauch wrote: > > > > On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG < > nanog@nanog.org> wrote: > > > > On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote: > > > In short, the idea is that optical networks are wasteful and routers > do a > > > better job making more use of a network's capacity than ROADMs. Take > the > > > extra router hop (or 3 or 8) versus short-cutting it with an optical > > > network because the silicon is so low-latency anyway that it hardly > makes a > > > difference now. Putting more GBs per second on fewer strands means > saving a > > > lot of money on infrastructure costs. > > > > This is a very convoluted way of backing into the ole packet-switched > > vs. circuit switched decision. > > > > I don't follow. > > While ROADMs can be thought of as circuit-switchers, > > the number of concurrent clients and switching latency put ROADMs on a > different operational level than packet switchers, right? > > > I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine > you are at a hop where in a long distance system solution, you would end up > with OEO, but instead you get directionality capability with an IP/MPLS > capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 > optics are the latest example of that. > > I know of a few companies that have looked at solutions like this, and can > expect there to be some interesting solutions that would appear as a > result. Optical line systems tend to have pretty low power requirements > compared to a router, but some of the routers are getting pretty low power > as well when it comes to the power OPEX/bit, and if you have the ability to > deliver services as an integrated packet optical you could see reduced > costs and simplified components/sparing. > > I’ll also say that I’ve not yet seen the price compression that I had > expected in the space yet, but I figure that’s coming. We are seeing the > bits/watt ratio improve though, so for the same or less power consumption > you get more bits. Some of this technology stuff is truly magical. > > - Jared
Re: Routed optical networks
> > I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine > you are at a hop where in a long distance system solution, you would end up > with OEO, but instead you get directionality capability with an IP/MPLS > capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 > optics are the latest example of that. > Jared, I understand your point in the above statement to be that directionality is cost-effectively implemented through label-switched paths, rather than (ROADM-enabled) optical path switching. Do I understand right? Thank you. Etienne On Tue, May 2, 2023 at 9:33 PM Jared Mauch wrote: > > > > On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG < > nanog@nanog.org> wrote: > > > > On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote: > > > In short, the idea is that optical networks are wasteful and routers > do a > > > better job making more use of a network's capacity than ROADMs. Take > the > > > extra router hop (or 3 or 8) versus short-cutting it with an optical > > > network because the silicon is so low-latency anyway that it hardly > makes a > > > difference now. Putting more GBs per second on fewer strands means > saving a > > > lot of money on infrastructure costs. > > > > This is a very convoluted way of backing into the ole packet-switched > > vs. circuit switched decision. > > > > I don't follow. > > While ROADMs can be thought of as circuit-switchers, > > the number of concurrent clients and switching latency put ROADMs on a > different operational level than packet switchers, right? > > > I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine > you are at a hop where in a long distance system solution, you would end up > with OEO, but instead you get directionality capability with an IP/MPLS > capable device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 > optics are the latest example of that. > > I know of a few companies that have looked at solutions like this, and can > expect there to be some interesting solutions that would appear as a > result. Optical line systems tend to have pretty low power requirements > compared to a router, but some of the routers are getting pretty low power > as well when it comes to the power OPEX/bit, and if you have the ability to > deliver services as an integrated packet optical you could see reduced > costs and simplified components/sparing. > > I’ll also say that I’ve not yet seen the price compression that I had > expected in the space yet, but I figure that’s coming. We are seeing the > bits/watt ratio improve though, so for the same or less power consumption > you get more bits. Some of this technology stuff is truly magical. > > - Jared -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
On 5/2/23 07:28, Vasilenko Eduard via NANOG wrote: The incumbent carrier typically has enough fiber strands to avoid any colored interfaces (that are 3x expensive compare to gray) in the Metro. Metro ring typically has 8-10 nodes (or similar). 16-20 strands of fiber were not possible to construct anyway – any cable is bigger. It is the same cost to lay down fiber on 16 strands or 32. Hence, PTT just does not need DWDM in Metro, not at all. Hence, the DWDM optimization that you are talking about below is not needed too. This may or may not always be the case. Especially for large carriers, where there could be a requirement to sell some of those dark fibre pairs to large customers (think the content folk coming into town, e.t.c.), they may no longer have the priviledge of having plenty of free fibre in the metro. Or if they did, the rate of traffic expansion means they burn through those fibre pairs pretty quick. 10Gbps isn't a lot nowadays, and 100Gbps may start to push the limits depending on the size of the operator, the scope of the Metro-E ring and the level of service that needs to be maintained during a re-route (two available paths in the ring could balance 100Gbps of traffic, but if one half of that ring breaks, the remaining path may need to carry a lot more than 100Gbps, and then packets start to fall flat on the floor). At that scale, DWDM in the metro will make sense, at least more sense than 400G-ZR, at the moment. If you rent a single pair of fiber then you need colored interfaces to multiplex 8-10 nodes into 1 pair on the ring. Then the movement of transponders from DWDM into the router would eliminate 2 gray interfaces on every node (4 per link): one on the router side, and another on the DWDM side. Overall, it is about a 25% cost cut of the whole “router+DWDM”. Some operators would also be selling Transport services in or along the metro, and customers paying for that may require that they do not cross a router device. It is still 2x more expensive compare to using additional fiber strands on YOUR fiber. There are plenty of DWDM pizza boxes that cost next to nothing. At scale, the price of these is not a stumbling block. And certainly, the price of these would be far lower than a router line card. By the way, about “well-defined stack of technologies”: NMS (polished by SDN our days) should be cross-layer: it should manage at the same time: ROADM/OADM in DWDM and colored laser in Router. It is a vendor lock up to now (no multi-vendor). Hence, 25% cost savings would go to the vendor that has such NMS, not to the carrier. Technology still does not make sense because no multivendor support between the NMS of one vendor and the router or DWDM of another. Looking at the NMS history, it would probably never be multi-vendor. For that reason, I am pessimistic about the future of the colored interfaces in routers (and alien lambdas in DWDM). Despite a potential 25% cost advantage in eliminating gray interfaces. OpenROADM is a good initiative. But it seems it's to be to Transport equipment vendors what IPv6 and DNSSEC is to the IP world :-). Mark.
Re: Routed optical networks
On 5/2/23 16:25, Izaac wrote: This is a very convoluted way of backing into the ole packet-switched vs. circuit switched decision. A fight that will never go away. There has been some compromise in recent years, with Transport-heavy customers accepting standard Ethernet services, but only if they are carried by a Transport device. Mark.
Re: Routed optical networks
On 5/2/23 21:32, Jared Mauch wrote: I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are at a hop where in a long distance system solution, you would end up with OEO, but instead you get directionality capability with an IP/MPLS capable device. My memory is rather fuzzy, but didn't Juniper attempt something like this in their PTX's after they picked up BTI? I think the plan was to co-locate the ROADM at the bottom of the PTX chassis, or something along those lines. I know Cisco (and Juniper) tried by integrating GMPLS into their code as a starting point, but that didn't go very far with customers. It just seemed impossible for the Transport teams to allow the IP/MPLS teams that level of access into their line system :-). As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest example of that. A rather high barrier to entry for most operators, but we have to start from somewhere. I know of a few companies that have looked at solutions like this, and can expect there to be some interesting solutions that would appear as a result. Optical line systems tend to have pretty low power requirements compared to a router, but some of the routers are getting pretty low power as well when it comes to the power OPEX/bit, and if you have the ability to deliver services as an integrated packet optical you could see reduced costs and simplified components/sparing. The main problem is distance. If you need to move that kind of capacity more than 50km, it's hard to avoid DWDM. I’ll also say that I’ve not yet seen the price compression that I had expected in the space yet, but I figure that’s coming. We are seeing the bits/watt ratio improve though, so for the same or less power consumption you get more bits. Some of this technology stuff is truly magical. I think for long spans, DWDM will not only be cheaper, but the only feasible solution. For the metro, it will come down to what motivates the business... plenty of features, or plenty of speed. Also, DWDM vendors are adding speed and distance faster and cheaper than the IP/MPLS vendors can. So they will always be one step ahead in that respect; and we have the submarine cable systems to thank for that. Mark.
Re: Routed optical networks
On 5/2/23 16:01, Eve Griliches wrote: Hi Etienne, Below is our (Cisco) definition of the Routed Optical Network. The goal, metro or long haul or subsea, is to reduce the number of control planes. By migration TDM traffic using CEM or PLE to the IP layer, you eliminate the OTN control plane and management. Eventually, when standards are settled the ultimate goal is to have a single control plane for the network. I'm not trying to be a commercial here, but you can read more in the resources section on this page: https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html HTH, Eve Routed optical networking, is an architecture that delivers improved operational efficiencies and simplicity. The solution works by merging IP and private line services onto a single layer where all the switching is done at Layer 3. Routers are connected with standardized 400G ZR/ZR+ coherent pluggable optics. With a single service layer based upon IP, flexible management tools can leverage telemetry and model-driven programmability to streamline lifecycle operations. This simplified architecture integrates open data models and standard APIs, enabling a provider to focus on automation initiatives for a simpler topology. To be honest, I've been hearing about this since as long as I can remember. IPoDWDM was another attempt at trying to make the above a reality. But for some reason, operators prefer to keep these networks separate, and many customers, especially very large ones, prefer to bypass routers for their Transport services. I think the effort will be appreciated, but if history is anything to go by, vendors are going to struggle to strip operators and customers away from some degree of separation. Mark.
Re: Routed optical networks
On 5/2/23 22:28, Eve Griliches wrote: So right Jaredmagic has been in the NPU capacity increase that's driven the cost per 100G down on 1RU routers; But that has only mainly solved for speed. Features have taken a hit, especially if the operator is motivated by the costs of merchant silicon. There has been a marked improvement of features from merchant silicon, both from their vendors as well as the router OEM's that implement them in clever ways to work around their restrictions, but there are still some things only in-house silicon can do, at a price point most operators are not comfortable to pay anymore. I think that as more of the Internet collapses into the hands of a few public cloud and content providers, operators are likely to place less and less importance on features, and just focus on speed, since the public Internet offers very little guarantees, if not none at all. I'm keen to see how this pans out. and the integration of DSPs and more into QSFP-DD form factors at much lower power than expected. Coherent has certainly changed the game, no doubt. If you grow steadily, you can wait for the evolution to make it into the IP/MPLS. If you need to move faster, you can't ignore the importance of Transport options in your network. Mark.
RE: Routed optical networks
> At that scale, DWDM in the metro will make sense I would risk to say a little more on this. Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services. But in general, packet services are enough these days for many carriers/regions. Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box. The colored interface is still very expensive. Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber. Yes, in the majority of cases, it is cheaper to lay down fiber. Hence, the importance of DWDM for the Metro is overestimated. Use only routers. Provision enough fiber. Have always 1 router hop to the aggregation (hub-spoke topology), no routers chaining in the ring. If fiber is not enough – then use normal DWDM with an external transponder. Routers would be still in hub-spoke topology. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On Behalf Of Mark Tinka Sent: Wednesday, May 3, 2023 7:09 AM To: nanog@nanog.org Subject: Re: Routed optical networks On 5/2/23 07:28, Vasilenko Eduard via NANOG wrote: The incumbent carrier typically has enough fiber strands to avoid any colored interfaces (that are 3x expensive compare to gray) in the Metro. Metro ring typically has 8-10 nodes (or similar). 16-20 strands of fiber were not possible to construct anyway – any cable is bigger. It is the same cost to lay down fiber on 16 strands or 32. Hence, PTT just does not need DWDM in Metro, not at all. Hence, the DWDM optimization that you are talking about below is not needed too. This may or may not always be the case. Especially for large carriers, where there could be a requirement to sell some of those dark fibre pairs to large customers (think the content folk coming into town, e.t.c.), they may no longer have the priviledge of having plenty of free fibre in the metro. Or if they did, the rate of traffic expansion means they burn through those fibre pairs pretty quick. 10Gbps isn't a lot nowadays, and 100Gbps may start to push the limits depending on the size of the operator, the scope of the Metro-E ring and the level of service that needs to be maintained during a re-route (two available paths in the ring could balance 100Gbps of traffic, but if one half of that ring breaks, the remaining path may need to carry a lot more than 100Gbps, and then packets start to fall flat on the floor). At that scale, DWDM in the metro will make sense, at least more sense than 400G-ZR, at the moment. If you rent a single pair of fiber then you need colored interfaces to multiplex 8-10 nodes into 1 pair on the ring. Then the movement of transponders from DWDM into the router would eliminate 2 gray interfaces on every node (4 per link): one on the router side, and another on the DWDM side. Overall, it is about a 25% cost cut of the whole “router+DWDM”. Some operators would also be selling Transport services in or along the metro, and customers paying for that may require that they do not cross a router device. It is still 2x more expensive compare to using additional fiber strands on YOUR fiber. There are plenty of DWDM pizza boxes that cost next to nothing. At scale, the price of these is not a stumbling block. And certainly, the price of these would be far lower than a router line card. By the way, about “well-defined stack of technologies”: NMS (polished by SDN our days) should be cross-layer: it should manage at the same time: ROADM/OADM in DWDM and colored laser in Router. It is a vendor lock up to now (no multi-vendor). Hence, 25% cost savings would go to the vendor that has such NMS, not to the carrier. Technology still does not make sense because no multivendor support between the NMS of one vendor and the router or DWDM of another. Looking at the NMS history, it would probably never be multi-vendor. For that reason, I am pessimistic about the future of the colored interfaces in routers (and alien lambdas in DWDM). Despite a potential 25% cost advantage in eliminating gray interfaces. OpenROADM is a good initiative. But it seems it's to be to Transport equipment vendors what IPv6 and DNSSEC is to the IP world :-). Mark.
Re: Routed optical networks
On 5/3/23 08:20, Vasilenko Eduard wrote: I would risk to say a little more on this. Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services. But in general, packet services are enough these days for many carriers/regions. There aren't enough TDM services to warrant DWDM, nowadays. The reason for DWDM is mainly being driven by Ethernet, and IP. At any reasonable scale, it's actually pretty hard to buy a TDM service, in most markets. Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box. I disagree. Existing fibre may be cheap because it was laid down a decade or more ago, en masse, by several operators. So the market would be experiencing a glut, not because it is cheap to open up the roads and plant more fibre, but because there is so much of it to begin with. At worst, there is still enough duct space that the operator can blow more fibre. But when that duct gets full, and there are no more free ducts available, or another route needs to get built for whatever reason, it is a rather costly affair to open up the roads and trunk some fibre, in any market. So no, DWDM is not more expensive, if you are delivering services at scale. It is actually cheaper. It is only more expensive if you are small scale, because in some markets, the fibre glut means you can buy dark fibre for cheaper than you can light it with DWDM. But this is a situation unique to small operators, not large ones. The colored interface is still very expensive. This only matters for the line side. For client-facing, it's not a drama. And you typically buy more optics for the client side than you do the line side. Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber. Yes, in the majority of cases, it is cheaper to lay down fiber. I think what you mean to say is that in the majority of cases where there is fibre glut, and dark fibre is a market option, buying fibre is cheaper than lighting it with DWDM. This is true. But I think that on a global scale, this is the exception, not the rule. In general, you are not likely to be able to buy dark fibre, cheaply or otherwise, if you look at all markets in the world. Hence, the importance of DWDM for the Metro is overestimated. Again, only if you are small scale. If you are a large scale operator with as many IP/Ethernet customers as you have Transport, DWDM is essential. Use only routers. Provision enough fiber. Have always 1 router hop to the aggregation (hub-spoke topology), no routers chaining in the ring. If fiber is not enough – then use normal DWDM with an external transponder. Routers would be still in hub-spoke topology. Yeah, you sound like an equipment vendor whose main customers are incumbent telco's in a few rich markets :-). The life of the average operator, around the world, is far less glamorous. Mark.
RE: Routed optical networks
> Yeah, you sound like an equipment vendor whose main customers are incumbent > telco's in a few rich markets :-). You are right. My message was pretty much geared toward incumbents. But the majority of the access/aggregation is in their possessions, isn’t it? They typically have ducts that were huge for copper that is already extracted. One more fiber cable would be easy. Agree that for competitive carriers DWDM would be more often needed. Even for competitive carriers, it makes sense to evaluate the cost to put fiber into the duct of incumbents. Especially because in some countries the price would be regulated. It would solve the problem forever – no need for the DWDM speed upgrade. I am calling to just not forget to evaluate this option too. Reminder: dark fiber is the best technical solution, for sure. Ed/ From: Mark Tinka [mailto:mark@tinka.africa] Sent: Wednesday, May 3, 2023 11:39 AM To: Vasilenko Eduard ; nanog@nanog.org Subject: Re: Routed optical networks On 5/3/23 08:20, Vasilenko Eduard wrote: I would risk to say a little more on this. Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services. But in general, packet services are enough these days for many carriers/regions. There aren't enough TDM services to warrant DWDM, nowadays. The reason for DWDM is mainly being driven by Ethernet, and IP. At any reasonable scale, it's actually pretty hard to buy a TDM service, in most markets. Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box. I disagree. Existing fibre may be cheap because it was laid down a decade or more ago, en masse, by several operators. So the market would be experiencing a glut, not because it is cheap to open up the roads and plant more fibre, but because there is so much of it to begin with. At worst, there is still enough duct space that the operator can blow more fibre. But when that duct gets full, and there are no more free ducts available, or another route needs to get built for whatever reason, it is a rather costly affair to open up the roads and trunk some fibre, in any market. So no, DWDM is not more expensive, if you are delivering services at scale. It is actually cheaper. It is only more expensive if you are small scale, because in some markets, the fibre glut means you can buy dark fibre for cheaper than you can light it with DWDM. But this is a situation unique to small operators, not large ones. The colored interface is still very expensive. This only matters for the line side. For client-facing, it's not a drama. And you typically buy more optics for the client side than you do the line side. Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber. Yes, in the majority of cases, it is cheaper to lay down fiber. I think what you mean to say is that in the majority of cases where there is fibre glut, and dark fibre is a market option, buying fibre is cheaper than lighting it with DWDM. This is true. But I think that on a global scale, this is the exception, not the rule. In general, you are not likely to be able to buy dark fibre, cheaply or otherwise, if you look at all markets in the world. Hence, the importance of DWDM for the Metro is overestimated. Again, only if you are small scale. If you are a large scale operator with as many IP/Ethernet customers as you have Transport, DWDM is essential. Use only routers. Provision enough fiber. Have always 1 router hop to the aggregation (hub-spoke topology), no routers chaining in the ring. If fiber is not enough – then use normal DWDM with an external transponder. Routers would be still in hub-spoke topology. Yeah, you sound like an equipment vendor whose main customers are incumbent telco's in a few rich markets :-). The life of the average operator, around the world, is far less glamorous. Mark.
Re: Routed optical networks
On 5/3/23 11:10, Vasilenko Eduard wrote: You are right. My message was pretty much geared toward incumbents. But the majority of the access/aggregation is in their possessions, isn’t it? Generally, I'd say yes. But to the OP's survey, the incumbents also have the majority installations of copper, and are least likely to use MPLS all the way into the Access. They would more typically use 802.1Q or Q-in-Q. Smaller operators green-fielding Access networks will rely mostly on fibre (particularly GPON) and MPLS all the way into the Access. It's just that FTTx services are growing at a much faster rate than copper-based services are. Depending on the market, it may not always be the incumbent witnessing this growth, although when they do finally get their act together, they can make up for lost time rather quickly. They typically have ducts that were huge for copper that is already extracted. One more fiber cable would be easy. Agreed. Agree that for competitive carriers DWDM would be more often needed. Even for competitive carriers, it makes sense to evaluate the cost to put fiber into the duct of incumbents. Especially because in some countries the price would be regulated. It would solve the problem forever – no need for the DWDM speed upgrade. Well, the only issue with that is that some markets make it more difficult to re-open up the roads for several years. Some worse than others. In such a case, DWDM is your best option. I am calling to just not forget to evaluate this option too. Reminder: dark fiber is the best technical solution, for sure. In general, most operators, regardless of size, will prefer dark fibre as a first option, especially for short spans like in the metro. But of course, real life is vastly different. Mark.
Re: Routed optical networks
Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a écrit : > > Additionally, I am sure that in many countries/Metro it is cheaper to lay > down a new fiber than to provision DWDM, even if it is a pizza box. The > colored interface is still very expensive. > Of course, there are some Cities (not “towns”) where it is very expensive or > maybe even impossible to lay down a new fiber. > Yes, in the majority of cases, it is cheaper to lay down fiber. > You may also take into account the time to deliver. Laying fiber takes much more time than plugging a colored optic.
Re: Routed optical networks
On 5/4/23 11:40, Denis Fondras wrote: You may also take into account the time to deliver. Laying fiber takes much more time than plugging a colored optic. Indeed - part of the expense of running new fibre is the time it takes to start making money from it. Mark.
RE: Routed optical networks
Well, ISP is typically plan something for a year. It is more than enough for both. Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch. Ed/ -Original Message- From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On Behalf Of Denis Fondras Sent: Thursday, May 4, 2023 12:41 PM To: nanog@nanog.org Subject: Re: Routed optical networks Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a écrit : > > Additionally, I am sure that in many countries/Metro it is cheaper to lay > down a new fiber than to provision DWDM, even if it is a pizza box. The > colored interface is still very expensive. > Of course, there are some Cities (not “towns”) where it is very expensive or > maybe even impossible to lay down a new fiber. > Yes, in the majority of cases, it is cheaper to lay down fiber. > You may also take into account the time to deliver. Laying fiber takes much more time than plugging a colored optic.
Re: Routed optical networks
On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote: Well, ISP is typically plan something for a year. It is more than enough for both. The real world is much less certain, especially in these economic times. Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch. Even in the same market, no two lays of fibre can be guaranteed to be completed in the same time. Mark.
Re: Routed optical networks
> > Well, ISP is typically plan something for a year. It is more than enough > for both. > s/more/should be/ The economics are such these days that in many circumstances, bean counters don't want to hear about payback in years, they want to hear it in quarters. Short term financial thinking is dominant. On Thu, May 4, 2023 at 6:59 AM Vasilenko Eduard via NANOG wrote: > Well, ISP is typically plan something for a year. It is more than enough > for both. > > Funny, that with the current lead times for electronics, Fiber could be > faster. > Of course, it is a temporary glitch. > > Ed/ > -Original Message- > From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] > On Behalf Of Denis Fondras > Sent: Thursday, May 4, 2023 12:41 PM > To: nanog@nanog.org > Subject: Re: Routed optical networks > > Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a > écrit : > > > > Additionally, I am sure that in many countries/Metro it is cheaper to > lay down a new fiber than to provision DWDM, even if it is a pizza box. The > colored interface is still very expensive. > > Of course, there are some Cities (not “towns”) where it is very > expensive or maybe even impossible to lay down a new fiber. > > Yes, in the majority of cases, it is cheaper to lay down fiber. > > > > You may also take into account the time to deliver. > Laying fiber takes much more time than plugging a colored optic. > >
RE: Routed optical networks
I had an experience in one big PTT. Fiber was easy in the majority of Metro places. Even faster than DWDM or router commissioning. It is just 1 PTT. Hence, an example could not be counted. Eduard -Original Message- From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On Behalf Of Mark Tinka Sent: Thursday, May 4, 2023 2:11 PM To: nanog@nanog.org Subject: Re: Routed optical networks On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote: > Well, ISP is typically plan something for a year. It is more than enough for > both. The real world is much less certain, especially in these economic times. > Funny, that with the current lead times for electronics, Fiber could be > faster. > Of course, it is a temporary glitch. Even in the same market, no two lays of fibre can be guaranteed to be completed in the same time. Mark.
RE: Routed optical networks
> The economics are such these days that in many circumstances, bean counters > don't want to hear about payback in years, they want to hear it in quarters. > Short term financial thinking is dominant. True. The industry is on decline. On the way to other utilities. But then any project is a challenge. Not just fiber that may be cheaper for Metro than DWDM. Eduard From: Tom Beecher [mailto:beec...@beecher.cc] Sent: Thursday, May 4, 2023 3:26 PM To: Vasilenko Eduard Cc: Denis Fondras ; nanog@nanog.org Subject: Re: Routed optical networks Well, ISP is typically plan something for a year. It is more than enough for both. s/more/should be/ The economics are such these days that in many circumstances, bean counters don't want to hear about payback in years, they want to hear it in quarters. Short term financial thinking is dominant. On Thu, May 4, 2023 at 6:59 AM Vasilenko Eduard via NANOG mailto:nanog@nanog.org>> wrote: Well, ISP is typically plan something for a year. It is more than enough for both. Funny, that with the current lead times for electronics, Fiber could be faster. Of course, it is a temporary glitch. Ed/ -Original Message- From: NANOG [mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei@nanog.org<mailto:huawei@nanog.org>] On Behalf Of Denis Fondras Sent: Thursday, May 4, 2023 12:41 PM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Routed optical networks Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a écrit : > > Additionally, I am sure that in many countries/Metro it is cheaper to lay > down a new fiber than to provision DWDM, even if it is a pizza box. The > colored interface is still very expensive. > Of course, there are some Cities (not “towns”) where it is very expensive or > maybe even impossible to lay down a new fiber. > Yes, in the majority of cases, it is cheaper to lay down fiber. > You may also take into account the time to deliver. Laying fiber takes much more time than plugging a colored optic.
Re: Routed optical networks
Many Big PTTs have a lot of ducts in many places, it is easy for them to lay fibers, especially in cities. Ge > Le 4 mai 2023 à 14:27, Vasilenko Eduard via NANOG a écrit : > > I had an experience in one big PTT. Fiber was easy in the majority of Metro > places. > Even faster than DWDM or router commissioning. > It is just 1 PTT. Hence, an example could not be counted. > Eduard > -Original Message- > From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On > Behalf Of Mark Tinka > Sent: Thursday, May 4, 2023 2:11 PM > To: nanog@nanog.org > Subject: Re: Routed optical networks > > > > On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote: > >> Well, ISP is typically plan something for a year. It is more than enough for >> both. > > The real world is much less certain, especially in these economic times. > > >> Funny, that with the current lead times for electronics, Fiber could be >> faster. >> Of course, it is a temporary glitch. > > Even in the same market, no two lays of fibre can be guaranteed to be > completed in the same time. > > Mark. >
Re: Routed optical networks
> On May 4, 2023, at 6:21 AM, Mark Tinka wrote: > > > > On 5/4/23 11:40, Denis Fondras wrote: > >> >> You may also take into account the time to deliver. >> Laying fiber takes much more time than plugging a colored optic. > > Indeed - part of the expense of running new fibre is the time it takes to > start making money from it. > One of the advantages in the US right now is that capital cost is being subsidized or reimbursed by state or federal government, leaving room to expand. Some providers have missed out on this, and others have capitalized on it. It’s very much a mix of people who are chasing that $ and those that are not. I will say that merchant silicon has it’s place, but so does the vendor silicon. At some point if you get the fiber to that location, unlocking the capacity becomes much easier with CEx or similar modules to overlay services. Making the choice to build a quality fiber first network is important, and why I have already had to take routes that I had planned lower count fibers on and upgrade them. I’m a bit shocked that I now need a 288F cable on some of my routes to support future expansion, but that fiber cost is still small compared to the labor. - Jared
Re: Routed optical networks
It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn’t exceed the capabilities of those optics. It’s been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG on behalf of Etienne-Victor Depasquale via NANOG Date: Monday, May 1, 2023 at 2:30 PM To: NANOG Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
On 5/4/23 14:27, Vasilenko Eduard wrote: I had an experience in one big PTT. Fiber was easy in the majority of Metro places. Even faster than DWDM or router commissioning. It is just 1 PTT. Hence, an example could not be counted. Yeah - I usually tend to look at what happens in the majority of cases. There is always at least one example in an exception, but it still remains an exception. Mark.
Re: Routed optical networks
On 5/4/23 15:03, Jared Mauch wrote: I’m a bit shocked that I now need a 288F cable on some of my routes to support future expansion, but that fiber cost is still small compared to the labor. Yes, labour is generally the cost. And then way-leaves add cost in terms of time and lost opportunities. Mark.
Re: Routed optical networks
On 5/4/23 19:32, Phil Bedard wrote: It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. I think every time the IP space gets close to running an all-packet network, the Transport folk come out with an easier way to do it, that it's too hard to ignore. Based on that, I think they will always be one step ahead, with the key advantage being reliability of capacity over the distance, for the cost. The farther your fibre has to run, the costlier it gets to do it without DWDM. I mean, it's only now that 100G-ZR is becoming a reality for packet networks, and we are talking thousands of US$ for optics to get us 80km - 120km distance. Meanwhile, DWDM vendors can get you 800Gbps per wavelength in the same distance (or 30X that distance) far less cheaply. I get the appeal of not needing DWDM gear to underlay your packet network... it's neater and offers fewer points of failure. But unless you are dealing with very short distances and can ride a reasonable balance between service features and switching/forwarding capacity in your router/switch, it's going to be hard to ignore the DWDM gear if you are trying to be a serious operation, at that scale, over a wide area. Mark.
RE: Routed optical networks
Disclaimer: Metaverse has not changed Metro traffic yet. Then ... I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one "2" for redundancy, another "2" for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits - it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change - it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale ; NANOG Subject: Re: Routed optical networks It's not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that's not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that's over a ROADM based optical network or not, especially at metro/regional distances. There are very large deployments of IPoDWDM over passive DWDM or dark fiber for access and aggregation networks where the aggregate required bandwidth doesn't exceed the capabilities of those optics. It's been done at 10G for many years. With the advent of pluggable EDFA amplifiers, you can even build links up to 120km* (perfect dark fiber) carrying tens of terabits of traffic without any additional active optical equipment. It's my personal opinion we aren't to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren't a good fit. Carrying high speed private line/wavelength type services as well is a different topic than interconnecting IP devices. Thanks, Phil From: NANOG mailto:nanog-bounces+bedard.phil=gmail@nanog.org>> on behalf of Etienne-Victor Depasquale via NANOG mailto:nanog@nanog.org>> Date: Monday, May 1, 2023 at 2:30 PM To: NANOG mailto:nanog@nanog.org>> Subject: Routed optical networks Hello folks, Simple question: does "routed optical networks" have a clear meaning in the metro area context, or not? Put differently: does it call to mind a well-defined stack of technologies in the control and data planes of metro-area networks? I'm asking because I'm having some thoughts about the clarity of this term, in the process of carrying out a qualitative survey of the results of the metro-area networks survey. Cheers, Etienne -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
On 5/5/23 07:57, Vasilenko Eduard via NANOG wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. So, it depends on what "metro" means to you. For an ISP selling connectivity to enterprise customers, it can be a bunch of Metro-E routers deployed in various commercial buildings within a city. For a content provider, it could be DCI. For a telco, it could interconnecting their Active-E/GPON/DSLAM/CMTS network. Whatever the case, the need for 100Gbps is going to be driven by the cost of optics over the distance required. Some operators run 2x 10Gbps for resilience/redundancy, while some others run 4x 10Gbps for the same. It all depends on the platform you are using. At some point, that capacity runs out, especially when you account for fibre outages, and you need something larger on one side of the ring mainly to provide sufficient bandwidth during failure events on the other side of the ring, and not necessarily because you are growing by that much. Also, if the optics are available and are reasonably priced, why muck around with 40Gbps when you can just go straight to 100Gbps? The equipment usually can support either. I'm unaware of any popularity around 50Gbps interfaces, but I also probably don't pay too much attention to such nuance :-). So, it's not that we are seeing organic growth that justifies 100Gbps over anything smaller. It's more that the optics are available, they are cheap, they can go the distance, and the routers/switches can do the speed. At least, for us anyway, that is what is driving the next phase of our Metro-E network... going straight from 10Gbps to 100Gbps links, not because that is the growth we are seeing from an organic traffic standpoint, but because the routers can do it, and it offers us peace of mind that we can handle any traffic re-route when one half of the ring fails, without dropping packets. The 400Gbps market will be restricted to mainly content folk linking up data centres, as well as some large telco's, for the time being. It is not likely to be the norm for the majority of operators who run some kind of metro network. Mark.
RE: Routed optical networks
You are right, my “Metro” definition is about ISP/Carriers. Mobile or Fixed, despite pure Mobiles would like to call it MBH – it has much less traffic (Mobile subscribers would always have 7x less than fixed). It is still the place where the majority of port capacity lives. Because all content and cache are after this link. 50GE is better just because it is half of the cost of 100GE and it is enough now for the great majority of cases. Money is very important these days for this industry. 100GE single mode is more expensive than the best router port itself. Routers have been deprecated 10x for the decade (almost 100x for 2 decades). Pluggable optics is not that much deprecated. I do not think that content provider guys call their DCI “Metro”, not very often. I agree that 100GE for DCI is the minimum, 400GE is probably already needed in some places. IMHO: it is a different story. Very interested too. PS: By the way, even if some ISP has 50% of revenue from Enterprise services (it is probably the biggest number, typically 30%-40%), it is still just 5% compare to residential traffic. Traffic to enterprises is still sold 4x-10x (depending on the country). Hence, Enterprise does not make sense to mention in the traffic discussion. It is a “rounding error”. Enterprise business created a huge demand for oversubscribed ports to connect Enterprises. And QoS/QoE. Not traffic. Eduard From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On Behalf Of Mark Tinka Sent: Friday, May 5, 2023 11:13 AM To: nanog@nanog.org Subject: Re: Routed optical networks On 5/5/23 07:57, Vasilenko Eduard via NANOG wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. So, it depends on what "metro" means to you. For an ISP selling connectivity to enterprise customers, it can be a bunch of Metro-E routers deployed in various commercial buildings within a city. For a content provider, it could be DCI. For a telco, it could interconnecting their Active-E/GPON/DSLAM/CMTS network. Whatever the case, the need for 100Gbps is going to be driven by the cost of optics over the distance required. Some operators run 2x 10Gbps for resilience/redundancy, while some others run 4x 10Gbps for the same. It all depends on the platform you are using. At some point, that capacity runs out, especially when you account for fibre outages, and you need something larger on one side of the ring mainly to provide sufficient bandwidth during failure events on the other side of the ring, and not necessarily because you are growing by that much. Also, if the optics are available and are reasonably priced, why muck around with 40Gbps when you can just go straight to 100Gbps? The equipment usually can support either. I'm unaware of any popularity around 50Gbps interfaces, but I also probably don't pay too much attention to such nuance :-). So, it's not that we are seeing organic growth that justifies 100Gbps over anything smaller. It's more that the optics are available, they are cheap, they can go the distance, and the routers/switches can do the speed. At least, for us anyway, that is what is driving the next phase of our Metro-E n
Re: Routed optical networks
On 5/5/23 10:54, Vasilenko Eduard wrote: 50GE is better just because it is half of the cost of 100GE and it is enough now for the great majority of cases. Money is very important these days for this industry. 100GE single mode is more expensive than the best router port itself. Routers have been deprecated 10x for the decade (almost 100x for 2 decades). Pluggable optics is not that much deprecated. Not sure where your pricing is coming from, but if I look at Flexoptix's 50Gbps QSFP28 optics pricing, I am getting: * EUR724 @ 10km. * EUR1,246 @ 40km. They are also selling an SFP56 LR for EUR925. Juxtapose that against 100Gbps pricing: * EUR473 @ 10km. * EUR1,300 @ 25km. * EUR1,500 @ 30km. * EUR2,600 @ 40km. * EUR3,925 @ 80km. Doesn't immediately seem to me that 50Gbps is cheaper than 100Gbps. There also don't seem to be as many deployments of 50Gbps in the metro (same could be said for 25Gbps and 40Gbps), but others on the list can chime in with what they are seeing/doing. I do not think that content provider guys call their DCI “Metro”, not very often. Well, whatever they call it, the concept is the same - move lots of traffic across town between data centres. I agree that 100GE for DCI is the minimum, 400GE is probably already needed in some places. IMHO: it is a different story. Very interested too. Most content providers have no choice but to run DWDM, for even very short spans between data centres. That is because it is just cheaper and simpler to pack Tbps of capacity in DWDM for the price than you can in a router. And besides, most routers don't need to carry Tbps of traffic in a single line card, which would be a waste of a fibre pair over that distance. In such cases, better to use DWDM and drop capacity on individual routers and/or line cards as you see fit. PS: By the way, even if some ISP has 50% of revenue from Enterprise services (it is probably the biggest number, typically 30%-40%), it is still just 5% compare to residential traffic. Traffic to enterprises is still sold 4x-10x (depending on the country). That is why residential Access networks tend to be 2nd class citizens :-). Hence, Enterprise does not make sense to mention in the traffic discussion. It is a “rounding error”. Enterprise business created a huge demand for oversubscribed ports to connect Enterprises. And QoS/QoE. Not traffic. Well, not all operators that offer enterprise services also do consumer broadband, or vice versa. To a network doing only one or the other, whatever traffic they are carrying means the world to them. It's not ours to decide what is high or low traffic... that priviledge always remains with the network operator. Mark.
RE: Routed optical networks
Hi Mark, Thanks a lot for many of your valuable comments I almost always agree. 1. I agree that 50GE has not got the same popularity as 100GE. Many vendors did ignore it for some time. Looks like not many ignore it now. 2. Even in your example for 40km, 100GE is about twice more expensive than 50GE. 3. Hence, I have to google for the cheaper proposition in 10km. For obvious reasons, I could not reference my employer (my assumption about the cost is based on this comparison). https://opticswave.com/collections/50g-qsfp28 https://www.compufox.com/50G_QSFP28_Transceivers_s/3036.htm https://www.genuinemodules.com/033030600050 Looks like I have found twice cheaper in public information. 4. Pay attention that bidirectional is available too. 5. The public price is not what you get in the real tender. We are talking about big networks, hence, big tenders. Eduard From: Mark Tinka [mailto:mark@tinka.africa] Sent: Friday, May 5, 2023 1:17 PM To: Vasilenko Eduard ; nanog@nanog.org Subject: Re: Routed optical networks On 5/5/23 10:54, Vasilenko Eduard wrote: 50GE is better just because it is half of the cost of 100GE and it is enough now for the great majority of cases. Money is very important these days for this industry. 100GE single mode is more expensive than the best router port itself. Routers have been deprecated 10x for the decade (almost 100x for 2 decades). Pluggable optics is not that much deprecated. Not sure where your pricing is coming from, but if I look at Flexoptix's 50Gbps QSFP28 optics pricing, I am getting: * EUR724 @ 10km. * EUR1,246 @ 40km. They are also selling an SFP56 LR for EUR925. Juxtapose that against 100Gbps pricing: * EUR473 @ 10km. * EUR1,300 @ 25km. * EUR1,500 @ 30km. * EUR2,600 @ 40km. * EUR3,925 @ 80km. Doesn't immediately seem to me that 50Gbps is cheaper than 100Gbps. There also don't seem to be as many deployments of 50Gbps in the metro (same could be said for 25Gbps and 40Gbps), but others on the list can chime in with what they are seeing/doing. I do not think that content provider guys call their DCI “Metro”, not very often. Well, whatever they call it, the concept is the same - move lots of traffic across town between data centres. I agree that 100GE for DCI is the minimum, 400GE is probably already needed in some places. IMHO: it is a different story. Very interested too. Most content providers have no choice but to run DWDM, for even very short spans between data centres. That is because it is just cheaper and simpler to pack Tbps of capacity in DWDM for the price than you can in a router. And besides, most routers don't need to carry Tbps of traffic in a single line card, which would be a waste of a fibre pair over that distance. In such cases, better to use DWDM and drop capacity on individual routers and/or line cards as you see fit. PS: By the way, even if some ISP has 50% of revenue from Enterprise services (it is probably the biggest number, typically 30%-40%), it is still just 5% compare to residential traffic. Traffic to enterprises is still sold 4x-10x (depending on the country). That is why residential Access networks tend to be 2nd class citizens :-). Hence, Enterprise does not make sense to mention in the traffic discussion. It is a “rounding error”. Enterprise business created a huge demand for oversubscribed ports to connect Enterprises. And QoS/QoE. Not traffic. Well, not all operators that offer enterprise services also do consumer broadband, or vice versa. To a network doing only one or the other, whatever traffic they are carrying means the world to them. It's not ours to decide what is high or low traffic... that priviledge always remains with the network operator. Mark.
Re: Routed optical networks
Incumbents are great at momentum. They're not great at innovation, customer experience, etc. They only reason most incumbents are still relevant is due to their prior market size. Around here, the incumbent telcos still have lead-sheathed cables in the ground, not removing anything. Often, things are abandoned in place, unless there's a good enough reason to remove it. I'm placing my own ducts into the ground and putting my own fiber in it. I still put in DWDM between my facilities to minimize the consumption of resources. A couple of hundred bucks for a DWDM optic is cheaper than a strand between two locations. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Vasilenko Eduard via NANOG" To: "Mark Tinka" , nanog@nanog.org Sent: Wednesday, May 3, 2023 4:10:19 AM Subject: RE: Routed optical networks > Yeah, you sound like an equipment vendor whose main customers are incumbent > telco's in a few rich markets :-). You are right. My message was pretty much geared toward incumbents. But the majority of the access/aggregation is in their possessions, isn’t it? They typically have ducts that were huge for copper that is already extracted. One more fiber cable would be easy. Agree that for competitive carriers DWDM would be more often needed. Even for competitive carriers, it makes sense to evaluate the cost to put fiber into the duct of incumbents. Especially because in some countries the price would be regulated. It would solve the problem forever – no need for the DWDM speed upgrade. I am calling to just not forget to evaluate this option too. Reminder: dark fiber is the best technical solution, for sure. Ed/ From: Mark Tinka [mailto:mark@tinka.africa] Sent: Wednesday, May 3, 2023 11:39 AM To: Vasilenko Eduard ; nanog@nanog.org Subject: Re: Routed optical networks On 5/3/23 08:20, Vasilenko Eduard wrote: I would risk to say a little more on this. Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services. But in general, packet services are enough these days for many carriers/regions. There aren't enough TDM services to warrant DWDM, nowadays. The reason for DWDM is mainly being driven by Ethernet, and IP. At any reasonable scale, it's actually pretty hard to buy a TDM service, in most markets. Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box. I disagree. Existing fibre may be cheap because it was laid down a decade or more ago, en masse, by several operators. So the market would be experiencing a glut, not because it is cheap to open up the roads and plant more fibre, but because there is so much of it to begin with. At worst, there is still enough duct space that the operator can blow more fibre. But when that duct gets full, and there are no more free ducts available, or another route needs to get built for whatever reason, it is a rather costly affair to open up the roads and trunk some fibre, in any market. So no, DWDM is not more expensive, if you are delivering services at scale. It is actually cheaper. It is only more expensive if you are small scale, because in some markets, the fibre glut means you can buy dark fibre for cheaper than you can light it with DWDM. But this is a situation unique to small operators, not large ones. The colored interface is still very expensive. This only matters for the line side. For client-facing, it's not a drama. And you typically buy more optics for the client side than you do the line side. Of course, there are some Cities (not “towns”) where it is very expensive or maybe even impossible to lay down a new fiber. Yes, in the majority of cases, it is cheaper to lay down fiber. I think what you mean to say is that in the majority of cases where there is fibre glut, and dark fibre is a market option, buying fibre is cheaper than lighting it with DWDM. This is true. But I think that on a global scale, this is the exception, not the rule. In general, you are not likely to be able to buy dark fibre, cheaply or otherwise, if you look at all markets in the world. Hence, the importance of DWDM for the Metro is overestimated. Again, only if you are small scale. If you are a large scale operator with as many IP/Ethernet customers as you have Transport, DWDM is essential. Use only routers. Provision enough fiber. Have always 1 router hop to the aggregation (hub-spoke topology), no routers chaining in the ring. If fiber is not enough – then use normal DWDM with an external transponder. Routers would be still in hub-spoke topology. Yeah, you sound like an equipment vendor whose main customers are incumbent telco's in a few rich markets :-).
Re: Routed optical networks
Anyone publicly traded doesn't plan longer than the current quarter. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Mark Tinka" To: nanog@nanog.org Sent: Thursday, May 4, 2023 6:10:36 AM Subject: Re: Routed optical networks On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote: > Well, ISP is typically plan something for a year. It is more than enough for > both. The real world is much less certain, especially in these economic times. > Funny, that with the current lead times for electronics, Fiber could be > faster. > Of course, it is a temporary glitch. Even in the same market, no two lays of fibre can be guaranteed to be completed in the same time. Mark.
RE: Routed optical networks
There are places in the world (like Middle East) where telephony system did not exist historically. Then no copper, then no ducts. Then new fiber is very difficult. But much more places where the Telephony system did exist. Ed/ From: Mike Hammett [mailto:na...@ics-il.net] Sent: Friday, May 5, 2023 4:50 PM To: Vasilenko Eduard Cc: Mark Tinka ; nanog@nanog.org Subject: Re: Routed optical networks Incumbents are great at momentum. They're not great at innovation, customer experience, etc. They only reason most incumbents are still relevant is due to their prior market size. Around here, the incumbent telcos still have lead-sheathed cables in the ground, not removing anything. Often, things are abandoned in place, unless there's a good enough reason to remove it. I'm placing my own ducts into the ground and putting my own fiber in it. I still put in DWDM between my facilities to minimize the consumption of resources. A couple of hundred bucks for a DWDM optic is cheaper than a strand between two locations. - Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL> Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix> The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg> From: "Vasilenko Eduard via NANOG" mailto:nanog@nanog.org>> To: "Mark Tinka" mailto:mark@tinka.africa>>, nanog@nanog.org<mailto:nanog@nanog.org> Sent: Wednesday, May 3, 2023 4:10:19 AM Subject: RE: Routed optical networks > Yeah, you sound like an equipment vendor whose main customers are incumbent > telco's in a few rich markets :-). You are right. My message was pretty much geared toward incumbents. But the majority of the access/aggregation is in their possessions, isn’t it? They typically have ducts that were huge for copper that is already extracted. One more fiber cable would be easy. Agree that for competitive carriers DWDM would be more often needed. Even for competitive carriers, it makes sense to evaluate the cost to put fiber into the duct of incumbents. Especially because in some countries the price would be regulated. It would solve the problem forever – no need for the DWDM speed upgrade. I am calling to just not forget to evaluate this option too. Reminder: dark fiber is the best technical solution, for sure. Ed/ From: Mark Tinka [mailto:mark@tinka.africa] Sent: Wednesday, May 3, 2023 11:39 AM To: Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>>; nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Routed optical networks On 5/3/23 08:20, Vasilenko Eduard wrote: I would risk to say a little more on this. Indeed, maybe the situation (in many countries) when the Carrier sells a lot of TDM services. But in general, packet services are enough these days for many carriers/regions. There aren't enough TDM services to warrant DWDM, nowadays. The reason for DWDM is mainly being driven by Ethernet, and IP. At any reasonable scale, it's actually pretty hard to buy a TDM service, in most markets. Additionally, I am sure that in many countries/Metro it is cheaper to lay down a new fiber than to provision DWDM, even if it is a pizza box. I disagree. Existing fibre may be cheap because it was laid down a decade or more ago, en masse, by several operators. So the market would be experiencing a glut, not because it is cheap to open up the roads and plant more fibre, but because there is so much of it to begin with. At worst, there is still enough duct space that the operator can blow more fibre. But when that duct gets full, and there are no more free ducts available, or another route needs to get built for whatever reason, it is a rather costly affair to open up the roads and trunk some fibre, in any market. So no, DWDM is not more expensive, if you are delivering services at scale. It is actually cheaper. It is only more expensive if you are small scale, because in some markets, the fibre glut means you can buy dark fibre for cheaper than you can light it with DWDM. But this is a situation u
Re: Routed optical networks
On 5/5/23, Mark Tinka wrote: > Juxtapose that against 100Gbps pricing: > > * EUR473 @ 10km. > * EUR1,300 @ 25km. > * EUR1,500 @ 30km. > * EUR2,600 @ 40km. > * EUR3,925 @ 80km. You can get 100G optics for less than half those prices. For reference, here are publicly listed prices for optics from an European vendor I have in production: 100G 4WDM & CLR4 QSFP28 * 10 km * 225€ 100GBASE-LR4 & OTU4 & 128GFC QSFP28 * 20km *305€ 100GBASE-ER4 Lite & OTU4 & 128GFC QSFP28 * 40 km * 1030€ 100GBASE-ZR4 QSFP28 * 80 km * 1650€ 100GBASE-ZR4+ QSFP28 * 100 km * 2100€ These prices are for single units without discounts. - Jared
Re: Routed optical networks
> On May 2, 2023, at 5:11 PM, Etienne-Victor Depasquale wrote: > > I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you > are at a hop where in a long distance system solution, you would end up with > OEO, but instead you get directionality capability with an IP/MPLS capable > device. As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the > latest example of that. > > Jared, I understand your point in the above statement to be that > directionality is cost-effectively implemented through label-switched paths, > rather than (ROADM-enabled) optical path switching. > > Do I understand right? It might be based on the distances and bandwidth required, and the need or desire to have diverse paths. Much of the economics of this vary based on the vendors and underlying either cost or pricing model. I do believe a few more engineers would be aided with a TCO for networking once you add up all the costs, either internal or external. We’ve seen the one-time-spend to build out the datacenter space exceed the cost of the equipment. (Based on the number of XC/patch panel positions needed for example - which might be a direct expense while the hardware may be capitalized and depreciated over a period, also TCO to power on a device - it can be quite common for that to exceed the Capex as well). - Jared
Re: Routed optical networks
On 5/5/23 13:04, Vasilenko Eduard wrote: Hi Mark, Thanks a lot for many of your valuable comments I almost always agree. 1.I agree that 50GE has not got the same popularity as 100GE. Many vendors did ignore it for some time. Looks like not many ignore it now. 2.Even in your example for 40km, 100GE is about twice more expensive than 50GE. 3.Hence, I have to google for the cheaper proposition in 10km. For obvious reasons, I could not reference my employer (my assumption about the cost is based on this comparison). https://opticswave.com/collections/50g-qsfp28 https://www.compufox.com/50G_QSFP28_Transceivers_s/3036.htm https://www.genuinemodules.com/033030600050 Looks like I have found twice cheaper in public information. 4.Pay attention that bidirectional is available too. In the real world, a 100Gbps network is very, very different from a 50Gbps network. A 100Gbps optic being twice the cost of a 50Gbps optic at 40km is worth it for most operators, because the value from 100Gbps far outweighs the cost when compared against the "savings" for a 50Gbps network. Also, most operators will buy a 40km optic yet they need something to cover the 11km - 30km range. With 25km and 30km optics costing as much as a 40km 50Gbps optic, this will satisfy many operators. 5.The public price is not what you get in the real tender. We are talking about big networks, hence, big tenders. I have zero sympathy for operators who choose to pay equipment vendors for their branded optics :-). Mark.
Re: Routed optical networks
On 5/5/23 15:49, Mike Hammett wrote: Incumbents are great at momentum. They're not great at innovation, customer experience, etc. They only reason most incumbents are still relevant is due to their prior market size. Most true, and even then, there is a visible change to the bottom line on how much money they print now vs. then, despite the fact that they are still printing money. This means that they all know that the dark day is looming, and even if they can see it drawing closer, telco's are stuck with not knowing what to do about it. I mean, if anyone remembers the days of SMS (2000 - 2008) - that was as free as money printing gets. Mark.
Re: Routed optical networks
On 5/5/23 19:02, Jared Brown wrote: You can get 100G optics for less than half those prices. For reference, here are publicly listed prices for optics from an European vendor I have in production: 100G 4WDM & CLR4 QSFP28 * 10 km * 225€ 100GBASE-LR4 & OTU4 & 128GFC QSFP28 * 20km *305€ 100GBASE-ER4 Lite & OTU4 & 128GFC QSFP28 * 40 km * 1030€ 100GBASE-ZR4 QSFP28 * 80 km * 1650€ 100GBASE-ZR4+ QSFP28 * 100 km * 2100€ These prices are for single units without discounts. It wasn't so much about the pricing that I was posting, but more about the value of 100Gbps over anything slower, if your main incentive is optics price. Mark.
Re: Routed optical networks
On 5/5/23 15:50, Mike Hammett wrote: Anyone publicly traded doesn't plan longer than the current quarter. A phenomenon not unique to them - private companies suffer the same issue. Mark.
Re: Routed optical networks
I guess let’s not confuse two things. The optical network is made up of the photonic portion and then the transponder/muxponder portion. A single term like “DWDM” can be confusing since it can refer to both. It will take a long time (maybe never) to remove the photonic switching part of the network. However, it’s always been cheap to deploy because optical vendors tended to subsidize that network using sales of the other portion, the transponders, which you buy more and more over time. Those photonic components are expensive. On the DWDM signal portion, I’m not talking about 100ZR compared to 100G on a transponder or DWDM line system. 100ZR has had to deal with the power limitations of QSFP28 ports, which QDD/OSFP do not suffer from. There are quite a few QDD pluggables in production today capable of supporting 100G signals over 1000s of km or 400G near 1500km. Now that’s not what you can get out of some external transponders, so those will still have their place in high performance applications. When you move to 800G, 1.2Tbps single channel they also have their own distance limitations. So it really depends on the application and the network. Phil From: NANOG on behalf of Mark Tinka Date: Friday, May 5, 2023 at 12:55 AM To: nanog@nanog.org Subject: Re: Routed optical networks On 5/4/23 19:32, Phil Bedard wrote: It’s my personal opinion we aren’t to the days yet of where we can simply build an all packet network with no photonic switching that carries all services, but eventually (random # of years) it gets there for many networks. There are also always going to be high performance applications for transponders where pluggable optics aren’t a good fit. I think every time the IP space gets close to running an all-packet network, the Transport folk come out with an easier way to do it, that it's too hard to ignore. Based on that, I think they will always be one step ahead, with the key advantage being reliability of capacity over the distance, for the cost. The farther your fibre has to run, the costlier it gets to do it without DWDM. I mean, it's only now that 100G-ZR is becoming a reality for packet networks, and we are talking thousands of US$ for optics to get us 80km - 120km distance. Meanwhile, DWDM vendors can get you 800Gbps per wavelength in the same distance (or 30X that distance) far less cheaply. I get the appeal of not needing DWDM gear to underlay your packet network... it's neater and offers fewer points of failure. But unless you are dealing with very short distances and can ride a reasonable balance between service features and switching/forwarding capacity in your router/switch, it's going to be hard to ignore the DWDM gear if you are trying to be a serious operation, at that scale, over a wide area. Mark.
Re: Routed optical networks
On 5/8/23 21:53, Phil Bedard wrote: I guess let’s not confuse two things. The optical network is made up of the photonic portion and then the transponder/muxponder portion. A single term like “DWDM” can be confusing since it can refer to both. Indeed. I am short-handing to mean DWDM on the line side and grey on the client side. It will take a long time (maybe never) to remove the photonic switching part of the network. There was a time when my intention was to do just that. But that was prior to expecting to ever run links larger than 10Gbps :-). However, it’s always been cheap to deploy because optical vendors tended to subsidize that network using sales of the other portion, the transponders, which you buy more and more over time. Those photonic components are expensive. It depends on the application... either you want performance, space and power cost optimization or functional integration of previously disparate system-level features. This will determine whether you focus on pluggables or embeddeds, with pluggables promoting cost reduction, while embeddeds push performance. On the DWDM signal portion, I’m not talking about 100ZR compared to 100G on a transponder or DWDM line system. 100ZR has had to deal with the power limitations of QSFP28 ports, which QDD/OSFP do not suffer from. Right - 100ZR is short-reach (80km). It's really for the metro edge where 400Gbps is not needed. There are quite a few QDD pluggables in production today capable of supporting 100G signals over 1000s of km or 400G near 1500km. I think the more interesting QSPF-DD (or OSFP) development is OpenZR+, which is an MSA project to standardize 400ZR+. The plan is to be able to support 100Gbps up to 5,800km, and 400Gbps up to 480km (EDFA) or about 1,000km (EDFA + Raman). With a 4x 100Gbps mode supported on QSFP28 router ports, you can have one muxponder talking to 4x routers at 100Gbps each. Now that’s not what you can get out of some external transponders, so those will still have their place in high performance applications. When you move to 800G, 1.2Tbps single channel they also have their own distance limitations. So it really depends on the application and the network. 800Gbps and 1.2Tbps applications are really for long haul use-cases, especially if you used 400Gbps pluggables before and run out of distance (so less than 1,000km). They are also preferred for submarine use-cases. I can't wait to see what happens when the CMOS gets down to 5nm and 3nm. Mark.
Re: Routed optical networks
> > The optical network is made up of the photonic portion and then the > transponder/muxponder portion. > Thank you for that direct definition. I'm serious (not sarcastic). One thing I've written about in papers is the nomenclature problem, and I'm in good company. Bill Norton had written explicitly "the lexicon is important", and dwelt on that theme, in his book "the internet peering playbook". This is the source of a lot of grief. Cheers, Etienne On Mon, May 8, 2023 at 9:57 PM Phil Bedard wrote: > I guess let’s not confuse two things. The optical network is made up of > the photonic portion and then the transponder/muxponder portion. A single > term like “DWDM” can be confusing since it can refer to both. It will > take a long time (maybe never) to remove the photonic switching part of the > network. However, it’s always been cheap to deploy because optical vendors > tended to subsidize that network using sales of the other portion, the > transponders, which you buy more and more over time. Those photonic > components are expensive. > > > > On the DWDM signal portion, I’m not talking about 100ZR compared to 100G > on a transponder or DWDM line system. 100ZR has had to deal with the power > limitations of QSFP28 ports, which QDD/OSFP do not suffer from. There are > quite a few QDD pluggables in production today capable of supporting 100G > signals over 1000s of km or 400G near 1500km. Now that’s not what you can > get out of some external transponders, so those will still have their place > in high performance applications. When you move to 800G, 1.2Tbps single > channel they also have their own distance limitations. So it really > depends on the application and the network. > > > > Phil > > > > *From: *NANOG on behalf > of Mark Tinka > *Date: *Friday, May 5, 2023 at 12:55 AM > *To: *nanog@nanog.org > *Subject: *Re: Routed optical networks > > > > On 5/4/23 19:32, Phil Bedard wrote: > > > > It’s my personal opinion we aren’t to the days yet of where we can simply > build an all packet network with no photonic switching that carries all > services, but eventually (random # of years) it gets there for many > networks. There are also always going to be high performance applications > for transponders where pluggable optics aren’t a good fit. > > > I think every time the IP space gets close to running an all-packet > network, the Transport folk come out with an easier way to do it, that it's > too hard to ignore. > > Based on that, I think they will always be one step ahead, with the key > advantage being reliability of capacity over the distance, for the cost. > > The farther your fibre has to run, the costlier it gets to do it without > DWDM. > > I mean, it's only now that 100G-ZR is becoming a reality for packet > networks, and we are talking thousands of US$ for optics to get us 80km - > 120km distance. Meanwhile, DWDM vendors can get you 800Gbps per wavelength > in the same distance (or 30X that distance) far less cheaply. > > I get the appeal of not needing DWDM gear to underlay your packet > network... it's neater and offers fewer points of failure. But unless you > are dealing with very short distances and can ride a reasonable balance > between service features and switching/forwarding capacity in your > router/switch, it's going to be hard to ignore the DWDM gear if you are > trying to be a serious operation, at that scale, over a wide area. > > Mark. > -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
Re: Routed optical networks
From: Mark Tinka Date: Tuesday, May 9, 2023 at 2:03 AM To: Phil Bedard , nanog@nanog.org Subject: Re: Routed optical networks On 5/8/23 21:53, Phil Bedard wrote: There are quite a few QDD pluggables in production today capable of supporting 100G signals over 1000s of km or 400G near 1500km. I think the more interesting QSPF-DD (or OSFP) development is OpenZR+, which is an MSA project to standardize 400ZR+. The plan is to be able to support 100Gbps up to 5,800km, and 400Gbps up to 480km (EDFA) or about 1,000km (EDFA + Raman). With a 4x 100Gbps mode supported on QSFP28 router ports, you can have one muxponder talking to 4x routers at 100Gbps each. [phil] These are already available today and have been for some time and in use in production networks for over a year now. This is with 400G links running up to 600km in routers with QDD ports. 400G-16QAM using 60Gbaud (the OpenZR+ standard) can reach around 1300km. These optics are being used in both routers and xponders on the line side. Now that’s not what you can get out of some external transponders, so those will still have their place in high performance applications. When you move to 800G, 1.2Tbps single channel they also have their own distance limitations. So it really depends on the application and the network. 800Gbps and 1.2Tbps applications are really for long haul use-cases, especially if you used 400Gbps pluggables before and run out of distance (so less than 1,000km). They are also preferred for submarine use-cases. [phil] 800Gbps and 1.2Tbps are not really positioned for long haul use cases yet. It requires lowering the modulation to something like QPSK which at those speeds requires baud rates which are not yet commercially available.
Re: Routed optical networks
Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG wrote: > Disclaimer: Metaverse has not changed Metro traffic yet. Then … > > > > I am puzzled when people talk about 400GE and Tbps in the Mero context. > > For historical reasons, Metro is still about 2*2*10GE (one “2” for > redundancy, another “2” for capacity) in the majority of cases worldwide. > > How many BRASes serve more than 4/1.5=27k users in the busy hour? > > It means that 50GE is the best interface now for the majority of cases. > 2*50GE=100Gbps is good room for growth. > > Of course, exceptions could be. I know BRAS that handles 86k subscribers > (do not recommend anybody to push the limits – it was so painful). > > > > We have just 2 eyes and look at video content about 22h per week (on > average). Our eyes do not permit us to see resolution better than > particular for chosen distance (4k for typical TV, HD for smartphones, and > so on). Color depth 10bits is enough for the majority, 12bits is sure > enough for everybody. 120 frames/sec is enough for everybody. It would > never change – it is our genetics. > > Fortunately for Carriers, the traffic has a limit. You have probably seen > that every year traffic growth % is decreasing. The Internet is stabilizing > and approaching the plateau. > > How much growth is still awaiting us? 1.5? 1.4? It needs separate > research. The result would be tailored for whom would pay for the research. > > IMHO: It is not mandatory that 100GE would become massive in the metro. (I > know that 100GE is already massive in the DC CLOS) > > > > Additionally, who would pay for this traffic growth? It also limits > traffic at some point. > > I hope it would happen after we would get our 22h/4k/12bit/120hz. > > > > Now, you could argue that Metaverse would jump and multiply traffic by an > additional 2x or 3x. Then 400GE may be needed. > > Sorry, but it is speculation yet. It is not a trend like the current > (declining) traffic growth. > > > > Ed/ > > *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] > *On Behalf Of *Phil Bedard > *Sent:* Thursday, May 4, 2023 8:32 PM > *To:* Etienne-Victor Depasquale ; NANOG > *Subject:* Re: Routed optical networks > > > > It’s not necessarily metro specific although the metro networks could lend > themselves to overall optimizations. > > > > The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your > adoption of 400G since today they require a QDD port. There are 100G QDD > ports but that’s not all that popular yet. Of course there is work to do > something similar in QSFP28 if the power can be reduced to what is > supported by an existing QSFP28 port in most devices. In larger networks > with higher speed requirements and moving to 400G with QDD, using the DCO > optics for connecting routers is kind of a no-brainer vs. a traditional > muxponder. Whether that’s over a ROADM based optical network or not, > especially at metro/regional distances. > > > > There are very large deployments of IPoDWDM over passive DWDM or dark > fiber for access and aggregation networks where the aggregate required > bandwidth doesn’t exceed the capabilities of those optics. It’s been done > at 10G for many years. With the advent of pluggable EDFA amplifiers, you > can even build links up to 120km* (perfect dark fiber) carrying tens of > terabits of traffic without any additional active optical equipment. > > > > It’s my personal opinion we aren’t to the days yet of where we can simply > build an all packet network with no photonic switching that carries all > services, but eventually (random # of years) it gets there for many > networks. There are also always going to be high performance applications > for transponders where pluggable optics aren’t a good fit. > > > > Carrying high speed private line/wavelength type services as well is a > different topic than interconnecting IP devices. > > > > Thanks, > > Phil > > &
Re: Routed optical networks
On 5/9/23 20:37, Phil Bedard wrote: [phil] These are already available today and have been for some time and in use in production networks for over a year now. This is with 400G links running up to 600km in routers with QDD ports. 400G-16QAM using 60Gbaud (the OpenZR+ standard) can reach around 1300km. These optics are being used in both routers and xponders on the line side. Right... I've not been keeping in touch with this development on the terrestrial side, as our focus is mainly on the submarine end of things. But this is good to know. In our case, the majority of our customers are still seeking 10Gbps and 100Gbps services over long distances (so 200km and over). Neither they (nor us) have a requirement for 400Gbps in the metro. That said, we do have the capability to do this on terrestrial spans where we are able to carry 600Gbps per channel across 700km or so. We are looking forward to the next generation of tech. with our vendor that could get us to about 700Gbps per channel for the same distance. [phil] 800Gbps and 1.2Tbps are not really positioned for long haul use cases yet. It requires lowering the modulation to something like QPSK which at those speeds requires baud rates which are not yet commercially available. I meant more in terms of the capabilities of the 800G/1.2T embedded option to improve spectral efficiency for more capacity over long haul use-cases, and not necessarily that they can actually get to 800Gbps or 1.2Tbps for that length of span. The next generation of 5nm CMOS with the potential for close to 150 Gbaud is quite intriguing, although I'm not expecting anything more than 15% - 18% improvement in performance compared to the current 7nm 100 Gbaud systems. Will keep you posted, especially on our submarine application of this. Mark.
Re: Routed optical networks
Dave, have you seen any recent output from Cisco's VNI on the matter of traffic growth? Or Sandvine's? How does it compare with your perception? Cheers, Etienne On Tue, May 9, 2023 at 10:40 PM Dave Taht wrote: > Up until this moment I was feeling that my take on the decline of traffic > growth was somewhat isolated, in that I have long felt that we are nearing > the top of the S curve of the data we humans can create and consume. About > the only source of future traffic growth I can think of comes from getting > more humans online, and that is a mere another doubling. > > On the other hand, predictions such as 640k should be enough for everyone > did not pan out. > > On the gripping hand, there has been an explosion of LLM stuff of late, > with enormous models being widely distributed in just the past month: > > https://lwn.net/Articles/930939/ > > Could the AIs takeoff lead to a resumption of traffic growth? I still > don´t think so... > > > On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG < > nanog@nanog.org> wrote: > >> Disclaimer: Metaverse has not changed Metro traffic yet. Then … >> >> >> >> I am puzzled when people talk about 400GE and Tbps in the Mero context. >> >> For historical reasons, Metro is still about 2*2*10GE (one “2” for >> redundancy, another “2” for capacity) in the majority of cases worldwide. >> >> How many BRASes serve more than 4/1.5=27k users in the busy hour? >> >> It means that 50GE is the best interface now for the majority of cases. >> 2*50GE=100Gbps is good room for growth. >> >> Of course, exceptions could be. I know BRAS that handles 86k subscribers >> (do not recommend anybody to push the limits – it was so painful). >> >> >> >> We have just 2 eyes and look at video content about 22h per week (on >> average). Our eyes do not permit us to see resolution better than >> particular for chosen distance (4k for typical TV, HD for smartphones, and >> so on). Color depth 10bits is enough for the majority, 12bits is sure >> enough for everybody. 120 frames/sec is enough for everybody. It would >> never change – it is our genetics. >> >> Fortunately for Carriers, the traffic has a limit. You have probably seen >> that every year traffic growth % is decreasing. The Internet is stabilizing >> and approaching the plateau. >> >> How much growth is still awaiting us? 1.5? 1.4? It needs separate >> research. The result would be tailored for whom would pay for the research. >> >> IMHO: It is not mandatory that 100GE would become massive in the metro. >> (I know that 100GE is already massive in the DC CLOS) >> >> >> >> Additionally, who would pay for this traffic growth? It also limits >> traffic at some point. >> >> I hope it would happen after we would get our 22h/4k/12bit/120hz. >> >> >> >> Now, you could argue that Metaverse would jump and multiply traffic by an >> additional 2x or 3x. Then 400GE may be needed. >> >> Sorry, but it is speculation yet. It is not a trend like the current >> (declining) traffic growth. >> >> >> >> Ed/ >> >> *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] >> *On Behalf Of *Phil Bedard >> *Sent:* Thursday, May 4, 2023 8:32 PM >> *To:* Etienne-Victor Depasquale ; NANOG >> *Subject:* Re: Routed optical networks >> >> >> >> It’s not necessarily metro specific although the metro networks could >> lend themselves to overall optimizations. >> >> >> >> The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your >> adoption of 400G since today they require a QDD port. There are 100G QDD >> ports but that’s not all that popular yet. Of course there is work to do >> something similar in QSFP28 if the power can be reduced to what is >> supported by an existing QSFP28 port in most devices. In larger networks >> with higher speed requirements and moving to 400G with QDD, using the DCO >> optics for connecting routers is kind of a no-brainer vs. a traditional >> muxponder. Whether that’s over a ROADM based optical network or not, >> especially at metro/regional distances. >> >> >> >> There are very large deployments of IPoDWDM over passive DWDM or dark >> fiber for access and aggregation networks where the aggregate required >> bandwidth doesn’t exceed the capabilities of those optics. It’s been done >> at 10G for many years. With the advent of pluggable EDFA amplifiers, you >> can even build links up to 120km*
RE: Routed optical networks
But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.t...@gmail.com] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard Cc: Phil Bedard ; Etienne-Victor Depasquale ; NANOG Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG mailto:nanog@nanog.org>> wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei@nanog.org<mailto:huawei@nanog.org>] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale mailto:ed...@ieee.org>>; NANOG mailto:nanog@nanog.org>> Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption of 400G since today they require a QDD port. There are 100G QDD ports but that’s not all that popular yet. Of course there is work to do something similar in QSFP28 if the power can be reduced to what is supported by an existing QSFP28 port in most devices. In larger networks with higher speed requirements and moving to 400G with QDD, using the DCO optics for connecting routers is kind of a no-brainer vs. a traditional muxponder. Whether that’s over a ROADM based opti
Re: Routed optical networks
Eduard, academics cite the VNI (and the Sandvine Global reports). Do you know of alternative sources that show traffic growth data you're more comfortable with? Cheers, Etienne On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard < vasilenko.edu...@huawei.com> wrote: > But it is speculation, not a trend yet. > > I remember 10y ago every presentation started from the claim that 100B of > IoT would drive XXX traffic. It did not happen. > > Now we see presentations that AI would be talking to AI that generates > traffic. > > Maybe some technology would push traffic next S-curve, maybe not. It is > still speculation. > > > > The traffic growth was stimulated (despite all VNIs) by 1) new > subscribers, 2) video quality for subscribers. Nothing else yet. > > It is almost finished for both trends. We are close to the plateau of > these S-curves. > > For some years (2013-2020) I was carefully looking at numbers for many > countries: it was always possible to split CAGR for these 2 components. The > video part was extremely consistent between countries. The subscriber part > was 100% proportional to subscriber CAGR. > > Everything else up to now was “marketing” to say it mildly. > > > > Reminder: nothing in nature could grow indefinitely. The limit always > exists. It is only a question of when. > > > > PS: Of course, marketing people could draw you any traffic growth – it > depends just on the marketing budget. > > > > Eduard > > *From:* Dave Taht [mailto:dave.t...@gmail.com] > *Sent:* Tuesday, May 9, 2023 11:41 PM > *To:* Vasilenko Eduard > *Cc:* Phil Bedard ; Etienne-Victor Depasquale < > ed...@ieee.org>; NANOG > *Subject:* Re: Routed optical networks > > > > Up until this moment I was feeling that my take on the decline of traffic > growth was somewhat isolated, in that I have long felt that we are nearing > the top of the S curve of the data we humans can create and consume. About > the only source of future traffic growth I can think of comes from getting > more humans online, and that is a mere another doubling. > > > > On the other hand, predictions such as 640k should be enough for everyone > did not pan out. > > > > On the gripping hand, there has been an explosion of LLM stuff of late, > with enormous models being widely distributed in just the past month: > > > > https://lwn.net/Articles/930939/ > > > > Could the AIs takeoff lead to a resumption of traffic growth? I still > don´t think so... > > > > > > On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG < > nanog@nanog.org> wrote: > > Disclaimer: Metaverse has not changed Metro traffic yet. Then … > > > > I am puzzled when people talk about 400GE and Tbps in the Mero context. > > For historical reasons, Metro is still about 2*2*10GE (one “2” for > redundancy, another “2” for capacity) in the majority of cases worldwide. > > How many BRASes serve more than 4/1.5=27k users in the busy hour? > > It means that 50GE is the best interface now for the majority of cases. > 2*50GE=100Gbps is good room for growth. > > Of course, exceptions could be. I know BRAS that handles 86k subscribers > (do not recommend anybody to push the limits – it was so painful). > > > > We have just 2 eyes and look at video content about 22h per week (on > average). Our eyes do not permit us to see resolution better than > particular for chosen distance (4k for typical TV, HD for smartphones, and > so on). Color depth 10bits is enough for the majority, 12bits is sure > enough for everybody. 120 frames/sec is enough for everybody. It would > never change – it is our genetics. > > Fortunately for Carriers, the traffic has a limit. You have probably seen > that every year traffic growth % is decreasing. The Internet is stabilizing > and approaching the plateau. > > How much growth is still awaiting us? 1.5? 1.4? It needs separate > research. The result would be tailored for whom would pay for the research. > > IMHO: It is not mandatory that 100GE would become massive in the metro. (I > know that 100GE is already massive in the DC CLOS) > > > > Additionally, who would pay for this traffic growth? It also limits > traffic at some point. > > I hope it would happen after we would get our 22h/4k/12bit/120hz. > > > > Now, you could argue that Metaverse would jump and multiply traffic by an > additional 2x or 3x. Then 400GE may be needed. > > Sorry, but it is speculation yet. It is not a trend like the current > (declining) traffic growth. > > > > Ed/ > > *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] > *On Behalf Of
RE: Routed optical networks
I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times). I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). Sometimes I talk to old connections and they confirm that it is even less now. In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber). But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth. In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it. Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing) Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:ed...@ieee.org] Sent: Thursday, May 11, 2023 12:48 PM To: Vasilenko Eduard Cc: Dave Taht ; Phil Bedard ; NANOG Subject: Re: Routed optical networks Eduard, academics cite the VNI (and the Sandvine Global reports). Do you know of alternative sources that show traffic growth data you're more comfortable with? Cheers, Etienne On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> wrote: But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.t...@gmail.com<mailto:dave.t...@gmail.com>] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> Cc: Phil Bedard mailto:bedard.p...@gmail.com>>; Etienne-Victor Depasquale mailto:ed...@ieee.org>>; NANOG mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG mailto:nanog@nanog.org>> wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It me
Re: Routed optical networks
Historically, this is what VNI has claimed <https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing> . Cheers, Etienne On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard < vasilenko.edu...@huawei.com> wrote: > I did investigate traffic for every Carrier while dealing with it as a > consultant (repeated many dozens of times). > > I have seen over a decade how traffic growth dropped year-over-year (from > 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked > travel). > > Sometimes I talk to old connections and they confirm that it is even less > now. > > In rear cases, It is typically possible to find this information on the > public Internet (I remember the case when Google disclosed traffic for > Pakistan at the conference with the explanation that 30% is attributed to > new subscribers, and an additional +30% is to more heavy content per > subscriber). > > But mostly, it was confidential information from a discussion with > Carriers – they all know very well their traffic growth. > > In general, traffic stat is pretty confidential. I did not have the > motivation to aggregate it. > > > > Sandvine is not representative of global traffic because DPI is installed > mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic > – it is not the biggest source. Moreover, Mobiles would look better growing > because the limiting factor was on technology (5G proposed more than 4G, 4G > proposed much more than 3G) – it would probably would less disruptive in > the future. > > Fixed Carriers do not pay DPI premiums. And rarely share their traffic > publicly. Sandvine could not see it. > > > > VNI is claiming so many things. Please show where exactly they show > traffic growth (I am not interested in prediction speculations). Is it > possible to understand CAGR for the 5 last years? Is it declining or > growing? (traffic itself is for sure still growing) > > > > Of course, the disruption could come at any year and add a new S-curve > (Metaverse?). But disruption is by definition not predictable. > > > > PS: Everything above and below in this thread is just my personal opinion. > > > > Eduard > > *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org] > *Sent:* Thursday, May 11, 2023 12:48 PM > *To:* Vasilenko Eduard > *Cc:* Dave Taht ; Phil Bedard ; > NANOG > *Subject:* Re: Routed optical networks > > > > Eduard, academics cite the VNI (and the Sandvine Global reports). > > > > Do you know of alternative sources that show traffic growth data you're > more comfortable with? > > > > Cheers, > > > > Etienne > > > > On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard < > vasilenko.edu...@huawei.com> wrote: > > But it is speculation, not a trend yet. > > I remember 10y ago every presentation started from the claim that 100B of > IoT would drive XXX traffic. It did not happen. > > Now we see presentations that AI would be talking to AI that generates > traffic. > > Maybe some technology would push traffic next S-curve, maybe not. It is > still speculation. > > > > The traffic growth was stimulated (despite all VNIs) by 1) new > subscribers, 2) video quality for subscribers. Nothing else yet. > > It is almost finished for both trends. We are close to the plateau of > these S-curves. > > For some years (2013-2020) I was carefully looking at numbers for many > countries: it was always possible to split CAGR for these 2 components. The > video part was extremely consistent between countries. The subscriber part > was 100% proportional to subscriber CAGR. > > Everything else up to now was “marketing” to say it mildly. > > > > Reminder: nothing in nature could grow indefinitely. The limit always > exists. It is only a question of when. > > > > PS: Of course, marketing people could draw you any traffic growth – it > depends just on the marketing budget. > > > > Eduard > > *From:* Dave Taht [mailto:dave.t...@gmail.com] > *Sent:* Tuesday, May 9, 2023 11:41 PM > *To:* Vasilenko Eduard > *Cc:* Phil Bedard ; Etienne-Victor Depasquale < > ed...@ieee.org>; NANOG > *Subject:* Re: Routed optical networks > > > > Up until this moment I was feeling that my take on the decline of traffic > growth was somewhat isolated, in that I have long felt that we are nearing > the top of the S curve of the data we humans can create and consume. About > the only source of future traffic growth I can think of comes from getting > more humans online, and that is a mere another doubling. > > > > On the other hand, predictions such as 640k s
Re: Routed optical networks
To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. Cheers, Etienne On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale wrote: > Historically, this is what VNI has claimed > <https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing> > . > > Cheers, > > Etienne > > On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard < > vasilenko.edu...@huawei.com> wrote: > >> I did investigate traffic for every Carrier while dealing with it as a >> consultant (repeated many dozens of times). >> >> I have seen over a decade how traffic growth dropped year-over-year (from >> 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked >> travel). >> >> Sometimes I talk to old connections and they confirm that it is even less >> now. >> >> In rear cases, It is typically possible to find this information on the >> public Internet (I remember the case when Google disclosed traffic for >> Pakistan at the conference with the explanation that 30% is attributed to >> new subscribers, and an additional +30% is to more heavy content per >> subscriber). >> >> But mostly, it was confidential information from a discussion with >> Carriers – they all know very well their traffic growth. >> >> In general, traffic stat is pretty confidential. I did not have the >> motivation to aggregate it. >> >> >> >> Sandvine is not representative of global traffic because DPI is installed >> mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic >> – it is not the biggest source. Moreover, Mobiles would look better growing >> because the limiting factor was on technology (5G proposed more than 4G, 4G >> proposed much more than 3G) – it would probably would less disruptive in >> the future. >> >> Fixed Carriers do not pay DPI premiums. And rarely share their traffic >> publicly. Sandvine could not see it. >> >> >> >> VNI is claiming so many things. Please show where exactly they show >> traffic growth (I am not interested in prediction speculations). Is it >> possible to understand CAGR for the 5 last years? Is it declining or >> growing? (traffic itself is for sure still growing) >> >> >> >> Of course, the disruption could come at any year and add a new S-curve >> (Metaverse?). But disruption is by definition not predictable. >> >> >> >> PS: Everything above and below in this thread is just my personal opinion. >> >> >> >> Eduard >> >> *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org] >> *Sent:* Thursday, May 11, 2023 12:48 PM >> *To:* Vasilenko Eduard >> *Cc:* Dave Taht ; Phil Bedard ; >> NANOG >> *Subject:* Re: Routed optical networks >> >> >> >> Eduard, academics cite the VNI (and the Sandvine Global reports). >> >> >> >> Do you know of alternative sources that show traffic growth data you're >> more comfortable with? >> >> >> >> Cheers, >> >> >> >> Etienne >> >> >> >> On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard < >> vasilenko.edu...@huawei.com> wrote: >> >> But it is speculation, not a trend yet. >> >> I remember 10y ago every presentation started from the claim that 100B of >> IoT would drive XXX traffic. It did not happen. >> >> Now we see presentations that AI would be talking to AI that generates >> traffic. >> >> Maybe some technology would push traffic next S-curve, maybe not. It is >> still speculation. >> >> >> >> The traffic growth was stimulated (despite all VNIs) by 1) new >> subscribers, 2) video quality for subscribers. Nothing else yet. >> >> It is almost finished for both trends. We are close to t
Re: Routed optical networks
> On May 11, 2023, at 7:45 AM, Etienne-Victor Depasquale via NANOG > wrote: > > To clarify the table I linked to in the previous email: > > Cisco estimates IP traffic exchanged over the access network by both > businesses and consumers with: > > • endpoints over managed networks and > • endpoints over unmanaged networks (“Internet traffic”). > > Both the mobile access network and the fixed access network are considered. > > Cisco considers IP traffic over managed networks to be characterized by > passage through a single service provider. > Without explicitly referring to quality of service (QoS), > the implication is clearly that the traffic is controlled to meet the QoS > demanded by the service level agreement (SLA). > > In contrast, “Internet traffic” crosses provider domains; > typically, this traffic is delivered on the basis of providers’ best effort. > These two kinds of traffic complement one another and collectively are > referred to as total global IP traffic. I think there’s a lot of problems here. While places like my employer will periodically disclose our traffic numbers, and DDoS providers, mitigation platforms and otherwise will disclose the peaks they see, much of this data is a bit opaque, and tools like AI that do in-metro or cross-metro datacenter-datacenter remote DMA type activities, those all count differently. We have seen a continued trend of the privatization of traffic and localization of that over time. I’ve watched all the big carriers retreat from their global network reaches to be more of regionalized networks. A decade ago you would have seen European national incumbents peering and with market in Asia, and the complete global networks continue to shrink. Meanwhile you have a mix of the content and cloud providers continue to build their business-purpose networks expanding into markets that the uppercase Internet may not need to reach. You can look at the proposals in the EU about fees, and I have dual thoughts on this which are MY OWN and don’t represent my employer or otherwise, but if you read this post from Petra Arts - https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major interconnection points like Frankfurt, which are important but double as problematic. The number of people that need to go to the near market (eg: Chicago, while I’m in Detroit area) for good connectivity is an issue, meanwhile there’s a robust need to keep traffic within the state of Michigan and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on the board). There need to be some aggregation points, so not everyone needs to be in Detroit, but also not everyone needs to be in Frankfurt - and content localization needs to continue to happen, but is also very regionalized in popularity. How to do this all and not have it all route via Chicago or Frankfurt is a challenge, but also not everyone will be in Berlin, Munich or these other markets. This is where having a robust optical network capability (or backbone) can come into play, that you can deliver deeper from those hub points, but at the same time, I’ve been in meetings where companies have their own challenges accepting that content in those downstream locations as their network was also built to get to/from the major hub cities, or IP space wasn’t allocated in a way that can provide consistent routing results or behaviors. (This is where IPv6 can be super helpful, it gives the chance to possibly Greenfield, aka not screw it up - at least initially). There’s huge volumes of IP traffic exchanged, but the largest volumes are being moved over private interconnects or a localized IX to those eyeball networks with the historical global backbones playing more of the long-distance carrier role, which is critical as you want a path to deliver those bits, without it following the ITU-style sender pays model, as the majority of IP traffic is actually requested by the customer of the end-user network. (All of it if you remove network scans, ddos, web bots/crawlers). Most networks have no SLA once things cross an unpaid boundary (SFI, or even private peering) - and if they are a customer and that path is congested, it’s up to the customer to upgrade that path. - Jared (many hats)
RE: Routed optical networks
Hi Etienne, Look carefully what you have shown to me. It is a only speculation again (“predictions”). It is just a table with a collection of all predictions in the past. Moreover, averaged between years. I was asking for real data from the past 5 years. Are you sure that VNI has it? If you would find real historical data in VNI, then we would be capable to check the table that you have shown: was the guessing right? I strongly suspect an answer. Eduard From: Etienne-Victor Depasquale [mailto:ed...@ieee.org] Sent: Thursday, May 11, 2023 2:46 PM To: Vasilenko Eduard Cc: Dave Taht ; Phil Bedard ; NANOG Subject: Re: Routed optical networks To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. Cheers, Etienne On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale mailto:ed...@ieee.org>> wrote: Historically, this is what VNI has claimed<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>. Cheers, Etienne On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> wrote: I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times). I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). Sometimes I talk to old connections and they confirm that it is even less now. In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber). But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth. In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it. Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing) Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:ed...@ieee.org<mailto:ed...@ieee.org>] Sent: Thursday, May 11, 2023 12:48 PM To: Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> Cc: Dave Taht mailto:dave.t...@gmail.com>>; Phil Bedard mailto:bedard.p...@gmail.com>>; NANOG mailto:nanog@nanog.org>> Subject: Re: Routed optical networks Eduard, academics cite the VNI (and the Sandvine Global reports). Do you know of alternative sources that show traffic growth data you're more comfortable with? Cheers, Etienne On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> wrote: But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020)
RE: Routed optical networks
Hi Jared, Could I make a conclusion from your comments: "only Carrier itself understand the traffic - see many examples in the text". I would very agree to this. Eduard -Original Message- From: Jared Mauch [mailto:ja...@puck.nether.net] Sent: Thursday, May 11, 2023 3:16 PM To: Etienne-Victor Depasquale Cc: Vasilenko Eduard ; NANOG Subject: Re: Routed optical networks > On May 11, 2023, at 7:45 AM, Etienne-Victor Depasquale via NANOG > wrote: > > To clarify the table I linked to in the previous email: > > Cisco estimates IP traffic exchanged over the access network by both > businesses and consumers with: > > • endpoints over managed networks and • endpoints over unmanaged > networks (“Internet traffic”). > > Both the mobile access network and the fixed access network are considered. > > Cisco considers IP traffic over managed networks to be characterized by > passage through a single service provider. > Without explicitly referring to quality of service (QoS), the > implication is clearly that the traffic is controlled to meet the QoS > demanded by the service level agreement (SLA). > > In contrast, “Internet traffic” crosses provider domains; typically, > this traffic is delivered on the basis of providers’ best effort. > These two kinds of traffic complement one another and collectively are > referred to as total global IP traffic. I think there’s a lot of problems here. While places like my employer will periodically disclose our traffic numbers, and DDoS providers, mitigation platforms and otherwise will disclose the peaks they see, much of this data is a bit opaque, and tools like AI that do in-metro or cross-metro datacenter-datacenter remote DMA type activities, those all count differently. We have seen a continued trend of the privatization of traffic and localization of that over time. I’ve watched all the big carriers retreat from their global network reaches to be more of regionalized networks. A decade ago you would have seen European national incumbents peering and with market in Asia, and the complete global networks continue to shrink. Meanwhile you have a mix of the content and cloud providers continue to build their business-purpose networks expanding into markets that the uppercase Internet may not need to reach. You can look at the proposals in the EU about fees, and I have dual thoughts on this which are MY OWN and don’t represent my employer or otherwise, but if you read this post from Petra Arts - https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major interconnection points like Frankfurt, which are important but double as problematic. The number of people that need to go to the near market (eg: Chicago, while I’m in Detroit area) for good connectivity is an issue, meanwhile there’s a robust need to keep traffic within the state of Michigan and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on the board). There need to be some aggregation points, so not everyone needs to be in Detroit, but also not everyone needs to be in Frankfurt - and content localization needs to continue to happen, but is also very regionalized in popularity. How to do this all and not have it all route via Chicago or Frankfurt is a challenge, but also not everyone will be in Berlin, Munich or these other markets. This is where having a robust optical network capability (or backbone) can come into play, that you can deliver deeper from those hub points, but at the same time, I’ve been in meetings where companies have their own challenges accepting that content in those downstream locations as their network was also built to get to/from the major hub cities, or IP space wasn’t allocated in a way that can provide consistent routing results or behaviors. (This is where IPv6 can be super helpful, it gives the chance to possibly Greenfield, aka not screw it up - at least initially). There’s huge volumes of IP traffic exchanged, but the largest volumes are being moved over private interconnects or a localized IX to those eyeball networks with the historical global backbones playing more of the long-distance carrier role, which is critical as you want a path to deliver those bits, without it following the ITU-style sender pays model, as the majority of IP traffic is actually requested by the customer of the end-user network. (All of it if you remove network scans, ddos, web bots/crawlers). Most networks have no SLA once things cross an unpaid boundary (SFI, or even private peering) - and if they are a customer and that path is congested, it’s up to the customer to upgrade that path. - Jared (many hats)
Re: Routed optical networks
On 5/11/23 13:45, Etienne-Victor Depasquale via NANOG wrote: To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). So either Cisco think/though that the only operators worth surveying/predicting were the large ones (NTT, Telia, Tata, Lumen, Cogent, e.t.c.), or that on-net MPLS/VPN traffic was more significant than public IP Transit both in terms of revenue and strategic direction of the operators they surveyed/predicated. Either way, I'd imagine any results based on those data points would be incomplete, at least from a real-world standpoint. In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. In the IP space, this traffic type is quickly exceeding any historically significant MPLS/VPN traffic, if it hasn't already. Mark.
Re: Routed optical networks
On 5/11/23 09:33, Vasilenko Eduard via NANOG wrote: But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. I think the place where we are seeing continued growth is in the content provider private network, for that DC-to-DC traffic, even across continents. For everything else, as you say, it's nice to predict. But what analysts and marketing departments say and what end users actually do are often vastly different. As long as the majority of the Internet's traffic is "invisible", we can't really ever know. We can only tell by how much kit the DWDM vendors are selling to non-telco customers (content folk), as well as how many submarine cables are being built by non-telco consortia (content folk). Mark.
Re: Routed optical networks
On 5/11/23 13:25, Vasilenko Eduard via NANOG wrote: Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. And QUIC is making the the DPI model so loved by MNO's quickly irrelevant. Mark.
Re: Routed optical networks
Eduard, you know the answer as well as I do, right :) ? Here's my answer: I think that Cisco can only estimate (let's not say speculate, it has pretty bad connotations) what comes out of access networks. No offence meant, I hope none is taken. Cheers, Etienne On Thu, May 11, 2023 at 3:47 PM Vasilenko Eduard < vasilenko.edu...@huawei.com> wrote: > Hi Etienne, > > Look carefully what you have shown to me. It is a only speculation again > (“predictions”). It is just a table with a collection of all predictions in > the past. Moreover, averaged between years. > > I was asking for real data from the past 5 years. Are you sure that VNI > has it? > > > > If you would find real historical data in VNI, then we would be capable to > check the table that you have shown: was the guessing right? > > I strongly suspect an answer. > > > > Eduard > > *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org] > *Sent:* Thursday, May 11, 2023 2:46 PM > *To:* Vasilenko Eduard > *Cc:* Dave Taht ; Phil Bedard ; > NANOG > *Subject:* Re: Routed optical networks > > > > To clarify the table I linked to in the previous email: > > > > Cisco estimates IP traffic exchanged over the access network by both > businesses and consumers with: > > > • endpoints over managed networks and > • endpoints over unmanaged networks (“Internet traffic”). > > > Both the mobile access network and the fixed access network are > considered. > > > > Cisco considers IP traffic over managed networks to be characterized by > passage through a single service provider. > > Without explicitly referring to quality of service (QoS), > > the implication is clearly that the traffic is controlled to meet the QoS > demanded by the service level agreement (SLA). > > > > In contrast, “Internet traffic” crosses provider domains; > > typically, this traffic is delivered on the basis of providers’ best > effort. > > These two kinds of traffic complement one another and collectively are > referred to as total global IP traffic. > > > > Cheers, > > > > Etienne > > > > On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale > wrote: > > Historically, this is what VNI has claimed > <https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing> > . > > > > Cheers, > > > Etienne > > > > On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard < > vasilenko.edu...@huawei.com> wrote: > > I did investigate traffic for every Carrier while dealing with it as a > consultant (repeated many dozens of times). > > I have seen over a decade how traffic growth dropped year-over-year (from > 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked > travel). > > Sometimes I talk to old connections and they confirm that it is even less > now. > > In rear cases, It is typically possible to find this information on the > public Internet (I remember the case when Google disclosed traffic for > Pakistan at the conference with the explanation that 30% is attributed to > new subscribers, and an additional +30% is to more heavy content per > subscriber). > > But mostly, it was confidential information from a discussion with > Carriers – they all know very well their traffic growth. > > In general, traffic stat is pretty confidential. I did not have the > motivation to aggregate it. > > > > Sandvine is not representative of global traffic because DPI is installed > mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic > – it is not the biggest source. Moreover, Mobiles would look better growing > because the limiting factor was on technology (5G proposed more than 4G, 4G > proposed much more than 3G) – it would probably would less disruptive in > the future. > > Fixed Carriers do not pay DPI premiums. And rarely share their traffic > publicly. Sandvine could not see it. > > > > VNI is claiming so many things. Please show where exactly they show > traffic growth (I am not interested in prediction speculations). Is it > possible to understand CAGR for the 5 last years? Is it declining or > growing? (traffic itself is for sure still growing) > > > > Of course, the disruption could come at any year and add a new S-curve > (Metaverse?). But disruption is by definition not predictable. > > > > PS: Everything above and below in this thread is just my personal opinion. > > > > Eduard > > *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org] > *Sent:* Thursday, May 11, 2023 12:48 PM > *To:* Vasilenko Eduard > *Cc:* Dave Taht ; Phil Bedard ; > NANOG > *Subject:*
Re: Routed optical networks
On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote: Hi Jared, Could I make a conclusion from your comments: "only Carrier itself understand the traffic - see many examples in the text". I would very agree to this. I wouldn't say only carriers understand the traffic as much as I would say carrier's traffic is more transparent, and perhaps, even more readily available. I just think that it is not relevant to try and lump network and content traffic into one growth pattern. For all intents & purposes, there are two Internets running side-by-side between network and content. They converge at some point, but really, they are very different. Mark.
Re: Routed optical networks
On 5/11/23 14:15, Jared Mauch wrote We have seen a continued trend of the privatization of traffic and localization of that over time. CDN's, submarine cables and exchange points are decentralizing the "core of the Internet", and relegating IP Transit providers to whom CDN's subscribe as sources of origin content, if they are unable to build their own backbones. Only in markets where CDN's are not as rife do you still see huge growth and stable pricing for IP Transit traffic. I’ve watched all the big carriers retreat from their global network reaches to be more of regionalized networks. A decade ago you would have seen European national incumbents peering and with market in Asia, and the complete global networks continue to shrink. Yeah - unless you serve a very tight niche of Enterprise customers that require stable (not necessarily fast) connectivity to some part of the word, it makes little sense, nowadays, for the large telco's of old to go poking in other markets that aren't theirs, something the top global exchange points are going to learn the hard way. Meanwhile you have a mix of the content and cloud providers continue to build their business-purpose networks expanding into markets that the uppercase Internet may not need to reach. At the moment, the only motivation I see for large telco's trying to enter markets they can't be strong in is their "global brand". Often times, they can't do very well in that market because a) there is sufficient local content and peering, b) they can't build a network as robust as what the local telco's can, and c) all they have to offer is access to traffic several milliseconds away that is not any faster than what the local telco's can, which forms less than 20% of what the local customers chase after anyway. Such telco's usually, then, turn to targeting global Enterprise customers with "worldwide service contracts" accompanied by marked-up pricing for "private global branch connectivity". It always sounds and looks good on PowerPoint slides, but 12 months after the big launch, a broken champagne bottle and some photos, the PoP is gathering quite a bit of dust in the data centre. You can look at the proposals in the EU about fees, and I have dual thoughts on this which are MY OWN and don’t represent my employer or otherwise, but if you read this post from Petra Arts -https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major interconnection points like Frankfurt, which are important but double as problematic. The number of people that need to go to the near market (eg: Chicago, while I’m in Detroit area) for good connectivity is an issue, meanwhile there’s a robust need to keep traffic within the state of Michigan and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on the board). There need to be some aggregation points, so not everyone needs to be in Detroit, but also not everyone needs to be in Frankfurt - and content localization needs to continue to happen, but is also very regionalized in popularity. Us telco's didn't learn our lesson 23 years ago when the content folk tried to do a deal with us. This will be deja vu, although this time, with some help from governments that we rather won't like. Telco's continue to strong-arm content because we own the customer. But, it's 2023... I'm not sure we want to see a battle of the customer between content and network. It might not end well. We already see glimpses of it with QUIC, so... How to do this all and not have it all route via Chicago or Frankfurt is a challenge, but also not everyone will be in Berlin, Munich or these other markets. This is where having a robust optical network capability (or backbone) can come into play, that you can deliver deeper from those hub points, but at the same time, I’ve been in meetings where companies have their own challenges accepting that content in those downstream locations as their network was also built to get to/from the major hub cities, or IP space wasn’t allocated in a way that can provide consistent routing results or behaviors. (This is where IPv6 can be super helpful, it gives the chance to possibly Greenfield, aka not screw it up - at least initially). As content continues to grow, PoP's built in far-flung locations away from a telco's core area of business will only remain as relevant as traffic that has not yet migrated to a syndicated cloud. As that tapers off- which it will - there will be more pressure to either shut those PoP's down, or compete for local IP Transit. The latter is much less likely. There’s huge volumes of IP traffic exchanged, but the largest volumes are being moved over private interconnects or a localized IX to those eyeball networks with the historical global backbones playing more of the long-distance carrier role, which is critical as you want a path to deliver those bits, without it followin
Re: Routed optical networks
On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote: Hi Jared, Could I make a conclusion from your comments: "only Carrier itself understand the traffic - see many examples in the text". I would very agree to this. I wouldn't say only carriers understand the traffic as much as I would say carrier's traffic is more transparent, and perhaps, even more readily available. I just think that it is not relevant to try and lump network and content traffic into one growth pattern. For all intents & purposes, there are two Internets running side-by-side between network and content. They converge at some point, but really, they are very different. Mark.
RE: Routed optical networks
Why you are so sure that they have access to traffic data for many access networks? Their BRASes market share is far from 100%. DSLAMs finished 20 years ago. For the cases where they support BRASes they could collect statistics. But are they doing it? Carriers have not given permission (not even requested). I do not have a clue about VNI arrangement – it is magic. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:ed...@ieee.org] Sent: Thursday, May 11, 2023 6:03 PM To: Vasilenko Eduard Cc: Dave Taht ; Phil Bedard ; NANOG Subject: Re: Routed optical networks Eduard, you know the answer as well as I do, right :) ? Here's my answer: I think that Cisco can only estimate (let's not say speculate, it has pretty bad connotations) what comes out of access networks. No offence meant, I hope none is taken. Cheers, Etienne On Thu, May 11, 2023 at 3:47 PM Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> wrote: Hi Etienne, Look carefully what you have shown to me. It is a only speculation again (“predictions”). It is just a table with a collection of all predictions in the past. Moreover, averaged between years. I was asking for real data from the past 5 years. Are you sure that VNI has it? If you would find real historical data in VNI, then we would be capable to check the table that you have shown: was the guessing right? I strongly suspect an answer. Eduard From: Etienne-Victor Depasquale [mailto:ed...@ieee.org<mailto:ed...@ieee.org>] Sent: Thursday, May 11, 2023 2:46 PM To: Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> Cc: Dave Taht mailto:dave.t...@gmail.com>>; Phil Bedard mailto:bedard.p...@gmail.com>>; NANOG mailto:nanog@nanog.org>> Subject: Re: Routed optical networks To clarify the table I linked to in the previous email: Cisco estimates IP traffic exchanged over the access network by both businesses and consumers with: • endpoints over managed networks and • endpoints over unmanaged networks (“Internet traffic”). Both the mobile access network and the fixed access network are considered. Cisco considers IP traffic over managed networks to be characterized by passage through a single service provider. Without explicitly referring to quality of service (QoS), the implication is clearly that the traffic is controlled to meet the QoS demanded by the service level agreement (SLA). In contrast, “Internet traffic” crosses provider domains; typically, this traffic is delivered on the basis of providers’ best effort. These two kinds of traffic complement one another and collectively are referred to as total global IP traffic. Cheers, Etienne On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale mailto:ed...@ieee.org>> wrote: Historically, this is what VNI has claimed<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>. Cheers, Etienne On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard mailto:vasilenko.edu...@huawei.com>> wrote: I did investigate traffic for every Carrier while dealing with it as a consultant (repeated many dozens of times). I have seen over a decade how traffic growth dropped year-over-year (from 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). Sometimes I talk to old connections and they confirm that it is even less now. In rear cases, It is typically possible to find this information on the public Internet (I remember the case when Google disclosed traffic for Pakistan at the conference with the explanation that 30% is attributed to new subscribers, and an additional +30% is to more heavy content per subscriber). But mostly, it was confidential information from a discussion with Carriers – they all know very well their traffic growth. In general, traffic stat is pretty confidential. I did not have the motivation to aggregate it. Sandvine is not representative of global traffic because DPI is installed mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – it is not the biggest source. Moreover, Mobiles would look better growing because the limiting factor was on technology (5G proposed more than 4G, 4G proposed much more than 3G) – it would probably would less disruptive in the future. Fixed Carriers do not pay DPI premiums. And rarely share their traffic publicly. Sandvine could not see it. VNI is claiming so many things. Please show where exactly they show traffic growth (I am not interested in prediction speculations). Is it possible to understand CAGR for the 5 last years? Is it declining or growing? (traffic itself is for sure still growing) Of course, the disruption could come at any year and add a new S-curve (Metaverse?). But disruption is by definition not predictable. PS: Everything above and below in this thread is just my personal opinion. Eduard From: Etienne-Victor Depasquale [mailto:ed...@
Re: Routed optical networks
> On May 11, 2023, at 11:11 AM, Mark Tinka wrote: > > > > On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote: > >> Hi Jared, >> Could I make a conclusion from your comments: "only Carrier itself >> understand the traffic - see many examples in the text". >> I would very agree to this. > > I wouldn't say only carriers understand the traffic as much as I would say > carrier's traffic is more transparent, and perhaps, even more readily > available. > > I just think that it is not relevant to try and lump network and content > traffic into one growth pattern. For all intents & purposes, there are two > Internets running side-by-side between network and content. They converge at > some point, but really, they are very different. And as I’ve seen, they continue to become a bit more divergent. As someone who has an access network I see where the majority of my bits go, which is to the content folks. There’s some to the other part, but mostly people want their MTV, but since it’s 2023, it’s not MTV, but people want their TikTok, Metaverse, Game downloads, Streaming service, Email and cloud ramps (enterprise). - jared
Re: Routed optical networks
On 5/11/23 17:26, Jared Mauch wrote: And as I’ve seen, they continue to become a bit more divergent. As someone who has an access network I see where the majority of my bits go, which is to the content folks. There’s some to the other part, but mostly people want their MTV, but since it’s 2023, it’s not MTV, but people want their TikTok, Metaverse, Game downloads, Streaming service, Email and cloud ramps (enterprise). 100% - and if the trend continues, Telegeography et al will have less raw growth to report on unless the content folk willingly open their skirts up to them for a peek - which they won't do. So yes, predicting stable or negative growth for the public Internet is not without merit. But that does not translate to what the content folk are recording. It might make more sense to start reporting on traffic growth in the edge, specifically, the peering edge, as content networks continue to become major on-ramp/off-ramp paths for telco's. But that prediction can be extrapolated from submarine cable builds, submarine cable upgrades as well as DWDM vendor sales, with some degree of reliability. Mark.
Re: Routed optical networks
> > Why you are so sure that they have access to traffic data for many access > networks? Well, no, I didn't say that I'm sure, actually I agree with you on everything you've said. But I can't say that it's purely speculation, as that would imply that I know more than I actually do. The overview of the method (from the 2016 - 2021 report) states the following: "The Cisco Visual Networking Index Forecast methodology has been developed based on a combination of analyst projections, in-house estimates and forecasts, and direct data collection. The analyst projections for broadband connections, video subscribers, mobile connections, and Internet application adoption come from SNL Kagan, Ovum, Informa Telecoms & Media, Infonetics, IDC, Gartner, AMI, Verto Analytics, Ookla Speedtest.net, Strategy Analytics, Screen Digest, Dell’Oro Group, Synergy, comScore, Nielsen, Maravedis, Machina Research, ACG Research, ABI Research, Media Partners Asia, IHS, International Telecommunications Union (ITU), CTIA, UN, telecommunications regulators, and others. Upon this foundation are layered Cisco’s own estimates for application adoption, minutes of use, and kilobytes per minute. The adoption, usage, and bit-rate assumptions are tied to fundamental enablers such as broadband speed and computing speed. All usage and traffic results are then validated using data shared with Cisco from service providers. Figure 1 shows the forecast methodology." Link to Figure 1 here <https://drive.google.com/file/d/1JYIXRFmlIR00qOftqXnX9Z5gKikTol7X/view?usp=sharing> . Cheers, Etienne On Thu, May 11, 2023 at 5:18 PM Vasilenko Eduard < vasilenko.edu...@huawei.com> wrote: > Why you are so sure that they have access to traffic data for many access > networks? > > Their BRASes market share is far from 100%. DSLAMs finished 20 years ago. > > For the cases where they support BRASes they could collect statistics. But > are they doing it? Carriers have not given permission (not even requested). > > I do not have a clue about VNI arrangement – it is magic. > > PS: Everything above and below in this thread is just my personal opinion. > > Eduard > > *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org] > *Sent:* Thursday, May 11, 2023 6:03 PM > *To:* Vasilenko Eduard > *Cc:* Dave Taht ; Phil Bedard ; > NANOG > *Subject:* Re: Routed optical networks > > > > Eduard, you know the answer as well as I do, right :) ? > > > > Here's my answer: I think that Cisco can only estimate (let's not say > speculate, it has pretty bad connotations) what comes out of access > networks. > > > > No offence meant, I hope none is taken. > > > > Cheers, > > > > Etienne > > > > On Thu, May 11, 2023 at 3:47 PM Vasilenko Eduard < > vasilenko.edu...@huawei.com> wrote: > > Hi Etienne, > > Look carefully what you have shown to me. It is a only speculation again > (“predictions”). It is just a table with a collection of all predictions in > the past. Moreover, averaged between years. > > I was asking for real data from the past 5 years. Are you sure that VNI > has it? > > > > If you would find real historical data in VNI, then we would be capable to > check the table that you have shown: was the guessing right? > > I strongly suspect an answer. > > > > Eduard > > *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org] > *Sent:* Thursday, May 11, 2023 2:46 PM > *To:* Vasilenko Eduard > *Cc:* Dave Taht ; Phil Bedard ; > NANOG > *Subject:* Re: Routed optical networks > > > > To clarify the table I linked to in the previous email: > > > > Cisco estimates IP traffic exchanged over the access network by both > businesses and consumers with: > > > • endpoints over managed networks and > • endpoints over unmanaged networks (“Internet traffic”). > > > Both the mobile access network and the fixed access network are > considered. > > > > Cisco considers IP traffic over managed networks to be characterized by > passage through a single service provider. > > Without explicitly referring to quality of service (QoS), > > the implication is clearly that the traffic is controlled to meet the QoS > demanded by the service level agreement (SLA). > > > > In contrast, “Internet traffic” crosses provider domains; > > typically, this traffic is delivered on the basis of providers’ best > effort. > > These two kinds of traffic complement one another and collectively are > referred to as total global IP traffic. > > > > Cheers, > > > > Etienne > > > > On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale > wrote: > > Historically, this is
Re: Routed optical networks
On Thu, May 11, 2023 at 4:25 AM Vasilenko Eduard wrote: > > I did investigate traffic for every Carrier while dealing with it as a > consultant (repeated many dozens of times). > > I have seen over a decade how traffic growth dropped year-over-year (from 60% > to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel). There was a covid spike, but the trendline appeared to be down to 5% projected this year in a british study for fixed residential that I cannot find right now. > Sometimes I talk to old connections and they confirm that it is even less now. > > In rear cases, It is typically possible to find this information on the > public Internet (I remember the case when Google disclosed traffic for > Pakistan at the conference with the explanation that 30% is attributed to new > subscribers, and an additional +30% is to more heavy content per subscriber). > > But mostly, it was confidential information from a discussion with Carriers – > they all know very well their traffic growth. > > In general, traffic stat is pretty confidential. I did not have the > motivation to aggregate it. > > > > Sandvine is not representative of global traffic because DPI is installed > mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – > it is not the biggest source. Moreover, Mobiles would look better growing > because the limiting factor was on technology (5G proposed more than 4G, 4G > proposed much more than 3G) – it would probably would less disruptive in the > future. > > Fixed Carriers do not pay DPI premiums. And rarely share their traffic > publicly. Sandvine could not see it. > > > > VNI is claiming so many things. Please show where exactly they show traffic > growth (I am not interested in prediction speculations). Is it possible to > understand CAGR for the 5 last years? Is it declining or growing? (traffic > itself is for sure still growing) > > > > Of course, the disruption could come at any year and add a new S-curve > (Metaverse?). But disruption is by definition not predictable. > > > > PS: Everything above and below in this thread is just my personal opinion. > > > > Eduard > > From: Etienne-Victor Depasquale [mailto:ed...@ieee.org] > Sent: Thursday, May 11, 2023 12:48 PM > To: Vasilenko Eduard > Cc: Dave Taht ; Phil Bedard ; > NANOG > Subject: Re: Routed optical networks > > > > Eduard, academics cite the VNI (and the Sandvine Global reports). > > > > Do you know of alternative sources that show traffic growth data you're more > comfortable with? > > > > Cheers, > > > > Etienne > > > > On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard > wrote: > > But it is speculation, not a trend yet. > > I remember 10y ago every presentation started from the claim that 100B of IoT > would drive XXX traffic. It did not happen. > > Now we see presentations that AI would be talking to AI that generates > traffic. > > Maybe some technology would push traffic next S-curve, maybe not. It is still > speculation. > > > > The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, > 2) video quality for subscribers. Nothing else yet. > > It is almost finished for both trends. We are close to the plateau of these > S-curves. > > For some years (2013-2020) I was carefully looking at numbers for many > countries: it was always possible to split CAGR for these 2 components. The > video part was extremely consistent between countries. The subscriber part > was 100% proportional to subscriber CAGR. > > Everything else up to now was “marketing” to say it mildly. > > > > Reminder: nothing in nature could grow indefinitely. The limit always exists. > It is only a question of when. > > > > PS: Of course, marketing people could draw you any traffic growth – it > depends just on the marketing budget. > > > > Eduard > > From: Dave Taht [mailto:dave.t...@gmail.com] > Sent: Tuesday, May 9, 2023 11:41 PM > To: Vasilenko Eduard > Cc: Phil Bedard ; Etienne-Victor Depasquale > ; NANOG > Subject: Re: Routed optical networks > > > > Up until this moment I was feeling that my take on the decline of traffic > growth was somewhat isolated, in that I have long felt that we are nearing > the top of the S curve of the data we humans can create and consume. About > the only source of future traffic growth I can think of comes from getting > more humans online, and that is a mere another doubling. > > > > On the other hand, predictions such as 640k should be enough for everyone did > not pan out. > > > > On the grippi
Re: Routed optical networks
On Thu, May 11, 2023 at 7:32 AM Mark Tinka wrote: > > > > On 5/11/23 13:25, Vasilenko Eduard via NANOG wrote: > > Sandvine is not representative of global traffic because DPI is installed > mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – > it is not the biggest source. Moreover, Mobiles would look better growing > because the limiting factor was on technology (5G proposed more than 4G, 4G > proposed much more than 3G) – it would probably would less disruptive in the > future. > > Fixed Carriers do not pay DPI premiums. And rarely share their traffic > publicly. Sandvine could not see it. > > > And QUIC is making the the DPI model so loved by MNO's quickly irrelevant. What we have been doing with libreqos is add in FQ + AQM (sch_cake), while pioneering some nifty new eBPF traffic monitoring abilities like being able to do "mice and elephants" plots, and more importantly sample TCP RTT statistics using a variant of kathie nichols' "passive ping" (pping) utility. This latter feature is proving very useful in diagnosing problems deeper in the network. Quic does respond to good ole queuing delays, packet drop, and in some cases ecn, but I am going to miss pping as it deploys more. It is the FQ that helps the most on videoconferencing and voip traffic. Libreqos is free software, working as a bridge, you can plug it in between any two points on your network, and on cheap (350 bucks off of ebay) xeon gold hardware easily cracks 25Gbits while shaping with a goal of cracking 100Gbits one day soon. demo here: https://payne.taht.net , running a variety of flent based stress tests, tcp up and downloads, the rrul test, etc. The interesting part to me, is that *real* traffic actually looks nothing like that on a per subscriber basis. Here´s a screen-movie example of real-world netflix queuing depth and delay: https://www.youtube.com/watch?v=C-2oSBr2200 Our new "piano roll" mice and elephants plotter is pretty neat, if I can say so myself. ... What we see from those ISPs that have shared their data from their Libreqos deployment is that the principal usage of major bandwidth is movie streaming, and there is essentially no difference in average usage between a 50Mbit residential plan and a gbit plan and only barely discernible at 25 vs 50. Given how new this codebase is we do not have any long term statistics for traffic growth or decline, or other patterns. But I kind of hope more folk here fire it up in their labs, at least. Please drop in on our chat channel #libreqos:matrix.org if you need help getting it setup. Sampling as we can at 10ms is a very different view of the net from 5 minute averages. Specifically plugging because I would like to understand CDN traffic patterns better, and do not have much data as yet on that side, collected this way. > Mark. -- Podcast: https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/ Dave Täht CSO, LibreQos
Re: Routed optical networks
On 5/12/23 15:03, Dave Taht wrote: Libreqos is free software, working as a bridge, you can plug it in between any two points on your network, and on cheap (350 bucks off of ebay) xeon gold hardware easily cracks 25Gbits while shaping with a goal of cracking 100Gbits one day soon. This is fantastic! I also found your post about it here: https://www.reddit.com/r/HomeNetworking/comments/11pmc9a/a_latency_on_the_internet_update_bufferbloat_sqm/ If you could throw more hardware at it, could it do several 100's of Gbps? Also, when you say "bridge", if the server dies, does it become a wire, or would that require specialized hardware builds? Mark.
Re: Routed optical networks
"I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen" Often the type of people making these kinds of predictions that a tire pressure sensor generates 20 gigabytes of traffic a day. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com - Original Message - From: "Vasilenko Eduard via NANOG" To: "Dave Taht" Cc: "NANOG" Sent: Thursday, May 11, 2023 2:33:58 AM Subject: RE: Routed optical networks But it is speculation, not a trend yet. I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen. Now we see presentations that AI would be talking to AI that generates traffic. Maybe some technology would push traffic next S-curve, maybe not. It is still speculation. The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) video quality for subscribers. Nothing else yet. It is almost finished for both trends. We are close to the plateau of these S-curves. For some years (2013-2020) I was carefully looking at numbers for many countries: it was always possible to split CAGR for these 2 components. The video part was extremely consistent between countries. The subscriber part was 100% proportional to subscriber CAGR. Everything else up to now was “marketing” to say it mildly. Reminder: nothing in nature could grow indefinitely. The limit always exists. It is only a question of when. PS: Of course, marketing people could draw you any traffic growth – it depends just on the marketing budget. Eduard From: Dave Taht [mailto:dave.t...@gmail.com] Sent: Tuesday, May 9, 2023 11:41 PM To: Vasilenko Eduard Cc: Phil Bedard ; Etienne-Victor Depasquale ; NANOG Subject: Re: Routed optical networks Up until this moment I was feeling that my take on the decline of traffic growth was somewhat isolated, in that I have long felt that we are nearing the top of the S curve of the data we humans can create and consume. About the only source of future traffic growth I can think of comes from getting more humans online, and that is a mere another doubling. On the other hand, predictions such as 640k should be enough for everyone did not pan out. On the gripping hand, there has been an explosion of LLM stuff of late, with enormous models being widely distributed in just the past month: https://lwn.net/Articles/930939/ Could the AIs takeoff lead to a resumption of traffic growth? I still don´t think so... On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG < nanog@nanog.org > wrote: Disclaimer: Metaverse has not changed Metro traffic yet. Then … I am puzzled when people talk about 400GE and Tbps in the Mero context. For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, another “2” for capacity) in the majority of cases worldwide. How many BRASes serve more than 4/1.5=27k users in the busy hour? It means that 50GE is the best interface now for the majority of cases. 2*50GE=100Gbps is good room for growth. Of course, exceptions could be. I know BRAS that handles 86k subscribers (do not recommend anybody to push the limits – it was so painful). We have just 2 eyes and look at video content about 22h per week (on average). Our eyes do not permit us to see resolution better than particular for chosen distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec is enough for everybody. It would never change – it is our genetics. Fortunately for Carriers, the traffic has a limit. You have probably seen that every year traffic growth % is decreasing. The Internet is stabilizing and approaching the plateau. How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The result would be tailored for whom would pay for the research. IMHO: It is not mandatory that 100GE would become massive in the metro. (I know that 100GE is already massive in the DC CLOS) Additionally, who would pay for this traffic growth? It also limits traffic at some point. I hope it would happen after we would get our 22h/4k/12bit/120hz. Now, you could argue that Metaverse would jump and multiply traffic by an additional 2x or 3x. Then 400GE may be needed. Sorry, but it is speculation yet. It is not a trend like the current (declining) traffic growth. Ed/ From: NANOG [mailto: nanog-bounces+vasilenko.eduard = huawei@nanog.org ] On Behalf Of Phil Bedard Sent: Thursday, May 4, 2023 8:32 PM To: Etienne-Victor Depasquale < ed...@ieee.org >; NANOG < nanog@nanog.org > Subject: Re: Routed optical networks It’s not necessarily metro specific although the metro networks could lend themselves to overall optimizations. Th
Re: Routed optical networks
On 5/12/23 22:14, Mike Hammett wrote: "I remember 10y ago every presentation started from the claim that 100B of IoT would drive XXX traffic. It did not happen" Often the type of people making these kinds of predictions that a tire pressure sensor generates 20 gigabytes of traffic a day. I like growing old... your BS detector becomes so slick, you know to ignore certain links, conferences, speakers, topics, meetings, slideware, e-mails, colleagues and announcements without fear of actually missing out on trends, because you know that in the end it will lead to nowhere real :-). Mark.
Re: Routed optical networks
> On May 13, 2023, at 4:03 AM, Mark Tinka wrote: > > > > On 5/12/23 22:14, Mike Hammett wrote: > >> "I remember 10y ago every presentation started from the claim that 100B of >> IoT would drive XXX traffic. It did not happen" >> >> Often the type of people making these kinds of predictions that a tire >> pressure sensor generates 20 gigabytes of traffic a day. > > I like growing old... your BS detector becomes so slick, you know to ignore > certain links, conferences, speakers, topics, meetings, slideware, e-mails, > colleagues and announcements without fear of actually missing out on trends, > because you know that in the end it will lead to nowhere real :-). As a security guy. The end of year “prediction for next year” papers are wearing me out. As an author of several of the big ones, I’m over it too