On Fri, Jan 9, 2026 at 4:15 PM Haoyu Song <[email protected]> wrote: > > Hi Dave, > > Thank you for the comments. We are on the same page that a compact header > with just enough address bits is critical in AI DCN (I would argue this also > applies to the scale-out networks). > > I want to further discuss two points:
Hi Haoyu, thanks for the discussion! > > 1. The variable size address isn't that "scary" actually. We have verified > the scheme with P4 and it's doable. Once it's realized in switch ASIC, > there's no performance implications at all. The size of addresses in lookups will have performance implications and cost effects as well. For instance, with 16 bit addresses a switch could do route lookup with a simple array lookup in SRAM, for 32 bit addresses we need a CAM or TCAM, for 128 bit addresses we need a CAM or TCAM 4x the size of the one for IPv4. > On the other hand, supporting different lengths have many advantages: it can > scale with the cluster size without any waste, it supports communication > between clusters with different sizes, it doesn't need to respin the chips > in case the network scale changes, and the same standard would be applied to > any scenarios as laid out in our paper "Adaptive Addresses for Next > Generation IP Protocol in Hierarchical Networks"(ICNP2020). Of course, > there's a tradeoff on how fine the address length step should be supported > (e.g., 1 bit, 2 bits, 4 bits, or 8 bits). This is subject to further study. Waste is relative and we get diminishing returns in both directions. For instance, if we halve an IPv6 address then we save sixteen bytes per packet. That's significant. But if we halve the sixteen bit addresses of SUNH we'd save a whole two bytes per packet and that's nothing to write home about. On the other hand, suppose we double the sixteen bit addresses of SUNH then we have addresses the same size as IPv4 addresses. Grant it, IPv4 header has some other stuff in the header, but at some point it starts to be a question of why not just use IPv4? As for the granularity, my strong preference is to first keep addresses in units of eight bits. It's unpleasant for a lot of processors to deal with anything smaller and is a long held convention in IP addresses, port numbers, and Ethernet addresses. I'd also prefer maintaining four byte alignment of the transport layer like IPv4 and IPv6, but I suppose alignment is mostly historical at this time so maybe it's not super critical. With all this in mind, if there were to be another address size in addition to 16 bits, I might opt for 24 bits. It breaks four byte alignment, but has the nice property that it maps 10/8 IPv4 addresses. SUNH with 24 bit addresses is 10 bytes, compared to 20 bytes IPv4 header. I suppose that might be worth it. > > 2. If we assume the scale up network would take ethernet as the L2 > technology, it can be envisioned that the scale up and scale out network > would eventually converge into a single network. Then we would consider that > the L3 should also have a common standard (strictly speaking, if we only have > a separate scale up network, we don't need L3 at all, because an L2 fabirc is > enough). Strictly speaking, yes, But people also want network layer functionality like TOS and Hop Limits so L3 enters the picture and we see people go down the path to reinventing L3 like AHF does. > Thus, the variable size address can support a hierarchical network naturally > mapping to the DCN topology and more important, it allows the seamlessly > connecting with the Internet which runs IPv4/IPv6 so the inter-DC > communication can be supported without any modification to the public > network. I think this is a reason we need an IP-like L3 header which can > translate into IPv4/v6. Note the SUNH proposal support this already, the only > issue is the 16-bit address is an overkill to the current cluster size, and > the fixed length is not flexible. Like I said, adding different address sizes could just be a matter of getting different EtherTypes for SUNH. But, I would only want to add support for more address sizes sparingly. Tom > > > Best regards, > Haoyu > > -----Original Message----- > From: dave seddon <[email protected]> > Sent: Friday, January 9, 2026 2:03 PM > To: [email protected] > Subject: [Int-area] Regarding the draft: Scale-Up Network Header (SUNH) > > G'day Tom and Haoyu, > > I'm trying to join the discussion about "draft: Scale-Up Network Header > (SUNH)", but I just joined the mail list, so I don't know if posting to the > subject line will do it. ( Apologies if this breaks threading ) > > Drafts: > https://datatracker.ietf.org/doc/draft-herbert-sunh/ > https://datatracker.ietf.org/doc/html/draft-song-ship-edge-05 > > It seems like the discussion centers on the address length. > > The SUNH "1.1. Problem statement" is very clear " > 8% overhead in a 256 byte packet, and the forty bytes of IPv6 header would be > about 16% overhead " > > Absolutely minimizing overhead makes sense currently, but for how long do we > expect this to be true? Tom, since you've been talking to people who run the > largest AI clusters in the world, you expect this to hold true for the > foreseeable future. > > > Tom - I wonder if draft-herbert-sunh would benefit from a small summary, > maybe with a table, that compares the proposed addressing to other protocols > that are common within data centers? > > For example, comparing protocols by their header, address lengths, and > "overhead" > - PCIe ( IEEE have paywalls, so it's hard to find a good source. > Maybe this: > https://www.pearsonhighered.com/assets/samplechapter/0/3/2/1/0321156307.pdf > ) > - Infiniband ( addressing scheme found here on page 625 > https://hjemmesider.diku.dk/~vinter/CC/Infinibandchap42.pdf ) > - Ethernet > - Ethernet with 802.1q ( and qnq ) > - IPv4 > - IPv6 > - SUNH > ... > > Now that the context is established, explain why 16 bits were chosen for the > source/destination address. I guess, but it's not in the document; You were > considering the number of hosts in the domain. > > Nit pick (sorry). "care must be taken to ensure the minimum packet size is > maintained". Might help to explain why. > > Re section "TCP and UDP in SUNH". I remember recently Stuart from Apple > saying something pretty interesting about UDP: "If IP had port numbers, you > wouldn't really need a UDP header at all." > > Multicast? It might be worth mentioning multicast and explaining why it > isn't discussed. e.g. No requirement for this, or it might be considered in > the future if a need arises. > > > > Haoyu - I really like your draft-song-ship-edge-05 Hierarchical addressing > stuff: > a) > This reminds me of good old fiber channel addressing, and I suppose the more > modern Infiniband/RDMA. > b) > The words "variable length" are scary because variability clearly isn't ideal > for hardware. I guess when you say "variable length" you don't actually mean > the addresses would vary dynamically, but that there could be a range of set > fixed length addressing that could be selected for different deployment > scenarios? > c) > One core concept of draft-song-ship-edge-05, is that traffic destined for IoT > devices needs a long, unique address, while the traffic _sourced_ from these > devices towards the data center can have a much smaller destination address. > I recall Geoff Huston discussing IPv6 at a recent NANGO, where he commented > that because of the pervasive use of anycast by a relatively small number of > CDNs, that the Internet might only need a /24 worth of addresses for 99% of > all traffic. > Other network protocols with asymmetric addresses include: > - PCIe (Requester vs Completer addressing) > - In InfiniBand / RDMA, requests carry full destination addressing (QPN + > LID/GID + path), while responses omit it and are routed implicitly using the > established queue-pair and path state, making the addressing directionally > asymmetric. > - QUIC has explicit directional asymmetry in connection IDs > > > -- > Regards, > Dave Seddon > > _______________________________________________ > Int-area mailing list -- [email protected] To unsubscribe send an email to > [email protected] > > _______________________________________________ > Int-area mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ Int-area mailing list -- [email protected] To unsubscribe send an email to [email protected]
