Re: [j-nsp] MX304 Port Layout
On Fri, 9 Jun 2023 at 20:37, Andrey Kostin wrote: > Sounds more like a datacenter setup, and for DC operator it could be > attractive to do at scale. For a traditional ISP with relatively small > PoPs spread across the country it may be not the case. Sure, not suggesting everyone is in the target market, but suggesting the target market includes people who are not developers with no interest in being one. For a typical access network with multiple pops, it may be the wrong optimisation point. -- ++ytti ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Hi Saku, Saku Ytti писал(а) 2023-06-09 12:09: On Fri, 9 Jun 2023 at 18:46, Andrey Kostin wrote: I'm not in this market, have no qualification and resources for development. The demand in such devices should be really massive to justify a process like this. Are you not? You use a lot of open source software, because someone else did the hard work, and you have something practical. The same would be the thesis here, You order the PCI NPU from newegg, and you have an ecosystem of practical software to pull from various sources. Maybe you'll contribute something back, maybe not. Well, technically maybe I could do it. But putting it in production is another story. I have to not only make it run but also make sure that there are people who can support it 24x7. I think you said it before and I agree that the cost of capital investment in routers is just a small fraction in expenses for service providers. Cable infrastructure, facilities, payroll, etc. make a bigger part, but risk of a router failure extends to business risks like reputation and financial loss and may have a catastrophic impact. We all know how long and difficult can be troubleshooting and fixing a complex issue with vendor's TAC but I consider the price we pay hardware vendors for their TAC support partially as a liability insurance. Very typical network is a border router or two, which needs features and performance, then switches to connect to compute. People who have no resources or competence to write software could still be users in this market. Sounds more like a datacenter setup, and for DC operator it could be attractive to do at scale. For a traditional ISP with relatively small PoPs spread across the country it may be not the case. Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Thank you very much, Jeff, for sharing your experience. Will watch closely Release Notes for upcoming Junos releases. And kudos to Juniper for finding and fixing it, 1,5 week is very fast reaction!. Kind regards, Andrey Litterick, Jeff (BIT) писал(а) 2023-06-09 12:41: This is why we got the MX304. It was a test to replace our MX10008 Chassis, which we bought a few of because we had to get at a reasonable price into 100G at high density at multiple sites a few years back now. Though we really only need 4 line cards, with 2 being for redundancy. The MX1004 was not available at the time back then (Wish it had been. The MX10008 is a heavy beast indeed and we had to use fork lifts to move them around into the data centers).But after handling the MX304 we will most likely for 400G go to the MX10004 line for the future and just use the MX304 at very small edge sites if needed. Mainly due to full FPC redundancy requirements at many of our locations. And yes we had multiple full FPC failures in the past on the MX10008 line. We went through at first an RMA cycle with multiple line cards which in the end was due to just 1 line cards causing full FPC failure on a different line card in the chassis around every 3 months or so. Only having everything redundant across both FPCs allowed us not to have serious downtime. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
This is why we got the MX304. It was a test to replace our MX10008 Chassis, which we bought a few of because we had to get at a reasonable price into 100G at high density at multiple sites a few years back now. Though we really only need 4 line cards, with 2 being for redundancy. The MX1004 was not available at the time back then (Wish it had been. The MX10008 is a heavy beast indeed and we had to use fork lifts to move them around into the data centers).But after handling the MX304 we will most likely for 400G go to the MX10004 line for the future and just use the MX304 at very small edge sites if needed. Mainly due to full FPC redundancy requirements at many of our locations. And yes we had multiple full FPC failures in the past on the MX10008 line. We went through at first an RMA cycle with multiple line cards which in the end was due to just 1 line cards causing full FPC failure on a different line card in the chassis around every 3 months or so. Only having everything redundant across both FPCs allowed us not to have serious downtime. -Original Message- From: juniper-nsp On Behalf Of Andrey Kostin via juniper-nsp Sent: Friday, June 9, 2023 11:09 AM To: Mark Tinka Cc: Saku Ytti ; juniper-nsp Subject: Re: [EXT] [j-nsp] MX304 Port Layout Mark Tinka писал(а) 2023-06-09 10:26: > On 6/9/23 16:12, Saku Ytti wrote: > >> I expect many people in this list have no need for more performance >> than single Trio YT in any pop at all, yet they need ports. And they >> are not adequately addressed by vendors. But they do need the deep >> features of NPU. > > This. > > There is sufficient performance in Trio today (even a single Trio chip > on the board) that people are willing to take an oversubscribed box or > line card because in real life, they will run out of ports long before > they run out of aggregate forwarding capacity. > > The MX204, even though it's a pizza box, is a good example of how it > could do with 8x 100Gbps ports, even though Trio on it will only > forward 400Gbps. Most use-cases will require another MX204 chassis, > just for ports, before the existing one has hit anywhere close to > capacity. Agree, there is a gap between 204 and 304, but don't forget that they belong to different generations. 304 is shiny new with a next level performance that's replacing MX10k3. The previous generation was announced to retire, but life of MX204 was extended because Juniper realized that they don't have anything atm to replace it and probably will lose revenue. Maybe this gap was caused by covid that slowed down the new platform. And possibly we may see a single NPU model based on the new gen chip, because chips for 204 are finite. At least it would be logical to make it, considering success of MX204. > > Really, folk are just chasing the Trio capability, otherwise they'd > have long solved their port-count problems by choosing any > Broadcom-based box on the market. Juniper know this, and they are > using it against their customers, knowingly or otherwise. Cisco was > good at this back in the day, over-subscribing line cards on their > switches and routers. Juniper have always been a little more purist, > but the market can't handle it because the rate of traffic growth is > being out-paced by what a single Trio chip can do for a couple of > ports, in the edge. I think that it's not rational to make another chipset with lower bandwidth, easier to limit an existing more powerful chip. Then it leads to MX5/MX10/MX40/MX80 hardware and licensing model. It could be a single Trio6 with up to 1.6T in access ports and 1.6T in uplink ports with low features. Maybe it will come, who knows, let's watch ;) Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On Fri, 9 Jun 2023 at 19:15, Andrey Kostin wrote: > Can anything else be inserted in this socket? If not, then what's the > point? For server CPUs there are many models with different clocking and > number of cores, so socket provides a flexibility. If there is only one > chip that fits the socket, then the socket is a redundant part. Not that I know. I think the point may be decouplement. BRCM doesn't want to do business with just everyone. This allows someone to build the switch, without providing the chips. Then customers can buy a switch from this vendor and chip directly from BRCM. I could imagine some big players like FB and AMZN designing their own switch, having some random shop actually build it. But Broadcom saying 'no, we don't do business with you'. This way they could actually get the switch from anywhere, while having a direct chip relationship with BRCM. -- ++ytti ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Not sure, but anything shipped before May most likely would be affected, if not into May a bit. Since we were one of the first if not the first customer to get the fixed applied to the equipment we got at the end of March. We never knew the real root cause outside that when it happened the primary RE would lock up-sold and not respond to any command or input or allow access to any management port and there were no crash dumps or logs. The Backup RE would NOT Take over and get stuck in a loop trying to take over the mastership, but the backup RE would still respond to management, but even a reboot of it would not allow it to take mastership. The only solution was a full power plug removal or RE removal from the chassis for a power reset. But they were able to find this in the lab at Juniper right after we reported it and they worked on a fix and got it to use about 1.5 weeks later.We got lucky in that one of the 3 boxes would never last more than 6 hours after a reboot before the lockup of the master RE (No matter if it was RE0 or RE1 as master) The other 2 boxes could go a week or more before locking up. So we were a good test to see if the fixed work since in the lab it would take up to 8 days before locking up. FYI: The one that would lock up inside 6 hours was our spare and had no traffic at all or even a optic plugged into any port and not even test traffic which the other 2 had going. We did not go into production until 2 weeks after the fix was applied to make sure. This problem would only surface also if you have more then one RE plugged into the system. Even if failover was not configured. It was just the presence of the 2nd RE that would trigger it. I understand that the engineering team is now fully regressive testing all releases with multiple REs now. I guess that was not true before we found the bug. -Original Message- From: juniper-nsp On Behalf Of Mark Tinka via juniper-nsp Sent: Thursday, June 8, 2023 10:53 PM To: juniper-nsp@puck.nether.net Subject: Re: [EXT] [j-nsp] MX304 Port Layout On 6/9/23 00:03, Litterick, Jeff (BIT) via juniper-nsp wrote: > The big issue we ran into is if you have redundant REs then there is a super > bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and > the other 2 would take a very long time) to 8 days will lock the entire > chassis up solid where we had to pull the REs physical out to reboot them. > It is fixed now, but they had to manually poke new firmware into the ASICs > on each RE when they were in a half-powered state, Was a very complex > procedure with tech support and the MX304 engineering team. It took about 3 > hours to do all 3 MX304s one RE at a time. We have not seen an update with > this built-in yet. (We just did this back at the end of April) Oh dear, that's pretty nasty. So did they say new units shipping today would come with the RE's already fixed? We've been suffering a somewhat similar issue on the PTX1000, where a bug was introduced via regression in Junos 21.4, 22.1 and 22.2 that causes CPU queues to get filled up by unknown MAC address frames, and are not cleared. It takes 64 days for this packet accumulation to grow to a point where the queues get exhausted, causing a host loopback wedge. You would see an error like this in the logs: alarmd[27630]: Alarm set: FPC id=150995048, color=RED, class=CHASSIS, reason=FPC 0 Major Errorsfpc0 Performing action cmalarm for error /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1] (0x20002) in module: Host Loopback with scope: pfe category: functional level: major fpc0 Cmerror Op Set: Host Loopback: HOST LOOPBACK WEDGE DETECTED IN PATH ID 1 (URI: /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1]) Apr 1 03:52:28 PTX1000 fpc0 CMError: /fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[3] (0x20004), in module: Host Loopback with scope: pfe category: functional level: major This causes the router to drop all control plane traffic, which, basically, makes it unusable. One has to reboot the box to get it back up and running, until it happens again 64 days later. The issue is resolved in Junos 21.4R3-S4, 22.4R2, 23.2R1 and 23.3R1. However, these releases are not shipping yet, so Juniper gave us a workaround SLAX script that automatically runs and clears the CPU queues before the 64 days are up. We are currently running Junos 22.1R3.9 on this platform, and will move to 22.4R2 in a few weeks to permanently fix this. Junos 20.2, 20.3 and 20.4 are not affected, nor is anything after 23.2R1. I understand it may also affect the QFX and MX, but I don't have details on that. Mark. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list
Re: [j-nsp] MX304 Port Layout
Saku Ytti писал(а) 2023-06-09 10:35: LGA8371 socketed BRCM TH4. Ostensibly this allows a lot more switches to appear in the market, as the switch maker doesn't need to be friendly with BRCM. They make the switch, the customer buys the chip and sockets it. Wouldn't surprise me if FB, AMZN and the likes would have pressed for something like this, so they could use cheaper sources to make the rest of the switch, sources which BRCM didn't want to play ball with. Can anything else be inserted in this socket? If not, then what's the point? For server CPUs there are many models with different clocking and number of cores, so socket provides a flexibility. If there is only one chip that fits the socket, then the socket is a redundant part. Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On Fri, 9 Jun 2023 at 18:46, Andrey Kostin wrote: > I'm not in this market, have no qualification and resources for > development. The demand in such devices should be really massive to > justify a process like this. Are you not? You use a lot of open source software, because someone else did the hard work, and you have something practical. The same would be the thesis here, You order the PCI NPU from newegg, and you have an ecosystem of practical software to pull from various sources. Maybe you'll contribute something back, maybe not. Very typical network is a border router or two, which needs features and performance, then switches to connect to compute. People who have no resources or competence to write software could still be users in this market. -- ++ytti ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Mark Tinka писал(а) 2023-06-09 10:26: On 6/9/23 16:12, Saku Ytti wrote: I expect many people in this list have no need for more performance than single Trio YT in any pop at all, yet they need ports. And they are not adequately addressed by vendors. But they do need the deep features of NPU. This. There is sufficient performance in Trio today (even a single Trio chip on the board) that people are willing to take an oversubscribed box or line card because in real life, they will run out of ports long before they run out of aggregate forwarding capacity. The MX204, even though it's a pizza box, is a good example of how it could do with 8x 100Gbps ports, even though Trio on it will only forward 400Gbps. Most use-cases will require another MX204 chassis, just for ports, before the existing one has hit anywhere close to capacity. Agree, there is a gap between 204 and 304, but don't forget that they belong to different generations. 304 is shiny new with a next level performance that's replacing MX10k3. The previous generation was announced to retire, but life of MX204 was extended because Juniper realized that they don't have anything atm to replace it and probably will lose revenue. Maybe this gap was caused by covid that slowed down the new platform. And possibly we may see a single NPU model based on the new gen chip, because chips for 204 are finite. At least it would be logical to make it, considering success of MX204. Really, folk are just chasing the Trio capability, otherwise they'd have long solved their port-count problems by choosing any Broadcom-based box on the market. Juniper know this, and they are using it against their customers, knowingly or otherwise. Cisco was good at this back in the day, over-subscribing line cards on their switches and routers. Juniper have always been a little more purist, but the market can't handle it because the rate of traffic growth is being out-paced by what a single Trio chip can do for a couple of ports, in the edge. I think that it's not rational to make another chipset with lower bandwidth, easier to limit an existing more powerful chip. Then it leads to MX5/MX10/MX40/MX80 hardware and licensing model. It could be a single Trio6 with up to 1.6T in access ports and 1.6T in uplink ports with low features. Maybe it will come, who knows, let's watch ;) Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Saku Ytti писал(а) 2023-06-09 10:12: On Fri, 9 Jun 2023 at 16:58, Andrey Kostin via juniper-nsp wrote: Not sure why it's eye-watering. The price of fully populated MX304 is basically the same as it's predecessor MX10003 but it provides 3.2T BW capacity vs 2.4T. If you compare with MX204, then MX304 is about 20% expensive for the same total BW, but MX204 doesn't have redundant RE and if you use it in redundant chassis configuration you will have to spend some BW on "fabric" links, effectively leveling the price if calculated for the same BW. I'm just comparing numbers, not considering any real That's not it, RE doesn't attach to fabric serdes. Sorry, I mixed two different points. I wanted to say that redundant RE adds more cost to MX304, unrelated to forwarding BW. But if you want to have MX204s in redundant configuration, some ports have to be sacrificed for connectivity between them. We have two MX204s running in pair with 2x100G taken for links between them and remaining BW is 6x100G for actual forwarding in/out. In this case it's kind of at the same level for price/100G value. I expect many people in this list have no need for more performance than single Trio YT in any pop at all, yet they need ports. And they are not adequately addressed by vendors. But they do need the deep features of NPU. I agree, and that's why I asked about HQoS experience, just to add more inexpensive low-speed switch ports via trunk but still be able to treat them more like separate ports from a router perspective. I keep hoping that someone is so disruptive that they take the nvidia/gpu approach to npu. That is, you can buy Trio PCI from newegg for 2 grand, and can program it as you wish. I think this market remains unidentified and even adjusting to cannibalization would increase market size. I can't understand why JNPR is not trying this, they've lost for 20 years to inflation in valuation, what do they have to lose? I'm not in this market, have no qualification and resources for development. The demand in such devices should be really massive to justify a process like this. Kind regards, Andrey ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On 6/9/23 16:35, Saku Ytti wrote: I'm not convinced at all that leaba is being sold. I think it's sold conditionally when customers would otherwise be lost. Probably - it's a "grain of salt" situation when you hear the news. I don't think Meta and Microsoft have not bought zero of the C8000... but not to the degree that they would ignore more primary options, I think. But NPU from newegg and community writes code that doesn't exist, and I think it should and there would be volume in it, but no large volume to any single customer. Not enough foresight from traditional OEM's to see the potential here. Mark. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On Fri, 9 Jun 2023 at 17:26, Mark Tinka wrote: > Well, the story is that Cisco are doing this with Meta and Microsoft on > their C8000 platform, and apparently, doing billions of US$ in business > on the back of that. I'm not convinced at all that leaba is being sold. I think it's sold conditionally when customers would otherwise be lost. I am reminder of this: https://www.servethehome.com/this-is-a-broadcom-tomahawk-4-64-port-400gbe-switch-chip-lga8371-intel-amd-ampere/ LGA8371 socketed BRCM TH4. Ostensibly this allows a lot more switches to appear in the market, as the switch maker doesn't need to be friendly with BRCM. They make the switch, the customer buys the chip and sockets it. Wouldn't surprise me if FB, AMZN and the likes would have pressed for something like this, so they could use cheaper sources to make the rest of the switch, sources which BRCM didn't want to play ball with. But NPU from newegg and community writes code that doesn't exist, and I think it should and there would be volume in it, but no large volume to any single customer. -- ++ytti ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On 6/9/23 16:12, Saku Ytti wrote: I expect many people in this list have no need for more performance than single Trio YT in any pop at all, yet they need ports. And they are not adequately addressed by vendors. But they do need the deep features of NPU. This. There is sufficient performance in Trio today (even a single Trio chip on the board) that people are willing to take an oversubscribed box or line card because in real life, they will run out of ports long before they run out of aggregate forwarding capacity. The MX204, even though it's a pizza box, is a good example of how it could do with 8x 100Gbps ports, even though Trio on it will only forward 400Gbps. Most use-cases will require another MX204 chassis, just for ports, before the existing one has hit anywhere close to capacity. Really, folk are just chasing the Trio capability, otherwise they'd have long solved their port-count problems by choosing any Broadcom-based box on the market. Juniper know this, and they are using it against their customers, knowingly or otherwise. Cisco was good at this back in the day, over-subscribing line cards on their switches and routers. Juniper have always been a little more purist, but the market can't handle it because the rate of traffic growth is being out-paced by what a single Trio chip can do for a couple of ports, in the edge. I keep hoping that someone is so disruptive that they take the nvidia/gpu approach to npu. That is, you can buy Trio PCI from newegg for 2 grand, and can program it as you wish. I think this market remains unidentified and even adjusting to cannibalization would increase market size. I can't understand why JNPR is not trying this, they've lost for 20 years to inflation in valuation, what do they have to lose? Well, the story is that Cisco are doing this with Meta and Microsoft on their C8000 platform, and apparently, doing billions of US$ in business on the back of that. Mark. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On 6/9/23 15:57, Andrey Kostin wrote: Hi Mark, Not sure why it's eye-watering. The price of fully populated MX304 is basically the same as it's predecessor MX10003 but it provides 3.2T BW capacity vs 2.4T. That's true, but the premium being paid for 400Gbps capability that some houses may not yet need is probably what is pushing that price up in comparison to the MX10003, which does not support 400Gbps. But to be fair, it will come down to the discounts you can negotiate with Juniper. I'm perhaps more concerned because we got good pricing on the MX10003, even when we did a like-for-somewhat-like comparison with the MX304. As much as we have struggled with Cisco in the past, this is forcing me to see what is available in their ASR99xx boxes. But off-the-bat, form factor and port density is poor on the Cisco side, compared to the MX304 and MX10003. If you compare with MX204, then MX304 is about 20% expensive for the same total BW, but MX204 doesn't have redundant RE and if you use it in redundant chassis configuration you will have to spend some BW on "fabric" links, effectively leveling the price if calculated for the same BW. I'm just comparing numbers, not considering any real topology, which is another can of worms. Most probably it's not worth to try to scale MX204s to more than a pair of devices, at least I wouldn't do it and consider it ;) The use-case for MX204 and MX304 is very very different. As you say, MX304 is a better alternative for the MX10003 (which I am getting conflicting information about re: sale availability from Juniper). We use the MX204 extensively, but only for peering and routing for value-added services too small to plug into a larger MX. I'd rather call eye-watering prices for MPC7 and MPC10 to upgrade existing MX480 routers if you still to use their low-speed ports. Two MPC10s with SCB3s upgrade cost more than MX304, but gives 30% less BW capacity. For MPC7 this ratio is even worse. Agreed - the MPC7 and MPC10's only make sense for large capacity aggregation or backbone links, not as an access port for 100Gbps customers. The MX10003 and MX304 are better boxes for 100Gbps access for customers. Conversely, trying to use the MX304 or MX10003 as a core box is too costly, since you are paying the premium for edge features in Trio, when all you need is basic Ethernet, IS-IS and MPLS. So the MPC7/MPC10 vs. MX304/10003 use-cases are clearly defined, if money is an object. This brings a question, does anybody have an experience with HQoS on MX304? I mean just per-subinterface queueing on an interface to a switch, not BNG subscribers CoS which is probably another big topic. At least I'm not dare yet to try MX304 in BNG role, maybe later ;) In this world where the kind of traffic you will be pushing through an MX304 most likely being majority off-net content, do you really need H-QoS :-)? Mark. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
On Fri, 9 Jun 2023 at 16:58, Andrey Kostin via juniper-nsp wrote: > Not sure why it's eye-watering. The price of fully populated MX304 is > basically the same as it's predecessor MX10003 but it provides 3.2T BW > capacity vs 2.4T. If you compare with MX204, then MX304 is about 20% > expensive for the same total BW, but MX204 doesn't have redundant RE and > if you use it in redundant chassis configuration you will have to spend > some BW on "fabric" links, effectively leveling the price if calculated > for the same BW. I'm just comparing numbers, not considering any real That's not it, RE doesn't attach to fabric serdes. You are right that the MX304 is the successor of MX10003 not MX201. MX80, M104 and MX201 are unique in that they are true pizzabox Trios. They have exactly 1 trio, and both WAN and FAB side connect to WAN ports (not sure if MX201 just leaves them unconnected) Therefore say 40G Trio in linecard mode is 80G Trio in pizza mode (albeit PPS stays the same) as you're not wasting capacity to non-revenue fabric ports. This single Trio design makes the box very cost effective, as not only do you just have one Trio and double the capacity per Trio, but you also don't have any fabric chip and fabric serdes. MX304 however has Trio in the linecard, so it really is very much a normal chassis box. And having multiple Trios it needs fabric. I do think Juniper and the rest of the vendors keep struggling to identify 'few to many' markets, and are only good at identifying 'many to few' markets. MX304 and ever denser 512x112G serdes chips represent this. I expect many people in this list have no need for more performance than single Trio YT in any pop at all, yet they need ports. And they are not adequately addressed by vendors. But they do need the deep features of NPU. I keep hoping that someone is so disruptive that they take the nvidia/gpu approach to npu. That is, you can buy Trio PCI from newegg for 2 grand, and can program it as you wish. I think this market remains unidentified and even adjusting to cannibalization would increase market size. I can't understand why JNPR is not trying this, they've lost for 20 years to inflation in valuation, what do they have to lose? -- ++ytti ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Hi Mark, Not sure why it's eye-watering. The price of fully populated MX304 is basically the same as it's predecessor MX10003 but it provides 3.2T BW capacity vs 2.4T. If you compare with MX204, then MX304 is about 20% expensive for the same total BW, but MX204 doesn't have redundant RE and if you use it in redundant chassis configuration you will have to spend some BW on "fabric" links, effectively leveling the price if calculated for the same BW. I'm just comparing numbers, not considering any real topology, which is another can of worms. Most probably it's not worth to try to scale MX204s to more than a pair of devices, at least I wouldn't do it and consider it ;) I'd rather call eye-watering prices for MPC7 and MPC10 to upgrade existing MX480 routers if you still to use their low-speed ports. Two MPC10s with SCB3s upgrade cost more than MX304, but gives 30% less BW capacity. For MPC7 this ratio is even worse. This brings a question, does anybody have an experience with HQoS on MX304? I mean just per-subinterface queueing on an interface to a switch, not BNG subscribers CoS which is probably another big topic. At least I'm not dare yet to try MX304 in BNG role, maybe later ;) Kind regards, Andrey Mark Tinka via juniper-nsp писал(а) 2023-06-08 12:04: Trio capacity aside, based on our experience with the MPC7E, MX204 and MX10003, we expect it to be fairly straight forward. What is holding us back is the cost. The license for each 16-port line card is eye-watering. While I don't see anything comparable in ASR99xx Cisco-land (in terms of form factor and 100Gbps port density), those prices are certainly going to force Juniper customers to look at other options. They would do well to get that under control. ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] MX304 Port Layout
Hi Jeff, Thank you very mush for sharing this information. Do you know in what publicly available release it's going to be fixed? Knowing PR number would be the best but I guess it may be internal-only. Kind regards, Andrey Litterick, Jeff (BIT) via juniper-nsp писал(а) 2023-06-08 18:03: No, that is not quite right. We have 2 chassis of MX304 in Production today and 1 spare all with Redundant REs You do not need all the ports filled in a port group. I know since we mixed in some 40G and 40G is ONLY supported on the bottom row of ports so we have a mix and had to break stuff out leaving empty ports because of that limitation, and it is running just fine.But you do have to be careful which type of optics get plugged into which ports. IE Port 0/2 vs Port 1/3 in a grouping if you are not using 100G optics. The big issue we ran into is if you have redundant REs then there is a super bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and the other 2 would take a very long time) to 8 days will lock the entire chassis up solid where we had to pull the REs physical out to reboot them. It is fixed now, but they had to manually poke new firmware into the ASICs on each RE when they were in a half-powered state, Was a very complex procedure with tech support and the MX304 engineering team. It took about 3 hours to do all 3 MX304s one RE at a time. We have not seen an update with this built-in yet. (We just did this back at the end of April) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
