Re: [j-nsp] MX304 Port Layout

[Mark Tinka via juniper-nsp]

On 6/9/23 00:03, Litterick, Jeff (BIT) via juniper-nsp wrote:


The big issue we ran into is if you have redundant REs then there is a super 
bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and 
the other 2 would take a very long time) to 8 days will lock the entire chassis 
up solid where we had to pull the REs physical out to reboot them. It is 
fixed now, but they had to manually poke new firmware into the ASICs on each RE 
when they were in a half-powered state,  Was a very complex procedure with tech 
support and the MX304 engineering team.  It took about 3 hours to do all 3 
MX304s  one RE at a time.   We have not seen an update with this built-in yet.  
(We just did this back at the end of April)


Oh dear, that's pretty nasty. So did they say new units shipping today 
would come with the RE's already fixed?


We've been suffering a somewhat similar issue on the PTX1000, where a 
bug was introduced via regression in Junos 21.4, 22.1 and 22.2 that 
causes CPU queues to get filled up by unknown MAC address frames, and 
are not cleared. It takes 64 days for this packet accumulation to grow 
to a point where the queues get exhausted, causing a host loopback wedge.


You would see an error like this in the logs:

   alarmd[27630]: Alarm set: FPC id=150995048, 
color=RED, class=CHASSIS, reason=FPC 0 Major Errors
   fpc0 Performing action cmalarm for error 
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1] 
(0x20002) in module: Host Loopback with scope: pfe category: functional 
level: major
   fpc0 Cmerror Op Set: Host Loopback: HOST 
LOOPBACK WEDGE DETECTED IN PATH ID 1  (URI: 
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[1])
Apr 1 03:52:28  PTX1000 fpc0 CMError: 
/fpc/0/pfe/0/cm/0/Host_Loopback/0/HOST_LOOPBACK_MAKE_CMERROR_ID[3] 
(0x20004), in module: Host Loopback with scope: pfe category: functional 
level: major


This causes the router to drop all control plane traffic, which, 
basically, makes it unusable. One has to reboot the box to get it back 
up and running, until it happens again 64 days later.


The issue is resolved in Junos 21.4R3-S4, 22.4R2, 23.2R1 and 23.3R1.

However, these releases are not shipping yet, so Juniper gave us a 
workaround SLAX script that automatically runs and clears the CPU queues 
before the 64 days are up.


We are currently running Junos 22.1R3.9 on this platform, and will move 
to 22.4R2 in a few weeks to permanently fix this.


Junos 20.2, 20.3 and 20.4 are not affected, nor is anything after 23.2R1.

I understand it may also affect the QFX and MX, but I don't have details 
on that.


Mark.

___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Litterick, Jeff (BIT) via juniper-nsp]

No, that is not quite right.  We have 2 chassis of MX304 in Production today 
and 1 spare all with Redundant REs   You do not need all the ports filled in a 
port group.   I know since we mixed in some 40G and 40G is ONLY supported on 
the bottom row of ports so we have a mix and had to break stuff out leaving 
empty ports because of that limitation, and it is running just fine.But you 
do have to be careful which type of optics get plugged into which ports.  IE 
Port 0/2 vs Port 1/3 in a grouping if you are not using 100G optics.

The big issue we ran into is if you have redundant REs then there is a super 
bad bug that after 6 hours (1 of our 3 would lock up after reboot quickly and 
the other 2 would take a very long time) to 8 days will lock the entire chassis 
up solid where we had to pull the REs physical out to reboot them. It is 
fixed now, but they had to manually poke new firmware into the ASICs on each RE 
when they were in a half-powered state,  Was a very complex procedure with tech 
support and the MX304 engineering team.  It took about 3 hours to do all 3 
MX304s  one RE at a time.   We have not seen an update with this built-in yet.  
(We just did this back at the end of April)


-Original Message-
From: juniper-nsp  On Behalf Of Thomas 
Bellman via juniper-nsp
Sent: Thursday, June 8, 2023 2:09 PM
To: juniper-nsp 
Subject: Re: [EXT] [j-nsp] MX304 Port Layout

On 2023-06-08 17:18, Kevin Shymkiw via juniper-nsp wrote:

> Along with this - I would suggest looking at Port Checker ( 
> https://apps.juniper.net/home/port-checker/index.html ) to make sure 
> your port combinations are valid.

The port checker claims an interresting "feature": if you have anything in port 
3, then *all* the other ports in that port group must also be occupied.  So if 
you use all those four ports for e.g. 100GE, everything is fine, but if you 
then want to stop using either of ports 0, 1 or 2, the configuration becomes 
invalid...

(And similarly for ports 5, 8 and 14 in their respective groups.)

I hope that's a bug in the port checker, not actual behaviour by the MX304...


--
Thomas Bellman,  National Supercomputer Centre,  Linköping Univ., Sweden "We 
don't understand the software, and sometimes we don't understand  the hardware, 
but we can *see* the blinking lights!"

___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Thomas Bellman via juniper-nsp]

On 2023-06-08 17:18, Kevin Shymkiw via juniper-nsp wrote:

> Along with this - I would suggest looking at Port Checker (
> https://apps.juniper.net/home/port-checker/index.html ) to make sure
> your port combinations are valid.

The port checker claims an interresting "feature": if you have
anything in port 3, then *all* the other ports in that port group
must also be occupied.  So if you use all those four ports for
e.g. 100GE, everything is fine, but if you then want to stop using
either of ports 0, 1 or 2, the configuration becomes invalid...

(And similarly for ports 5, 8 and 14 in their respective groups.)

I hope that's a bug in the port checker, not actual behaviour by
the MX304...


-- 
Thomas Bellman,  National Supercomputer Centre,  Linköping Univ., Sweden
"We don't understand the software, and sometimes we don't understand
 the hardware, but we can *see* the blinking lights!"



signature.asc
Description: OpenPGP digital signature
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Mark Tinka via juniper-nsp]

On 6/8/23 18:39, Giuliano C. Medalha wrote:

but you have the flex model. With license for capacity and features. 
 Advanced and Premium.


Which isn't a new thing with vendors. The prices are just terrible, even 
with discounts.




fib is better now - 12M

sampling rate for ipfix is better too.

but you have other parameters for mpls and bng too


Yes, like I said, increased Trio capacity aside, it's straight forward. 
Just not sure the price is justified. I expect a premium for custom 
silicon over Broadcom, but that seems excessive.




But you need the correct drivers on Junos


Yes, that is assumed, of course, especially if you want to talk to a 
ROADM. There is varying support amongst vendors, but as with everything, 
they will converge in time.



Juniper now has good prices ( common optics ) for 400G ( JCO part 
numbers )


Mixing $vendor_name and "good optics prices" has always ended in tears.



Low 40 km or maximum 80 km direct with ZR high power ( end of the year )


Okay.

Our use-case is 100Gbps customer edge, so within the data centre.

We operate an optical network for anything longer than 80km.

Mark.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Giuliano C. Medalha via juniper-nsp]

> Hello good afternoon.
>
> Please have a look at the following documentation:
>
>
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.juniper.net%2Fblogs%2Freema-ray%2F2023%2F03%2F28%2Fmx304-deepdive=05%7C01%7Cgiuliano%40wztech.com.br%7Cc7b64b057b6b488bff0d08db683a0b28%7C584787b077bd4312bf8815412b8ae504%7C1%7C0%7C638218370748741098%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=fhF2D3TSWDQljPw0jepN5ZsB%2FdOSUq4zIx%2F9w1TkkhE%3D=0

Thanks, this is most useful!


> It will have everything you need to do with it, including the pictures.
>
> Our first boxes are arriving this next month in Brazil.
>
> By the specs of the new chipset (TRIO6) it's a very good box. A lot of 
> enhancements.

Trio capacity aside, based on our experience with the MPC7E, MX204 and
MX10003, we expect it to be fairly straight forward.

What is holding us back is the cost. The license for each 16-port line
card is eye-watering. While I don't see anything comparable in ASR99xx
Cisco-land (in terms of form factor and 100Gbps port density), those
prices are certainly going to force Juniper customers to look at other
options. They would do well to get that under control.


but you have the flex model. With license for capacity and features.  Advanced 
and Premium.

fib is better now - 12M

sampling rate for ipfix is better too.

but you have other parameters for mpls and bng too




> And it supports 400G already ( ZR and ZR+ need to check ) ( 16 x 100 or 4 x 
> 400 ) per LMIC.

The LMIC won't care whether it's ZR, ZR+, FR4 or DR4. It will be compatible 
with whatever pluggable is used, as long as it can do 400Gbps.



But you need the correct drivers on Junos

And the chassis must support temperature and power budgets

Juniper now has good prices ( common optics ) for 400G ( JCO part numbers )



Unless, of course, you mean whether Juniper provide an interface into
the optic for use-cases where you are plugging into a ROADM... that, I
don't know.

Are you intending to use this router for long-distance applications?



Low 40 km or maximum 80 km direct with ZR high power ( end of the year )

Thanks

Mark.

WZTECH is registered trademark of WZTECH NETWORKS.
Copyright (c) 2023 WZTECH NETWORKS. All Rights Reserved.

IMPORTANTE:
As informa??es deste e-mail e o conte?do dos eventuais documentos anexos s?o 
confidenciais e para conhecimento exclusivo do destinat?rio. Se o leitor desta 
mensagem n?o for o seu destinat?rio, fica desde j? notificado de que n?o poder? 
divulgar, distribuir ou, sob qualquer forma, dar conhecimento a terceiros das 
informa??es e do conte?do dos documentos anexos. Neste caso, favor comunicar 
imediatamente o remetente, respondendo este e-mail ou telefonando ao mesmo, e 
em seguida apague-o.

CONFIDENTIALITY NOTICE:
The information transmitted in this email message and any attachments are 
solely for the intended recipient and may contain confidential or privileged 
information. If you are not the intended recipient, any review, transmission, 
dissemination or other use of this information is prohibited. If you have 
received this communication in error, please notify the sender immediately and 
delete the material from any computer, including any copies.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Mark Tinka via juniper-nsp]

On 6/8/23 17:35, Giuliano C. Medalha wrote:


Hello good afternoon.

Please have a look at the following documentation:


https://community.juniper.net/blogs/reema-ray/2023/03/28/mx304-deepdive


Thanks, this is most useful!



It will have everything you need to do with it, including the pictures.

Our first boxes are arriving this next month in Brazil.

By the specs of the new chipset (TRIO6) it's a very good box. A lot of 
enhancements.


Trio capacity aside, based on our experience with the MPC7E, MX204 and 
MX10003, we expect it to be fairly straight forward.


What is holding us back is the cost. The license for each 16-port line 
card is eye-watering. While I don't see anything comparable in ASR99xx 
Cisco-land (in terms of form factor and 100Gbps port density), those 
prices are certainly going to force Juniper customers to look at other 
options. They would do well to get that under control.




And it supports 400G already ( ZR and ZR+ need to check ) ( 16 x 100 or 4 x 400 
) per LMIC.


The LMIC won't care whether it's ZR, ZR+, FR4 or DR4. It will be 
compatible with whatever pluggable is used, as long as it can do 400Gbps.


Unless, of course, you mean whether Juniper provide an interface into 
the optic for use-cases where you are plugging into a ROADM... that, I 
don't know.


Are you intending to use this router for long-distance applications?

Mark.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Giuliano C. Medalha via juniper-nsp]

Hello good afternoon.

Please have a look at the following documentation:


https://community.juniper.net/blogs/reema-ray/2023/03/28/mx304-deepdive


It will have everything you need to do with it, including the pictures.

Our first boxes are arriving this next month in Brazil.

By the specs of the new chipset (TRIO6) it's a very good box. A lot of 
enhancements.

And it supports 400G already ( ZR and ZR+ need to check ) ( 16 x 100 or 4 x 400 
) per LMIC.

Best regards

Giuliano



-Original Message-
From: juniper-nsp  On Behalf Of Mark Tinka 
via juniper-nsp
Sent: Thursday, June 8, 2023 12:25 PM
To: Kevin Shymkiw 
Cc: juniper-nsp 
Subject: Re: [j-nsp] MX304 Port Layout



On 6/8/23 17:18, Kevin Shymkiw wrote:
> Along with this - I would suggest looking at Port Checker (
> https://apps/
> .juniper.net%2Fhome%2Fport-checker%2Findex.html=05%7C01%7Cgiuliano%40wztech.com.br%7C28f16ddcdd4440a77d9d08db68348d0b%7C584787b077bd4312bf8815412b8ae504%7C1%7C0%7C638218347183708615%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=KbDjdprQEMTa1yXQSPrF%2BytprEB0JAtWWRJtkX2QYAs%3D=0
>  ) to make sure your port combinations are valid.

We've had ample experience with Juniper's MPC7E, MX204, PTX1000 and
PTX10001 to know how they structure this from a philosophical standpoint. So 
not a major drama there.

It's just interesting to me that the data sheet does not mention needing to 
sacrifice an RE to get to the chassis' advertised full port compliment. Unless 
the data sheet was updated and I missed it.

Mark.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

WZTECH is registered trademark of WZTECH NETWORKS.
Copyright © 2023 WZTECH NETWORKS. All Rights Reserved.

IMPORTANTE:
As informações deste e-mail e o conteúdo dos eventuais documentos anexos são 
confidenciais e para conhecimento exclusivo do destinatário. Se o leitor desta 
mensagem não for o seu destinatário, fica desde já notificado de que não poderá 
divulgar, distribuir ou, sob qualquer forma, dar conhecimento a terceiros das 
informações e do conteúdo dos documentos anexos. Neste caso, favor comunicar 
imediatamente o remetente, respondendo este e-mail ou telefonando ao mesmo, e 
em seguida apague-o.

CONFIDENTIALITY NOTICE:
The information transmitted in this email message and any attachments are 
solely for the intended recipient and may contain confidential or privileged 
information. If you are not the intended recipient, any review, transmission, 
dissemination or other use of this information is prohibited. If you have 
received this communication in error, please notify the sender immediately and 
delete the material from any computer, including any copies.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Mark Tinka via juniper-nsp]

On 6/8/23 17:18, Kevin Shymkiw wrote:

Along with this - I would suggest looking at Port Checker (
https://apps.juniper.net/home/port-checker/index.html ) to make sure
your port combinations are valid.


We've had ample experience with Juniper's MPC7E, MX204, PTX1000 and 
PTX10001 to know how they structure this from a philosophical 
standpoint. So not a major drama there.


It's just interesting to me that the data sheet does not mention needing 
to sacrifice an RE to get to the chassis' advertised full port 
compliment. Unless the data sheet was updated and I missed it.


Mark.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] MX304 Port Layout

[Kevin Shymkiw via juniper-nsp]

Along with this - I would suggest looking at Port Checker (
https://apps.juniper.net/home/port-checker/index.html ) to make sure
your port combinations are valid.

Kevin

On Thu, Jun 8, 2023 at 9:16 AM Mark Tinka via juniper-nsp
 wrote:
>
> So, we decided to give the MX304 another sniff, and needed to find out
> why Juniper charge a license for 16x 100Gbps ports per line card, and
> yet the data sheet suggests the box can handle 48x 100Gbps ports
> chassis-wide.
>
> Well, turns out that if you deploy it with redundant RE's, you get 32x
> 100Gbps ports (2x line cards of 16x ports each).
>
> However, to take another 16 ports that gets you to 48x 100Gbps ports,
> you need to sacrifice one RE to use its slot :-).
>
> Mark.
> ___
> juniper-nsp mailing list juniper-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

[j-nsp] MX304 Port Layout

[Mark Tinka via juniper-nsp]

So, we decided to give the MX304 another sniff, and needed to find out 
why Juniper charge a license for 16x 100Gbps ports per line card, and 
yet the data sheet suggests the box can handle 48x 100Gbps ports 
chassis-wide.


Well, turns out that if you deploy it with redundant RE's, you get 32x 
100Gbps ports (2x line cards of 16x ports each).


However, to take another 16 ports that gets you to 48x 100Gbps ports, 
you need to sacrifice one RE to use its slot :-).


Mark.
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp