Re: [j-nsp] OSPF LFA and LDP LSPs
Hello, An update on this thread for anyone who might find it in the future. It appears the problem (or design) is with the traceroute mpls ldp command. It will test both the primary and backup LSP paths setup by OSPF LFA. Per JTAC, I simulated customer traffic over my lab network and the backup LSP path was never used. As a resolution to the case JTAC has agreed to update the traceroute command documentation. Serge - Original Message From: Serge Vautour sergevaut...@yahoo.ca To: Alex alex.arsen...@gmail.com; juniper-nsp@puck.nether.net Sent: Tue, March 16, 2010 3:25:38 PM Subject: Re: [j-nsp] OSPF LFA and LDP LSPs Hello, Well that command explains a few things. Thanks! I do want the LDP LSPs to follow the OSPF shortest path. I had never really noticed that the inet.3 table entries had a different metric than my inet.0 entries. That helped make the metrics match however I still have the same problem. Note the matching metrics: - se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) *OSPF Preference: 10 Next hop type: Router Next-hop reference count: 26 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 State: Active Int Local AS: 855 Age: 8:04 Metric: 2 Area: 0.0.0.0 Task: OSPF Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2 AS path: I inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) State: FlashAll *LDPPreference: 9 Next hop type: Router Next-hop reference count: 3 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Label operation: Push 300064 Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 Label operation: Push 30 State: Active Int Local AS: 855 Age: 8:04 Metric: 2 --- Matches inet.0 Task: LDP Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2 AS path: I - The same problem exists: se36...@pe1-stjhlab-re0 show ldp route 10.10.80.2 logical-system PE10 Destination Next-hop intf/lspNext-hop address 10.10.80.2/32 xe-0/3/0.0 10.10.81.10 ge-1/3/3.0 10.10.81.23 se36...@pe1-stjhlab-re0 traceroute 10.10.80.2 no-resolve logical-system PE10 traceroute to 10.10.80.2 (10.10.80.2), 30 hops max, 40 byte packets 1 10.10.81.10 0.347 ms 0.268 ms 0.279 ms 2 10.10.80.2 36.471 ms 0.480 ms 0.436 ms se36...@pe1-stjhlab-re0 traceroute mpls ldp 10.10.80.2 no-resolve logical-system PE10 Probe options: ttl 64, retries 3, wait 10, paths 16, exp 7, fanout 16 ttlLabel ProtocolAddress Previous Hop Probe Status 1 300064 LDP 10.10.81.10 (null) Success 23 LDP 10.10.81.1 10.10.81.10 Egress Path 1 via xe-0/3/0.0 destination 127.0.0.64 ttlLabel ProtocolAddress Previous Hop Probe Status 1 30 LDP 10.10.81.23 (null) Success 23 LDP 10.10.81.20 10.10.81.23 Egress Path 2 via ge-1/3/3.0 destination 127.0.1.64 - Note that the LDP LSP is still taking both paths?!? Thanks, Serge - Original Message From: Alex alex.arsen...@gmail.com To: Serge Vautour se...@nbnet.nb.ca; juniper-nsp@puck.nether.net Sent: Tue, March 16, 2010 1:44:13 PM Subject: Re: [j-nsp] OSPF LFA and LDP LSPs Serge, Do you have track-igp-metric configured under LDP? I am pretty sure it does not since your OSPF and LDP metrics are different but I'd like to confirm. Please try to configure track-igp-metric for LDP and re-run MPLS traceroute. Rgds Alex - Original Message - From: Serge Vautour sergevaut...@yahoo.ca To: Clarke Morledge chm...@wm.edu; juniper-nsp@puck.nether.net Sent: Tuesday, March 16, 2010 2:54 PM Subject: Re: [j-nsp] OSPF LFA and LDP LSPs Hello, Thanks for your comments. I agree with your theory. It's my understanding as well. What I don't get is why the traceroute mpls ldp command uses both paths and yet traceroute ip uses 1 path. It's like the backup path is being used by the LDP LSP when it shouldn't be... Serge - Original Message From: Clarke Morledge chm...@wm.edu To: juniper-nsp@puck.nether.net Cc: Serge Vautour sergevaut...@yahoo.ca Sent: Tue, March 16, 2010 11:35:30 AM Subject: RE
[j-nsp] OSPF LFA and LDP LSPs
Hello,
We are testing MPLS in our lab using two MX960s with a few LS to add more
routers. I'm using OSPF as the IGP and LDP for transport labels. We're on 10.0
code so I've been testing OSPF LFA. It appears to be doing something strange
and I can't tell if it's per design or a bug.
Here's a PE's route to a destination PE loopback:
--
se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail
inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden)
10.10.80.2/32 (1 entry, 1 announced)
*OSPF Preference: 10
Next hop type: Router, Next hop index: 1192
Next-hop reference count: 40
Next hop: 10.10.81.10 via xe-0/3/0.0, selected
State: Active Int
Local AS: 855
Age: 2 Metric: 2
Area: 0.0.0.0
Task: OSPF
Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2
AS path: I
inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
10.10.80.2/32 (1 entry, 1 announced)
State: FlashAll
*LDPPreference: 9
Next hop type: Router
Next-hop reference count: 3
Next hop: 10.10.81.10 via xe-0/3/0.0, selected
Label operation: Push 299776
State: Active Int
Local AS: 855
Age: 2 Metric: 1
Task: LDP
Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2
AS path: I
{master}
se36...@pe1-stjhlab-re0 show ldp route logical-system PE10 10.10.80.2
extensive
Destination Next-hop intf/lspNext-hop address
10.10.80.2/32 xe-0/3/0.0 10.10.81.10
Session ID 10.10.80.10:0--10.10.80.1:0
Bound to outgoing label 299840, Topology entry: 0x8e69980
Ingress route status: Active, Last modified: 00:00:10 ago
se36...@pe1-stjhlab-re0 traceroute 10.10.80.2 logical-system PE10 no-resolve
traceroute to 10.10.80.2 (10.10.80.2), 30 hops max, 40 byte packets
1 10.10.81.10 0.353 ms 0.274 ms 0.279 ms
2 10.10.80.2 0.460 ms 0.455 ms 0.437 ms
{master}
se36...@pe1-stjhlab-re0 traceroute mpls ldp 10.10.80.2 logical-system PE10
no-resolve
Probe options: ttl 64, retries 3, wait 10, paths 16, exp 7, fanout 16
ttlLabel ProtocolAddress Previous Hop Probe Status
1 299776 LDP 10.10.81.10 (null) Success
23 LDP 10.10.81.1 10.10.81.10 Egress
Path 1 via xe-0/3/0.0 destination 127.0.0.64
--
Everything is normal. IP packets are following the OSPF shortest path per the
costs I've set. The LDP LSP is following OSPF.
Now I turn on OSPF LFA link-protection on the links and re-run the same tests:
---
se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail
inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden)
10.10.80.2/32 (1 entry, 1 announced)
*OSPF Preference: 10
Next hop type: Router
Next-hop reference count: 26
Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected
Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 -
Huh???
State: Active Int
Local AS: 855
Age: 4 Metric: 2
Area: 0.0.0.0
Task: OSPF
Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2
AS path: I
inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
10.10.80.2/32 (1 entry, 1 announced)
State: FlashAll
*LDPPreference: 9
Next hop type: Router
Next-hop reference count: 3
Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected
Label operation: Push 299776
Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 -
Huh???
Label operation: Push 299856
State: Active Int
Local AS: 855
Age: 4 Metric: 1
Task: LDP
Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2
AS path: I
{master}
se36...@pe1-stjhlab-re0 show ldp route logical-system PE10 10.10.80.2
extensive
Destination Next-hop intf/lspNext-hop address
10.10.80.2/32 xe-0/3/0.0 10.10.81.10
Session ID 10.10.80.10:0--10.10.80.1:0
ge-1/3/3.0 10.10.81.23
Session ID 10.10.80.10:0--10.10.80.14:0
Bound to outgoing label 299840, Topology entry: 0x8e69980
Ingress route status: Active, Last modified: 00:00:09 ago
{master}
se36...@pe1-stjhlab-re0 traceroute 10.10.80.2 logical-system PE10 no-resolve
traceroute to 10.10.80.2
Re: [j-nsp] OSPF LFA and LDP LSPs
Serge, Part of what you wrote included this: Now I turn on OSPF LFA link-protection on the links and re-run the same tests: --- se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) *OSPF Preference: 10 Next hop type: Router Next-hop reference count: 26 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 - Huh??? State: Active Int Local AS: 855 Age: 4 Metric: 2 Area: 0.0.0.0 Task: OSPF Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2 AS path: I inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) State: FlashAll *LDPPreference: 9 Next hop type: Router Next-hop reference count: 3 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Label operation: Push 299776 Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 - Huh??? Label operation: Push 299856 State: Active Int Local AS: 855 Age: 4 Metric: 1 Task: LDP Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2 AS path: I I can not speak to your traceroute issue, but my understanding is that the next-hop 10.10.81.23 references are the alternate paths that get put in your routing table by the LFA algorithm. These routes exist but only in stand-by mode. So if the 10.10.81.10 next-hop ever goes away, traffic can immediately use this stand-by routing entry to forward the traffic while OSPF is recalculating new routes under the covers. This is loosely analogous to how Detours work in RSVP/MPLS Fast ReRoute -- though, admittedly, Fast ReRoute is much more involved. Then, once the OSPF recalculations are done in LFA, the routing table is updated with a new primary routing entry and another stand-by entry. Therefore, LFA effectively doubles the size of your routing table to accommodate all of the stand-by routes. Unless, I'm missing something, that is at least my understanding of how LFA actually works --- or at least how it is supposed to work. In other words, per your routing table it is working as designed. However, this does not necessarily mean that OSPF LFA currently solves all of the problems with microloops in some topologies. If someone has a better explanation, I'd like to know, too. Clarke Morledge College of William and Mary Information Technology - Network Engineering Jones Hall (Room 18) Williamsburg VA 23187 ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] OSPF LFA and LDP LSPs
Hello, Thanks for your comments. I agree with your theory. It's my understanding as well. What I don't get is why the traceroute mpls ldp command uses both paths and yet traceroute ip uses 1 path. It's like the backup path is being used by the LDP LSP when it shouldn't be... Serge - Original Message From: Clarke Morledge chm...@wm.edu To: juniper-nsp@puck.nether.net Cc: Serge Vautour sergevaut...@yahoo.ca Sent: Tue, March 16, 2010 11:35:30 AM Subject: RE: [j-nsp] OSPF LFA and LDP LSPs Serge, Part of what you wrote included this: Now I turn on OSPF LFA link-protection on the links and re-run the same tests: --- se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) *OSPF Preference: 10 Next hop type: Router Next-hop reference count: 26 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 - Huh??? State: Active Int Local AS: 855 Age: 4 Metric: 2 Area: 0.0.0.0 Task: OSPF Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2 AS path: I inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) State: FlashAll *LDPPreference: 9 Next hop type: Router Next-hop reference count: 3 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Label operation: Push 299776 Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 - Huh??? Label operation: Push 299856 State: Active Int Local AS: 855 Age: 4 Metric: 1 Task: LDP Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2 AS path: I I can not speak to your traceroute issue, but my understanding is that the next-hop 10.10.81.23 references are the alternate paths that get put in your routing table by the LFA algorithm. These routes exist but only in stand-by mode. So if the 10.10.81.10 next-hop ever goes away, traffic can immediately use this stand-by routing entry to forward the traffic while OSPF is recalculating new routes under the covers. This is loosely analogous to how Detours work in RSVP/MPLS Fast ReRoute -- though, admittedly, Fast ReRoute is much more involved. Then, once the OSPF recalculations are done in LFA, the routing table is updated with a new primary routing entry and another stand-by entry. Therefore, LFA effectively doubles the size of your routing table to accommodate all of the stand-by routes. Unless, I'm missing something, that is at least my understanding of how LFA actually works --- or at least how it is supposed to work. In other words, per your routing table it is working as designed. However, this does not necessarily mean that OSPF LFA currently solves all of the problems with microloops in some topologies. If someone has a better explanation, I'd like to know, too. Clarke Morledge College of William and Mary Information Technology - Network Engineering Jones Hall (Room 18) Williamsburg VA 23187 __ Looking for the perfect gift? Give the gift of Flickr! http://www.flickr.com/gift/ ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] OSPF LFA and LDP LSPs
Serge, Do you have track-igp-metric configured under LDP? I am pretty sure it does not since your OSPF and LDP metrics are different but I'd like to confirm. Please try to configure track-igp-metric for LDP and re-run MPLS traceroute. Rgds Alex - Original Message - From: Serge Vautour sergevaut...@yahoo.ca To: Clarke Morledge chm...@wm.edu; juniper-nsp@puck.nether.net Sent: Tuesday, March 16, 2010 2:54 PM Subject: Re: [j-nsp] OSPF LFA and LDP LSPs Hello, Thanks for your comments. I agree with your theory. It's my understanding as well. What I don't get is why the traceroute mpls ldp command uses both paths and yet traceroute ip uses 1 path. It's like the backup path is being used by the LDP LSP when it shouldn't be... Serge - Original Message From: Clarke Morledge chm...@wm.edu To: juniper-nsp@puck.nether.net Cc: Serge Vautour sergevaut...@yahoo.ca Sent: Tue, March 16, 2010 11:35:30 AM Subject: RE: [j-nsp] OSPF LFA and LDP LSPs Serge, Part of what you wrote included this: Now I turn on OSPF LFA link-protection on the links and re-run the same tests: --- se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) *OSPF Preference: 10 Next hop type: Router Next-hop reference count: 26 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 - Huh??? State: Active Int Local AS: 855 Age: 4 Metric: 2 Area: 0.0.0.0 Task: OSPF Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2 AS path: I inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) State: FlashAll *LDPPreference: 9 Next hop type: Router Next-hop reference count: 3 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Label operation: Push 299776 Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 - Huh??? Label operation: Push 299856 State: Active Int Local AS: 855 Age: 4 Metric: 1 Task: LDP Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2 AS path: I I can not speak to your traceroute issue, but my understanding is that the next-hop 10.10.81.23 references are the alternate paths that get put in your routing table by the LFA algorithm. These routes exist but only in stand-by mode. So if the 10.10.81.10 next-hop ever goes away, traffic can immediately use this stand-by routing entry to forward the traffic while OSPF is recalculating new routes under the covers. This is loosely analogous to how Detours work in RSVP/MPLS Fast ReRoute -- though, admittedly, Fast ReRoute is much more involved. Then, once the OSPF recalculations are done in LFA, the routing table is updated with a new primary routing entry and another stand-by entry. Therefore, LFA effectively doubles the size of your routing table to accommodate all of the stand-by routes. Unless, I'm missing something, that is at least my understanding of how LFA actually works --- or at least how it is supposed to work. In other words, per your routing table it is working as designed. However, this does not necessarily mean that OSPF LFA currently solves all of the problems with microloops in some topologies. If someone has a better explanation, I'd like to know, too. Clarke Morledge College of William and Mary Information Technology - Network Engineering Jones Hall (Room 18) Williamsburg VA 23187 __ Looking for the perfect gift? Give the gift of Flickr! http://www.flickr.com/gift/ ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] OSPF LFA and LDP LSPs
Hello, Well that command explains a few things. Thanks! I do want the LDP LSPs to follow the OSPF shortest path. I had never really noticed that the inet.3 table entries had a different metric than my inet.0 entries. That helped make the metrics match however I still have the same problem. Note the matching metrics: - se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) *OSPF Preference: 10 Next hop type: Router Next-hop reference count: 26 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 State: Active Int Local AS: 855 Age: 8:04 Metric: 2 Area: 0.0.0.0 Task: OSPF Announcement bits (3): 2-LDP 3-KRT 5-Resolve tree 2 AS path: I inet.3: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) State: FlashAll *LDPPreference: 9 Next hop type: Router Next-hop reference count: 3 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Label operation: Push 300064 Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000 Label operation: Push 30 State: Active Int Local AS: 855 Age: 8:04 Metric: 2 --- Matches inet.0 Task: LDP Announcement bits (2): 2-Resolve tree 1 3-Resolve tree 2 AS path: I - The same problem exists: se36...@pe1-stjhlab-re0 show ldp route 10.10.80.2 logical-system PE10 Destination Next-hop intf/lspNext-hop address 10.10.80.2/32 xe-0/3/0.0 10.10.81.10 ge-1/3/3.0 10.10.81.23 se36...@pe1-stjhlab-re0 traceroute 10.10.80.2 no-resolve logical-system PE10 traceroute to 10.10.80.2 (10.10.80.2), 30 hops max, 40 byte packets 1 10.10.81.10 0.347 ms 0.268 ms 0.279 ms 2 10.10.80.2 36.471 ms 0.480 ms 0.436 ms se36...@pe1-stjhlab-re0 traceroute mpls ldp 10.10.80.2 no-resolve logical-system PE10 Probe options: ttl 64, retries 3, wait 10, paths 16, exp 7, fanout 16 ttlLabel ProtocolAddress Previous Hop Probe Status 1 300064 LDP 10.10.81.10 (null) Success 23 LDP 10.10.81.1 10.10.81.10 Egress Path 1 via xe-0/3/0.0 destination 127.0.0.64 ttlLabel ProtocolAddress Previous Hop Probe Status 1 30 LDP 10.10.81.23 (null) Success 23 LDP 10.10.81.20 10.10.81.23 Egress Path 2 via ge-1/3/3.0 destination 127.0.1.64 - Note that the LDP LSP is still taking both paths?!? Thanks, Serge - Original Message From: Alex alex.arsen...@gmail.com To: Serge Vautour se...@nbnet.nb.ca; juniper-nsp@puck.nether.net Sent: Tue, March 16, 2010 1:44:13 PM Subject: Re: [j-nsp] OSPF LFA and LDP LSPs Serge, Do you have track-igp-metric configured under LDP? I am pretty sure it does not since your OSPF and LDP metrics are different but I'd like to confirm. Please try to configure track-igp-metric for LDP and re-run MPLS traceroute. Rgds Alex - Original Message - From: Serge Vautour sergevaut...@yahoo.ca To: Clarke Morledge chm...@wm.edu; juniper-nsp@puck.nether.net Sent: Tuesday, March 16, 2010 2:54 PM Subject: Re: [j-nsp] OSPF LFA and LDP LSPs Hello, Thanks for your comments. I agree with your theory. It's my understanding as well. What I don't get is why the traceroute mpls ldp command uses both paths and yet traceroute ip uses 1 path. It's like the backup path is being used by the LDP LSP when it shouldn't be... Serge - Original Message From: Clarke Morledge chm...@wm.edu To: juniper-nsp@puck.nether.net Cc: Serge Vautour sergevaut...@yahoo.ca Sent: Tue, March 16, 2010 11:35:30 AM Subject: RE: [j-nsp] OSPF LFA and LDP LSPs Serge, Part of what you wrote included this: Now I turn on OSPF LFA link-protection on the links and re-run the same tests: --- se36...@pe1-stjhlab-re0 show route 10.10.80.2 logical-system PE10 detail inet.0: 34 destinations, 34 routes (34 active, 0 holddown, 0 hidden) 10.10.80.2/32 (1 entry, 1 announced) *OSPF Preference: 10 Next hop type: Router Next-hop reference count: 26 Next hop: 10.10.81.10 via xe-0/3/0.0 weight 0x1, selected Next hop: 10.10.81.23 via ge-1/3/3.0 weight 0xf000
