Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[CiscoNSP List]

Wow...thanks - okthought this was referring to the "other" memory leak bug, 
but this looks to be a new one, no IOS fix yetworkaround is 
"interesting"

Workaround:
Issue 'no shut' on all the interface in the system, consumed memory is released.





From: Erik Sundberg <esundb...@nitelusa.com>
Sent: Friday, 3 June 2016 5:41 AM
To: CiscoNSP List; Mark Tinka; James Jun
Cc: cisco-nsp@puck.nether.net
Subject: RE: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working 
through?

We have been using ASR920's for a couple months now.

I have an outstanding Memory leak issue in 
asr920-universalk9_npe.03.16.01a.S.155-3.S1a-ext.bin

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy87268

The work around doesn't work for me, I just tried it. TAC gave me the 
following. We have to reboot the ASR920 to lower the memory back down.

The fix for CSCuy87268 has been committed and would be available in the 
following releases:

XE 3.16.4S/15.5(3)S4 - planned for September 2016.
XE 3.18.S1/15.6(2)S1 - planned for mid June 2016.



Issue RP0 memory usage at 98% and growing, status is in the warning state. I 
had to reboot the device to get the memory back down, however it still grows.


Model: ASR-920-24SZ-M

ASR920 - #1
98% used memory and uptime is 24 weeks.
ASR920#sh platform software status control-processor bri
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.04   0.07   0.04

Memory (kB)
 Slot  StatusTotal Used (Pct) Free (Pct) Committed (Pct)
  RP0 Warning  3438048  3361672 (98%)76376 ( 2%)   3345388 (97%)

CPU Utilization
 Slot  CPU   User System   Nice   IdleIRQ   SIRQ IOwait
  RP00  11.93   7.22   0.00  80.64   0.00   0.20   0.00
 1   9.60   8.30   0.00  81.78   0.00   0.30   0.00


ASR920 - #2 Before Reboot (Uptime around 10 Weeks)

ASR920#sh platform software status control-processor bri
Load Average
  Slot  Status  1-Min  5-Min 15-Min
   RP0 Healthy   0.00   0.00   0.00

Memory (kB)
  Slot  StatusTotal Used (Pct) Free (Pct) Committed (Pct)
  RP0 Warning  3438048  3360436 (98%)77612 ( 2%)   3341328 (97%)

CPU Utilization
  Slot  CPU   User System   Nice   IdleIRQ   SIRQ IOwait
   RP00   6.60   3.70   0.00  89.38   0.00   0.30   0.00
1   1.40   0.80   0.00  97.80   0.00   0.00   0.00


ASR920 - #2 Post Reboot

EAR1.ATL1#sh platform software status control-processor bri
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.00   0.02   0.00

Memory (kB)
 Slot  StatusTotal Used (Pct) Free (Pct) Committed (Pct)
  RP0 Healthy  3438048  1855832 (54%)  1582216 (46%)   1512524 (44%)

CPU Utilization
 Slot  CPU   User System   Nice   IdleIRQ   SIRQ IOwait
  RP00  10.68  10.88   0.00  78.02   0.00   0.39   0.00
 1   6.29   6.49   0.00  86.91   0.00   0.29   0.00




The memory leaks in the process SPA_XCVR_OIR and  enqueue_oir_msg  from what 
tac has told me.

EAR1.ATL1#  show platform software memory iomd 0/0 brief
  module  allocated requested allocsfrees
  --
  DEVOBJ  38496 37792 880
  IOMd intr   1392  1104  360
  Summary 1198313647679413327 496656324 431793784
  appsess_ctx 1736  1728  1 0
  appsess_timer   56403 1
  bsess_hdl   40321 0
  cdh-shim7260  5940  165   0
  cdllib  1668  1660  7 6
  chunk   5573  5517  7 0
  enqueue_oir_msg 506580480 253290240 49236333  17575053
  env_wheel   20108 20100 1 0
  eventutil   310794307834386   16
  fpd_sb  208   200   1 0
  fpd_upg 5636  5548  110
  geim_esb5040  4816  280
  geim_hwidb  16352 16128 280
  geim_instance   16800 16576 280
  geim_spa_instance   13440 13216 280
  geim_spa_plugin 764   756   2 1
  ipc_shim_pak0 0 17057467  17057467
  null_spa_plugin 104   961 0
  oir_create  80721 0
  oir_enqueue_event   0 0 1 1
  oir_processing  24161 0
  queue   240   200   5 0
  spa_

Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[Mark Tinka]

On 2/Jun/16 21:41, Erik Sundberg wrote:

> We have been using ASR920's for a couple months now.
>
> I have an outstanding Memory leak issue in 
> asr920-universalk9_npe.03.16.01a.S.155-3.S1a-ext.bin
>
> https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy87268
>
> The work around doesn't work for me, I just tried it. TAC gave me the 
> following. We have to reboot the ASR920 to lower the memory back down.
>
> The fix for CSCuy87268 has been committed and would be available in the 
> following releases:
>
> XE 3.16.4S/15.5(3)S4 - planned for September 2016.
> XE 3.18.S1/15.6(2)S1 - planned for mid June 2016.

Have yo considered 3.16(2a)S meanwhile? We're running fine on those and
haven't had memory leak issues.

Mark.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[Erik Sundberg]

  tdl 0 0 36500281  36500281
  tdl_hdl 640   472   210
  tdl_ping_stats  5096  5024  9 0
  trccfg  256   248   5 4
  uipeer  9288  9264  1512
  unknown 0 0 181   181
  unknown 120   112   1 0
  watched_bitfield36281 0
  xcvr5084  4852  290
  xcvr_oir_timer  672   448   280



-Original Message-
From: cisco-nsp [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of 
CiscoNSP List
Sent: Thursday, May 26, 2016 4:50 AM
To: Mark Tinka; James Jun
Cc: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working 
through?

Cheers Guys - Have 03.16.02aS on the ones ready for deployment, so fingers 
crossed, it remains a stable release :)

Thanks for the feedback.


From: Mark Tinka <mark.ti...@seacom.mu>
Sent: Thursday, 26 May 2016 3:35 PM
To: James Jun; CiscoNSP List
Cc: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working 
through?

On 26/May/16 06:23, James Jun wrote:

> I believe there is an mpls label-range bug that Mark noted (CSCuy29638) which 
> is outstanding, but that can be worked around by staying out of high label 
> range.

Actually, Cisco normally allocate low-range labels. So this is fine if the box 
adjacent to the ASR920 is a Cisco.

Juniper tend to allocate labels in the 300,000 - 400,000+ range. This is not an 
issue if the ASR920 is not adjacent to the Juniper. In our case, all situations 
where we hit this bug is when the ASR920 is adjacent to a Juniper. We are not 
in a position to change the topology to avoid the adjacency between the ASR920 
and a Juniper.

Unfortunately, it is not possible to set a label allocation range in Junos 
today, the same way you can on a Cisco. That capability is only slated for 
Junos 16.

For us, the workaround is a test image that includes the fix that the BU built 
specially for us.

But not to worry, the BU say the fixed image is still on schedule for release 
7th June, this year :-).

>
>
> Thankfully, this bug is also fixed on 03.16.02aS.  920s upgraded to this 
> release appear to be fine so far.

Save for my MPLS label range issue, this release is very stable.

Mark.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net 
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/



CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or 
previous e-mail messages attached to it may contain confidential information 
that is legally privileged. If you are not the intended recipient, or a person 
responsible for delivering it to the intended recipient, you are hereby 
notified that any disclosure, copying, distribution or use of any of the 
information contained in or attached to this transmission is STRICTLY 
PROHIBITED. If you have received this transmission in error please notify the 
sender immediately by replying to this e-mail. You must destroy the original 
transmission and its attachments without reading or saving in any manner. Thank 
you.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[CiscoNSP List]

Cheers Guys - Have 03.16.02aS on the ones ready for deployment, so fingers 
crossed, it remains a stable release :)

Thanks for the feedback.


From: Mark Tinka <mark.ti...@seacom.mu>
Sent: Thursday, 26 May 2016 3:35 PM
To: James Jun; CiscoNSP List
Cc: cisco-nsp@puck.nether.net
Subject: Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working 
through?

On 26/May/16 06:23, James Jun wrote:

> I believe there is an mpls label-range bug that Mark noted (CSCuy29638) which 
> is outstanding, but that can be worked around by staying out of high label 
> range.

Actually, Cisco normally allocate low-range labels. So this is fine if
the box adjacent to the ASR920 is a Cisco.

Juniper tend to allocate labels in the 300,000 - 400,000+ range. This is
not an issue if the ASR920 is not adjacent to the Juniper. In our case,
all situations where we hit this bug is when the ASR920 is adjacent to a
Juniper. We are not in a position to change the topology to avoid the
adjacency between the ASR920 and a Juniper.

Unfortunately, it is not possible to set a label allocation range in
Junos today, the same way you can on a Cisco. That capability is only
slated for Junos 16.

For us, the workaround is a test image that includes the fix that the BU
built specially for us.

But not to worry, the BU say the fixed image is still on schedule for
release 7th June, this year :-).

>
>
> Thankfully, this bug is also fixed on 03.16.02aS.  920s upgraded to this 
> release appear to be fine so far.

Save for my MPLS label range issue, this release is very stable.

Mark.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[Mark Tinka]

On 26/May/16 06:23, James Jun wrote:

> I believe there is an mpls label-range bug that Mark noted (CSCuy29638) which 
> is outstanding, but that can be worked around by staying out of high label 
> range.

Actually, Cisco normally allocate low-range labels. So this is fine if
the box adjacent to the ASR920 is a Cisco.

Juniper tend to allocate labels in the 300,000 - 400,000+ range. This is
not an issue if the ASR920 is not adjacent to the Juniper. In our case,
all situations where we hit this bug is when the ASR920 is adjacent to a
Juniper. We are not in a position to change the topology to avoid the
adjacency between the ASR920 and a Juniper.

Unfortunately, it is not possible to set a label allocation range in
Junos today, the same way you can on a Cisco. That capability is only
slated for Junos 16.

For us, the workaround is a test image that includes the fix that the BU
built specially for us.

But not to worry, the BU say the fixed image is still on schedule for
release 7th June, this year :-).

>
>
> Thankfully, this bug is also fixed on 03.16.02aS.  920s upgraded to this 
> release appear to be fine so far.

Save for my MPLS label range issue, this release is very stable.

Mark.

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[James Jun]

On Wed, May 25, 2016 at 09:54:11AM +, CiscoNSP List wrote:
> 
> Just reading through all the recent threads, and as we are about to deploy a 
> number of these boxes in production, wanted to be 100% on any potential 
> issues we may face (couple of recent posts, mention ASR920, ISIS stopping 
> working(And routing unexpectedly), and reboot "resolves" the 
> issue(temporarily)...this was also an issue on the ME3600's (As reported by 
> Mark Tinka), and a work-around provided, but no actual fix by Cisco (Even 
> though the bug has been present for some years?)not too sure if this is 
> the case on the ASR920?  Nor if it is ISIS specific?

I believe there is an mpls label-range bug that Mark noted (CSCuy29638) which 
is outstanding, but that can be worked around by staying out of high label 
range.

The most serious bug for everyone was CSCux98051 (ASR920 stops forwarding bug), 
but it is fixed on 3.16.02aS -- it's been stable for me so far.  I have two 
ASR920-24SZ-IM boxes upgraded to 3.16.02aS that're doing roughly 3 Gbps of 
EoMPLS (l2circuit over ldp over TE tunnels) each and they've been stable so far.

> 
> I am aware of the following, and have applied the relevant "recommended" IOS 
> update:
> 
> bug ID CSCux91894, 
> which causes a memory leak with the default configuration.

Hahah, yup.  I got about a dozen in the field that look like this: 
http://twdx.co/jZoM8v  
These 920s don't do any memory-hungry stuff (like BGP full table w/ selective 
download, etc) -- they have small IGP routes and all they're doing is EoMPLS.

Thankfully, this bug is also fixed on 03.16.02aS.  920s upgraded to this 
release appear to be fine so far.


> Are there any others people are currently working through on these boxes?
>
> Just a little hesitant to deploy them atmwhen we originally deployed the 
> ME3600's, we got hit with a number of rather nasty issues, so am a little gun 
> shy given the ASR920 is the "replacement" for the ME3600, and is also a 
> relatively new box..

Knock on the wood.  So far, I'm finding 03.16.02aS (15.5-3.S2a) as the most 
stable build for production use.  It's been a bit of an excitement, especially 
on ASR 902 and 903 boxes (I must admit, we started replacing them with ASR9006 
+ MOD80 spam :p)  Some funny bug I ran into with that platform was CSCuq32410 
(router instantly crashes upon applying L2ACL on an EFP); another funny one was 
CSCuw21323 (SFP+ not switching on, IMA insert/remove dance required to achieve 
limited success).  These are fixed now though.

James
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working through?

[Mal]

Hiding one,

Have you thought about logging a TAC case ?   That's generally what 
professionals do when they have issues or concerns.. 

Good luck !

Mal




-Original Message-
From: cisco-nsp [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of 
CiscoNSP List
Sent: Wednesday, 25 May 2016 7:24 PM
To: cisco-nsp@puck.nether.net
Subject: [c-nsp] ASR920 - Any "outstanding" TAC cases people are working 
through?

Hi Everyone,


Just reading through all the recent threads, and as we are about to deploy a 
number of these boxes in production, wanted to be 100% on any potential issues 
we may face (couple of recent posts, mention ASR920, ISIS stopping working(And 
routing unexpectedly), and reboot "resolves" the issue(temporarily)...this was 
also an issue on the ME3600's (As reported by Mark Tinka), and a work-around 
provided, but no actual fix by Cisco (Even though the bug has been present for 
some years?)not too sure if this is the case on the ASR920?  Nor if it is 
ISIS specific?


I am aware of the following, and have applied the relevant "recommended" IOS 
update:

bug ID CSCux91894, 
which causes a memory leak with the default configuration.
Problem Symptoms
A memory leak on the ASR 900 Series routers is seen under the iomd process and 
even when the node state is idle.
When executed, this command shows the increased memory leak under spa_xcvr_oir.
#show platform software memory iomd 0/0 brief The increased memory leak might 
cause the router to crash and reboot multiple times.


Are there any others people are currently working through on these boxes?



Our ASR920's will be used as "PEs"...runninging MPLS/BGP/VRF/OSPF, and 
terminating cust tails.


Just a little hesitant to deploy them atmwhen we originally deployed the 
ME3600's, we got hit with a number of rather nasty issues, so am a little gun 
shy given the ASR920 is the "replacement" for the ME3600, and is also a 
relatively new box..


Thanks in advance.

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net 
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/