Re: [c-nsp] Cisco ACE Context

Fri, 22 Aug 2008 12:19:52 -0700

Which time? :-) I didn't work the FWSM problem but from what I was told there was some problem with the internal vs overall VLAN structure having to do with VLANs being added or removed while already (or still) being assigned to the FWSM, etc. I can try to locate the case notes if needed. The problem was IOS-related, not FWSM-specific. That was the FWSM fix.
The IPSec SPA problem was horrible. I beat my head on the wall for months on that one, went through numerous TAC engineers too. I finally got that one TAC VPN Specialist and he knocked it out of the park. I'd turn up a new VLAN and within seconds the CPU would hit 100%. Come to find out that the VLAN I enabled was permitted as part of a range of VLANs that the SMEs configured on both sides of the IPSec SPA. Frames were getting looped through the SPA and in a never-ending circle. The VLAN I'm referring to had HSRP configured on it. The Sup was bombarded with HSRP packets which I assume are processed in SW because the CPU was thoroughly hammered. I did a packet capture on the IPSec SPA ints. Within a second of running tcpdump I'd ctrl-c it. tcpdump had caught roughly 250k of packets and missed 500k. That's a lot of packets for a processor to chew on! I had some VLANs without a FHRP configured. Mirroring an IPSec SPA GigE int showed all sorts of traffic that shouldn't have been on the IPSec SPA at all. And in the end I came to find out that the VLAN ranges on those ports were causing the problem. 


I got conflicting answers from TAC on what those 2 ints did.  Some 
engineers said that the first int was L2 and the second was L3 (made no 
sense but I couldn't argue).  Some said that all VLANs should be 
permitted on both ints and that the SPA would sort it out (that would 
have been fun!).  Others had it partially right, that one was the 
encrypted outside of the VLAN and the other was the unencrypted inside. 
 However they were on to state that the VLAN list for each int should 
match what you have configured with your crypto engine commands.  Ie, 
all 'crypto engine slot x/y outside' VLANs were to be allowed on the 
first 1Q trunk and all the inside VLANs were to be on the second.  Some 
engineers added to that saying that the inside VLANs should only be on 
the 2nd int but that the outside VLANs should be on both.  It was a real 
cluster and resulted in some of the longest downtime this phone company 
and our class-5 switches had ever experienced.  That VPN Specialist 
bailed us out though.  I'm always coming up with really weird, baffling 
problems like that.



One last thing.  Our 7600s have single Sups.  It was a design choice 
that I plan on someday correcting.  However in the 3 TAC cases where 
we've been told to reboot the chassis to correct the problem I always 
make a point to ask if we had 2 Sups would failing over the Sups fix the 
problem.  In none of the 3 cases was the answer ever yes.  It took 
reboots to fix all 3 major problems (the 3rd had to do with VRFs that 
would not disappear after deleting them).


Good luck
 Justin


Teller, Robert wrote:

Hmmm that is really weird rebooting the chassis fixed it. Any idea what

could have happened? 


Is it just me or do the ace modules take FOREVER to boot up?

I also noticed that if I configure the ANM software to administer the
ace modules when I type show run on the active context it just hangs and
nothing happens. Anyone else experience this?

-----Original Message-----

From: Justin Shore [mailto:[EMAIL PROTECTED] 
Sent: Friday, August 22, 2008 11:23 AM

To: Teller, Robert
Cc: Tony Varriale; [email protected]
Subject: Re: [c-nsp] Cisco ACE Context

I haven't worked with an ACE yet but I have two possibly related stories

to relay.


Our FWSM internal 1Q trunks (firewall-group) got hosed up shortly after 
their deployment in our 7600s (SR code).  We'd add a VLAN and it would 
show up in the firewall-group config line and it would appear in the 
FWSM sys context but it would not come up/up in the context.  No data 
could be passed by the FWSM on those VLANs.  TAC determined that a 
reboot of the FWSM was necessary.  We rebooted the FWSM to no avail. 
When that failed TAC instructed us to power cycle the chassis.  Doing 
that resolved the VLAN issue.  IIRC we were on a SRAn release at the 
time.  I later upgraded to SRB.  Prior to the mentioning of the 10G 
interface this fit you problem more but I didn't have time to write it 
up at the time.


The second story has to do with the special 10G internal interfaces.  We


had a couple SMEs out to install and configure a pair of IPSec SPAs in 
the SSC-400 carriers in our 7600s.  The SMEs manually configured the 2 
internal GigE ints on the SPAs with the VLANs that they thought so be on



them.  The virtual ints were 1Q trunks.  A few months later after 
battling extremely weird problems (traffic from VLAN x appearing on VLAN



y with a significant delay in the middle, dupe frames, packet loss, 
7600s crashing, etc) I found a TAC engineer who could explain how the 
IPSec SPA ints were supposed to be configured.  As it turns out you are 
not supposed to touch the virtual ints when running in VRF Mode, period.



  Under no circumstances do you touch the ints when in VRF Mode.  The 
inside and outside VLANs are configured automatically as you configure 
VRF in crypto statements.  Turns out that the SMEs had configured 
numerous VLANs on both virtual ints and in many cases the VLANs 
overlapped.  Ie, you had the same VLANs on both sides of the SPA, both 
the encrypted side and the unencrypted side.  The auto config stopped as


soon as they modified the interface config manually.  My TAC engineer (a


VPN specialist) couldn't believe it actually worked, even a little.  He 
helped me fix the problem though.  I had to pull the SPAs, reboot both 
7600s, reinsert the SPAs, and reconfigure crypto from the ground up 
without touching the 1 GigE internal ints.  I mention this story in case



these internal 10G ints aren't supposed to be manually configured but 
are instead supposed to be configured automatically based on the svclc 
group commands.  None of this may be related though.  Good luck.


FYI
  Justin

Teller, Robert wrote:

So it looks like the problem is that the interface associated to the

ace

is configurable. Does anyone know how to remove it without rebuilding
the chassis?

-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Teller, Robert
Sent: Friday, August 22, 2008 9:08 AM
To: Tony Varriale; [email protected]
Subject: Re: [c-nsp] Cisco ACE Context

So on Chassis-B interface tengig 7/1 is configured differently then
chassis-A. And I can't even get into chassis-a tengig 7/1 to make any
changes to it.

interface TenGigabitEthernet7/1
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan
100,120,138,150,190,200,210,235,238,555,575
 switchport trunk allowed vlan add 801-804,999
 switchport mode trunk
 switchport nonegotiate
 mls qos trust cos
 flowcontrol receive on
 no cdp enable
end

-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Tony Varriale
Sent: Thursday, August 21, 2008 5:22 PM
To: [email protected]
Subject: Re: [c-nsp] Cisco ACE Context

I'm partially confused as you are missing a number of vlans not just
138.

Can you remove it and reapply?

The only other thing I can think of is sh int trunk and see if the

vlan

is 
getting pruned back.


tv

----- Original Message ----- 
From: "Teller, Robert" <[EMAIL PROTECTED]>

To: "Christian Koch" <[EMAIL PROTECTED]>
Cc: "Tony Varriale" <[EMAIL PROTECTED]>;

<[email protected]>

Sent: Thursday, August 21, 2008 6:53 PM
Subject: RE: [c-nsp] Cisco ACE Context


Sea-6509-B#sh svclc vlan-group
Display vlan-groups created by both ACE module and FWSM commands

Group    Created by      vlans
-----    ----------      -----
 9706          FWSM
100,120,138,150,190,200,210,235,238,555,575,801-804,999

-----Original Message-----
From: Christian Koch [mailto:[EMAIL PROTECTED]
Sent: Thursday, August 21, 2008 4:53 PM
To: Teller, Robert
Cc: Tony Varriale; [email protected]
Subject: Re: [c-nsp] Cisco ACE Context

what do you see when you do a 'sh svclc vlan-group' on the  6500 that
ace-b is installed in?


On Thu, Aug 21, 2008 at 7:32 PM, Teller, Robert
<[EMAIL PROTECTED]> wrote:

That is correct. But if I do show vlan on the ace module it doesn't

show

up even though it is associated to vlan group 9706

Sea-ACE-A/Admin# show vlans
Vlans configured on SUP for this module
 vlan100  vlan120  vlan138  vlan150  vlan190  vlan200  vlan210

vlan235

vlan238  vlan555  vlan801-803  vlan999

Sea-ACE-B/Admin# show vlans
Vlans configured on SUP for this module
 vlan100  vlan200  vlan210  vlan235  vlan238  vlan555  vlan801-803



-----Original Message-----
From: Tony Varriale [mailto:[EMAIL PROTECTED]
Sent: Thursday, August 21, 2008 4:16 PM
To: Teller, Robert; [email protected]
Subject: Re: [c-nsp] Cisco ACE Context

Would you do a sh vlan b on sup-b?

Is 138 there?

tv
----- Original Message -----
From: "Teller, Robert" <[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Thursday, August 21, 2008 5:47 PM
Subject: [c-nsp] Cisco ACE Context



I have two cisco 6509 chassis with ace and fwsm modules. I have
configured the ace blades to use an internal and external conext. On
ACE-A I am able to bring up both contexts and everything talks just

fine

but on ACE-B I can't bring up vlan 138. Is there something I'm

missing?




------------------------------------------------------------------------

-----------------------------------------

svclc autostate

svclc multiple-vlan-interfaces

svclc module 7 vlan-group 9706,

firewall autostate

firewall multiple-vlan-interfaces

firewall module 3 vlan-group 9706,

firewall vlan-group 9706
100,120,138,150,190,200,210,235,238,555,575,801-804

firewall vlan-group 9706  999



------------------------------------------------------------------------

-----------------------------------------



ADMIN Context



------------------------------------------------------------------------

-----------------------------------------

ft interface vlan 801

 ip address XXX.XXX.XXX.145 255.255.255.252

 peer ip address XXX.XXX.XXX.146 255.255.255.252

 no shutdown



ft peer 1

 heartbeat interval 300

 heartbeat count 20

 ft-interface vlan 801

ft group 1

 peer 1

 priority 200

 associate-context Admin

 inservice



context WDS-External

 allocate-interface vlan 138

context WDS-Internal

 allocate-interface vlan 238



ft group 2

 peer 1

 priority 200

 associate-context WDS-Internal

 inservice

ft group 3

 peer 1

 priority 200

 associate-context WDS-External

 inservice



------------------------------------------------------------------------

-----------------------------------------



context WDS-External



------------------------------------------------------------------------

-----------------------------------------

interface vlan 138

 ip address XXX.XXX.XXX.150 255.255.255.192

 alias XXX.XXX.XXX.188 255.255.255.192

 peer ip address XXX.XXX.XXX.189 255.255.255.192

 access-group input any

 service-policy input REMOTE_MGMT_ALLOW_POLICY

 no shutdown



vlan138 is down, VLAN not assigned from the supervisor

 Hardware type is VLAN

 MAC address is 00:1f:6c:89:0c:33

 Mode : routed

 IP address is XXX.XXX.XXX.150 netmask is 255.255.255.192

 FT status is standby

 Description:not set

 MTU: 1500 bytes

 Last cleared: never

 Alias IP address is XXX.XXX.XXX.188 netmask is 255.255.255.192

 Peer IP address is XXX.XXX.XXX.189 Peer IP netmask is

255.255.255.192

 Not assigned from the Supervisor, down on Supervisor

 Service-policy download failures : 3

    0 unicast packets input, 0 bytes

    0 multicast, 0 broadcast

    0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops

    0 unicast packets output, 0 bytes

    0 multicast, 0 broadcast

    0 output errors, 0 ignored



------------------------------------------------------------------------

-----------------------------------------



Robert Teller
Washington Dental Service
Network Administrator
(206) 528-2371
[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>




#########################################################
The information contained in this e-mail and subsequent attachments

may be

privileged,
confidential and protected from disclosure.  This transmission is

intended

for the sole
use of the individual and entity to whom it is addressed.  If you

are

not

the intended
recipient, any dissemination, distribution or copying is strictly
prohibited.  If you
think that you have received this message in error, please e-mail

the

sender at the above
e-mail address.
#########################################################
_______________________________________________
cisco-nsp mailing list  [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


_______________________________________________
cisco-nsp mailing list  [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/



_______________________________________________
cisco-nsp mailing list  [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


_______________________________________________
cisco-nsp mailing list  [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


_______________________________________________
cisco-nsp mailing list  [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/




_______________________________________________
cisco-nsp mailing list  [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/























					Reply via email to