THE BEST way to work with TAC and possibly anticipate failure in your case is (assuming you don't have backup 6500 to load the intended image in lab) :
1. Open up the TAC case 3 - 4 hrs before upgrading through the web (open w/ P2) 2. Provide all necessary information through web (these information will go through system before reaching the correct guy) 3. If an engineer has been assigned, call him up and tell him about your upgrade plan. (In this time, you can ensure yourself there's not any communication issue) If you're not happy with the assigned engineer, call duty manager to get native speaker. 4. Only after you've found TAC engineer you're comfortable (tech & language) with, get him to understand your plan (details), and get him remote access to console your 6500. Don't forget to get his desk number as well. If everything is settled, ask him to wait when the maintenance window comes and leave the case as P2. 5. You upgrade 6500 during the window, when problems come. Call up the engineer, tell your problem and if necessary ask him to console in to your core switch.(Or ask him right away) I'm sure you'll get appropriate TAC help within minutes. Asa Powered by Telkomsel BlackBerry® -----Original Message----- From: "Steve Fischer" <sfischer1...@gmail.com> Date: Sun, 20 Sep 2009 17:41:08 To: <cisco-nsp@puck.nether.net> Subject: [c-nsp] Cisco TAC issues - can someone from Cisco enlighten me on this? Last Thursday evening, at around midnight, in the course of my organizations network maintenance, we had not one but two of our core 6500 switches go into ROMMON (after being rebooted with new code, and being operational for approximately 45 minutes)at the same time and for no apparent reason. Attempts to reboot the devices were in vain, and attempts to roll-back also appeared to be in vain, so I called the Cisco TAC and opened a P1 case. Immediately, the call was routed over to India. I was in a loud data center, and the engineers accent was very thick, to the point I could not hear him over the background noise, much less understand him. Other than asking for a webex session - made impossible by the fact that the network core is down, he offers nothing in the way of assistance. I asked to have the case transferred to a native-English speaking engineer. Call transferred to Indian engineer #2, and the communications issues persist. I have two core switches down, and am becoming more than a little concerned. Same result - engineer really offers nothing in the way of assistance, and I again, request the call to be transferred to a native-English speaking engineer. Enter Indian engineer #3. Now let me state here for the record that I am in no way questioning the competence of the three gentlemen I spoke to, nor do I have any xenophobic tendencies, but I would like to make a few points here: 1. If I cannot understand the support engineer, it will be difficult for him to assist me, regardless of his skill level. 2. Having a native-English speaking engineer available would have been at this time very disarming, and calming in the midst of for what was for me a crisis. In the medical field, they call it bed-side manner, which would have been of immense value given the crisis I was facing. 3. My organization spends well over $100K annually in Cisco maintenance. Case transferred to Indian engineer #4. Now, while this was occurring, I called Cisco's TAC and asked the case be re-queued to an engineer in North America. I was told that there were no support engineers on duty in North America. Now, I'm getting upset, and more than just a little. Also, in the meantime, it was suggested that I remove one of the CompactFlash cards from one of the 6500's that was still working (we have 4 total), and try to boot from the IOS image on it. Upon ejecting the Flash card, that 6500 too, went immediately into ROMMON. So, now, we have 3 of 4 core switches down. The entire data center is down, and are one step away from the phone system going down as well - which indeed did happen. As we now have all four cores down, the options of rebooting them with the old code. One by one, through all four cores, they are rolled back, and finally the network comes up. Let me say the fourth engineer suggested this, by prior to that, I had concluded this was going to be the best course of action. Now, back up two weeks. I had a Cisco Works issue at around 3:00PM EST, and open a case for it. The call is transferred to.wait for it.India. So, it doesn't appear that the time of an issue completely influences to what Cisco support center a call is routed. As a matter of fact, the support engineer for that particular call informed me it was 2:00AM where he was. This leads me to several questions that perhaps someone from Cisco monitoring this forum could answer. 1. Given the stature of the 6500 platform within Cisco's product line, and given the severity of the issue I was experiencing, is there no Cisco Support Center in North America staffed 24X7X365 to deal with such issues? 2. Was the only option available to me a webex session? It seemed strange that the engineer would even ask for that, given that this was reported as a network down emergency. 3. Upon asking the call be transferred to a native English speaking engineer, why was the call routed to 3 more Indian engineers? Rather than informing me that no native English-speaking engineers were available, why did I have to request this three more times? I find it INCREDIBLE that an organization of Cisco's size could not find a single native English-speaking engineer to assist me in this crisis concerning their flagship product. 4. Does Cisco as a company understand the value of being disarming to a stressed-out client in the midst of a crisis, and how much providing a support engineer who can clearly and effectively communicate with the customer? If I were to grade Cisco on this support call, and to my knowledge, this was the first network down call I've opened since I've been with this organization, I'd give Cisco an F.no, an F-. This was an epic-fail for Cisco, to the point that my organization is considering dumping their support all together, and going with a third party. There is just no way I should have been bounced to four Indian engineers when after the first one, and the second one, I calmly explained the rationale behind requesting a native English-speaking engineer. What can Cisco do to make this right? 1. A written letter of apology. I don't blame the engineers for this - they are likely highly competent support engineers. Cisco failed to provide an engineer who could communicate with me clearly and concisely. 2. Provide an additional one year of maintenance coverage on all four of our 6500's free of charge, as that pales by comparison to the amount of money lost by my company for the two hours our network was down, and I was unable to reach an engineer who could effectively communicate with me. 3. Staff a support center in North America 24X7X365 geared to deal with exactly the issue I encountered. A P1 case should be treated differently than a P3 case, and this case wasn't, at least not from my perspective. 4. When it became clear that the communications issues that existed between the support engineer and myself precluded progress being made to resolve the issue, and I requested the case be escalated, escalate the case. I DO blame the engineer for that - not escalating the case when requested to do so. 5. Stop making the first course of action being requesting a webex. There will be scenarios where that won't be possible, as was the case Thursday evening. Engineer 1 seemed to give up when webex wasn't an option. I am interested in any and all feedback from the community on this. If there is someone within Cisco (other than my salesperson, who's heard this before from me.on more than one occasion) who I can send this to, and can respond to it, it would also be appreciated. _______________________________________________ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ _______________________________________________ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/