Re: [cisco-voip] UCCX 11.5 Upgrade Disaster -- SOLVED

2017-01-18 Thread Abhiram Kramadhati (akramadh) 
Hi Anthony,

This is the one: CSCvc72933

Regards,
Abhiram Kramadhati
Technical Solutions Manager, CCBU
CCIE Collaboration # 40065


From: cisco-voip <cisco-voip-boun...@puck.nether.net> on behalf of Anthony 
Holloway <avholloway+cisco-v...@gmail.com>
Date: Thursday, 19 January 2017 at 12:36 AM
To: "Ayoub, Gregory" <g...@ufl.edu>, "cisco-voip@puck.nether.net" 
<cisco-voip@puck.nether.net>
Subject: Re: [cisco-voip] UCCX 11.5 Upgrade Disaster -- SOLVED

Thanks for the follow up and good job guessing the root cause.

What's the bug ID?

On Wed, Jan 18, 2017 at 6:32 AM Ayoub,Gregory 
<g...@ufl.edu<mailto:g...@ufl.edu>> wrote:













Just in case anyone is curious, the cause of our 11.5 IPCC failure was finally 
solved by Cisco TAC.



The license file issued to us after the upgrade did not contain proper 
licensing for outbound dialing.   Because there was no outbound licensing, and 
because

the campaign was carried forward by the upgrade from 10.6, it was still able to 
run even though it appeared as if the outbound campaign was not installed (you 
have to have outbound license to even see the outbound campaign menu).   When 
the outbound campaign

kicked off, this caused a lock on all of the inbound CTI ports which resulted 
in the busy signals.   Because of the severity of this failure scenario, a bug 
ID was created to track resolution so the outbound campaign will check for 
proper licensing before

starting.



The bright side was that my guess as to the source of the failure was correct 
:-)





From: Ayoub,Gregory




Sent: Monday, November 14, 2016 2:53 PM


To: cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>


Subject: UCCX 11.5 Upgrade Disaster





We recently attempted to upgrade UCCX 10.5 -- > 11.5.  The deployment was HA, 
and we didn’t see much risk.   The upgrade went fairly smooth, and we hit a 
finesse

bug which required the ECDSA COP file.  While minor, it’s still not mentioned 
in the release notes.



The real shocker was 4 hours later when the system stopped accepting calls and 
just handed out fast busys.  Failing over to the secondary would fix the issue

for a few minutes, but then fast busy.



Our entire contact center, which is HA, was entirely down.  Primary down, 
Secondary Down, and TAC was unable to resolve after hours and hours.  It was a 
total

unmitigated Cisco disaster.



Rolling back fixed the problem.  And then rolling forward again to 11.5 the 
system worked great again – but only for 4 hours.  Then endless fast busys.  We

rolled back and are on 10.5 working fine.  But we are at a loss what could be 
causing this problem.  Cisco TAC is in the same boat.



If I had to guess, that seems like more of a licensing failure, because TAC 
even tried replacing our license.   Anyone have a similar experience?



Thanks Greg.














___

cisco-voip mailing list

cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>

https://puck.nether.net/mailman/listinfo/cisco-voip
___
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

Re: [cisco-voip] UCCX 11.5 Upgrade Disaster -- SOLVED

2017-01-18 Thread Anthony Holloway 
Thanks for the follow up and good job guessing the root cause.

What's the bug ID?

On Wed, Jan 18, 2017 at 6:32 AM Ayoub,Gregory  wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Just in case anyone is curious, the cause of our 11.5 IPCC failure was
> finally solved by Cisco TAC.
>
>
>
>
>
>
> The license file issued to us after the upgrade did not contain proper
> licensing for outbound dialing.   Because there was no outbound licensing,
> and because
>
> the campaign was carried forward by the upgrade from 10.6, it was still
> able to run even though it appeared as if the outbound campaign was not
> installed (you have to have outbound license to even see the outbound
> campaign menu).   When the outbound campaign
>
> kicked off, this caused a lock on all of the inbound CTI ports which
> resulted in the busy signals.   Because of the severity of this failure
> scenario, a bug ID was created to track resolution so the outbound campaign
> will check for proper licensing before
>
> starting.
>
>
>
>
>
> The bright side was that my guess as to the source of the failure was
> correct :-)
>
>
>
>
>
>
>
>
>
> *From:* Ayoub,Gregory
>
>
>
>
> *Sent:* Monday, November 14, 2016 2:53 PM
>
>
> *To:* cisco-voip@puck.nether.net
>
>
> *Subject:* UCCX 11.5 Upgrade Disaster
>
>
>
>
>
>
>
>
>
> We recently attempted to upgrade UCCX 10.5 -- > 11.5.  The deployment was
> HA, and we didn’t see much risk.   The upgrade went fairly smooth, and we
> hit a finesse
>
> bug which required the ECDSA COP file.  While minor, it’s still not
> mentioned in the release notes.
>
>
>
>
>
> The real shocker was 4 hours later when the system stopped accepting calls
> and just handed out fast busys.  Failing over to the secondary would fix
> the issue
>
> for a few minutes, but then fast busy.
>
>
>
>
>
> Our entire contact center, which is HA, was entirely down.  Primary down,
> Secondary Down, and TAC was unable to resolve after hours and hours.  It
> was a total
>
> unmitigated Cisco disaster.
>
>
>
>
>
> Rolling back fixed the problem.  And then rolling forward again to 11.5
> the system worked great again – but only for 4 hours.  Then endless fast
> busys.  We
>
> rolled back and are on 10.5 working fine.  But we are at a loss what could
> be causing this problem.  Cisco TAC is in the same boat.
>
>
>
>
>
> If I had to guess, that seems like more of a licensing failure, because
> TAC even tried replacing our license.   Anyone have a similar experience?
>
>
>
>
>
> Thanks Greg.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
>
> cisco-voip mailing list
>
> cisco-voip@puck.nether.net
>
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
___
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

Re: [cisco-voip] UCCX 11.5 Upgrade Disaster -- SOLVED

2017-01-18 Thread Ayoub,Gregory 
Just in case anyone is curious, the cause of our 11.5 IPCC failure was finally 
solved by Cisco TAC.

The license file issued to us after the upgrade did not contain proper 
licensing for outbound dialing.   Because there was no outbound licensing, and 
because the campaign was carried forward by the upgrade from 10.6, it was still 
able to run even though it appeared as if the outbound campaign was not 
installed (you have to have outbound license to even see the outbound campaign 
menu).   When the outbound campaign kicked off, this caused a lock on all of 
the inbound CTI ports which resulted in the busy signals.   Because of the 
severity of this failure scenario, a bug ID was created to track resolution so 
the outbound campaign will check for proper licensing before starting.

The bright side was that my guess as to the source of the failure was correct 
:-)

From: Ayoub,Gregory
Sent: Monday, November 14, 2016 2:53 PM
To: cisco-voip@puck.nether.net
Subject: UCCX 11.5 Upgrade Disaster

We recently attempted to upgrade UCCX 10.5 -- > 11.5.  The deployment was HA, 
and we didn't see much risk.   The upgrade went fairly smooth, and we hit a 
finesse bug which required the ECDSA COP file.  While minor, it's still not 
mentioned in the release notes.

The real shocker was 4 hours later when the system stopped accepting calls and 
just handed out fast busys.  Failing over to the secondary would fix the issue 
for a few minutes, but then fast busy.

Our entire contact center, which is HA, was entirely down.  Primary down, 
Secondary Down, and TAC was unable to resolve after hours and hours.  It was a 
total unmitigated Cisco disaster.

Rolling back fixed the problem.  And then rolling forward again to 11.5 the 
system worked great again - but only for 4 hours.  Then endless fast busys.  We 
rolled back and are on 10.5 working fine.  But we are at a loss what could be 
causing this problem.  Cisco TAC is in the same boat.

If I had to guess, that seems like more of a licensing failure, because TAC 
even tried replacing our license.   Anyone have a similar experience?

Thanks Greg.




___
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip