Re: [asterisk-dev] Asterisk Load Performance

[Richard Mudgett]

On Tue, Jul 5, 2016 at 4:03 PM, Jonathan Rose <
jonathan.r...@motorolasolutions.com> wrote:

>
> On Tue, Jul 5, 2016 at 3:43 PM, Michael Petruzzello <
> michael.petruzze...@civi.com> wrote:
>
>> On Wed, Jun 29 at 11:14:04 AM, Richard Mudgett> >
>> wrote:
>> > Each softmix bridge has only one thread performing all of the media
>> mixing
>> > for the bridge.  To
>> > get better mixing performance for such a large conference, you will
>> need to
>> > create several
>> > softmix bridges in a hierarchy with the bridges linked by local
>> channels.
>>
>> A bridge is only able to handle around 2000-2500 channels, so I created
>> 15 bridges with 14 channels bridging the bridges together.
>>
>> When doing this an error I see a lot is WARNING[98920]: channel.c:1101
>> __ast_queue_frame: Exceptionally long voice queue length queuing to
>> Local/**@default-;2, which then turns into WARNING[47525]:
>> pjproject:0 :  sip_transactio .Unable to register INVITE transaction
>> (key exists) and ERROR[47525]: res_pjsip.c:2777 ast_sip_create_dialog_uas:
>> Could not create dialog with endpoint sippeer. Object already exists
>> (PJ_EEXISTS). Finally the following repeats over and over again, [Jun 30
>> 12:22:21] ERROR[84189][C-0958]: netsock2.c:305 ast_sockaddr_resolve:
>> getaddrinfo("domain.name
>> ",
>> "(null)", ...): Temporary failure in name resolution
>> [Jun 30 12:22:21] WARNING[84189][C-0958]: acl.c:800 resolve_first:
>> Unable to lookup 'domain.name
>> 
>> '.
>>
>
The exceptionally long voice queue length messages can be a symptom of
thread
starvation as the Local channels frame queue has developed an excessive
backlog.

The forthcoming v13.10.0 release should indirectly take care of the EEXISTS
messages
as part of the https://issues.asterisk.org/jira/browse/ASTERISK-26088 fix.
Working on
that issue I saw the EEXISTS messages for REGISTER and SUBSCRIBE message
processing.  The issue was a result of the original message and
retransmissions getting
backlogged in the serializer/taskprocessor and responses sent using another
serializer.

Looks like your system's DNS resolver has gotten overwhelmed.


>
>> The last error just keeps on repeating and calls can no longer join (only
>> around 3,500 make it on before this starts to occur). Calling in manually I
>> receive an "all circuits are busy" message.
>>
>> I'm going to try halving the number of bridges, but is there anything
>> else I can do to improve performance? This seems to be the last hurdle to
>> use one server for 10,000 callers.
>>
>
> If you don't need all of your participants actually to be speaking at a
> time (and I hope not with that kind of volume), you could use holding
> bridges for the vast majority of the partipants. Link the bridges using a
> local channel with the Hold bridge side being set to use the 'announcer'
> bridge role and the hold bridge will effectively just be voiceless
> conference participants. If you want, you can listen for DTMF events to
> move the participants back and forth between the different bridges.
>

Wow.  Thanks Jonathan.  I hadn't thought of doing it that way.  That should
really drop the mixing load.
Probably should allow only ulaw or alaw (pick one) for all participants to
minimize translation costs.
One additional thing I should add is that those linking Local channel
bridges should just allow the
chosen alaw/ulaw to reduce translation to each participant in the holding
bridge.  The forthcoming
v13.10.0 adds the ability to specify formats when ARI originates a channel
(Local in this case) and
an originator channel is not available.  (See CHANGES file)

Richard
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Asterisk Load Performance

[Jonathan Rose]

On Tue, Jul 5, 2016 at 3:43 PM, Michael Petruzzello <
michael.petruzze...@civi.com> wrote:

> On Wed, Jun 29 at 11:14:04 AM, Richard Mudgett >
> wrote:
> > Each softmix bridge has only one thread performing all of the media
> mixing
> > for the bridge.  To
> > get better mixing performance for such a large conference, you will need
> to
> > create several
> > softmix bridges in a hierarchy with the bridges linked by local channels.
>
> A bridge is only able to handle around 2000-2500 channels, so I created 15
> bridges with 14 channels bridging the bridges together.
>
> When doing this an error I see a lot is WARNING[98920]: channel.c:1101
> __ast_queue_frame: Exceptionally long voice queue length queuing to
> Local/**@default-;2, which then turns into WARNING[47525]:
> pjproject:0 :  sip_transactio .Unable to register INVITE transaction
> (key exists) and ERROR[47525]: res_pjsip.c:2777 ast_sip_create_dialog_uas:
> Could not create dialog with endpoint sippeer. Object already exists
> (PJ_EEXISTS). Finally the following repeats over and over again, [Jun 30
> 12:22:21] ERROR[84189][C-0958]: netsock2.c:305 ast_sockaddr_resolve:
> getaddrinfo("domain.name
> ",
> "(null)", ...): Temporary failure in name resolution
> [Jun 30 12:22:21] WARNING[84189][C-0958]: acl.c:800 resolve_first:
> Unable to lookup 'domain.name
> 
> '.
>
> The last error just keeps on repeating and calls can no longer join (only
> around 3,500 make it on before this starts to occur). Calling in manually I
> receive an "all circuits are busy" message.
>
> I'm going to try halving the number of bridges, but is there anything else
> I can do to improve performance? This seems to be the last hurdle to use
> one server for 10,000 callers.
>

If you don't need all of your participants actually to be speaking at a
time (and I hope not with that kind of volume), you could use holding
bridges for the vast majority of the partipants. Link the bridges using a
local channel with the Hold bridge side being set to use the 'announcer'
bridge role and the hold bridge will effectively just be voiceless
conference participants. If you want, you can listen for DTMF events to
move the participants back and forth between the different bridges.

-- 

*Jonathan R. Rose*Senior Systems Engineer

Emergency CallWorks
Motorola Solutions

email: jonathan.r...@motorolasolutions.com
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Asterisk Load Performance

[Michael Petruzzello]

On Wed, Jun 29 at 11:14:04 AM, Richard Mudgett
wrote:
> Each softmix bridge has only one thread performing all of the media mixing
> for the bridge.  To
> get better mixing performance for such a large conference, you will need
to
> create several
> softmix bridges in a hierarchy with the bridges linked by local channels.

A bridge is only able to handle around 2000-2500 channels, so I created 15
bridges with 14 channels bridging the bridges together.

When doing this an error I see a lot is WARNING[98920]: channel.c:1101
__ast_queue_frame: Exceptionally long voice queue length queuing to
Local/**@default-;2, which then turns into WARNING[47525]:
pjproject:0 :  sip_transactio .Unable to register INVITE transaction
(key exists) and ERROR[47525]: res_pjsip.c:2777 ast_sip_create_dialog_uas:
Could not create dialog with endpoint sippeer. Object already exists
(PJ_EEXISTS). Finally the following repeats over and over again, [Jun 30
12:22:21] ERROR[84189][C-0958]: netsock2.c:305 ast_sockaddr_resolve:
getaddrinfo("domain.name", "(null)", ...): Temporary failure in name
resolution
[Jun 30 12:22:21] WARNING[84189][C-0958]: acl.c:800 resolve_first:
Unable to lookup 'domain.name'.

The last error just keeps on repeating and calls can no longer join (only
around 3,500 make it on before this starts to occur). Calling in manually I
receive an "all circuits are busy" message.

I'm going to try halving the number of bridges, but is there anything else
I can do to improve performance? This seems to be the last hurdle to use
one server for 10,000 callers.




*Michael J. Petruzzello*
Software Engineer
P.O. Box 4689
Greenwich, CT 06831
203-618-1811 ext.289 (office)
www.civi.com

On Wed, Jun 29, 2016 at 10:55 AM, Michael Petruzzello <
michael.petruzze...@civi.com> wrote:

> It is very interesting how threading issues on both a stasis application
> and Asterisk escalate each other. Using 15 websockets in one stasis
> application and removing all thread locking from the application have made
> the ARI messages flow smoothly. Right now I am using about 900 threads to
> process messages from Asterisk and Asterisk has at least 320 in stasis,
> though that can increase to infinity.
>
> I have also disabled the channel_varset from stasis because it becomes
> really unwieldy. When having thousands of callers in a bridge, every time a
> channel is added to a bridge or removed, every channel receives a channel
> var set message because of the BridgePeer variable.
>
> As of now, I have two remaining problems:
>
> 1. At around having 5,000 channels in a bridge (whether majority are muted
> or not), the audio breaks down. Anyone talking can only be heard in 3
> second bursts approximately every 5-10 seconds. At 10,000 channels only
> static can be heard in these 3 second bursts.
>
> Is there anything I can optimize so that Asterisk can handle all these
> channels in a bridge?
>
> 2. Every time a channel joins the bridge, the websocket responsible for
> that channel is then subscribed to the bridge. Then any events that occur
> on that bridge (such as another channel entering or exiting it) are sent to
> that websocket. Because every websocket then ends up receiving these
> messages, it defeats the point of having multiple websockets. To get around
> this I have been unsubscribing the websockets from the bridge anytime a
> channel from that websocket enters the bridge, but this isn't perfect as
> timing is an issue.
>
> Is there anyway to disable this automatic subscription behavior to a
> bridge?
>
>
> *Michael J. Petruzzello*
> Software Engineer
> P.O. Box 4689
> Greenwich, CT 06831
> 203-618-1811 ext.289 (office)
> www.civi.com
>
> On Tue, Jun 21, 2016 at 3:29 PM, Michael Petruzzello <
> michael.petruzze...@civi.com> wrote:
>
>> On Tue, Jun 21, 2016 at 12:16 PM, Richard Mudgett 
>> wrote:
>> > The subm:devService-test-0038 taskprocessor is servicing the stasis
>> > message bus
>> > communication with your devService-test ARI application.  Since each
>> > taskprocessor is
>> > executed by one thread, that is going to be a bottleneck.  One thing you
>> > can try is to
>> > register multiple copies of your ARI application and randomly spread the
>> > calls to the
>> > different copies of the application.  (devService-test1,
>> > devService-test2,...)
>>
>> Ah, that explains it! Everything else has been running well in Asterisk
>> as far as handling the actual channels and the SIP messaging with the large
>> calls / second.
>>
>> I was thinking about the potential of parallelizing the stasis message
>> bus communication to use multiple task processors, but that would introduce
>> other issues. Messages would be sent out of order, and the ARI application
>> would need to handle that.
>>
>> Your suggestion sounds like the best approach. That way I still have only
>> one application with multiple connections to Asterisk. No need to have
>> multiple applications and servers that would need to communicate 

[asterisk-dev] Asterisk 13.10.0-rc2 Now Available

[Asterisk Development Team]

The Asterisk Development Team has announced the second release candidate of
Asterisk 13.10.0. This release candidate is available for immediate
download at http://downloads.asterisk.org/pub/telephony/asterisk

The release of Asterisk 13.10.0-rc2 resolves an issue reported by the
community and would have not been possible without your participation.
Thank you!

The following is the issue resolved in this release candidate:

Bugs fixed in this release:
---
 * ASTERISK-26099 - res_pjsip_pubsub: Crash when sending request
  due to server timeout (Reported by Ross Beer)

For a full list of changes in this release candidate, please see the ChangeLog:

http://downloads.asterisk.org/pub/telephony/asterisk/ChangeLog-13.10.0-rc2

Thank you for your continued support of Asterisk!

-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev