Re: [Project Clearwater] Is Bono a multi-thread program?

[王俊中]

Hi , Richard

I know that is not very concerned about Clearwater , Thanks for your time
to answer my question .

Thank you very much .

Morris

2016-09-07 0:58 GMT+08:00 Richard Whitehouse (projectclearwater.org) <
richard.whiteho...@projectclearwater.org>:

> Morris,
>
>
>
> Yes that is possible. You could either spawn a new thread for each
> request, or dispatch it to a worker pool. What you choose to do is really
> an implementation detail of the application server you write, and not
> really related to how Clearwater is structured.
>
>
>
> Richard
>
>
>
> *From:* yan morris [mailto:morrisyan...@gmail.com]
> *Sent:* 06 September 2016 05:30
> *To:* Richard Whitehouse (projectclearwater.org)  projectclearwater.org>
> *Subject:* Re: [Project Clearwater] Is Bono a multi-thread program?
>
>
>
> Hi , Richard
>
> Thanks for your help .
>
>
> I understand the architecture of modules .
>
> However I have a question , I want to my stateful_proxy_server can process
> multiple request or response in the same time.
>
> Could I just create threads , and distribute incoming request to
> particular thread?
>
> That is , without creating new module to do multi-thread .
>
>
>
> Is that possible ?
>
> Thank you very much.
>
> Morris
>
>
>
> 2016-09-02 22:31 GMT+08:00 Richard Whitehouse (projectclearwater.org) <
> richard.whiteho...@projectclearwater.org>:
>
> Morris,
>
>
>
> The thread dispatcher calls pjsip_endpt_process_rx_data to continue
> processing of the message - http://www.pjsip.org/pjsip/
> docs/html/group__PJSIP__ENDPT.htm - this passes it to the other
> registered modules e.g. the stateful proxy module.
>
>
>
> If you are creating a thread and it immediately dies, then it’s possible
> that it’s run out of work to do. Normally threads will programmed to run in
> a loop handling events until terminated.
>
>
>
> Thanks,
>
>
>
> Richard
>
>
>
> *From:* yan morris [mailto:morrisyan...@gmail.com]
> *Sent:* 02 September 2016 07:05
> *To:* Richard Whitehouse (projectclearwater.org)  projectclearwater.org>
> *Subject:* Re: [Project Clearwater] Is Bono a multi-thread program?
>
>
>
> Hi , Richard
>
> Please forgive me that I misunderstand your reply.
>
> I find the thread_dispatcher.cpp file , however , I cannot understand how
> the mod_thread_dispatcher communicate with stateful_proxy_module. Could you
> explain it for me?
>
> In addition , I create the threads , but it would end immediately , do you
> know the reason about this problem?
>
> Thank for your time !
>
> Morris
>
>
>
> 2016-09-01 1:27 GMT+08:00 yan morris :
>
> Hi , Richard
>
> Thank for your reply.
>
> Could you please tell me more detailed about implementing multi-thread on
> Bono ?
>
> Where I can find the codes about creating thread? Because I see the
> Bono.cpp , I only find the high level process
>
> request or response function . Does Clearwater use PJSIP API to make it
> multi-threaded or "mod_stateful_proxy"
>
> is implemented by multi-thread already?
>
> I am trying to use PJSIP write my own sip server as Clearwater's
> Application Server , and I want it multi-threaded .
>
> Therefore I think Bono's source code may help me , but I don't see the
> codes about this.
>
>
> Hope you can give me a hand.
>
> Thank you very much.
>
> Morris
>
>
>
> 2016-09-01 0:16 GMT+08:00 Richard Whitehouse (projectclearwater.org) <
> richard.whiteho...@projectclearwater.org>:
>
> Morris,
>
>
>
> Yes, Bono is multi-threaded. It’s architected with a transport thread
> which handles incoming PJSIP requests and a number of worker threads, which
> SIP messages are dispatched onto.
>
>
>
> The worker threads are created by thread_dispatcher. By default bono has
> one worker thread per CPU core.
>
>
>
> The threads are created using PJSIP’s pj_thread functions.
>
>
>
> Thanks,
>
>
>
> Richard
>
>
>
> *From:* Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org]
> *On Behalf Of *yan morris
> *Sent:* 31 August 2016 17:05
> *To:* clearwater@lists.projectclearwater.org
> *Subject:* [Project Clearwater] Is Bono a multi-thread program?
>
>
>
> Hi ,
>
> I want to know , if Bono is a multi-thread program or not.
>
> I see the source code of Bono , I look at the function
> "proxy_on_rx_request"
>
>
>
> which process incoming request , but I don't see any code is creating a
> new thread when request income .
>
>
> Therefore ,  I want to know if Bono processes request using multi-thread ?
>
> If it does , could you tell where I can find the codes which concerned
> about?
>
> (or It is implemented by PJSIP , please let me know , too)
>
> Thank you very much
>
> Morris
>
>
>
>
>
>
>
>
>
>
> ___
> Clearwater mailing list
> Clearwater@lists.projectclearwater.org
> http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.
> projectclearwater.org
>
>
___
Clearwater mailing list

Re: [Project Clearwater] Is Bono a multi-thread program?

[Richard Whitehouse (projectclearwater.org)]

Morris,

Yes that is possible. You could either spawn a new thread for each request, or 
dispatch it to a worker pool. What you choose to do is really an implementation 
detail of the application server you write, and not really related to how 
Clearwater is structured.

Richard

From: yan morris [mailto:morrisyan...@gmail.com]
Sent: 06 September 2016 05:30
To: Richard Whitehouse (projectclearwater.org) 

Subject: Re: [Project Clearwater] Is Bono a multi-thread program?

Hi , Richard

Thanks for your help .

I understand the architecture of modules .

However I have a question , I want to my stateful_proxy_server can process 
multiple request or response in the same time.

Could I just create threads , and distribute incoming request to particular 
thread?

That is , without creating new module to do multi-thread .

Is that possible ?

Thank you very much.

Morris

2016-09-02 22:31 GMT+08:00 Richard Whitehouse 
(projectclearwater.org) 
>:
Morris,

The thread dispatcher calls pjsip_endpt_process_rx_data to continue processing 
of the message - http://www.pjsip.org/pjsip/docs/html/group__PJSIP__ENDPT.htm - 
this passes it to the other registered modules e.g. the stateful proxy module.

If you are creating a thread and it immediately dies, then it’s possible that 
it’s run out of work to do. Normally threads will programmed to run in a loop 
handling events until terminated.

Thanks,

Richard

From: yan morris [mailto:morrisyan...@gmail.com]
Sent: 02 September 2016 07:05
To: Richard Whitehouse (projectclearwater.org) 
>
Subject: Re: [Project Clearwater] Is Bono a multi-thread program?

Hi , Richard

Please forgive me that I misunderstand your reply.

I find the thread_dispatcher.cpp file , however , I cannot understand how the 
mod_thread_dispatcher communicate with stateful_proxy_module. Could you explain 
it for me?

In addition , I create the threads , but it would end immediately , do you know 
the reason about this problem?

Thank for your time !

Morris


2016-09-01 1:27 GMT+08:00 yan morris 
>:
Hi , Richard

Thank for your reply.

Could you please tell me more detailed about implementing multi-thread on Bono ?

Where I can find the codes about creating thread? Because I see the Bono.cpp , 
I only find the high level process

request or response function . Does Clearwater use PJSIP API to make it 
multi-threaded or "mod_stateful_proxy"

is implemented by multi-thread already?

I am trying to use PJSIP write my own sip server as Clearwater's Application 
Server , and I want it multi-threaded .

Therefore I think Bono's source code may help me , but I don't see the codes 
about this.


Hope you can give me a hand.

Thank you very much.

Morris

2016-09-01 0:16 GMT+08:00 Richard Whitehouse 
(projectclearwater.org) 
>:
Morris,

Yes, Bono is multi-threaded. It’s architected with a transport thread which 
handles incoming PJSIP requests and a number of worker threads, which SIP 
messages are dispatched onto.

The worker threads are created by thread_dispatcher. By default bono has one 
worker thread per CPU core.

The threads are created using PJSIP’s pj_thread functions.

Thanks,

Richard

From: Clearwater 
[mailto:clearwater-boun...@lists.projectclearwater.org]
 On Behalf Of yan morris
Sent: 31 August 2016 17:05
To: 
clearwater@lists.projectclearwater.org
Subject: [Project Clearwater] Is Bono a multi-thread program?

Hi ,

I want to know , if Bono is a multi-thread program or not.

I see the source code of Bono , I look at the function "proxy_on_rx_request"

which process incoming request , but I don't see any code is creating a new 
thread when request income .

Therefore ,  I want to know if Bono processes request using multi-thread ?

If it does , could you tell where I can find the codes which concerned about?

(or It is implemented by PJSIP , please let me know , too)

Thank you very much

Morris






___
Clearwater mailing list
Clearwater@lists.projectclearwater.org
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org

Re: [Project Clearwater] Registration Timeout in 300 sec

[Richard Whitehouse (projectclearwater.org)]

Aravind,

The maximum expiry time that a subscriber can register for is configurable. By 
default, it's 300 seconds, but it can be configured by changing the value of 
reg_max_expires which can be shared in Clearwater's shared configuration. 
There's further documentation on this at 
http://clearwater.readthedocs.io/en/stable/Clearwater_Configuration_Options_Reference.html.

Registering clients can provide a registration of any time up to the maximum 
configured registration expiry.

Thanks,

Richard

From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On 
Behalf Of Shirabur, Aravind Ashok (CMS)
Sent: 06 September 2016 08:32
To: Clearwater@lists.projectclearwater.org
Subject: [Project Clearwater] Registration Timeout in 300 sec

Hi,

We are testing project clear water (clearwater-infrastructure  
1.0-160518.173307) , we observed that IMS core network registration expiry time 
is always 300 sec; it's not considering the Expiry timer in the successful 
Registration message. Any configuration to set the Registration Expiry timer 
value.

Thanks
Aravind
___
Clearwater mailing list
Clearwater@lists.projectclearwater.org
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org

Re: [Project Clearwater] Sprout/Bono/Chronos crashes under stress test

[Richard Whitehouse (projectclearwater.org)]

Stanislav,

Sorry to hear you are having these problems. We run stress regularly against 
Project Clearwater deployments, and we don't see such problems, so it's down to 
a difference in your setup.

We wouldn't ever expect to run stress against an all in one node - it's not 
designed for any particular capacity, it's designed to be use to manual trying 
out Clearwater performance, and trying it out initially.

Instead, as per the Stress Testing instructions, we'd expect stress to either 
be run against a deployment done using Chef, or using a Manual Install, with 
Clearwater deployed on at least six separate boxes with Sprout, Homer, 
Homestead, Ralf, Bono and Ellis all instead on separate servers, with separate 
servers for the Sip Stress Node. We'd expect all of the VMs to have around 
1VCPU and 2GB of RAM as documented in the Manual Install instructions. If 
greater performance is required, we'd expect this to be achieved by increasing 
the number of deployed VMs, based on where the system was stressed, rather than 
increasing the resources assigned to each VM.

The latest dumps you have sent don't represent crashes, they represent 
processes being killed because they become unresponsive to the polling 
mechanism, due to the amount of load on the VM, and the process being unable to 
serve the monitoring request in time.

Can you provide some background on what you are trying to find out from the 
testing you are doing?


Thanks,

Richard

From: Stanislav Khalup [mailto:skha...@virtuozzo.com]
Sent: 06 September 2016 16:14
To: Richard Whitehouse ; 
clearwater@lists.projectclearwater.org
Cc: Denis Plotnikov 
Subject: RE: Sprout/Bono/Chronos crashes under stress test

Hello again,

I'm sorry for spamming  the mail list but we continue our testing and come 
across new crashes. This time we took all-in-one VM and limited CPU to 1 core 
(to avoid Bono crashes). We found out that crashes begin when we hit ~15k open 
live sockets. The thing is we expect that Clearwater IMS will stop processing 
registration/call requests but will preserve ongoing calls but as we saw during 
sipp test but instead most connections are forcefully closed and dump is 
generated. I am adding some more dumps (homer this time) and sip logs. Could 
you please tell me if this kind of behavior during testing is kind of normal, 
expected behavior?

Dumps: https://www.dropbox.com/sh/bl5ghgwrpum6pq9/AAA_UQV7v9NfG3y8q0lOP0Xra?dl=0

BR,
Stanislav Khalup

From: Stanislav Khalup
Sent: Monday, September 5, 2016 5:29 PM
To: 'Richard Whitehouse' 
>; 
'clearwater@lists.projectclearwater.org' 
>
Cc: Denis Plotnikov >
Subject: RE: Sprout/Bono/Chronos crashes under stress test

Hello all,

We continue our stress tests to understand some dependencies behind crashes but 
it seems there is no such thing. This time we installed the latest all-in-one 
node and limited it to 1 CPU. Then we ran many sipp stress tests. We believed 
that in this case we will get application crashes only when load hits some 
considerable level but it seems crashes and actual number of registration 
attempts/calls ongoing are not or poorly related. The latest dumps are place 
here: https://www.dropbox.com/sh/ckjr8oi3rll5y78/AAArJDu7WItDxJjipdDMx7TVa?dl=0 
Is this kind of re-occurring crashes expected from all-in-one node?

BR,
Stanislav Khalup

From: Stanislav Khalup
Sent: Friday, September 2, 2016 7:21 PM
To: 'Richard Whitehouse' 
>; 
clearwater@lists.projectclearwater.org
Cc: Denis Plotnikov >
Subject: RE: Sprout/Bono/Chronos crashes under stress test

Richard,

Thank you very much for your response. Let me add some details.  Initially we 
tested all component in VMs with 2 vCPUs each but after reading the list we 
changed bono config to 1 vCPU, for sprout we left 2 vCPUs. As for bono the 
crashes were somehow resolved but for sprout the situation is the same. We 
experience the same kind of crashes with all-in-one image (with 8vCPU).
As for SNMP statistics I don't know whether this is related or not but we 
couldn't get bono/sprout functional statistics like: The number of incoming 
requests, indexed by time period or The number of requests rejected due to 
overload, indexed by time period. - those metrics were always zero.
I've added a pair of chronos dumps to dropbox folder.  Maybe they can shed some 
more light on the problem: 
https://www.dropbox.com/sh/qjdja9eowgvo1zc/AADm25_pwKNs3gWwBmb0Pzhpa?dl=0

BR,
Stanislav Khalup

From: Richard Whitehouse [mailto:richard.whiteho...@metaswitch.com]
Sent: Friday, September 2, 

Re: [Project Clearwater] Sprout/Bono/Chronos crashes under stress test

[Stanislav Khalup]

Hello again,

I'm sorry for spamming  the mail list but we continue our testing and come 
across new crashes. This time we took all-in-one VM and limited CPU to 1 core 
(to avoid Bono crashes). We found out that crashes begin when we hit ~15k open 
live sockets. The thing is we expect that Clearwater IMS will stop processing 
registration/call requests but will preserve ongoing calls but as we saw during 
sipp test but instead most connections are forcefully closed and dump is 
generated. I am adding some more dumps (homer this time) and sip logs. Could 
you please tell me if this kind of behavior during testing is kind of normal, 
expected behavior?

Dumps: https://www.dropbox.com/sh/bl5ghgwrpum6pq9/AAA_UQV7v9NfG3y8q0lOP0Xra?dl=0

BR,
Stanislav Khalup

From: Stanislav Khalup
Sent: Monday, September 5, 2016 5:29 PM
To: 'Richard Whitehouse' ; 
'clearwater@lists.projectclearwater.org' 

Cc: Denis Plotnikov 
Subject: RE: Sprout/Bono/Chronos crashes under stress test

Hello all,

We continue our stress tests to understand some dependencies behind crashes but 
it seems there is no such thing. This time we installed the latest all-in-one 
node and limited it to 1 CPU. Then we ran many sipp stress tests. We believed 
that in this case we will get application crashes only when load hits some 
considerable level but it seems crashes and actual number of registration 
attempts/calls ongoing are not or poorly related. The latest dumps are place 
here: https://www.dropbox.com/sh/ckjr8oi3rll5y78/AAArJDu7WItDxJjipdDMx7TVa?dl=0 
Is this kind of re-occurring crashes expected from all-in-one node?

BR,
Stanislav Khalup

From: Stanislav Khalup
Sent: Friday, September 2, 2016 7:21 PM
To: 'Richard Whitehouse' 
>; 
clearwater@lists.projectclearwater.org
Cc: Denis Plotnikov >
Subject: RE: Sprout/Bono/Chronos crashes under stress test

Richard,

Thank you very much for your response. Let me add some details.  Initially we 
tested all component in VMs with 2 vCPUs each but after reading the list we 
changed bono config to 1 vCPU, for sprout we left 2 vCPUs. As for bono the 
crashes were somehow resolved but for sprout the situation is the same. We 
experience the same kind of crashes with all-in-one image (with 8vCPU).
As for SNMP statistics I don't know whether this is related or not but we 
couldn't get bono/sprout functional statistics like: The number of incoming 
requests, indexed by time period or The number of requests rejected due to 
overload, indexed by time period. - those metrics were always zero.
I've added a pair of chronos dumps to dropbox folder.  Maybe they can shed some 
more light on the problem: 
https://www.dropbox.com/sh/qjdja9eowgvo1zc/AADm25_pwKNs3gWwBmb0Pzhpa?dl=0

BR,
Stanislav Khalup

From: Richard Whitehouse [mailto:richard.whiteho...@metaswitch.com]
Sent: Friday, September 2, 2016 5:19 PM
To: 
clearwater@lists.projectclearwater.org;
 Stanislav Khalup >
Cc: Denis Plotnikov >
Subject: RE: Sprout/Bono/Chronos crashes under stress test

Stanislav,

I've taken a look at the Sprout crash. It looks like you have are hitting a 
crash in the Net SNMP library we use for alarms and statistics. I've raised an 
issue to track this - https://github.com/Metaswitch/sprout/issues/1527

We've seen similar looking stacks for Bono before on multi-core VMs - e.g. 
under 
http://lists.projectclearwater.org/pipermail/clearwater_lists.projectclearwater.org/2015-January/001986.html

Historically we've scaled up Sprout and Bono by running many single or 
dual-core instances rather than running fewer larger instances - this is because
- we've seen virtualization environments impose per-VM limits on TCP connection 
counts, and obviously Bono has large numbers of TCP connections in a real-world 
scenario
- we only support a single transport thread and, since Bono performs relatively 
little processing per message, and Sprout needs to perform some processing per 
message, it is this that ends up being the bottleneck quite quickly.

Generally we've run single core Bono nodes, and dual core Sprout nodes.

Having said that, we should look into why it's crashing when you're running 
more cores.

Can you give us some description of the scenario you are running under when you 
see this?

You might also find it useful to subscribe to the mailing list so that you 
receive updates when we push out updated releases.

Thanks,

Richard

From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On 
Behalf Of Stanislav Khalup
Sent: 01 September 2016 10:49
To: 

[Project Clearwater] Registration Timeout in 300 sec

[Shirabur, Aravind Ashok (CMS)]

Hi,

We are testing project clear water (clearwater-infrastructure  
1.0-160518.173307) , we observed that IMS core network registration expiry time 
is always 300 sec; it's not considering the Expiry timer in the successful 
Registration message. Any configuration to set the Registration Expiry timer 
value.

Thanks
Aravind
___
Clearwater mailing list
Clearwater@lists.projectclearwater.org
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org