Re: [OpenSIPS-Users] Major performance issues question

2016-12-21 Thread Schneur Rosenberg 
No this server is running only OpenSIPS and mysql, here is a report of
iotop which I took when the resources was going down

Actual DISK READ:   0.00 B/s | Actual DISK WRITE:  78.77 K/s
  TID  PRIO  USER DISK READ  DISK WRITE  SWAPIN IO>COMMAND
  153 be/3 root0.00 B/s0.00 B/s  0.00 %  0.08 % [jbd2/vda1-8]
 1681 be/4 mysql   0.00 B/s7.88 K/s  0.00 %  0.06 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
 1711 be/4 mysql   0.00 B/s3.94 K/s  0.00 %  0.05 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
  647 be/4 root0.00 B/s3.94 K/s  0.00 %  0.00 % rsyslogd -n
[rs:main Q:Reg]
 2048 be/4 mysql   0.00 B/s0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
1 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % init
2 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kthreadd]
3 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
 2052 be/4 mysql   0.00 B/s0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
5 be/0 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/0:0H]
6 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/u16:0]
7 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [rcu_sched]
8 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [rcu_bh]
9 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [migration/0]
   10 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/0]
   11 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/1]
   12 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [migration/1]
   13 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
 1038 be/4 mysql   0.00 B/s0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
   15 be/0 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/1:0H]
   16 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/2]
   17 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [migration/2]
   18 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
 2051 be/4 opensips0.00 B/s0.00 B/s  0.00 %  0.00 % opensips -P
/var/run/opensips/opensips.pid -m 128 -M 4 -u opensips -g opensips -f
/etc/opensips/opensips.cfg
   20 be/0 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/2:0H]
   21 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/3]
   22 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [migration/3]
   23 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
 2072 be/4 mysql   0.00 B/s0.00 B/s  0.00 %  0.00 % mysqld
--basedir=/usr --datadir=/var/lib/mysql --plug~d.pid
--socket=/var/run/mysqld/mysqld.sock --port=3306
   25 be/0 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/3:0H]
   26 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/4]
   27 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [migration/4]
   28 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
 2077 be/4 opensips0.00 B/s0.00 B/s  0.00 %  0.00 % opensips -P
/var/run/opensips/opensips.pid -m 128 -M 4 -u opensips -g opensips -f
/etc/opensips/opensips.cfg
   30 be/0 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/4:0H]
   31 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/5]
   32 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [migration/5]
   33 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [ksoftirqd/5]
   34 be/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/5:0]
   35 be/0 root0.00 B/s0.00 B/s  0.00 %  0.00 % [kworker/5:0H]
   36 rt/4 root0.00 B/s0.00 B/s  0.00 %  0.00 % [watchdog/6]


On Wed, Dec 21, 2016 at 7:25 PM, Liviu Chircu  wrote:

> Okay, so it seems lots of CPU cycles are spent in the kernel!
>
> Are you sure there isn't any intensive I/O going on? For example, maybe
> you are pushing lots of gateway pings. You could run "iostat 1" to assess
> this.
>
> Regards,
>
> Liviu Chircu
> OpenSIPS Developerhttp://www.opensips-solutions.com
>
> On 21.12.2016 18:58, Schneur Rosenberg wrote:
>
> procs ---memory-- ---swap-- -io -system--
> --cpu-
>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
> wa st
>  7  0  0 9966412 296728 523915200 36000  0  0
> 99  0  0
>  6  0  0 9966684 296728 523915200 020 2075 1569 12 64
> 24  0  0
>  7  0  0 9966588 296728 523915200 020 2023 1408 12 64
> 24  0  0
>  4  0  0 9966492 296728 523915200 0   108 2027 1434 11 63

Re: [OpenSIPS-Users] Major performance issues question

2016-12-21 Thread Schneur Rosenberg 
top - 12:26:45 up 23 days,  7:32,  2 users,  load average: 1.91, 2.08, 2.58
Tasks: 160 total,   5 running, 155 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.9 us, 63.9 sy,  0.0 ni, 23.2 id,  0.0 wa,  0.0 hi,  0.0 si,
 0.0 st
KiB Mem:  16469164 total,  6479024 used,  9990140 free,   296740 buffers
KiB Swap:0 total,0 used,0 free.  5268140 cached Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
 6759 opensips  20   0  335392  19512  17552 R 100.0  0.1  13:47.21 opensips
 6746 opensips  20   0  335396  19548  17604 R  99.9  0.1  12:08.76 opensips
 6760 opensips  20   0  335396  19728  17736 R  99.5  0.1  16:17.60 opensips
 6743 opensips  20   0  335404  19704  17740 S  57.6  0.1   9:05.29 opensips
 6753 opensips  20   0  335404  19576  17640 S  57.6  0.1  11:53.10 opensips
 6745 opensips  20   0  335388  19608  17648 S  57.3  0.1  11:07.58 opensips
 6750 opensips  20   0  335400  19696  17740 S  57.3  0.1   9:09.70 opensips
 6762 opensips  20   0  335388  19604  17652 S  57.3  0.1  12:50.60 opensips


On Wed, Dec 21, 2016 at 6:58 PM, Schneur Rosenberg  wrote:

> procs ---memory-- ---swap-- -io -system--
> --cpu-
>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
> wa st
>  7  0  0 9966412 296728 523915200 36000  0  0
> 99  0  0
>  6  0  0 9966684 296728 523915200 020 2075 1569 12 64
> 24  0  0
>  7  0  0 9966588 296728 523915200 020 2023 1408 12 64
> 24  0  0
>  4  0  0 9966492 296728 523915200 0   108 2027 1434 11 63
> 26  0  0
>  5  0  0 9966588 296728 523915200 044 1687 1232 11 52
> 38  0  0
>  5  0  0 9966620 296728 523915600 072 1704 1136  9 53
> 37  0  0
>  6  0  0 9966492 296728 523915600 040 1874 1133 11 64
> 25  0  0
>  6  0  0 9966524 296728 523915600 020 1794  853 11 65
> 25  0  0
>  6  0  0 9966556 296728 523915600 056 1810  951 11 64
> 25  0  0
>  6  0  0 9966620 296728 523915600 0 0 1802  934 10 65
> 25  0  0
>  7  0  0 9966556 296728 523916000 0  2328 2356 1489 12 76
> 12  0  0
>  7  0  0 9966460 296728 523916000 0 0 2074  929 12 76
> 12  0  0
>  7  0  0 9966588 296728 523916400 0 0 2022  728 11 76
> 12  0  0
>  7  0  0 9966620 296728 523916400 0 0 2015  802 14 74
> 12  0  0
>  5  0  0 9966652 296728 523916400 072 2275 1049 14 81
>  5  0  0
>
>
> On Wed, Dec 21, 2016 at 6:42 PM, Liviu Chircu  wrote:
>
>> Hi Schneur,
>>
>> Those warnings are due to a rather harmless bug, which was fixed and
>> backported ~ two weeks ago.
>>
>> Regarding 100% CPU usage though, could you please run "vmstat 1" for 10
>> seconds and paste the full output? Thanks!
>>
>> Regards,
>>
>> Liviu Chircu
>> OpenSIPS Developerhttp://www.opensips-solutions.com
>>
>> On 21.12.2016 18:17, Schneur Rosenberg wrote:
>>
>> Hi, I'm running a instance of OpenSIPS (just signalling no RTP on this
>> machine) on a DigitalOcean VM, it was running fine for a while and it does
>> not process lots of CPS (4-5 max) and today I was getting complaints of
>> dropped calls, I checked the logs and I saw lots of
>>  WARNING:core:utimer_ticker: utimer task  already scheduled
>> for 3956760 ms (now 3956860 ms), it may overlap
>> and
>>  WARNING:core:handle_timer_job: utimer job  has a 3 us
>> delay in execution
>>
>> I had 4 children and I upped it to 15, I think it helped but not
>> completely solved the problem.
>>
>> When running top I see all of the sudden a few opensips processes jumping
>> to 100% and the idle resources slowly going down sometimes even to 0% then
>> it recovers, I haven't changed anything in my cfg file lately and the
>> traffic has not increased etc.
>>
>> Is there anyway to see in the logs whats taking so much resources? I'm
>> also wondering if its can be a DigitalOcean issue where other users on the
>> same Hyperviser are using very high resources and its stealing my resources.
>>
>> thank you
>>
>>
>> ___
>> Users mailing 
>> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>>
>> ___
>> Users mailing list
>> Users@lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>>
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] Major performance issues question

2016-12-21 Thread Liviu Chircu 

Okay, so it seems lots of CPU cycles are spent in the kernel!

Are you sure there isn't any intensive I/O going on? For example, maybe 
you are pushing lots of gateway pings. You could run "iostat 1" to 
assess this.


Regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 21.12.2016 18:58, Schneur Rosenberg wrote:
procs ---memory-- ---swap-- -io -system-- 
--cpu-
 r  b   swpd   free   buff  cache   si   sobibo in   cs us sy 
id wa st
 7  0  0 9966412 296728 523915200 360  00  0 
 0 99  0  0
 6  0  0 9966684 296728 523915200 020 2075 1569 12 
64 24  0  0
 7  0  0 9966588 296728 523915200 020 2023 1408 12 
64 24  0  0
 4  0  0 9966492 296728 523915200 0   108 2027 1434 11 
63 26  0  0
 5  0  0 9966588 296728 523915200 044 1687 1232 11 
52 38  0  0
 5  0  0 9966620 296728 523915600 072 1704 1136  9 
53 37  0  0
 6  0  0 9966492 296728 523915600 040 1874 1133 11 
64 25  0  0
 6  0  0 9966524 296728 523915600 020 1794  853 11 
65 25  0  0
 6  0  0 9966556 296728 523915600 056 1810  951 11 
64 25  0  0
 6  0  0 9966620 296728 523915600 0 0 1802  934 10 
65 25  0  0
 7  0  0 9966556 296728 523916000 0  2328 2356 1489 12 
76 12  0  0
 7  0  0 9966460 296728 523916000 0 0 2074  929 12 
76 12  0  0
 7  0  0 9966588 296728 523916400 0 0 2022  728 11 
76 12  0  0
 7  0  0 9966620 296728 523916400 0 0 2015  802 14 
74 12  0  0
 5  0  0 9966652 296728 523916400 072 2275 1049 14 
81  5  0  0



On Wed, Dec 21, 2016 at 6:42 PM, Liviu Chircu > wrote:


Hi Schneur,

Those warnings are due to a rather harmless bug, which was fixed
and backported ~ two weeks ago.

Regarding 100% CPU usage though, could you please run "vmstat 1"
for 10 seconds and paste the full output? Thanks!

Regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com 

On 21.12.2016 18:17, Schneur Rosenberg wrote:

Hi, I'm running a instance of OpenSIPS (just signalling no RTP on
this machine) on a DigitalOcean VM, it was running fine for a
while and it does not process lots of CPS (4-5 max) and today I
was getting complaints of dropped calls, I checked the logs and I
saw lots of  WARNING:core:utimer_ticker: utimer task 
already scheduled for 3956760 ms (now 3956860 ms), it may overlap
and  WARNING:core:handle_timer_job: utimer job  has a
3 us delay in execution
I had 4 children and I upped it to 15, I think it helped but not
completely solved the problem. When running top I see all of the
sudden a few opensips processes jumping to 100% and the idle
resources slowly going down sometimes even to 0% then it
recovers, I haven't changed anything in my cfg file lately and
the traffic has not increased etc. Is there anyway to see in the
logs whats taking so much resources? I'm also wondering if its
can be a DigitalOcean issue where other users on the same
Hyperviser are using very high resources and its stealing my
resources.
thank you

___
Users mailing list
Users@lists.opensips.org 
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


___ Users mailing list
Users@lists.opensips.org 
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
 


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] Major performance issues question

2016-12-21 Thread Schneur Rosenberg 
procs ---memory-- ---swap-- -io -system--
--cpu-
 r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
wa st
 7  0  0 9966412 296728 523915200 36000  0  0
99  0  0
 6  0  0 9966684 296728 523915200 020 2075 1569 12 64
24  0  0
 7  0  0 9966588 296728 523915200 020 2023 1408 12 64
24  0  0
 4  0  0 9966492 296728 523915200 0   108 2027 1434 11 63
26  0  0
 5  0  0 9966588 296728 523915200 044 1687 1232 11 52
38  0  0
 5  0  0 9966620 296728 523915600 072 1704 1136  9 53
37  0  0
 6  0  0 9966492 296728 523915600 040 1874 1133 11 64
25  0  0
 6  0  0 9966524 296728 523915600 020 1794  853 11 65
25  0  0
 6  0  0 9966556 296728 523915600 056 1810  951 11 64
25  0  0
 6  0  0 9966620 296728 523915600 0 0 1802  934 10 65
25  0  0
 7  0  0 9966556 296728 523916000 0  2328 2356 1489 12 76
12  0  0
 7  0  0 9966460 296728 523916000 0 0 2074  929 12 76
12  0  0
 7  0  0 9966588 296728 523916400 0 0 2022  728 11 76
12  0  0
 7  0  0 9966620 296728 523916400 0 0 2015  802 14 74
12  0  0
 5  0  0 9966652 296728 523916400 072 2275 1049 14 81
 5  0  0


On Wed, Dec 21, 2016 at 6:42 PM, Liviu Chircu  wrote:

> Hi Schneur,
>
> Those warnings are due to a rather harmless bug, which was fixed and
> backported ~ two weeks ago.
>
> Regarding 100% CPU usage though, could you please run "vmstat 1" for 10
> seconds and paste the full output? Thanks!
>
> Regards,
>
> Liviu Chircu
> OpenSIPS Developerhttp://www.opensips-solutions.com
>
> On 21.12.2016 18:17, Schneur Rosenberg wrote:
>
> Hi, I'm running a instance of OpenSIPS (just signalling no RTP on this
> machine) on a DigitalOcean VM, it was running fine for a while and it does
> not process lots of CPS (4-5 max) and today I was getting complaints of
> dropped calls, I checked the logs and I saw lots of
>  WARNING:core:utimer_ticker: utimer task  already scheduled for
> 3956760 ms (now 3956860 ms), it may overlap
> and
>  WARNING:core:handle_timer_job: utimer job  has a 3 us
> delay in execution
>
> I had 4 children and I upped it to 15, I think it helped but not
> completely solved the problem.
>
> When running top I see all of the sudden a few opensips processes jumping
> to 100% and the idle resources slowly going down sometimes even to 0% then
> it recovers, I haven't changed anything in my cfg file lately and the
> traffic has not increased etc.
>
> Is there anyway to see in the logs whats taking so much resources? I'm
> also wondering if its can be a DigitalOcean issue where other users on the
> same Hyperviser are using very high resources and its stealing my resources.
>
> thank you
>
>
> ___
> Users mailing 
> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] Major performance issues question

2016-12-21 Thread Schneur Rosenberg 
procs ---memory-- ---swap-- -io -system--
--cpu-
 r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id
wa st
 7  0  0 9966412 296728 523915200 36000  0  0
99  0  0
 6  0  0 9966684 296728 523915200 020 2075 1569 12 64
24  0  0
 7  0  0 9966588 296728 523915200 020 2023 1408 12 64
24  0  0
 4  0  0 9966492 296728 523915200 0   108 2027 1434 11 63
26  0  0
 5  0  0 9966588 296728 523915200 044 1687 1232 11 52
38  0  0
 5  0  0 9966620 296728 523915600 072 1704 1136  9 53
37  0  0
 6  0  0 9966492 296728 523915600 040 1874 1133 11 64
25  0  0
 6  0  0 9966524 296728 523915600 020 1794  853 11 65
25  0  0
 6  0  0 9966556 296728 523915600 056 1810  951 11 64
25  0  0
 6  0  0 9966620 296728 523915600 0 0 1802  934 10 65
25  0  0
 7  0  0 9966556 296728 523916000 0  2328 2356 1489 12 76
12  0  0
 7  0  0 9966460 296728 523916000 0 0 2074  929 12 76
12  0  0
 7  0  0 9966588 296728 523916400 0 0 2022  728 11 76
12  0  0
 7  0  0 9966620 296728 523916400 0 0 2015  802 14 74
12  0  0
 5  0  0 9966652 296728 523916400 072 2275 1049 14 81
 5  0  0


On Wed, Dec 21, 2016 at 6:17 PM, Schneur Rosenberg  wrote:

> Hi, I'm running a instance of OpenSIPS (just signalling no RTP on this
> machine) on a DigitalOcean VM, it was running fine for a while and it does
> not process lots of CPS (4-5 max) and today I was getting complaints of
> dropped calls, I checked the logs and I saw lots of
>  WARNING:core:utimer_ticker: utimer task  already scheduled for
> 3956760 ms (now 3956860 ms), it may overlap
> and
>  WARNING:core:handle_timer_job: utimer job  has a 3 us
> delay in execution
>
> I had 4 children and I upped it to 15, I think it helped but not
> completely solved the problem.
>
> When running top I see all of the sudden a few opensips processes jumping
> to 100% and the idle resources slowly going down sometimes even to 0% then
> it recovers, I haven't changed anything in my cfg file lately and the
> traffic has not increased etc.
>
> Is there anyway to see in the logs whats taking so much resources? I'm
> also wondering if its can be a DigitalOcean issue where other users on the
> same Hyperviser are using very high resources and its stealing my resources.
>
> thank you
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] Major performance issues question

2016-12-21 Thread Liviu Chircu 

Hi Schneur,

Those warnings are due to a rather harmless bug, which was fixed and 
backported ~ two weeks ago.


Regarding 100% CPU usage though, could you please run "vmstat 1" for 10 
seconds and paste the full output? Thanks!


Regards,

Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 21.12.2016 18:17, Schneur Rosenberg wrote:
Hi, I'm running a instance of OpenSIPS (just signalling no RTP on this 
machine) on a DigitalOcean VM, it was running fine for a while and it 
does not process lots of CPS (4-5 max) and today I was getting 
complaints of dropped calls, I checked the logs and I saw lots of
 WARNING:core:utimer_ticker: utimer task  already scheduled 
for 3956760 ms (now 3956860 ms), it may overlap

and
 WARNING:core:handle_timer_job: utimer job  has a 3 us 
delay in execution


I had 4 children and I upped it to 15, I think it helped but not 
completely solved the problem.


When running top I see all of the sudden a few opensips processes 
jumping to 100% and the idle resources slowly going down sometimes 
even to 0% then it recovers, I haven't changed anything in my cfg file 
lately and the traffic has not increased etc.


Is there anyway to see in the logs whats taking so much resources? I'm 
also wondering if its can be a DigitalOcean issue where other users on 
the same Hyperviser are using very high resources and its stealing my 
resources.


thank you


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users