Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Yes. I understand and I check that point specifically.
Thank you for your quick solution.

--
regards,

abdul basit


On Wed, 3 Oct 2018 at 00:37, Bogdan-Andrei Iancu 
wrote:

> Hi Abdul,
>
> Thanks for the feedback. I just want to be sure you understand that the
> change I did does not speed up the loading process, but it simply decouples
> the startup sequence from the data loading process. Shortly OpenSIPS does
> not have to wait for the DB data to be loaded in order to start up.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/02/2018 07:22 PM, Abdul Basit wrote:
>
> Thanks a lot.
> Earlier status
> Opensips service was started in *(real 1m6.474s)*
>
> # time opensipsctl fifo reload_blacklist
> WARNING: database engine not found - tried 'MYSQL'
>
> real0m3.948s
> user0m0.039s
> sys 0m0.013s
>
> time systemctl stop opensips
>
> real0m1.424s
> user0m0.004s
> sys 0m0.000s
>
> *I pulled source from master.*
>
> # time systemctl start opensips
> Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast
> and flexible SIP (RFC3261) server...
> Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on
> Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020
> Oct  2 15:57:41 srouter-215 opensips[15906]: udp:
> x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020
> Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases:
> Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:*
>
> real0m0.066s
> user0m0.003s
> sys 0m0.002s
> root@srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started
> OpenSIPS is a very fast and flexible SIP (RFC3261) server.
>
> /var/log/opensips.log is here ... https://pastebin.com/8md6sBiP
>
> # time opensipsctl fifo reload_blacklist
> WARNING: database engine not found - tried 'MYSQL'
> Oct  2 16:17:24 srouter-215 SIPSrv[16185]:
> INFO:userblacklist:reload_sources: got 1386901 entries from
> 'globalblacklist'
>
> real0m41.551s
> user0m0.037s
> sys 0m0.015s
>
> # time systemctl stop opensips
> Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast
> and flexible SIP (RFC3261) server...
> Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast
> and flexible SIP (RFC3261) server.
>
> real0m12.134s
> user0m0.003s
> sys 0m0.002s
>
>
> Looks like issue has been fixed.. :)
>
> --
> regards,
>
> abdul basit
>
>
> On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu 
> wrote:
>
>> OK, I pushed on master a patch to move the initial data load after the
>> startup sequence. The patch should apply to 2.4 also.
>>
>>
>> https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728
>>
>> While checking the module, I found some issues with it:
>>
>> * before my patch, the load (for same data) was independently done in
>> each opensips process (even if the data is shared) :)
>>
>> * the "reload" is no error pron - if the new data load fails, the old
>> existing data is lost
>>
>> * during the data reload the module has no data, so it cannot operate
>> (it will actually block) - normally it should continue with the old data
>> until the new data is fully loaded and validated.
>>
>> Let me know if the patch makes a different for you and I will backport it
>> to 2.4 too.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
>>
>> Hi,
>>
>> Taking a closer look at the module (to be honest I do not user it, as
>> drouting does a similar job) I see the module is loading the data into
>> memory. It is doing this during the init stage, holding back the entire
>> OpenSIPS startup.
>> Module doing similar heavy db load were changed to do the load
>> post-startup. Let patch the module to decouple the loading from the
>> startup, so that OpenSIPS will start without delays.
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/02/2018 04:04 PM, Jon Abrams wrote:
>>
>> OpenSIPs is loading the global blacklist into a digit trie structure into
>> shared memory on startup (or reload_blacklist command). That causes the
>> startup delay.
>>
>> - Jon
>>
>> On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit  wrote:
>>
>>> Hi Bogdan,
>>>
>>> I don't think so its translating to DB query on run time, instead its
>>> caching the blacklist.
>>> I am saying so because opensips response is quick, where as when I query
>>> manually from DB, it take some time even prefix col. is indexed.
>>> Moreover, while opensips restart i get following warnings in syslog:
>>>
>>>
>>>
>>
>

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Bogdan-Andrei Iancu]

Hi Abdul,

Thanks for the feedback. I just want to be sure you understand that the 
change I did does not speed up the loading process, but it simply 
decouples the startup sequence from the data loading process. Shortly 
OpenSIPS does not have to wait for the DB data to be loaded in order to 
start up.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/02/2018 07:22 PM, Abdul Basit wrote:

Thanks a lot.
Earlier status
Opensips service was started in *(real 1m6.474s)*

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'

real   0m3.948s
user   0m0.039s
sys0m0.013s

time systemctl stop opensips

real0m1.424s
user   0m0.004s
sys0m0.000s

*I pulled source from master.*

# time systemctl start opensips
Oct 2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very 
fast and flexible SIP (RFC3261) server...

Oct 2 15:57:41 srouter-215 opensips[15906]: Listening on
Oct 2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020
Oct 2 15:57:41 srouter-215 opensips[15906]: udp: 
x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020

Oct 2 15:57:41 srouter-215 opensips[15906]: Aliases:
Oct 2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:*

real   0m0.066s
user   0m0.003s
sys0m0.002s
root@srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started 
OpenSIPS is a very fast and flexible SIP (RFC3261) server.


/var/log/opensips.log is here ... https://pastebin.com/8md6sBiP

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'
Oct 2 16:17:24 srouter-215 SIPSrv[16185]: 
INFO:userblacklist:reload_sources: got 1386901 entries from 
'globalblacklist'


real   0m41.551s
user   0m0.037s
sys0m0.015s

# time systemctl stop opensips
Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very 
fast and flexible SIP (RFC3261) server...
Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very 
fast and flexible SIP (RFC3261) server.


real0m12.134s
user   0m0.003s
sys0m0.002s


Looks like issue has been fixed.. :)

--
regards,

abdul basit


On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu > wrote:


OK, I pushed on master a patch to move the initial data load after
the startup sequence. The patch should apply to 2.4 also.


https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728

While checking the module, I found some issues with it:

* before my patch, the load (for same data) was independently
done in each opensips process (even if the data is shared) :)

* the "reload" is no error pron - if the new data load fails,
the old existing data is lost

* during the data reload the module has no data, so it cannot
operate (it will actually block) - normally it should continue
with the old data until the new data is fully loaded and validated.

Let me know if the patch makes a different for you and I will
backport it to 2.4 too.

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:

Hi,

Taking a closer look at the module (to be honest I do not user
it, as drouting does a similar job) I see the module is loading
the data into memory. It is doing this during the init stage,
holding back the entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load
post-startup. Let patch the module to decouple the loading from
the startup, so that OpenSIPS will start without delays.

Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:

OpenSIPs is loading the global blacklist into a digit trie
structure into shared memory on startup (or reload_blacklist
command). That causes the startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit mailto:basit.e...@gmail.com>> wrote:

Hi Bogdan,

I don't think so its translating to DB query on run time,
instead its caching the blacklist.
I am saying so because opensips response is quick, where as
when I query manually from DB, it take some time even prefix
col. is indexed.
Moreover, while opensips restart i get following warnings in
syslog:






___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Thanks a lot.
Earlier status
Opensips service was started in *(real 1m6.474s)*

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'

real0m3.948s
user0m0.039s
sys 0m0.013s

time systemctl stop opensips

real0m1.424s
user0m0.004s
sys 0m0.000s

*I pulled source from master.*

# time systemctl start opensips
Oct  2 15:57:41 srouter-215 systemd[1]: Starting OpenSIPS is a very fast
and flexible SIP (RFC3261) server...
Oct  2 15:57:41 srouter-215 opensips[15906]: Listening on
Oct  2 15:57:41 srouter-215 opensips[15906]: udp: x.x.x.x [x.x.x.x]:9020
Oct  2 15:57:41 srouter-215 opensips[15906]: udp:
x6:x6:x6:x6:x6:x6 [[x6:x6:0:0:x6:x6:x6:x6]]:9020
Oct  2 15:57:41 srouter-215 opensips[15906]: Aliases:
Oct  2 15:57:41 srouter-215 opensips[15906]: *: x.x.x.x:*

real0m0.066s
user0m0.003s
sys 0m0.002s
root@srouter-215:~# Oct  2 15:57:41 srouter-215 systemd[1]: Started
OpenSIPS is a very fast and flexible SIP (RFC3261) server.

/var/log/opensips.log is here ... https://pastebin.com/8md6sBiP

# time opensipsctl fifo reload_blacklist
WARNING: database engine not found - tried 'MYSQL'
Oct  2 16:17:24 srouter-215 SIPSrv[16185]:
INFO:userblacklist:reload_sources: got 1386901 entries from
'globalblacklist'

real0m41.551s
user0m0.037s
sys 0m0.015s

# time systemctl stop opensips
Oct  2 16:06:38 srouter-215 systemd[1]: Stopping OpenSIPS is a very fast and
flexible SIP (RFC3261) server...
Oct  2 16:06:50 srouter-215 systemd[1]: Stopped OpenSIPS is a very fast and
flexible SIP (RFC3261) server.

real0m12.134s
user0m0.003s
sys 0m0.002s


Looks like issue has been fixed.. :)

--
regards,

abdul basit


On Tue, 2 Oct 2018 at 18:55, Bogdan-Andrei Iancu 
wrote:

> OK, I pushed on master a patch to move the initial data load after the
> startup sequence. The patch should apply to 2.4 also.
>
>
> https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728
>
> While checking the module, I found some issues with it:
>
> * before my patch, the load (for same data) was independently done in
> each opensips process (even if the data is shared) :)
>
> * the "reload" is no error pron - if the new data load fails, the old
> existing data is lost
>
> * during the data reload the module has no data, so it cannot operate
> (it will actually block) - normally it should continue with the old data
> until the new data is fully loaded and validated.
>
> Let me know if the patch makes a different for you and I will backport it
> to 2.4 too.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:
>
> Hi,
>
> Taking a closer look at the module (to be honest I do not user it, as
> drouting does a similar job) I see the module is loading the data into
> memory. It is doing this during the init stage, holding back the entire
> OpenSIPS startup.
> Module doing similar heavy db load were changed to do the load
> post-startup. Let patch the module to decouple the loading from the
> startup, so that OpenSIPS will start without delays.
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/02/2018 04:04 PM, Jon Abrams wrote:
>
> OpenSIPs is loading the global blacklist into a digit trie structure into
> shared memory on startup (or reload_blacklist command). That causes the
> startup delay.
>
> - Jon
>
> On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit  wrote:
>
>> Hi Bogdan,
>>
>> I don't think so its translating to DB query on run time, instead its
>> caching the blacklist.
>> I am saying so because opensips response is quick, where as when I query
>> manually from DB, it take some time even prefix col. is indexed.
>> Moreover, while opensips restart i get following warnings in syslog:
>>
>>
>>
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Bogdan-Andrei Iancu]

OK, I pushed on master a patch to move the initial data load after the 
startup sequence. The patch should apply to 2.4 also.


https://github.com/OpenSIPS/opensips/commit/32cba1d751ee5a5b705da97ac4333fcad3d67728

While checking the module, I found some issues with it:

* before my patch, the load (for same data) was independently done 
in each opensips process (even if the data is shared) :)


* the "reload" is no error pron - if the new data load fails, the 
old existing data is lost


* during the data reload the module has no data, so it cannot 
operate (it will actually block) - normally it should continue with the 
old data until the new data is fully loaded and validated.


Let me know if the patch makes a different for you and I will backport 
it to 2.4 too.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/02/2018 04:18 PM, Bogdan-Andrei Iancu wrote:

Hi,

Taking a closer look at the module (to be honest I do not user it, as 
drouting does a similar job) I see the module is loading the data into 
memory. It is doing this during the init stage, holding back the 
entire OpenSIPS startup.
Module doing similar heavy db load were changed to do the load 
post-startup. Let patch the module to decouple the loading from the 
startup, so that OpenSIPS will start without delays.


Regards,
Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure 
into shared memory on startup (or reload_blacklist command). That 
causes the startup delay.


- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit > wrote:


Hi Bogdan,

I don't think so its translating to DB query on run time, instead
its caching the blacklist.
I am saying so because opensips response is quick, where as when
I query manually from DB, it take some time even prefix col. is
indexed.
Moreover, while opensips restart i get following warnings in syslog:




___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Bogdan-Andrei Iancu]

Hi,

Taking a closer look at the module (to be honest I do not user it, as 
drouting does a similar job) I see the module is loading the data into 
memory. It is doing this during the init stage, holding back the entire 
OpenSIPS startup.
Module doing similar heavy db load were changed to do the load 
post-startup. Let patch the module to decouple the loading from the 
startup, so that OpenSIPS will start without delays.


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/02/2018 04:04 PM, Jon Abrams wrote:
OpenSIPs is loading the global blacklist into a digit trie structure 
into shared memory on startup (or reload_blacklist command). That 
causes the startup delay.


- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit > wrote:


Hi Bogdan,

I don't think so its translating to DB query on run time, instead
its caching the blacklist.
I am saying so because opensips response is quick, where as when I
query manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog:

Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:timer_ticker: timer task  already scheduled
for 990 ms (now 17920 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:timer_ticker: timer task 
already scheduled for 990 ms (now 17920 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:timer_ticker: timer task  already
scheduled for 990 ms (now 17920 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:timer_ticker: timer task  already
scheduled for 990 ms (now 17920 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:timer_ticker: timer task  already scheduled
for 990 ms (now 17920 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:timer_ticker: timer task  already
scheduled for 990 ms (now 17920 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:utimer_ticker: utimer task  already
scheduled for 100 ms (now 18020 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:utimer_ticker: utimer task  already
scheduled for 100 ms (now 18120 ms), it may overlap..
Oct 2 12:35:28 srouter-215 SIPSrv[20876]:
WARNING:core:utimer_ticker: utimer task  already
scheduled for 100 ms (now 18220 ms), it may overlap..
*Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
INFO:userblacklist:reload_sources: got 1386901 entries from
'globalblacklist' *
Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
WARNING:core:handle_timer_job: utimer job  has a
1817 us delay in execution
Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
WARNING:core:handle_timer_job: timer job  has a 1728
us delay in execution
Oct 2 12:35:28 srouter-215 SIPSrv[20882]:
WARNING:core:handle_timer_job: timer job  has
a 1728 us delay in execution

Once globalblacklist loaded *1386901 entries* , opensips behave
normally.
When I will add more records to globalblacklist, opensips wouldn't
start due to service timeout.
One way might be to increase service timeout to allow opensips
keep loading the entries but it may take ages.

--
regards,

abdul basit

On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu
mailto:bog...@opensips.org>> wrote:

Hi Abdul,

OK, but what is the blacklist related operation you do in
startup route ?? As far as I know, the blacklist module is not
doing any kind of caching, so any check you do against the
blacklists will translate into a DB query, right ?

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/01/2018 06:58 PM, Abdul Basit wrote:

Yes. That's right.
The code I am using is as below. globalblacklist table has
complete number as prefixwith and whitelist status.

route {
.
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not
Allowed. Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu
mailto:bog...@opensips.org>> wrote:

Hi Abdul,

IF I understand correctly, you use startup route to load

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Jon Abrams]

OpenSIPs is loading the global blacklist into a digit trie structure into
shared memory on startup (or reload_blacklist command). That causes the
startup delay.

- Jon

On Tue, Oct 2, 2018 at 7:55 AM Abdul Basit  wrote:

> Hi Bogdan,
>
> I don't think so its translating to DB query on run time, instead its
> caching the blacklist.
> I am saying so because opensips response is quick, where as when I query
> manually from DB, it take some time even prefix col. is indexed.
> Moreover, while opensips restart i get following warnings in syslog:
>
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task  already scheduled for 990 ms (now 17920 ms), it may
> overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task  already scheduled for 990 ms (now 17920
> ms), it may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task  already scheduled for 990 ms (now 17920
> ms), it may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task  already scheduled for 990 ms (now 17920 ms), it
> may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task  already scheduled for 990 ms (now 17920 ms), it may
> overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker:
> timer task  already scheduled for 990 ms (now 17920 ms),
> it may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
> utimer task  already scheduled for 100 ms (now 18020 ms), it
> may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
> utimer task  already scheduled for 100 ms (now 18120 ms), it
> may overlap..
> Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
> utimer task  already scheduled for 100 ms (now 18220 ms), it
> may overlap..
> *Oct  2 12:35:28 srouter-215 SIPSrv[20882]:
> INFO:userblacklist:reload_sources: got 1386901 entries from
> 'globalblacklist' *
> Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
> utimer job  has a 1817 us delay in execution
> Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
> timer job  has a 1728 us delay in execution
> Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
> timer job  has a 1728 us delay in execution
>
> Once globalblacklist loaded *1386901 entries* , opensips behave normally.
> When I will add more records to globalblacklist, opensips wouldn't start
> due to service timeout.
> One way might be to increase service timeout to allow opensips keep
> loading the entries but it may take ages.
>
> --
> regards,
>
> abdul basit
>
> On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu 
> wrote:
>
>> Hi Abdul,
>>
>> OK, but what is the blacklist related operation you do in startup route
>> ?? As far as I know, the blacklist module is not doing any kind of caching,
>> so any check you do against the blacklists will translate into a DB query,
>> right ?
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 10/01/2018 06:58 PM, Abdul Basit wrote:
>>
>> Yes. That's right.
>> The code I am using is as below. globalblacklist table has complete
>> number as prefix with and whitelist status.
>>
>> route {
>> .
>> if ( !check_blacklist("globalblacklist") ) {
>> xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed.
>> Dropping call.\n");
>> sl_send_reply("403", "Forbidden");
>> exit;
>> }
>> else {
>> xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
>> route(10);
>> exit;
>> }
>> .
>> }
>>
>>
>> --
>> regards,
>>
>> abdul basit
>>
>>
>> On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu 
>> wrote:
>>
>>> Hi Abdul,
>>>
>>> IF I understand correctly, you use startup route to load (from DB?) some
>>> blacklists into localcache ?
>>>
>>> Regards,
>>>
>>> Bogdan-Andrei Iancu
>>>
>>> OpenSIPS Founder and Developer
>>>   http://www.opensips-solutions.com
>>> OpenSIPS Bootcamp 2018
>>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>>
>>> On 09/28/2018 04:10 AM, Abdul Basit wrote:
>>>
>>> Hi team,
>>>
>>> I have a opensips 2.4 deployed having 4 children and with mysql
>>> configured for loading user blacklist to opensips local cache on opensips
>>> startup.
>>> Opensips work fine on filtering the call based on user blacklist records
>>> loaded into local cache.
>>>
>>> While blacklist was growing upto 1.3m records, opensips startup was
>>> taking ~60 sec on VM of 16GB RAM and 6 CPU.
>>> I increased shared memory to *6144* and package memory to *256.*
>>> Filter works fine here yet. Call load is not that much only blacklist
>>> entries are growing.
>>>
>>> But now blacklist has grown to very huge number, up to 10m records in
>>> mysql. Opensips is not starting successful

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Hi Bogdan,

I don't think so its translating to DB query on run time, instead its
caching the blacklist.
I am saying so because opensips response is quick, where as when I query
manually from DB, it take some time even prefix col. is indexed.
Moreover, while opensips restart i get following warnings in syslog:

Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer
task  already scheduled for 990 ms (now 17920 ms), it may
overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer
task  already scheduled for 990 ms (now 17920 ms), it
may overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer
task  already scheduled for 990 ms (now 17920 ms), it
may overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer
task  already scheduled for 990 ms (now 17920 ms), it may
overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer
task  already scheduled for 990 ms (now 17920 ms), it may
overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:timer_ticker: timer
task  already scheduled for 990 ms (now 17920 ms), it may
overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
utimer task  already scheduled for 100 ms (now 18020 ms), it may
overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
utimer task  already scheduled for 100 ms (now 18120 ms), it may
overlap..
Oct  2 12:35:28 srouter-215 SIPSrv[20876]: WARNING:core:utimer_ticker:
utimer task  already scheduled for 100 ms (now 18220 ms), it may
overlap..
*Oct  2 12:35:28 srouter-215 SIPSrv[20882]:
INFO:userblacklist:reload_sources: got 1386901 entries from
'globalblacklist' *
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
utimer job  has a 1817 us delay in execution
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
timer job  has a 1728 us delay in execution
Oct  2 12:35:28 srouter-215 SIPSrv[20882]: WARNING:core:handle_timer_job:
timer job  has a 1728 us delay in execution

Once globalblacklist loaded *1386901 entries* , opensips behave normally.
When I will add more records to globalblacklist, opensips wouldn't start
due to service timeout.
One way might be to increase service timeout to allow opensips keep loading
the entries but it may take ages.

--
regards,

abdul basit

On Tue, 2 Oct 2018 at 13:19, Bogdan-Andrei Iancu 
wrote:

> Hi Abdul,
>
> OK, but what is the blacklist related operation you do in startup route ??
> As far as I know, the blacklist module is not doing any kind of caching, so
> any check you do against the blacklists will translate into a DB query,
> right ?
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 10/01/2018 06:58 PM, Abdul Basit wrote:
>
> Yes. That's right.
> The code I am using is as below. globalblacklist table has complete
> number as prefix with and whitelist status.
>
> route {
> .
> if ( !check_blacklist("globalblacklist") ) {
> xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed.
> Dropping call.\n");
> sl_send_reply("403", "Forbidden");
> exit;
> }
> else {
> xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
> route(10);
> exit;
> }
> .
> }
>
>
> --
> regards,
>
> abdul basit
>
>
> On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu 
> wrote:
>
>> Hi Abdul,
>>
>> IF I understand correctly, you use startup route to load (from DB?) some
>> blacklists into localcache ?
>>
>> Regards,
>>
>> Bogdan-Andrei Iancu
>>
>> OpenSIPS Founder and Developer
>>   http://www.opensips-solutions.com
>> OpenSIPS Bootcamp 2018
>>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>>
>> On 09/28/2018 04:10 AM, Abdul Basit wrote:
>>
>> Hi team,
>>
>> I have a opensips 2.4 deployed having 4 children and with mysql
>> configured for loading user blacklist to opensips local cache on opensips
>> startup.
>> Opensips work fine on filtering the call based on user blacklist records
>> loaded into local cache.
>>
>> While blacklist was growing upto 1.3m records, opensips startup was
>> taking ~60 sec on VM of 16GB RAM and 6 CPU.
>> I increased shared memory to *6144* and package memory to *256.*
>> Filter works fine here yet. Call load is not that much only blacklist
>> entries are growing.
>>
>> But now blacklist has grown to very huge number, up to 10m records in
>> mysql. Opensips is not starting successfully and service startup requests
>> timeout after huge CPU spikes while trying to populate local cache from
>> mysql.
>> I increased shared memory to *12288* and package memory to *1024* but no
>> success yet.
>>
>> I indexed mysql blacklist table but no luck.
>>
>> Please advise what settings I should add to opensips for successful
>> startup.
>>
>> --
>> regards,
>>
>> abdul basit
>>
>>
>> __

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Bogdan-Andrei Iancu]

Hi Abdul,

OK, but what is the blacklist related operation you do in startup route 
?? As far as I know, the blacklist module is not doing any kind of 
caching, so any check you do against the blacklists will translate into 
a DB query, right ?


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 10/01/2018 06:58 PM, Abdul Basit wrote:

Yes. That's right.
The code I am using is as below. globalblacklist table has complete 
number as prefixwith and whitelist status.


route {
.
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed. 
Dropping call.\n");

sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu > wrote:


Hi Abdul,

IF I understand correctly, you use startup route to load (from
DB?) some blacklists into localcache ?

Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
   http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
   http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 09/28/2018 04:10 AM, Abdul Basit wrote:

Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql
configured for loading user blacklist to opensips local cache on
opensips startup.
Opensips work fine on filtering the call based on user blacklist
records loaded into local cache.

While blacklist was growing upto 1.3m records, opensips startup
was taking ~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to *6144* and package memory to *256.*
Filter works fine here yet. Call load is not that much only
blacklist entries are growing.

But now blacklist has grown to very huge number, up to 10m
records in mysql. Opensips is not starting successfully and
service startup requests timeout after huge CPU spikes while
trying to populate local cache from mysql.
I increased shared memory to *12288* and package memory to *1024*
but no success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for
successful startup.

--
regards,

abdul basit


___
Users mailing list
Users@lists.opensips.org 
http://lists.opensips.org/cgi-bin/mailman/listinfo/users




___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Yes. That's right.
The code I am using is as below. globalblacklist table has complete number
as prefix with and whitelist status.

route {
.
if ( !check_blacklist("globalblacklist") ) {
xlog("L_NOTICE","[$fU/$tU/$ci] [$rm] Destination [$rU] is not Allowed.
Dropping call.\n");
sl_send_reply("403", "Forbidden");
exit;
}
else {
xlog("L_INFO", "[$fU/$tU/$ci] [$rm] INBOUND CALL for [$ru]");
route(10);
exit;
}
.
}


--
regards,

abdul basit


On Mon, 1 Oct 2018 at 19:07, Bogdan-Andrei Iancu 
wrote:

> Hi Abdul,
>
> IF I understand correctly, you use startup route to load (from DB?) some
> blacklists into localcache ?
>
> Regards,
>
> Bogdan-Andrei Iancu
>
> OpenSIPS Founder and Developer
>   http://www.opensips-solutions.com
> OpenSIPS Bootcamp 2018
>   http://opensips.org/training/OpenSIPS_Bootcamp_2018/
>
> On 09/28/2018 04:10 AM, Abdul Basit wrote:
>
> Hi team,
>
> I have a opensips 2.4 deployed having 4 children and with mysql configured
> for loading user blacklist to opensips local cache on opensips startup.
> Opensips work fine on filtering the call based on user blacklist records
> loaded into local cache.
>
> While blacklist was growing upto 1.3m records, opensips startup was taking
> ~60 sec on VM of 16GB RAM and 6 CPU.
> I increased shared memory to *6144* and package memory to *256.*
> Filter works fine here yet. Call load is not that much only blacklist
> entries are growing.
>
> But now blacklist has grown to very huge number, up to 10m records in
> mysql. Opensips is not starting successfully and service startup requests
> timeout after huge CPU spikes while trying to populate local cache from
> mysql.
> I increased shared memory to *12288* and package memory to *1024* but no
> success yet.
>
> I indexed mysql blacklist table but no luck.
>
> Please advise what settings I should add to opensips for successful
> startup.
>
> --
> regards,
>
> abdul basit
>
>
> ___
> Users mailing 
> listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Bogdan-Andrei Iancu]

Hi Abdul,

IF I understand correctly, you use startup route to load (from DB?) some 
blacklists into localcache ?


Regards,

Bogdan-Andrei Iancu

OpenSIPS Founder and Developer
  http://www.opensips-solutions.com
OpenSIPS Bootcamp 2018
  http://opensips.org/training/OpenSIPS_Bootcamp_2018/

On 09/28/2018 04:10 AM, Abdul Basit wrote:

Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql 
configured for loading user blacklist to opensips local cache on 
opensips startup.
Opensips work fine on filtering the call based on user blacklist 
records loaded into local cache.


While blacklist was growing upto 1.3m records, opensips startup was 
taking ~60 sec on VM of 16GB RAM and 6 CPU.

I increased shared memory to *6144* and package memory to *256.*
Filter works fine here yet. Call load is not that much only blacklist 
entries are growing.


But now blacklist has grown to very huge number, up to 10m records in 
mysql. Opensips is not starting successfully and service startup 
requests timeout after huge CPU spikes while trying to populate local 
cache from mysql.
I increased shared memory to *12288* and package memory to *1024* but 
no success yet.


I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful 
startup.


--
regards,

abdul basit


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Hi Jon,

Thanks, I'll check it out with memcached and Redis both.
Current disturbance is opensips restart due to localcache loading from
mysql. Once loaded, it works like a charm.

Any idea about *-m* and *-M* vales? How much RAM was allocated to the box?

Opensips performed better when I increased shared and package memory.
I want to avoid overkill of resources.

--
regards,

abdul basit

On Sun, 30 Sep 2018 at 20:00, Jon Abrams  wrote:

> On the high CPS box its an old OpenSIPs 1.9 instance running 64 children
> IIRC on a pair of old 6 core Westmere Core Xeons. On a 2.2 box the code
> looks sort of like this:
>
> # memcached config
> modparam("cachedb_memcached", "cachedb_url", "memcached:blacklist://
> 127.0.0.1:11211/")
>
>
> # in the main route block
> if (cache_fetch("memcached: blacklist ", "$avp(key)", $avp(value)))
> {
>#reject the call or something
> }
>
> The only other possible downside to memcached is the lack of longest
> prefix match, which you may not need. You also need to build the Memcached.
>
> I don't have any OpenSIPs Redis code samples handy, as I think I've only
> used it with that other open source SIP server.
>
> Since the data is not being loaded into OpenSIPs memory, startup times are
> considerably faster. With memcached you need to remember or script it to
> reload the blacklist into memory on reboot or restart of memcached.
>
> With Redis you can persist it, although make sure check the performance
> hit when you turn on persistence. AOF isn't bad, especially with a SSD
> disk. The background save can add a bunch of latency when it kicks off.
> If its a lot of data, I usually do a BGSAVE once a day during off-peak
> times with a cron job. But I don't think 20-30 million entries will cause
> much disturbance.
>
> - Jon
>
> On Sun, Sep 30, 2018 at 9:13 AM Abdul Basit  wrote:
>
>> Hi Jon,
>>
>> Good to know.
>> If possible, can you share the settings you have for opensips?
>> Whats the server hardware specification?
>> Whats opensips service startup time?
>>
>> --
>> regards,
>>
>> abdul basit
>>
>> On Sun, 30 Sep 2018 at 05:17, Jon Abrams  wrote:
>>
>>> I have used memached to store 20-30 million entry blacklists while
>>> maintaining 6000-9000 responses per second on a dipping server. The
>>> downside is its not persistent.
>>>
>>> I've used redis as well to store the whole North American LNP database
>>> in memory, but I've only tried pushing it to a couple thousand CPS so far.
>>> I would imagine it would perform at least as well as the memcached if
>>> needed.
>>>
>>> I wouldn't worry about the speed difference between local cache speed
>>> and using Redis/Memcached over a socket in a normal deployment scenario.
>>>
>>> - Jon
>>>
>>> On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit 
>>> wrote:
>>>
 Hi team,

 I am thinking to do horizontal scaling through dispatcher module and
 sharding the blacklist to 2 or 3 opensips nodes local cache.
 or using redis for blacklist repository but local cache is even more
 quick.

 Any comments or experience to share about the problem I am facing
 currently?

 --
 regards,

 abdul basit


 On Fri, 28 Sep 2018 at 06:10, Abdul Basit  wrote:

> Hi team,
>
> I have a opensips 2.4 deployed having 4 children and with mysql
> configured for loading user blacklist to opensips local cache on opensips
> startup.
> Opensips work fine on filtering the call based on user blacklist
> records loaded into local cache.
>
> While blacklist was growing upto 1.3m records, opensips startup was
> taking ~60 sec on VM of 16GB RAM and 6 CPU.
> I increased shared memory to *6144* and package memory to *256.*
> Filter works fine here yet. Call load is not that much only blacklist
> entries are growing.
>
> But now blacklist has grown to very huge number, up to 10m records in
> mysql. Opensips is not starting successfully and service startup requests
> timeout after huge CPU spikes while trying to populate local cache from
> mysql.
> I increased shared memory to *12288* and package memory to *1024* but
> no success yet.
>
> I indexed mysql blacklist table but no luck.
>
> Please advise what settings I should add to opensips for successful
> startup.
>
> --
> regards,
>
> abdul basit
>
 ___
 Users mailing list
 Users@lists.opensips.org
 http://lists.opensips.org/cgi-bin/mailman/listinfo/users

>>> ___
>>> Users mailing list
>>> Users@lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Jon Abrams]

On the high CPS box its an old OpenSIPs 1.9 instance running 64 children
IIRC on a pair of old 6 core Westmere Core Xeons. On a 2.2 box the code
looks sort of like this:

# memcached config
modparam("cachedb_memcached", "cachedb_url", "memcached:blacklist://
127.0.0.1:11211/")


# in the main route block
if (cache_fetch("memcached: blacklist ", "$avp(key)", $avp(value)))
{
   #reject the call or something
}

The only other possible downside to memcached is the lack of longest prefix
match, which you may not need. You also need to build the Memcached.

I don't have any OpenSIPs Redis code samples handy, as I think I've only
used it with that other open source SIP server.

Since the data is not being loaded into OpenSIPs memory, startup times are
considerably faster. With memcached you need to remember or script it to
reload the blacklist into memory on reboot or restart of memcached.

With Redis you can persist it, although make sure check the performance hit
when you turn on persistence. AOF isn't bad, especially with a SSD disk.
The background save can add a bunch of latency when it kicks off.
If its a lot of data, I usually do a BGSAVE once a day during off-peak
times with a cron job. But I don't think 20-30 million entries will cause
much disturbance.

- Jon

On Sun, Sep 30, 2018 at 9:13 AM Abdul Basit  wrote:

> Hi Jon,
>
> Good to know.
> If possible, can you share the settings you have for opensips?
> Whats the server hardware specification?
> Whats opensips service startup time?
>
> --
> regards,
>
> abdul basit
>
> On Sun, 30 Sep 2018 at 05:17, Jon Abrams  wrote:
>
>> I have used memached to store 20-30 million entry blacklists while
>> maintaining 6000-9000 responses per second on a dipping server. The
>> downside is its not persistent.
>>
>> I've used redis as well to store the whole North American LNP database in
>> memory, but I've only tried pushing it to a couple thousand CPS so far. I
>> would imagine it would perform at least as well as the memcached if needed.
>>
>> I wouldn't worry about the speed difference between local cache speed and
>> using Redis/Memcached over a socket in a normal deployment scenario.
>>
>> - Jon
>>
>> On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit  wrote:
>>
>>> Hi team,
>>>
>>> I am thinking to do horizontal scaling through dispatcher module and
>>> sharding the blacklist to 2 or 3 opensips nodes local cache.
>>> or using redis for blacklist repository but local cache is even more
>>> quick.
>>>
>>> Any comments or experience to share about the problem I am facing
>>> currently?
>>>
>>> --
>>> regards,
>>>
>>> abdul basit
>>>
>>>
>>> On Fri, 28 Sep 2018 at 06:10, Abdul Basit  wrote:
>>>
 Hi team,

 I have a opensips 2.4 deployed having 4 children and with mysql
 configured for loading user blacklist to opensips local cache on opensips
 startup.
 Opensips work fine on filtering the call based on user blacklist
 records loaded into local cache.

 While blacklist was growing upto 1.3m records, opensips startup was
 taking ~60 sec on VM of 16GB RAM and 6 CPU.
 I increased shared memory to *6144* and package memory to *256.*
 Filter works fine here yet. Call load is not that much only blacklist
 entries are growing.

 But now blacklist has grown to very huge number, up to 10m records in
 mysql. Opensips is not starting successfully and service startup requests
 timeout after huge CPU spikes while trying to populate local cache from
 mysql.
 I increased shared memory to *12288* and package memory to *1024* but
 no success yet.

 I indexed mysql blacklist table but no luck.

 Please advise what settings I should add to opensips for successful
 startup.

 --
 regards,

 abdul basit

>>> ___
>>> Users mailing list
>>> Users@lists.opensips.org
>>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>>
>> ___
>> Users mailing list
>> Users@lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Hi Jon,

Good to know.
If possible, can you share the settings you have for opensips?
Whats the server hardware specification?
Whats opensips service startup time?

--
regards,

abdul basit

On Sun, 30 Sep 2018 at 05:17, Jon Abrams  wrote:

> I have used memached to store 20-30 million entry blacklists while
> maintaining 6000-9000 responses per second on a dipping server. The
> downside is its not persistent.
>
> I've used redis as well to store the whole North American LNP database in
> memory, but I've only tried pushing it to a couple thousand CPS so far. I
> would imagine it would perform at least as well as the memcached if needed.
>
> I wouldn't worry about the speed difference between local cache speed and
> using Redis/Memcached over a socket in a normal deployment scenario.
>
> - Jon
>
> On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit  wrote:
>
>> Hi team,
>>
>> I am thinking to do horizontal scaling through dispatcher module and
>> sharding the blacklist to 2 or 3 opensips nodes local cache.
>> or using redis for blacklist repository but local cache is even more
>> quick.
>>
>> Any comments or experience to share about the problem I am facing
>> currently?
>>
>> --
>> regards,
>>
>> abdul basit
>>
>>
>> On Fri, 28 Sep 2018 at 06:10, Abdul Basit  wrote:
>>
>>> Hi team,
>>>
>>> I have a opensips 2.4 deployed having 4 children and with mysql
>>> configured for loading user blacklist to opensips local cache on opensips
>>> startup.
>>> Opensips work fine on filtering the call based on user blacklist records
>>> loaded into local cache.
>>>
>>> While blacklist was growing upto 1.3m records, opensips startup was
>>> taking ~60 sec on VM of 16GB RAM and 6 CPU.
>>> I increased shared memory to *6144* and package memory to *256.*
>>> Filter works fine here yet. Call load is not that much only blacklist
>>> entries are growing.
>>>
>>> But now blacklist has grown to very huge number, up to 10m records in
>>> mysql. Opensips is not starting successfully and service startup requests
>>> timeout after huge CPU spikes while trying to populate local cache from
>>> mysql.
>>> I increased shared memory to *12288* and package memory to *1024* but
>>> no success yet.
>>>
>>> I indexed mysql blacklist table but no luck.
>>>
>>> Please advise what settings I should add to opensips for successful
>>> startup.
>>>
>>> --
>>> regards,
>>>
>>> abdul basit
>>>
>> ___
>> Users mailing list
>> Users@lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Jon Abrams]

I have used memached to store 20-30 million entry blacklists while
maintaining 6000-9000 responses per second on a dipping server. The
downside is its not persistent.

I've used redis as well to store the whole North American LNP database in
memory, but I've only tried pushing it to a couple thousand CPS so far. I
would imagine it would perform at least as well as the memcached if needed.

I wouldn't worry about the speed difference between local cache speed and
using Redis/Memcached over a socket in a normal deployment scenario.

- Jon

On Sat, Sep 29, 2018 at 6:17 PM Abdul Basit  wrote:

> Hi team,
>
> I am thinking to do horizontal scaling through dispatcher module and
> sharding the blacklist to 2 or 3 opensips nodes local cache.
> or using redis for blacklist repository but local cache is even more
> quick.
>
> Any comments or experience to share about the problem I am facing
> currently?
>
> --
> regards,
>
> abdul basit
>
>
> On Fri, 28 Sep 2018 at 06:10, Abdul Basit  wrote:
>
>> Hi team,
>>
>> I have a opensips 2.4 deployed having 4 children and with mysql
>> configured for loading user blacklist to opensips local cache on opensips
>> startup.
>> Opensips work fine on filtering the call based on user blacklist records
>> loaded into local cache.
>>
>> While blacklist was growing upto 1.3m records, opensips startup was
>> taking ~60 sec on VM of 16GB RAM and 6 CPU.
>> I increased shared memory to *6144* and package memory to *256.*
>> Filter works fine here yet. Call load is not that much only blacklist
>> entries are growing.
>>
>> But now blacklist has grown to very huge number, up to 10m records in
>> mysql. Opensips is not starting successfully and service startup requests
>> timeout after huge CPU spikes while trying to populate local cache from
>> mysql.
>> I increased shared memory to *12288* and package memory to *1024* but no
>> success yet.
>>
>> I indexed mysql blacklist table but no luck.
>>
>> Please advise what settings I should add to opensips for successful
>> startup.
>>
>> --
>> regards,
>>
>> abdul basit
>>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Hi team,

I am thinking to do horizontal scaling through dispatcher module and
sharding the blacklist to 2 or 3 opensips nodes local cache.
or using redis for blacklist repository but local cache is even more quick.

Any comments or experience to share about the problem I am facing currently?

--
regards,

abdul basit


On Fri, 28 Sep 2018 at 06:10, Abdul Basit  wrote:

> Hi team,
>
> I have a opensips 2.4 deployed having 4 children and with mysql configured
> for loading user blacklist to opensips local cache on opensips startup.
> Opensips work fine on filtering the call based on user blacklist records
> loaded into local cache.
>
> While blacklist was growing upto 1.3m records, opensips startup was taking
> ~60 sec on VM of 16GB RAM and 6 CPU.
> I increased shared memory to *6144* and package memory to *256.*
> Filter works fine here yet. Call load is not that much only blacklist
> entries are growing.
>
> But now blacklist has grown to very huge number, up to 10m records in
> mysql. Opensips is not starting successfully and service startup requests
> timeout after huge CPU spikes while trying to populate local cache from
> mysql.
> I increased shared memory to *12288* and package memory to *1024* but no
> success yet.
>
> I indexed mysql blacklist table but no luck.
>
> Please advise what settings I should add to opensips for successful
> startup.
>
> --
> regards,
>
> abdul basit
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

[OpenSIPS-Users] opensips will not start while loading huge blacklist

[Abdul Basit]

Hi team,

I have a opensips 2.4 deployed having 4 children and with mysql configured
for loading user blacklist to opensips local cache on opensips startup.
Opensips work fine on filtering the call based on user blacklist records
loaded into local cache.

While blacklist was growing upto 1.3m records, opensips startup was taking
~60 sec on VM of 16GB RAM and 6 CPU.
I increased shared memory to *6144* and package memory to *256.*
Filter works fine here yet. Call load is not that much only blacklist
entries are growing.

But now blacklist has grown to very huge number, up to 10m records in
mysql. Opensips is not starting successfully and service startup requests
timeout after huge CPU spikes while trying to populate local cache from
mysql.
I increased shared memory to *12288* and package memory to *1024* but no
success yet.

I indexed mysql blacklist table but no luck.

Please advise what settings I should add to opensips for successful
startup.

--
regards,

abdul basit
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users