Re: [OpenSIPS-Users] Working with a dataset larger than dialplan will load.

2019-04-22 Thread John Kiniston 
Thanks guys, I was moving down the path of using redis as an alternative to
the dialplan module and it sounds like that is the right direction to go.

Liviu, There were not any additional errors, I'll see if I saved a copy of
the VM I was exploring using dialplan on and look at the MySQL side of
things.

On Sun, Apr 21, 2019 at 11:51 PM Liviu Chircu  wrote:

> I second this -- strictly discussing implementation, dialplan performs a
> simple (and costly) iteration over up-to-all matching rules for each
> lookup, while drouting builds an internal trie structure which will
> dramatically improve lookup latencies.
>
> Still, regarding the original problem:  are there no additional errors
> which may be relevant?  If yes, then you may have a MySQL server
> configuration issue (notice how it's not able to return the data).  Tuning
> settings like "max_allowed_packet" might fix this.
>
> Liviu Chircu
> OpenSIPS Developerhttp://www.opensips-solutions.com
>
> On 17.04.2019 15:59, Jon Abrams wrote:
>
> If that's the LCAD or LERG databases, I'd put the data in Redis and query
> that. That will give you faster startup times, and won't add much of any
> noticeable latency.
>
> Alternatively you could try to do something with the drouting module,
> which is designed for very large data sets.
>
> - Jon Abrams
>
> On Mon, Apr 8, 2019 at 12:58 PM John Kiniston 
> wrote:
>
>> Good Morning,
>>
>> I am attempting to use dialplan to lookup rate centers on dialed calls to
>> determine if a caller is dialing outside it's local area.
>>
>> If I attempt to use the full data set with 167424 rows opensips fails to
>> start with a failure to query the database.
>>
>> When I use a smaller data set of only 500 rows I'm able to start opensips
>> without errors.
>>
>> I'm not seeing the query time out, if I run the query by hand I get
>> results back from MySQL?
>>
>> Is there a different module I should be attempting to do this with? I was
>> planning to look up the NPANXX pairing for the source and destination of
>> the call and check against the attrs column where I stored the ratecenter.
>>
>>
>> pr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> INFO:db_mysql:connect_with_retry: re-connected successful for
>> 0x7f33bef25a18
>> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> DBG:db_mysql:mysql_raise_event: MySQL status has not changed: connected
>> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> CRITICAL:db_mysql:db_mysql_submit_query: too many mysql server reconnection
>> failures
>> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> ERROR:core:db_do_query: error while submitting query - [select
>> dpid,pr,match_op,match_exp,match_flags,subst_exp,repl_exp,attrs,timerec
>> from dialplan where disabled=0 order by pr]
>> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> ERROR:dialplan:dp_load_db: failed to query database!
>> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> ERROR:dialplan:dp_load_all_db: unable to load dialplan table
>> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
>> ERROR:dialplan:mi_reload_rules: failed to reload database
>> Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
>> DBG:mi_fifo:mi_parse_tree: adding node <> ; val <9>
>> Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
>> DBG:mi_fifo:mi_parse_node: end of input tree
>> Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
>> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
>> Apr  8 13:39:57 federated-sip /sbin/opensips[4666]:
>> DBG:mi_fifo:mi_parse_node: end of input tree
>> Apr  8 13:39:57 federated-sip /sbin/opensips[4666]:
>> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> INFO:db_mysql:switch_state_to_disconnected: disconnect event for
>> 0x7f33bef25a18
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> INFO:db_mysql:reset_all_statements: resetting all statements on
>> connection: (0x7f33bef26538) 0x7f33bef25a18
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> DBG:db_mysql:db_mysql_connect: opening connection: mysql://
>> :@172.16.52.35/opensips
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> DBG:db_mysql:db_mysql_connect: connection type is 172.16.52.35 via TCP/IP
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> DBG:db_mysql:db_mysql_connect: protocol version is 10
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> DBG:db_mysql:db_mysql_connect: server version is 5.5.60-MariaDB
>> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
>> INFO:db_mysql:connect_with_retry: re-connected successful for
>> 0x7f33bef25a18
>> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
>> INFO:db_mysql:switch_state_to_disconnected: disconnect event for
>> 0x7f33bef25a18
>> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
>> INFO:db_mysql:reset_all_statements: resetting all statements on
>> connection: (0x7f33bef26538) 0x7f33bef25a18
>> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
>> DBG:db_mysql:db_mysql

Re: [OpenSIPS-Users] Working with a dataset larger than dialplan will load.

2019-04-21 Thread Liviu Chircu 
I second this -- strictly discussing implementation, dialplan performs a 
simple (and costly) iteration over up-to-all matching rules for each 
lookup, while drouting builds an internal trie structure which will 
dramatically improve lookup latencies.


Still, regarding the original problem:  are there no additional errors 
which may be relevant?  If yes, then you may have a MySQL server 
configuration issue (notice how it's not able to return the data).  
Tuning settings like "max_allowed_packet" might fix this.


Liviu Chircu
OpenSIPS Developer
http://www.opensips-solutions.com

On 17.04.2019 15:59, Jon Abrams wrote:
If that's the LCAD or LERG databases, I'd put the data in Redis and 
query that. That will give you faster startup times, and won't add 
much of any noticeable latency.


Alternatively you could try to do something with the drouting module, 
which is designed for very large data sets.


- Jon Abrams

On Mon, Apr 8, 2019 at 12:58 PM John Kiniston > wrote:


Good Morning,

I am attempting to use dialplan to lookup rate centers on dialed
calls to determine if a caller is dialing outside it's local area.

If I attempt to use the full data set with 167424 rows opensips
fails to start with a failure to query the database.

When I use a smaller data set of only 500 rows I'm able to start
opensips without errors.

I'm not seeing the query time out, if I run the query by hand I
get results back from MySQL?

Is there a different module I should be attempting to do this
with? I was planning to look up the NPANXX pairing for the source
and destination of the call and check against the attrs column
where I stored the ratecenter.


pr  8 13:37:50 federated-sip /sbin/opensips[4666]:
INFO:db_mysql:connect_with_retry: re-connected successful for
0x7f33bef25a18
Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:mysql_raise_event: MySQL status has not changed:
connected
Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
CRITICAL:db_mysql:db_mysql_submit_query: too many mysql server
reconnection failures
Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
ERROR:core:db_do_query: error while submitting query - [select
dpid,pr,match_op,match_exp,match_flags,subst_exp,repl_exp,attrs,timerec
from dialplan where disabled=0 order by pr]
Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
ERROR:dialplan:dp_load_db: failed to query database!
Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
ERROR:dialplan:dp_load_all_db: unable to load dialplan table
Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
ERROR:dialplan:mi_reload_rules: failed to reload database
Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
DBG:mi_fifo:mi_parse_tree: adding node <> ; val <9>
Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
DBG:mi_fifo:mi_parse_node: end of input tree
Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
Apr  8 13:39:57 federated-sip /sbin/opensips[4666]:
DBG:mi_fifo:mi_parse_node: end of input tree
Apr  8 13:39:57 federated-sip /sbin/opensips[4666]:
DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
INFO:db_mysql:switch_state_to_disconnected: disconnect event for
0x7f33bef25a18
Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
INFO:db_mysql:reset_all_statements: resetting all statements on
connection: (0x7f33bef26538) 0x7f33bef25a18
Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:db_mysql_connect: opening connection:
mysql://:@172.16.52.35/opensips

Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:db_mysql_connect: connection type is 172.16.52.35 via
TCP/IP
Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:db_mysql_connect: protocol version is 10
Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:db_mysql_connect: server version is 5.5.60-MariaDB
Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
INFO:db_mysql:connect_with_retry: re-connected successful for
0x7f33bef25a18
Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
INFO:db_mysql:switch_state_to_disconnected: disconnect event for
0x7f33bef25a18
Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
INFO:db_mysql:reset_all_statements: resetting all statements on
connection: (0x7f33bef26538) 0x7f33bef25a18
Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:db_mysql_connect: opening connection:
mysql://:@172.16.52.35/opensips

Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
DBG:db_mysql:db_mysql_connect: connection type is 172.16.52.35 via
TCP

Re: [OpenSIPS-Users] Working with a dataset larger than dialplan will load.

2019-04-17 Thread Jon Abrams 
If that's the LCAD or LERG databases, I'd put the data in Redis and query
that. That will give you faster startup times, and won't add much of any
noticeable latency.

Alternatively you could try to do something with the drouting module, which
is designed for very large data sets.

- Jon Abrams

On Mon, Apr 8, 2019 at 12:58 PM John Kiniston 
wrote:

> Good Morning,
>
> I am attempting to use dialplan to lookup rate centers on dialed calls to
> determine if a caller is dialing outside it's local area.
>
> If I attempt to use the full data set with 167424 rows opensips fails to
> start with a failure to query the database.
>
> When I use a smaller data set of only 500 rows I'm able to start opensips
> without errors.
>
> I'm not seeing the query time out, if I run the query by hand I get
> results back from MySQL?
>
> Is there a different module I should be attempting to do this with? I was
> planning to look up the NPANXX pairing for the source and destination of
> the call and check against the attrs column where I stored the ratecenter.
>
>
> pr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:connect_with_retry: re-connected successful for 0x7f33bef25a18
> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:mysql_raise_event: MySQL status has not changed: connected
> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> CRITICAL:db_mysql:db_mysql_submit_query: too many mysql server reconnection
> failures
> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> ERROR:core:db_do_query: error while submitting query - [select
> dpid,pr,match_op,match_exp,match_flags,subst_exp,repl_exp,attrs,timerec
> from dialplan where disabled=0 order by pr]
> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> ERROR:dialplan:dp_load_db: failed to query database!
> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> ERROR:dialplan:dp_load_all_db: unable to load dialplan table
> Apr  8 13:37:50 federated-sip /sbin/opensips[4666]:
> ERROR:dialplan:mi_reload_rules: failed to reload database
> Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
> DBG:mi_fifo:mi_parse_tree: adding node <> ; val <9>
> Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
> DBG:mi_fifo:mi_parse_node: end of input tree
> Apr  8 13:39:44 federated-sip /sbin/opensips[4666]:
> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
> Apr  8 13:39:57 federated-sip /sbin/opensips[4666]:
> DBG:mi_fifo:mi_parse_node: end of input tree
> Apr  8 13:39:57 federated-sip /sbin/opensips[4666]:
> DBG:mi_fifo:mi_fifo_server: done parsing the mi tree
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:switch_state_to_disconnected: disconnect event for
> 0x7f33bef25a18
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:reset_all_statements: resetting all statements on connection:
> (0x7f33bef26538) 0x7f33bef25a18
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: opening connection: mysql://
> :@172.16.52.35/opensips
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: connection type is 172.16.52.35 via TCP/IP
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: protocol version is 10
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: server version is 5.5.60-MariaDB
> Apr  8 13:39:59 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:connect_with_retry: re-connected successful for 0x7f33bef25a18
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:switch_state_to_disconnected: disconnect event for
> 0x7f33bef25a18
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:reset_all_statements: resetting all statements on connection:
> (0x7f33bef26538) 0x7f33bef25a18
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: opening connection: mysql://
> :@172.16.52.35/opensips
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: connection type is 172.16.52.35 via TCP/IP
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: protocol version is 10
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:db_mysql_connect: server version is 5.5.60-MariaDB
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> INFO:db_mysql:connect_with_retry: re-connected successful for 0x7f33bef25a18
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> DBG:db_mysql:mysql_raise_event: MySQL status has not changed: connected
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> CRITICAL:db_mysql:db_mysql_submit_query: too many mysql server reconnection
> failures
> Apr  8 13:40:01 federated-sip /sbin/opensips[4666]:
> ERROR:core:db_do_query: error while submitting query - [select
> dpid,pr,match_op,match_exp,match_flags,subst_exp,repl_exp,attrs,timerec
> from dialplan where disabled=0 order by pr]
> Apr  8 13:40:01 federated