Re: [OpenSIPS-Users] Help with big amount of data for routing
ok tnx so much Bogdan! br miha On 12/10/2016 14:39, Bogdan-Andrei Iancu wrote: Give it a try. Just be sure you properly adjust the shared memory and note that the loading will take a bit of a time. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:37, Miha wrote: this is great! tnx :) I was thinking that but as it was so much data i did not even want to try it :) I will try and let you know! br miha On 12/10/2016 14:33, Bogdan-Andrei Iancu wrote: Why don't you use DR for that translation. Make a routing group where you put all DIDs (as prefixes) in dr_rules and have the NET_ID as attribute for the rule. And when looking it up: do_routing("group","LC"); See : http://www.opensips.org/html/docs/modules/2.2.x/drouting.html#id295067 DR is more memory efficient and much much faster in the lookup. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:30, Miha wrote: yes. I have in redis like: KEY (DID) and VALUE (NETID), than I am doing lookup in opensips script. I was looking what is the most a appropriate way to do this. As redis is quite good in this cases I choose it but the issue is memory :( tnx miha On 12/10/2016 14:25, Bogdan-Andrei Iancu wrote: So basically you need to determine the NET_ID based on the DID number ? this is what you do in REDIS now ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:09, Miha wrote: Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Give it a try. Just be sure you properly adjust the shared memory and note that the loading will take a bit of a time. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:37, Miha wrote: this is great! tnx :) I was thinking that but as it was so much data i did not even want to try it :) I will try and let you know! br miha On 12/10/2016 14:33, Bogdan-Andrei Iancu wrote: Why don't you use DR for that translation. Make a routing group where you put all DIDs (as prefixes) in dr_rules and have the NET_ID as attribute for the rule. And when looking it up: do_routing("group","LC"); See : http://www.opensips.org/html/docs/modules/2.2.x/drouting.html#id295067 DR is more memory efficient and much much faster in the lookup. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:30, Miha wrote: yes. I have in redis like: KEY (DID) and VALUE (NETID), than I am doing lookup in opensips script. I was looking what is the most a appropriate way to do this. As redis is quite good in this cases I choose it but the issue is memory :( tnx miha On 12/10/2016 14:25, Bogdan-Andrei Iancu wrote: So basically you need to determine the NET_ID based on the DID number ? this is what you do in REDIS now ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:09, Miha wrote: Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
this is great! tnx :) I was thinking that but as it was so much data i did not even want to try it :) I will try and let you know! br miha On 12/10/2016 14:33, Bogdan-Andrei Iancu wrote: Why don't you use DR for that translation. Make a routing group where you put all DIDs (as prefixes) in dr_rules and have the NET_ID as attribute for the rule. And when looking it up: do_routing("group","LC"); See : http://www.opensips.org/html/docs/modules/2.2.x/drouting.html#id295067 DR is more memory efficient and much much faster in the lookup. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:30, Miha wrote: yes. I have in redis like: KEY (DID) and VALUE (NETID), than I am doing lookup in opensips script. I was looking what is the most a appropriate way to do this. As redis is quite good in this cases I choose it but the issue is memory :( tnx miha On 12/10/2016 14:25, Bogdan-Andrei Iancu wrote: So basically you need to determine the NET_ID based on the DID number ? this is what you do in REDIS now ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:09, Miha wrote: Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Why don't you use DR for that translation. Make a routing group where you put all DIDs (as prefixes) in dr_rules and have the NET_ID as attribute for the rule. And when looking it up: do_routing("group","LC"); See : http://www.opensips.org/html/docs/modules/2.2.x/drouting.html#id295067 DR is more memory efficient and much much faster in the lookup. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:30, Miha wrote: yes. I have in redis like: KEY (DID) and VALUE (NETID), than I am doing lookup in opensips script. I was looking what is the most a appropriate way to do this. As redis is quite good in this cases I choose it but the issue is memory :( tnx miha On 12/10/2016 14:25, Bogdan-Andrei Iancu wrote: So basically you need to determine the NET_ID based on the DID number ? this is what you do in REDIS now ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:09, Miha wrote: Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
yes. I have in redis like: KEY (DID) and VALUE (NETID), than I am doing lookup in opensips script. I was looking what is the most a appropriate way to do this. As redis is quite good in this cases I choose it but the issue is memory :( tnx miha On 12/10/2016 14:25, Bogdan-Andrei Iancu wrote: So basically you need to determine the NET_ID based on the DID number ? this is what you do in REDIS now ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:09, Miha wrote: Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
So basically you need to determine the NET_ID based on the DID number ? this is what you do in REDIS now ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 12.10.2016 15:09, Miha wrote: Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Hi Bogdan i missed your email, sorry... Operator need to have number like NET_ID + DID. So routing is based on NET_ID. I manage to get all number in redis (quite a big amount of memory is used :) ).So first I do lookup in redis, get NET_ID and then I am using d_routing based od NET_IDs. br miha On 05/10/2016 15:17, Bogdan-Andrei Iancu wrote: Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Hi Miha, so you have 120M records (NET_ID + DID) - how do you use them from OpenSIPS ? As I fail to understand what are the operations you want to perform over this data. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 09:44, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Hi Miha I have a similar situation, but around 20 M routes. The native routing mecanims wasn't performing well, so I developed a custom mecanism using Opensips scripting. Everything is stored on MySQL database. The best approach was use avp_db_query to get the route, the primary key (and index) of the table is the route prefix and stored as BIG INT, so you have up to 19 digits of routes, which is OK to me. I could achieve more than 100 cps with this method. You have to find the longest route "by hand", so I developed this procedure: DELIMITER $$ CREATE DEFINER=`root`@`localhost` PROCEDURE `getLongestRoute`(IN route VARCHAR(50), OUT bestroute BIGINT, OUT regionid INT) BEGIN DECLARE rotatemp VARCHAR(50); DECLARE tempprefix BIGINT; CREATE TEMPORARY TABLE IF NOT EXISTS temptabrotas ( prefix BIGINT UNSIGNED) ENGINE=HEAP; SET rotatemp = SUBSTRING(route, 1, LENGTH(route)); INSERT INTO temptabrotas (prefix) VALUES (rotatemp); WHILE (LENGTH(rotatemp) > 1) DO SET rotatemp = SUBSTRING(route, 1, LENGTH(rotatemp)-1); INSERT INTO temptabrotas (prefix) VALUES (rotatemp); END WHILE; SELECT routes.prefix, routes.regionid FROM routes INNER JOIN temptabrotas ON routes.prefix = temptabrotas.prefix ORDER BY routes.prefix DESC LIMIT 1 INTO bestroute, regionid; DROP TABLE temptabrotas; END$$ DELIMITER ; Hope it helps. Regards On Wed, Oct 5, 2016 at 4:16 AM, Mihawrote: > Hi Alex > > i tried, but mysql takes so long time for every select. What do u have in > mind? > > > tnx > > miha > > > > On 05/10/2016 08:46, Alex Balashov wrote: > >> Why do you believe that using a traditional RDBM necessarily means slow >> lookups? >> >> On 10/05/2016 02:44 AM, Miha wrote: >> >> HI >>> >>> the is not really opensips issue:) I need somehow to store big amount of >>> data for routing. >>> >>> To every telephone operator I must send RURI like >>> Net_ID+Telephone_number (value indicates to who number belongs to). In >>> this country they have around 120 millions of numbers. >>> >>> After i have all NET_IDs with numbers I will use drouting for routing >>> numbers to right operator based on NET_ID. >>> >>> Here is the issue: >>> - I tried this with redis (lookup must be quick) but this takes so much >>> memory that basically redis brakes everytime in between 50 millions and >>> 70 millions entries >>> - I tried with hash (hset) in redis but did not do any good >>> >>> >>> Do you have any suggestion how to deal with this, what would be the best >>> thing to use? >>> >>> >>> >>> tnx >>> >>> miha >>> >>> >>> ___ >>> Users mailing list >>> Users@lists.opensips.org >>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users >>> >> >> >> > > ___ > Users mailing list > Users@lists.opensips.org > http://lists.opensips.org/cgi-bin/mailman/listinfo/users > ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Hi Alex i tried, but mysql takes so long time for every select. What do u have in mind? tnx miha On 05/10/2016 08:46, Alex Balashov wrote: Why do you believe that using a traditional RDBM necessarily means slow lookups? On 10/05/2016 02:44 AM, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Help with big amount of data for routing
Why do you believe that using a traditional RDBM necessarily means slow lookups? On 10/05/2016 02:44 AM, Miha wrote: HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 (direct) / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
[OpenSIPS-Users] Help with big amount of data for routing
HI the is not really opensips issue:) I need somehow to store big amount of data for routing. To every telephone operator I must send RURI like Net_ID+Telephone_number (value indicates to who number belongs to). In this country they have around 120 millions of numbers. After i have all NET_IDs with numbers I will use drouting for routing numbers to right operator based on NET_ID. Here is the issue: - I tried this with redis (lookup must be quick) but this takes so much memory that basically redis brakes everytime in between 50 millions and 70 millions entries - I tried with hash (hset) in redis but did not do any good Do you have any suggestion how to deal with this, what would be the best thing to use? tnx miha ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users