Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
++. This might also be useful in ipam. I'll have a look at the patch regardless for the short term. (I had actually suggested something similar on Eugene's patch review.) Carl On Jun 26, 2014 8:14 AM, "Zang MingJie" wrote: > it would be better to make the range increase dynamic, instead of create > all entries at initialize. > > for example, if vxlan range 1~1M is configured, only initialize 1~1K, when > it has been used up, extend the range to 1K~2K, and so on. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
it would be better to make the range increase dynamic, instead of create all entries at initialize. for example, if vxlan range 1~1M is configured, only initialize 1~1K, when it has been used up, extend the range to 1K~2K, and so on. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi everyone, A new change (https://review.openstack.org/101982) has been proposed to improve vxlan pool initiation with an improvement on delete of obsolete unallocated vnis using a unique delete SQL command. I've tested performance with the following (delete only) scenario: vxlan range is changed from 0:10 to 5:10. The scenario code is available here: http://paste.openstack.org/show/84882 50k vnis to deletePostgresql MySQL Sqlite current code 6,0 5,5 5,1 proposed code 3,2 3,3 3,2 The gain is from 40% to 50%. Raw results: http://paste.openstack.org/show/84890 On Mon, Jun 9, 2014 at 3:38 PM, Eugene Nikanorov wrote: > Mike, > > Thanks a lot for your response! > Some comments: > > There's some in-Python filtering following it which does not seem > necessary; the "alloc.vxlan_vni not in vxlan_vnis" phrase > > could just as well be a SQL "NOT IN" expression. > There we have to do specific set intersection between configured ranges > and existing allocation. That could be done in sql, > but that certainly would lead to a huge sql query text as full vxlan range > could consist of 16 millions of ids. > > > The synchronize_session="fetch" is certainly a huge part of the time > spent here > You've actually made a good point about synchronize_session="fetch" which > was obviously misused by me. > It seems to save up to 40% of plain deleting time. > > I've fixed that and get some speedup with deletes for both mysql and > postgress that reduced difference between chunked/non-chunked version: > > 50k vnis to add/deletePg adding vnisPg deleting vnis Pg TotalMysql > adding vnis Mysql deleting vnisMysql totalnon-chunked sql 221537 151530 chuked > in 10020 133314 1428 > > Results of chunked and non-chunked version look closer, but gap increases > with vni range size (based on few tests of 150k vni range) > > So I'm going to fix chunked version that is on review now. If you think > that the benefit doesn't worth complexity - please let me know. > > Thanks, > Eugene. > > On Mon, Jun 9, 2014 at 1:33 AM, Mike Bayer wrote: > >> >> On Jun 7, 2014, at 4:38 PM, Eugene Nikanorov >> wrote: >> >> Hi folks, >> >> There was a small discussion about the better way of doing sql operations >> for vni synchronization with the config. >> Initial proposal was to handle those in chunks. Carl also suggested to >> issue a single sql query. >> I've did some testing with my sql and postgress. >> I've tested the following scenario: vxlan range is changed from >> 5:15 to 0:10 and vice versa. >> That involves adding and deleting 5 vni in each test. >> >> Here are the numbers: >> 50k vnis to add/deletePg adding vnisPg deleting vnis Pg TotalMysql >> adding vnis Mysql deleting vnisMysql totalnon-chunked sql 232245 142034 >> chunked >> in 10020 173714 1731 >> >> I've done about 5 tries to get each number to minimize random floating >> factor (due to swaps, disc or cpu activity or other factors) >> That might be surprising that issuing multiple sql statements instead one >> big is little bit more efficient, so I would appreciate if someone could >> reproduce those numbers. >> Also I'd like to note that part of code that iterates over vnis fetched >> from db is taking 10 seconds both on mysql and postgress and is a part of >> "deleting vnis" numbers. >> In other words, difference between multiple DELETE sql statements and >> single one is even bigger (in percent) than these numbers show. >> >> The code which I used to test is here: >> http://paste.openstack.org/show/83298/ >> Right now the chunked version is commented out, so to switch between >> versions some lines should be commented and some - uncommented. >> >> >> I've taken a look at this, though I'm not at the point where I have >> things set up to run things like this within full context, and I don't know >> that I have any definitive statements to make, but I do have some >> suggestions: >> >> 1. I do tend to chunk things a lot, selects, deletes, inserts, though the >> chunk size I work with is typically more like 1000, rather than 100. When >> chunking, we're looking to select a size that doesn't tend to overload the >> things that are receiving the data (query buffers, structures internal to >> both SQLAlchemy as well as the DBAPI and the relational database), but at >> the same time doesn't lead to too much repetition on the Python side (where >> of course there's a lot of slowness). >> >> 2. Specifically regarding "WHERE x IN (.)", I always chunk those. When >> we use IN with a list of values, we're building an actual SQL string that >> becomes enormous. This puts strain on the database's query engine that is >> not optimized for SQL strings that are hundreds of thousands of characters >> long, and on some backends this size is limited; on Oracle, there's a limit >> of 1000 items. So I'd always chunk this kind of thing. >> >> 3. I'm not sure of the broader context of this code, but in fact placing >> a literal list of items in the IN i
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Mike, Thanks a lot for your response! Some comments: > There’s some in-Python filtering following it which does not seem necessary; the "alloc.vxlan_vni not in vxlan_vnis” phrase > could just as well be a SQL “NOT IN” expression. There we have to do specific set intersection between configured ranges and existing allocation. That could be done in sql, but that certainly would lead to a huge sql query text as full vxlan range could consist of 16 millions of ids. > The synchronize_session=“fetch” is certainly a huge part of the time spent here You've actually made a good point about synchronize_session=“fetch” which was obviously misused by me. It seems to save up to 40% of plain deleting time. I've fixed that and get some speedup with deletes for both mysql and postgress that reduced difference between chunked/non-chunked version: 50k vnis to add/deletePg adding vnisPg deleting vnisPg TotalMysql adding vnisMysql deleting vnisMysql totalnon-chunked sql221537151530chuked in 10020 1333141428 Results of chunked and non-chunked version look closer, but gap increases with vni range size (based on few tests of 150k vni range) So I'm going to fix chunked version that is on review now. If you think that the benefit doesn't worth complexity - please let me know. Thanks, Eugene. On Mon, Jun 9, 2014 at 1:33 AM, Mike Bayer wrote: > > On Jun 7, 2014, at 4:38 PM, Eugene Nikanorov > wrote: > > Hi folks, > > There was a small discussion about the better way of doing sql operations > for vni synchronization with the config. > Initial proposal was to handle those in chunks. Carl also suggested to > issue a single sql query. > I've did some testing with my sql and postgress. > I've tested the following scenario: vxlan range is changed from > 5:15 to 0:10 and vice versa. > That involves adding and deleting 5 vni in each test. > > Here are the numbers: > 50k vnis to add/deletePg adding vnisPg deleting vnis Pg TotalMysql > adding vnis Mysql deleting vnisMysql totalnon-chunked sql 232245 142034 > chunked > in 10020 173714 1731 > > I've done about 5 tries to get each number to minimize random floating > factor (due to swaps, disc or cpu activity or other factors) > That might be surprising that issuing multiple sql statements instead one > big is little bit more efficient, so I would appreciate if someone could > reproduce those numbers. > Also I'd like to note that part of code that iterates over vnis fetched > from db is taking 10 seconds both on mysql and postgress and is a part of > "deleting vnis" numbers. > In other words, difference between multiple DELETE sql statements and > single one is even bigger (in percent) than these numbers show. > > The code which I used to test is here: > http://paste.openstack.org/show/83298/ > Right now the chunked version is commented out, so to switch between > versions some lines should be commented and some - uncommented. > > > I’ve taken a look at this, though I’m not at the point where I have things > set up to run things like this within full context, and I don’t know that I > have any definitive statements to make, but I do have some suggestions: > > 1. I do tend to chunk things a lot, selects, deletes, inserts, though the > chunk size I work with is typically more like 1000, rather than 100. When > chunking, we’re looking to select a size that doesn’t tend to overload the > things that are receiving the data (query buffers, structures internal to > both SQLAlchemy as well as the DBAPI and the relational database), but at > the same time doesn’t lead to too much repetition on the Python side (where > of course there’s a lot of slowness). > > 2. Specifically regarding “WHERE x IN (…..)”, I always chunk those. When > we use IN with a list of values, we’re building an actual SQL string that > becomes enormous. This puts strain on the database’s query engine that is > not optimized for SQL strings that are hundreds of thousands of characters > long, and on some backends this size is limited; on Oracle, there’s a limit > of 1000 items. So I’d always chunk this kind of thing. > > 3. I’m not sure of the broader context of this code, but in fact placing a > literal list of items in the IN in this case seems unnecessary; the > “vmis_to_remove” list itself was just SELECTed two lines above. There’s > some in-Python filtering following it which does not seem necessary; the " > alloc.vxlan_vni not in vxlan_vnis” phrase could just as well be a SQL > “NOT IN” expression. Not sure if determination of the “.allocated” flag > can be done in SQL, if that’s a plain column, then certainly.Again not > sure if this is just an artifact of how the test is done here, but if the > goal is to optimize this code for speed, doing a DELETE…WHERE .. IN (SELECT > ..) is probably better. I see that the SELECT is using a lockmode, but it > would seem that if just the rows we care to DELETE are inlined within the > DELETE itself this wouldn’t be needed either. > > It’s likely
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
On Jun 7, 2014, at 4:38 PM, Eugene Nikanorov wrote: > Hi folks, > > There was a small discussion about the better way of doing sql operations for > vni synchronization with the config. > Initial proposal was to handle those in chunks. Carl also suggested to issue > a single sql query. > I've did some testing with my sql and postgress. > I've tested the following scenario: vxlan range is changed from 5:15 > to 0:10 and vice versa. > That involves adding and deleting 5 vni in each test. > > Here are the numbers: > 50k vnis to add/deletePg adding vnis Pg deleting vnisPg > TotalMysql adding vnisMysql deleting vnisMysql total > non-chunked sql 23 22 45 14 20 34 > chunked in 10020 17 37 14 17 31 > > I've done about 5 tries to get each number to minimize random floating factor > (due to swaps, disc or cpu activity or other factors) > That might be surprising that issuing multiple sql statements instead one big > is little bit more efficient, so I would appreciate if someone could > reproduce those numbers. > Also I'd like to note that part of code that iterates over vnis fetched from > db is taking 10 seconds both on mysql and postgress and is a part of > "deleting vnis" numbers. > In other words, difference between multiple DELETE sql statements and single > one is even bigger (in percent) than these numbers show. > > The code which I used to test is here: http://paste.openstack.org/show/83298/ > Right now the chunked version is commented out, so to switch between versions > some lines should be commented and some - uncommented. I’ve taken a look at this, though I’m not at the point where I have things set up to run things like this within full context, and I don’t know that I have any definitive statements to make, but I do have some suggestions: 1. I do tend to chunk things a lot, selects, deletes, inserts, though the chunk size I work with is typically more like 1000, rather than 100. When chunking, we’re looking to select a size that doesn’t tend to overload the things that are receiving the data (query buffers, structures internal to both SQLAlchemy as well as the DBAPI and the relational database), but at the same time doesn’t lead to too much repetition on the Python side (where of course there’s a lot of slowness). 2. Specifically regarding “WHERE x IN (…..)”, I always chunk those. When we use IN with a list of values, we’re building an actual SQL string that becomes enormous. This puts strain on the database’s query engine that is not optimized for SQL strings that are hundreds of thousands of characters long, and on some backends this size is limited; on Oracle, there’s a limit of 1000 items. So I’d always chunk this kind of thing. 3. I’m not sure of the broader context of this code, but in fact placing a literal list of items in the IN in this case seems unnecessary; the “vmis_to_remove” list itself was just SELECTed two lines above. There’s some in-Python filtering following it which does not seem necessary; the "alloc.vxlan_vni not in vxlan_vnis” phrase could just as well be a SQL “NOT IN” expression. Not sure if determination of the “.allocated” flag can be done in SQL, if that’s a plain column, then certainly.Again not sure if this is just an artifact of how the test is done here, but if the goal is to optimize this code for speed, doing a DELETE…WHERE .. IN (SELECT ..) is probably better. I see that the SELECT is using a lockmode, but it would seem that if just the rows we care to DELETE are inlined within the DELETE itself this wouldn’t be needed either. It’s likely that everything in #3 is pretty obvious already and there’s reasons it’s the way it is, but I’m just learning all of these codebases so feel free to point out more of the background for me. 4. The synchronize_session=“fetch” is certainly a huge part of the time spent here, and it seems unclear why this synchronize is necessary. When I use query.delete() I never use “fetch”; I either have synchronization turned off, as the operation is not dealing with any set of objects already in play, or I use “evaluate” which here is not possible with the IN (though there is a SQLAlchemy ticket for many years to implement “evaluate” using "IN (values)" that is pretty easy to implement, but if the query became an "IN (SELECT …)” that again would not be feasible). 5. I don’t have a great theory on why chunking does better here on the INSERT. My vague notion here is that as with the DELETE, the systems in play do better when they aren’t tasked with building up very large internal buffers for operations, but that’s not something I have the background to prove. These are all just some impressions and as I’m totally new to this code base I may be way off, so please feel to help me get up to speed ! - mike __
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi folks, There was a small discussion about the better way of doing sql operations for vni synchronization with the config. Initial proposal was to handle those in chunks. Carl also suggested to issue a single sql query. I've did some testing with my sql and postgress. I've tested the following scenario: vxlan range is changed from 5:15 to 0:10 and vice versa. That involves adding and deleting 5 vni in each test. Here are the numbers: 50k vnis to add/deletePg adding vnisPg deleting vnisPg TotalMysql adding vnisMysql deleting vnisMysql totalnon-chunked sql232245142034chunked in 100 201737141731 I've done about 5 tries to get each number to minimize random floating factor (due to swaps, disc or cpu activity or other factors) That might be surprising that issuing multiple sql statements instead one big is little bit more efficient, so I would appreciate if someone could reproduce those numbers. Also I'd like to note that part of code that iterates over vnis fetched from db is taking 10 seconds both on mysql and postgress and is a part of "deleting vnis" numbers. In other words, difference between multiple DELETE sql statements and single one is even bigger (in percent) than these numbers show. The code which I used to test is here: http://paste.openstack.org/show/83298/ Right now the chunked version is commented out, so to switch between versions some lines should be commented and some - uncommented. Thanks, Eugene. P.S. I'm also afraid that issuing one big sql statement (and it will be megabytes big) could lead to timeouts/deadlocks just because it will take too much time, how ever I'm not 100% sure about that, it's just my bare concern. On Thu, Jun 5, 2014 at 1:06 PM, Xurong Yang wrote: > great. > I will do more test base on Eugene Nikanorov's modification. > > *Thanks,* > > > 2014-06-05 11:01 GMT+08:00 Isaku Yamahata : > > Wow great. >> I think the same applies to gre type driver. >> so we should create similar one after vxlan case is resolved. >> >> thanks, >> >> >> On Thu, Jun 05, 2014 at 12:36:54AM +0400, >> Eugene Nikanorov wrote: >> >> > We hijacked the vxlan initialization performance thread with ipam! :) >> > I've tried to address initial problem with some simple sqla stuff: >> > https://review.openstack.org/97774 >> > With sqlite it gives ~3x benefit over existing code in master. >> > Need to do a little bit more testing with real backends to make sure >> > parameters are optimal. >> > >> > Thanks, >> > Eugene. >> > >> > >> > On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin >> wrote: >> > >> > > Yes, memcached is a candidate that looks promising. First things >> first, >> > > though. I think we need the abstraction of an ipam interface merged. >> That >> > > will take some more discussion and work on its own. >> > > >> > > Carl >> > > On May 30, 2014 4:37 PM, "Eugene Nikanorov" >> > > wrote: >> > > >> > >> > I was thinking it would be a separate process that would >> communicate over >> > >> the RPC channel or something. >> > >> memcached? >> > >> >> > >> Eugene. >> > >> >> > >> >> > >> On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin >> wrote: >> > >> >> > >>> Eugene, >> > >>> >> > >>> That was part of the "whole new set of complications" that I >> > >>> dismissively waved my hands at. :) >> > >>> >> > >>> I was thinking it would be a separate process that would communicate >> > >>> over the RPC channel or something. More complications come when you >> > >>> think about making this process HA, etc. It would mean going over >> RPC >> > >>> to rabbit to get an allocation which would be slow. But the current >> > >>> implementation is slow. At least going over RPC is greenthread >> > >>> friendly where going to the database doesn't seem to be. >> > >>> >> > >>> Carl >> > >>> >> > >>> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov >> > >>> wrote: >> > >>> > Hi Carl, >> > >>> > >> > >>> > The idea of in-memory storage was discussed for similar problem, >> but >> > >>> might >> > >>> > not work for multiple server deployment. >> > >>> > Some hybrid approach though may be used, I think. >> > >>> > >> > >>> > Thanks, >> > >>> > Eugene. >> > >>> > >> > >>> > >> > >>> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin > > >> > >>> wrote: >> > >>> >> >> > >>> >> This is very similar to IPAM... There is a space of possible >> ids or >> > >>> >> addresses that can grow very large. We need to track the >> allocation >> > >>> >> of individual ids or addresses from that space and be able to >> quickly >> > >>> >> come up with a new allocations and recycle old ones. I've had >> this in >> > >>> >> the back of my mind for a week or two now. >> > >>> >> >> > >>> >> A similar problem came up when the database would get populated >> with >> > >>> >> the entire free space worth of ip addresses to reflect the >> > >>> >> availability of all of the individual addresses. With a large >> space >> > >>> >> (like an ip4 /8 or practically any ip6 subnet) this would take a >> very >> > >>
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
great. I will do more test base on Eugene Nikanorov's modification. *Thanks,* 2014-06-05 11:01 GMT+08:00 Isaku Yamahata : > Wow great. > I think the same applies to gre type driver. > so we should create similar one after vxlan case is resolved. > > thanks, > > > On Thu, Jun 05, 2014 at 12:36:54AM +0400, > Eugene Nikanorov wrote: > > > We hijacked the vxlan initialization performance thread with ipam! :) > > I've tried to address initial problem with some simple sqla stuff: > > https://review.openstack.org/97774 > > With sqlite it gives ~3x benefit over existing code in master. > > Need to do a little bit more testing with real backends to make sure > > parameters are optimal. > > > > Thanks, > > Eugene. > > > > > > On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin > wrote: > > > > > Yes, memcached is a candidate that looks promising. First things > first, > > > though. I think we need the abstraction of an ipam interface merged. > That > > > will take some more discussion and work on its own. > > > > > > Carl > > > On May 30, 2014 4:37 PM, "Eugene Nikanorov" > > > wrote: > > > > > >> > I was thinking it would be a separate process that would > communicate over > > >> the RPC channel or something. > > >> memcached? > > >> > > >> Eugene. > > >> > > >> > > >> On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin > wrote: > > >> > > >>> Eugene, > > >>> > > >>> That was part of the "whole new set of complications" that I > > >>> dismissively waved my hands at. :) > > >>> > > >>> I was thinking it would be a separate process that would communicate > > >>> over the RPC channel or something. More complications come when you > > >>> think about making this process HA, etc. It would mean going over > RPC > > >>> to rabbit to get an allocation which would be slow. But the current > > >>> implementation is slow. At least going over RPC is greenthread > > >>> friendly where going to the database doesn't seem to be. > > >>> > > >>> Carl > > >>> > > >>> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov > > >>> wrote: > > >>> > Hi Carl, > > >>> > > > >>> > The idea of in-memory storage was discussed for similar problem, > but > > >>> might > > >>> > not work for multiple server deployment. > > >>> > Some hybrid approach though may be used, I think. > > >>> > > > >>> > Thanks, > > >>> > Eugene. > > >>> > > > >>> > > > >>> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin > > >>> wrote: > > >>> >> > > >>> >> This is very similar to IPAM... There is a space of possible ids > or > > >>> >> addresses that can grow very large. We need to track the > allocation > > >>> >> of individual ids or addresses from that space and be able to > quickly > > >>> >> come up with a new allocations and recycle old ones. I've had > this in > > >>> >> the back of my mind for a week or two now. > > >>> >> > > >>> >> A similar problem came up when the database would get populated > with > > >>> >> the entire free space worth of ip addresses to reflect the > > >>> >> availability of all of the individual addresses. With a large > space > > >>> >> (like an ip4 /8 or practically any ip6 subnet) this would take a > very > > >>> >> long time or never finish. > > >>> >> > > >>> >> Neutron was a little smarter about this. It compressed > availability > > >>> >> in to availability ranges in a separate table. This solved the > > >>> >> original problem but is not problem free. It turns out that > writing > > >>> >> database operations to manipulate both the allocations table and > the > > >>> >> availability table atomically is very difficult and ends up being > very > > >>> >> slow and has caused us some grief. The free space also gets > > >>> >> fragmented which degrades performance. This is what led me -- > > >>> >> somewhat reluctantly -- to change how IPs get recycled back in to > the > > >>> >> free pool which hasn't been very popular. > > >>> >> > > >>> >> I wonder if we can discuss a good pattern for handling allocations > > >>> >> where the free space can grow very large. We could use the > pattern > > >>> >> for the allocation of both IP addresses, VXlan ids, and other > similar > > >>> >> resource spaces. > > >>> >> > > >>> >> For IPAM, I have been entertaining the idea of creating an > allocation > > >>> >> agent that would manage the availability of IPs in memory rather > than > > >>> >> in the database. I hesitate, because that brings up a whole new > set > > >>> >> of complications. I'm sure there are other potential solutions > that I > > >>> >> haven't yet considered. > > >>> >> > > >>> >> The L3 subteam is currently working on a pluggable IPAM model. > Once > > >>> >> the initial framework for this is done, we can more easily play > around > > >>> >> with changing the underlying IPAM implementation. > > >>> >> > > >>> >> Thoughts? > > >>> >> > > >>> >> Carl > > >>> >> > > >>> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang > > >>> wrote: > > >>> >> > Hi, Folks, > > >>> >> > > > >>> >> > When we configure VXLAN range [1,
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Wow great. I think the same applies to gre type driver. so we should create similar one after vxlan case is resolved. thanks, On Thu, Jun 05, 2014 at 12:36:54AM +0400, Eugene Nikanorov wrote: > We hijacked the vxlan initialization performance thread with ipam! :) > I've tried to address initial problem with some simple sqla stuff: > https://review.openstack.org/97774 > With sqlite it gives ~3x benefit over existing code in master. > Need to do a little bit more testing with real backends to make sure > parameters are optimal. > > Thanks, > Eugene. > > > On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin wrote: > > > Yes, memcached is a candidate that looks promising. First things first, > > though. I think we need the abstraction of an ipam interface merged. That > > will take some more discussion and work on its own. > > > > Carl > > On May 30, 2014 4:37 PM, "Eugene Nikanorov" > > wrote: > > > >> > I was thinking it would be a separate process that would communicate over > >> the RPC channel or something. > >> memcached? > >> > >> Eugene. > >> > >> > >> On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin wrote: > >> > >>> Eugene, > >>> > >>> That was part of the "whole new set of complications" that I > >>> dismissively waved my hands at. :) > >>> > >>> I was thinking it would be a separate process that would communicate > >>> over the RPC channel or something. More complications come when you > >>> think about making this process HA, etc. It would mean going over RPC > >>> to rabbit to get an allocation which would be slow. But the current > >>> implementation is slow. At least going over RPC is greenthread > >>> friendly where going to the database doesn't seem to be. > >>> > >>> Carl > >>> > >>> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov > >>> wrote: > >>> > Hi Carl, > >>> > > >>> > The idea of in-memory storage was discussed for similar problem, but > >>> might > >>> > not work for multiple server deployment. > >>> > Some hybrid approach though may be used, I think. > >>> > > >>> > Thanks, > >>> > Eugene. > >>> > > >>> > > >>> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin > >>> wrote: > >>> >> > >>> >> This is very similar to IPAM... There is a space of possible ids or > >>> >> addresses that can grow very large. We need to track the allocation > >>> >> of individual ids or addresses from that space and be able to quickly > >>> >> come up with a new allocations and recycle old ones. I've had this in > >>> >> the back of my mind for a week or two now. > >>> >> > >>> >> A similar problem came up when the database would get populated with > >>> >> the entire free space worth of ip addresses to reflect the > >>> >> availability of all of the individual addresses. With a large space > >>> >> (like an ip4 /8 or practically any ip6 subnet) this would take a very > >>> >> long time or never finish. > >>> >> > >>> >> Neutron was a little smarter about this. It compressed availability > >>> >> in to availability ranges in a separate table. This solved the > >>> >> original problem but is not problem free. It turns out that writing > >>> >> database operations to manipulate both the allocations table and the > >>> >> availability table atomically is very difficult and ends up being very > >>> >> slow and has caused us some grief. The free space also gets > >>> >> fragmented which degrades performance. This is what led me -- > >>> >> somewhat reluctantly -- to change how IPs get recycled back in to the > >>> >> free pool which hasn't been very popular. > >>> >> > >>> >> I wonder if we can discuss a good pattern for handling allocations > >>> >> where the free space can grow very large. We could use the pattern > >>> >> for the allocation of both IP addresses, VXlan ids, and other similar > >>> >> resource spaces. > >>> >> > >>> >> For IPAM, I have been entertaining the idea of creating an allocation > >>> >> agent that would manage the availability of IPs in memory rather than > >>> >> in the database. I hesitate, because that brings up a whole new set > >>> >> of complications. I'm sure there are other potential solutions that I > >>> >> haven't yet considered. > >>> >> > >>> >> The L3 subteam is currently working on a pluggable IPAM model. Once > >>> >> the initial framework for this is done, we can more easily play around > >>> >> with changing the underlying IPAM implementation. > >>> >> > >>> >> Thoughts? > >>> >> > >>> >> Carl > >>> >> > >>> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang > >>> wrote: > >>> >> > Hi, Folks, > >>> >> > > >>> >> > When we configure VXLAN range [1,16M], neutron-server service costs > >>> long > >>> >> > time and cpu rate is very high(100%) when initiation. One test base > >>> on > >>> >> > postgresql has been verified: more than 1h when VXLAN range is [1, > >>> 1M]. > >>> >> > > >>> >> > So, any good solution about this performance issue? > >>> >> > > >>> >> > Thanks, > >>> >> > Xurong Yang > >>> >> > > >>> >> > > >>> >> > > >>> >> > __
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
This patch packages 500 DB ‘add’ operations into 1. And in my own test, time costed reduces from 90s to 30s. boost! /Yalei From: Eugene Nikanorov [mailto:enikano...@mirantis.com] Sent: Thursday, June 05, 2014 4:37 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation We hijacked the vxlan initialization performance thread with ipam! :) I've tried to address initial problem with some simple sqla stuff: https://review.openstack.org/97774 With sqlite it gives ~3x benefit over existing code in master. Need to do a little bit more testing with real backends to make sure parameters are optimal. Thanks, Eugene. On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin mailto:c...@ecbaldwin.net>> wrote: Yes, memcached is a candidate that looks promising. First things first, though. I think we need the abstraction of an ipam interface merged. That will take some more discussion and work on its own. Carl On May 30, 2014 4:37 PM, "Eugene Nikanorov" mailto:enikano...@mirantis.com>> wrote: > I was thinking it would be a separate process that would communicate over the > RPC channel or something. memcached? Eugene. On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin mailto:c...@ecbaldwin.net>> wrote: Eugene, That was part of the "whole new set of complications" that I dismissively waved my hands at. :) I was thinking it would be a separate process that would communicate over the RPC channel or something. More complications come when you think about making this process HA, etc. It would mean going over RPC to rabbit to get an allocation which would be slow. But the current implementation is slow. At least going over RPC is greenthread friendly where going to the database doesn't seem to be. Carl On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov mailto:enikano...@mirantis.com>> wrote: > Hi Carl, > > The idea of in-memory storage was discussed for similar problem, but might > not work for multiple server deployment. > Some hybrid approach though may be used, I think. > > Thanks, > Eugene. > > > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin > mailto:c...@ecbaldwin.net>> wrote: >> >> This is very similar to IPAM... There is a space of possible ids or >> addresses that can grow very large. We need to track the allocation >> of individual ids or addresses from that space and be able to quickly >> come up with a new allocations and recycle old ones. I've had this in >> the back of my mind for a week or two now. >> >> A similar problem came up when the database would get populated with >> the entire free space worth of ip addresses to reflect the >> availability of all of the individual addresses. With a large space >> (like an ip4 /8 or practically any ip6 subnet) this would take a very >> long time or never finish. >> >> Neutron was a little smarter about this. It compressed availability >> in to availability ranges in a separate table. This solved the >> original problem but is not problem free. It turns out that writing >> database operations to manipulate both the allocations table and the >> availability table atomically is very difficult and ends up being very >> slow and has caused us some grief. The free space also gets >> fragmented which degrades performance. This is what led me -- >> somewhat reluctantly -- to change how IPs get recycled back in to the >> free pool which hasn't been very popular. >> >> I wonder if we can discuss a good pattern for handling allocations >> where the free space can grow very large. We could use the pattern >> for the allocation of both IP addresses, VXlan ids, and other similar >> resource spaces. >> >> For IPAM, I have been entertaining the idea of creating an allocation >> agent that would manage the availability of IPs in memory rather than >> in the database. I hesitate, because that brings up a whole new set >> of complications. I'm sure there are other potential solutions that I >> haven't yet considered. >> >> The L3 subteam is currently working on a pluggable IPAM model. Once >> the initial framework for this is done, we can more easily play around >> with changing the underlying IPAM implementation. >> >> Thoughts? >> >> Carl >> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang >> mailto:ido...@gmail.com>> wrote: >> > Hi, Folks, >> > >> > When we configure VXLAN range [1,16M], neutron-server service costs long >> > time and cpu rate is very high(100%) when initiation. One test base
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
You are right. I did feel a bit bad about hijacking the thread. But, most of discussion was related closely enough that I never decided to fork in to a newer thread. I think I'm done now. I'll have a look at your review and we'll put IPAM to rest for now. :) Carl On Wed, Jun 4, 2014 at 2:36 PM, Eugene Nikanorov wrote: > We hijacked the vxlan initialization performance thread with ipam! :) > I've tried to address initial problem with some simple sqla stuff: > https://review.openstack.org/97774 > With sqlite it gives ~3x benefit over existing code in master. > Need to do a little bit more testing with real backends to make sure > parameters are optimal. > > Thanks, > Eugene. > > > On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin wrote: >> >> Yes, memcached is a candidate that looks promising. First things first, >> though. I think we need the abstraction of an ipam interface merged. That >> will take some more discussion and work on its own. >> >> Carl >> >> On May 30, 2014 4:37 PM, "Eugene Nikanorov" >> wrote: >>> >>> > I was thinking it would be a separate process that would communicate >>> > over the RPC channel or something. >>> memcached? >>> >>> Eugene. >>> >>> >>> On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin wrote: Eugene, That was part of the "whole new set of complications" that I dismissively waved my hands at. :) I was thinking it would be a separate process that would communicate over the RPC channel or something. More complications come when you think about making this process HA, etc. It would mean going over RPC to rabbit to get an allocation which would be slow. But the current implementation is slow. At least going over RPC is greenthread friendly where going to the database doesn't seem to be. Carl On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov wrote: > Hi Carl, > > The idea of in-memory storage was discussed for similar problem, but > might > not work for multiple server deployment. > Some hybrid approach though may be used, I think. > > Thanks, > Eugene. > > > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin > wrote: >> >> This is very similar to IPAM... There is a space of possible ids or >> addresses that can grow very large. We need to track the allocation >> of individual ids or addresses from that space and be able to quickly >> come up with a new allocations and recycle old ones. I've had this >> in >> the back of my mind for a week or two now. >> >> A similar problem came up when the database would get populated with >> the entire free space worth of ip addresses to reflect the >> availability of all of the individual addresses. With a large space >> (like an ip4 /8 or practically any ip6 subnet) this would take a very >> long time or never finish. >> >> Neutron was a little smarter about this. It compressed availability >> in to availability ranges in a separate table. This solved the >> original problem but is not problem free. It turns out that writing >> database operations to manipulate both the allocations table and the >> availability table atomically is very difficult and ends up being >> very >> slow and has caused us some grief. The free space also gets >> fragmented which degrades performance. This is what led me -- >> somewhat reluctantly -- to change how IPs get recycled back in to the >> free pool which hasn't been very popular. >> >> I wonder if we can discuss a good pattern for handling allocations >> where the free space can grow very large. We could use the pattern >> for the allocation of both IP addresses, VXlan ids, and other similar >> resource spaces. >> >> For IPAM, I have been entertaining the idea of creating an allocation >> agent that would manage the availability of IPs in memory rather than >> in the database. I hesitate, because that brings up a whole new set >> of complications. I'm sure there are other potential solutions that >> I >> haven't yet considered. >> >> The L3 subteam is currently working on a pluggable IPAM model. Once >> the initial framework for this is done, we can more easily play >> around >> with changing the underlying IPAM implementation. >> >> Thoughts? >> >> Carl >> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang >> wrote: >> > Hi, Folks, >> > >> > When we configure VXLAN range [1,16M], neutron-server service costs >> > long >> > time and cpu rate is very high(100%) when initiation. One test base >> > on >> > postgresql has been verified: more than 1h when VXLAN range is [1, >> > 1M]. >> > >> > So, any good solution about this performance issue? >> > >>
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
We hijacked the vxlan initialization performance thread with ipam! :) I've tried to address initial problem with some simple sqla stuff: https://review.openstack.org/97774 With sqlite it gives ~3x benefit over existing code in master. Need to do a little bit more testing with real backends to make sure parameters are optimal. Thanks, Eugene. On Thu, Jun 5, 2014 at 12:29 AM, Carl Baldwin wrote: > Yes, memcached is a candidate that looks promising. First things first, > though. I think we need the abstraction of an ipam interface merged. That > will take some more discussion and work on its own. > > Carl > On May 30, 2014 4:37 PM, "Eugene Nikanorov" > wrote: > >> > I was thinking it would be a separate process that would communicate over >> the RPC channel or something. >> memcached? >> >> Eugene. >> >> >> On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin wrote: >> >>> Eugene, >>> >>> That was part of the "whole new set of complications" that I >>> dismissively waved my hands at. :) >>> >>> I was thinking it would be a separate process that would communicate >>> over the RPC channel or something. More complications come when you >>> think about making this process HA, etc. It would mean going over RPC >>> to rabbit to get an allocation which would be slow. But the current >>> implementation is slow. At least going over RPC is greenthread >>> friendly where going to the database doesn't seem to be. >>> >>> Carl >>> >>> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov >>> wrote: >>> > Hi Carl, >>> > >>> > The idea of in-memory storage was discussed for similar problem, but >>> might >>> > not work for multiple server deployment. >>> > Some hybrid approach though may be used, I think. >>> > >>> > Thanks, >>> > Eugene. >>> > >>> > >>> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin >>> wrote: >>> >> >>> >> This is very similar to IPAM... There is a space of possible ids or >>> >> addresses that can grow very large. We need to track the allocation >>> >> of individual ids or addresses from that space and be able to quickly >>> >> come up with a new allocations and recycle old ones. I've had this in >>> >> the back of my mind for a week or two now. >>> >> >>> >> A similar problem came up when the database would get populated with >>> >> the entire free space worth of ip addresses to reflect the >>> >> availability of all of the individual addresses. With a large space >>> >> (like an ip4 /8 or practically any ip6 subnet) this would take a very >>> >> long time or never finish. >>> >> >>> >> Neutron was a little smarter about this. It compressed availability >>> >> in to availability ranges in a separate table. This solved the >>> >> original problem but is not problem free. It turns out that writing >>> >> database operations to manipulate both the allocations table and the >>> >> availability table atomically is very difficult and ends up being very >>> >> slow and has caused us some grief. The free space also gets >>> >> fragmented which degrades performance. This is what led me -- >>> >> somewhat reluctantly -- to change how IPs get recycled back in to the >>> >> free pool which hasn't been very popular. >>> >> >>> >> I wonder if we can discuss a good pattern for handling allocations >>> >> where the free space can grow very large. We could use the pattern >>> >> for the allocation of both IP addresses, VXlan ids, and other similar >>> >> resource spaces. >>> >> >>> >> For IPAM, I have been entertaining the idea of creating an allocation >>> >> agent that would manage the availability of IPs in memory rather than >>> >> in the database. I hesitate, because that brings up a whole new set >>> >> of complications. I'm sure there are other potential solutions that I >>> >> haven't yet considered. >>> >> >>> >> The L3 subteam is currently working on a pluggable IPAM model. Once >>> >> the initial framework for this is done, we can more easily play around >>> >> with changing the underlying IPAM implementation. >>> >> >>> >> Thoughts? >>> >> >>> >> Carl >>> >> >>> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang >>> wrote: >>> >> > Hi, Folks, >>> >> > >>> >> > When we configure VXLAN range [1,16M], neutron-server service costs >>> long >>> >> > time and cpu rate is very high(100%) when initiation. One test base >>> on >>> >> > postgresql has been verified: more than 1h when VXLAN range is [1, >>> 1M]. >>> >> > >>> >> > So, any good solution about this performance issue? >>> >> > >>> >> > Thanks, >>> >> > Xurong Yang >>> >> > >>> >> > >>> >> > >>> >> > ___ >>> >> > OpenStack-dev mailing list >>> >> > OpenStack-dev@lists.openstack.org >>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> > >>> >> >>> >> ___ >>> >> OpenStack-dev mailing list >>> >> OpenStack-dev@lists.openstack.org >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> > >>> > >>> > >>> > __
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Yes, memcached is a candidate that looks promising. First things first, though. I think we need the abstraction of an ipam interface merged. That will take some more discussion and work on its own. Carl On May 30, 2014 4:37 PM, "Eugene Nikanorov" wrote: > > I was thinking it would be a separate process that would communicate over > the RPC channel or something. > memcached? > > Eugene. > > > On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin wrote: > >> Eugene, >> >> That was part of the "whole new set of complications" that I >> dismissively waved my hands at. :) >> >> I was thinking it would be a separate process that would communicate >> over the RPC channel or something. More complications come when you >> think about making this process HA, etc. It would mean going over RPC >> to rabbit to get an allocation which would be slow. But the current >> implementation is slow. At least going over RPC is greenthread >> friendly where going to the database doesn't seem to be. >> >> Carl >> >> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov >> wrote: >> > Hi Carl, >> > >> > The idea of in-memory storage was discussed for similar problem, but >> might >> > not work for multiple server deployment. >> > Some hybrid approach though may be used, I think. >> > >> > Thanks, >> > Eugene. >> > >> > >> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin >> wrote: >> >> >> >> This is very similar to IPAM... There is a space of possible ids or >> >> addresses that can grow very large. We need to track the allocation >> >> of individual ids or addresses from that space and be able to quickly >> >> come up with a new allocations and recycle old ones. I've had this in >> >> the back of my mind for a week or two now. >> >> >> >> A similar problem came up when the database would get populated with >> >> the entire free space worth of ip addresses to reflect the >> >> availability of all of the individual addresses. With a large space >> >> (like an ip4 /8 or practically any ip6 subnet) this would take a very >> >> long time or never finish. >> >> >> >> Neutron was a little smarter about this. It compressed availability >> >> in to availability ranges in a separate table. This solved the >> >> original problem but is not problem free. It turns out that writing >> >> database operations to manipulate both the allocations table and the >> >> availability table atomically is very difficult and ends up being very >> >> slow and has caused us some grief. The free space also gets >> >> fragmented which degrades performance. This is what led me -- >> >> somewhat reluctantly -- to change how IPs get recycled back in to the >> >> free pool which hasn't been very popular. >> >> >> >> I wonder if we can discuss a good pattern for handling allocations >> >> where the free space can grow very large. We could use the pattern >> >> for the allocation of both IP addresses, VXlan ids, and other similar >> >> resource spaces. >> >> >> >> For IPAM, I have been entertaining the idea of creating an allocation >> >> agent that would manage the availability of IPs in memory rather than >> >> in the database. I hesitate, because that brings up a whole new set >> >> of complications. I'm sure there are other potential solutions that I >> >> haven't yet considered. >> >> >> >> The L3 subteam is currently working on a pluggable IPAM model. Once >> >> the initial framework for this is done, we can more easily play around >> >> with changing the underlying IPAM implementation. >> >> >> >> Thoughts? >> >> >> >> Carl >> >> >> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang wrote: >> >> > Hi, Folks, >> >> > >> >> > When we configure VXLAN range [1,16M], neutron-server service costs >> long >> >> > time and cpu rate is very high(100%) when initiation. One test base >> on >> >> > postgresql has been verified: more than 1h when VXLAN range is [1, >> 1M]. >> >> > >> >> > So, any good solution about this performance issue? >> >> > >> >> > Thanks, >> >> > Xurong Yang >> >> > >> >> > >> >> > >> >> > ___ >> >> > OpenStack-dev mailing list >> >> > OpenStack-dev@lists.openstack.org >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >> >> >> >> ___ >> >> OpenStack-dev mailing list >> >> OpenStack-dev@lists.openstack.org >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.o
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Unit tests use sqlite db backend, so it might be much faster than production environment where DB is on different server. Eugene. On Wed, Jun 4, 2014 at 11:14 AM, Wang, Yalei wrote: > Hi, Xurong, > > > > How do you test it in postgresql? Do you use tox to do a unittest and get > the result(1h)? > > > > > > /Yalei > > > > *From:* Xurong Yang [mailto:ido...@gmail.com] > *Sent:* Thursday, May 29, 2014 6:01 PM > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* [openstack-dev] [Neutron] One performance issue about VXLAN > pool initiation > > > > Hi, Folks, > > > > When we configure VXLAN range [1,16M], neutron-server service costs long > time and cpu rate is very high(100%) when initiation. One test base on > postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. > > > > So, any good solution about this performance issue? > > > > Thanks, > > Xurong Yang > > > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi, Xurong, How do you test it in postgresql? Do you use tox to do a unittest and get the result(1h)? /Yalei From: Xurong Yang [mailto:ido...@gmail.com] Sent: Thursday, May 29, 2014 6:01 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation Hi, Folks, When we configure VXLAN range [1,16M], neutron-server service costs long time and cpu rate is very high(100%) when initiation. One test base on postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. So, any good solution about this performance issue? Thanks, Xurong Yang ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi, i have reported a bug[1] [1]https://bugs.launchpad.net/neutron/+bug/1324875 but no better idea about this issue now, maybe need more discussion. any thoughts? :) Xurong Yang 2014-05-31 6:33 GMT+08:00 Eugene Nikanorov : > > I was thinking it would be a separate process that would communicate over > the RPC channel or something. > memcached? > > Eugene. > > > On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin wrote: > >> Eugene, >> >> That was part of the "whole new set of complications" that I >> dismissively waved my hands at. :) >> >> I was thinking it would be a separate process that would communicate >> over the RPC channel or something. More complications come when you >> think about making this process HA, etc. It would mean going over RPC >> to rabbit to get an allocation which would be slow. But the current >> implementation is slow. At least going over RPC is greenthread >> friendly where going to the database doesn't seem to be. >> >> Carl >> >> On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov >> wrote: >> > Hi Carl, >> > >> > The idea of in-memory storage was discussed for similar problem, but >> might >> > not work for multiple server deployment. >> > Some hybrid approach though may be used, I think. >> > >> > Thanks, >> > Eugene. >> > >> > >> > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin >> wrote: >> >> >> >> This is very similar to IPAM... There is a space of possible ids or >> >> addresses that can grow very large. We need to track the allocation >> >> of individual ids or addresses from that space and be able to quickly >> >> come up with a new allocations and recycle old ones. I've had this in >> >> the back of my mind for a week or two now. >> >> >> >> A similar problem came up when the database would get populated with >> >> the entire free space worth of ip addresses to reflect the >> >> availability of all of the individual addresses. With a large space >> >> (like an ip4 /8 or practically any ip6 subnet) this would take a very >> >> long time or never finish. >> >> >> >> Neutron was a little smarter about this. It compressed availability >> >> in to availability ranges in a separate table. This solved the >> >> original problem but is not problem free. It turns out that writing >> >> database operations to manipulate both the allocations table and the >> >> availability table atomically is very difficult and ends up being very >> >> slow and has caused us some grief. The free space also gets >> >> fragmented which degrades performance. This is what led me -- >> >> somewhat reluctantly -- to change how IPs get recycled back in to the >> >> free pool which hasn't been very popular. >> >> >> >> I wonder if we can discuss a good pattern for handling allocations >> >> where the free space can grow very large. We could use the pattern >> >> for the allocation of both IP addresses, VXlan ids, and other similar >> >> resource spaces. >> >> >> >> For IPAM, I have been entertaining the idea of creating an allocation >> >> agent that would manage the availability of IPs in memory rather than >> >> in the database. I hesitate, because that brings up a whole new set >> >> of complications. I'm sure there are other potential solutions that I >> >> haven't yet considered. >> >> >> >> The L3 subteam is currently working on a pluggable IPAM model. Once >> >> the initial framework for this is done, we can more easily play around >> >> with changing the underlying IPAM implementation. >> >> >> >> Thoughts? >> >> >> >> Carl >> >> >> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang wrote: >> >> > Hi, Folks, >> >> > >> >> > When we configure VXLAN range [1,16M], neutron-server service costs >> long >> >> > time and cpu rate is very high(100%) when initiation. One test base >> on >> >> > postgresql has been verified: more than 1h when VXLAN range is [1, >> 1M]. >> >> > >> >> > So, any good solution about this performance issue? >> >> > >> >> > Thanks, >> >> > Xurong Yang >> >> > >> >> > >> >> > >> >> > ___ >> >> > OpenStack-dev mailing list >> >> > OpenStack-dev@lists.openstack.org >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > >> >> >> >> ___ >> >> OpenStack-dev mailing list >> >> OpenStack-dev@lists.openstack.org >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/ope
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
> I was thinking it would be a separate process that would communicate over the RPC channel or something. memcached? Eugene. On Sat, May 31, 2014 at 2:27 AM, Carl Baldwin wrote: > Eugene, > > That was part of the "whole new set of complications" that I > dismissively waved my hands at. :) > > I was thinking it would be a separate process that would communicate > over the RPC channel or something. More complications come when you > think about making this process HA, etc. It would mean going over RPC > to rabbit to get an allocation which would be slow. But the current > implementation is slow. At least going over RPC is greenthread > friendly where going to the database doesn't seem to be. > > Carl > > On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov > wrote: > > Hi Carl, > > > > The idea of in-memory storage was discussed for similar problem, but > might > > not work for multiple server deployment. > > Some hybrid approach though may be used, I think. > > > > Thanks, > > Eugene. > > > > > > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin > wrote: > >> > >> This is very similar to IPAM... There is a space of possible ids or > >> addresses that can grow very large. We need to track the allocation > >> of individual ids or addresses from that space and be able to quickly > >> come up with a new allocations and recycle old ones. I've had this in > >> the back of my mind for a week or two now. > >> > >> A similar problem came up when the database would get populated with > >> the entire free space worth of ip addresses to reflect the > >> availability of all of the individual addresses. With a large space > >> (like an ip4 /8 or practically any ip6 subnet) this would take a very > >> long time or never finish. > >> > >> Neutron was a little smarter about this. It compressed availability > >> in to availability ranges in a separate table. This solved the > >> original problem but is not problem free. It turns out that writing > >> database operations to manipulate both the allocations table and the > >> availability table atomically is very difficult and ends up being very > >> slow and has caused us some grief. The free space also gets > >> fragmented which degrades performance. This is what led me -- > >> somewhat reluctantly -- to change how IPs get recycled back in to the > >> free pool which hasn't been very popular. > >> > >> I wonder if we can discuss a good pattern for handling allocations > >> where the free space can grow very large. We could use the pattern > >> for the allocation of both IP addresses, VXlan ids, and other similar > >> resource spaces. > >> > >> For IPAM, I have been entertaining the idea of creating an allocation > >> agent that would manage the availability of IPs in memory rather than > >> in the database. I hesitate, because that brings up a whole new set > >> of complications. I'm sure there are other potential solutions that I > >> haven't yet considered. > >> > >> The L3 subteam is currently working on a pluggable IPAM model. Once > >> the initial framework for this is done, we can more easily play around > >> with changing the underlying IPAM implementation. > >> > >> Thoughts? > >> > >> Carl > >> > >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang wrote: > >> > Hi, Folks, > >> > > >> > When we configure VXLAN range [1,16M], neutron-server service costs > long > >> > time and cpu rate is very high(100%) when initiation. One test base on > >> > postgresql has been verified: more than 1h when VXLAN range is [1, > 1M]. > >> > > >> > So, any good solution about this performance issue? > >> > > >> > Thanks, > >> > Xurong Yang > >> > > >> > > >> > > >> > ___ > >> > OpenStack-dev mailing list > >> > OpenStack-dev@lists.openstack.org > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Eugene, That was part of the "whole new set of complications" that I dismissively waved my hands at. :) I was thinking it would be a separate process that would communicate over the RPC channel or something. More complications come when you think about making this process HA, etc. It would mean going over RPC to rabbit to get an allocation which would be slow. But the current implementation is slow. At least going over RPC is greenthread friendly where going to the database doesn't seem to be. Carl On Fri, May 30, 2014 at 2:56 PM, Eugene Nikanorov wrote: > Hi Carl, > > The idea of in-memory storage was discussed for similar problem, but might > not work for multiple server deployment. > Some hybrid approach though may be used, I think. > > Thanks, > Eugene. > > > On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin wrote: >> >> This is very similar to IPAM... There is a space of possible ids or >> addresses that can grow very large. We need to track the allocation >> of individual ids or addresses from that space and be able to quickly >> come up with a new allocations and recycle old ones. I've had this in >> the back of my mind for a week or two now. >> >> A similar problem came up when the database would get populated with >> the entire free space worth of ip addresses to reflect the >> availability of all of the individual addresses. With a large space >> (like an ip4 /8 or practically any ip6 subnet) this would take a very >> long time or never finish. >> >> Neutron was a little smarter about this. It compressed availability >> in to availability ranges in a separate table. This solved the >> original problem but is not problem free. It turns out that writing >> database operations to manipulate both the allocations table and the >> availability table atomically is very difficult and ends up being very >> slow and has caused us some grief. The free space also gets >> fragmented which degrades performance. This is what led me -- >> somewhat reluctantly -- to change how IPs get recycled back in to the >> free pool which hasn't been very popular. >> >> I wonder if we can discuss a good pattern for handling allocations >> where the free space can grow very large. We could use the pattern >> for the allocation of both IP addresses, VXlan ids, and other similar >> resource spaces. >> >> For IPAM, I have been entertaining the idea of creating an allocation >> agent that would manage the availability of IPs in memory rather than >> in the database. I hesitate, because that brings up a whole new set >> of complications. I'm sure there are other potential solutions that I >> haven't yet considered. >> >> The L3 subteam is currently working on a pluggable IPAM model. Once >> the initial framework for this is done, we can more easily play around >> with changing the underlying IPAM implementation. >> >> Thoughts? >> >> Carl >> >> On Thu, May 29, 2014 at 4:01 AM, Xurong Yang wrote: >> > Hi, Folks, >> > >> > When we configure VXLAN range [1,16M], neutron-server service costs long >> > time and cpu rate is very high(100%) when initiation. One test base on >> > postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. >> > >> > So, any good solution about this performance issue? >> > >> > Thanks, >> > Xurong Yang >> > >> > >> > >> > ___ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi Carl, The idea of in-memory storage was discussed for similar problem, but might not work for multiple server deployment. Some hybrid approach though may be used, I think. Thanks, Eugene. On Fri, May 30, 2014 at 8:53 PM, Carl Baldwin wrote: > This is very similar to IPAM... There is a space of possible ids or > addresses that can grow very large. We need to track the allocation > of individual ids or addresses from that space and be able to quickly > come up with a new allocations and recycle old ones. I've had this in > the back of my mind for a week or two now. > > A similar problem came up when the database would get populated with > the entire free space worth of ip addresses to reflect the > availability of all of the individual addresses. With a large space > (like an ip4 /8 or practically any ip6 subnet) this would take a very > long time or never finish. > > Neutron was a little smarter about this. It compressed availability > in to availability ranges in a separate table. This solved the > original problem but is not problem free. It turns out that writing > database operations to manipulate both the allocations table and the > availability table atomically is very difficult and ends up being very > slow and has caused us some grief. The free space also gets > fragmented which degrades performance. This is what led me -- > somewhat reluctantly -- to change how IPs get recycled back in to the > free pool which hasn't been very popular. > > I wonder if we can discuss a good pattern for handling allocations > where the free space can grow very large. We could use the pattern > for the allocation of both IP addresses, VXlan ids, and other similar > resource spaces. > > For IPAM, I have been entertaining the idea of creating an allocation > agent that would manage the availability of IPs in memory rather than > in the database. I hesitate, because that brings up a whole new set > of complications. I'm sure there are other potential solutions that I > haven't yet considered. > > The L3 subteam is currently working on a pluggable IPAM model. Once > the initial framework for this is done, we can more easily play around > with changing the underlying IPAM implementation. > > Thoughts? > > Carl > > On Thu, May 29, 2014 at 4:01 AM, Xurong Yang wrote: > > Hi, Folks, > > > > When we configure VXLAN range [1,16M], neutron-server service costs long > > time and cpu rate is very high(100%) when initiation. One test base on > > postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. > > > > So, any good solution about this performance issue? > > > > Thanks, > > Xurong Yang > > > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
This is very similar to IPAM... There is a space of possible ids or addresses that can grow very large. We need to track the allocation of individual ids or addresses from that space and be able to quickly come up with a new allocations and recycle old ones. I've had this in the back of my mind for a week or two now. A similar problem came up when the database would get populated with the entire free space worth of ip addresses to reflect the availability of all of the individual addresses. With a large space (like an ip4 /8 or practically any ip6 subnet) this would take a very long time or never finish. Neutron was a little smarter about this. It compressed availability in to availability ranges in a separate table. This solved the original problem but is not problem free. It turns out that writing database operations to manipulate both the allocations table and the availability table atomically is very difficult and ends up being very slow and has caused us some grief. The free space also gets fragmented which degrades performance. This is what led me -- somewhat reluctantly -- to change how IPs get recycled back in to the free pool which hasn't been very popular. I wonder if we can discuss a good pattern for handling allocations where the free space can grow very large. We could use the pattern for the allocation of both IP addresses, VXlan ids, and other similar resource spaces. For IPAM, I have been entertaining the idea of creating an allocation agent that would manage the availability of IPs in memory rather than in the database. I hesitate, because that brings up a whole new set of complications. I'm sure there are other potential solutions that I haven't yet considered. The L3 subteam is currently working on a pluggable IPAM model. Once the initial framework for this is done, we can more easily play around with changing the underlying IPAM implementation. Thoughts? Carl On Thu, May 29, 2014 at 4:01 AM, Xurong Yang wrote: > Hi, Folks, > > When we configure VXLAN range [1,16M], neutron-server service costs long > time and cpu rate is very high(100%) when initiation. One test base on > postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. > > So, any good solution about this performance issue? > > Thanks, > Xurong Yang > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
I agree with Salvatore, I don't the optimization of that method (and possibly others) requires a BP, but rather a bug. Can you please file one Xurong? Thanks, Kyle On Fri, May 30, 2014 at 3:39 AM, Salvatore Orlando wrote: > It seems that method has some room for optimization, and I suspect the same > logic has been used in other type drivers as well. > If optimization is possible, it might be the case to open a bug for it. > > Salvatore > > > On 30 May 2014 04:58, Xurong Yang wrote: >> >> Hi, >> Thanks for your response, yes, i get the reason, so, That's why i question >> that whether one good solution can have a high performance with a large >> vxlan range? if possible, a blueprint is deserved to consider. >> >> Tanks, >> Xurong Yang >> >> >> 2014-05-29 18:12 GMT+08:00 ZZelle : >> >>> Hi, >>> >>> >>> vxlan network are inserted/verified in DB one by one, which could explain >>> the time required >>> >>> >>> https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/type_vxlan.py#L138-L172 >>> >>> Cédric >>> >>> >>> >>> On Thu, May 29, 2014 at 12:01 PM, Xurong Yang wrote: Hi, Folks, When we configure VXLAN range [1,16M], neutron-server service costs long time and cpu rate is very high(100%) when initiation. One test base on postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. So, any good solution about this performance issue? Thanks, Xurong Yang ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
It seems that method has some room for optimization, and I suspect the same logic has been used in other type drivers as well. If optimization is possible, it might be the case to open a bug for it. Salvatore On 30 May 2014 04:58, Xurong Yang wrote: > Hi, > Thanks for your response, yes, i get the reason, so, That's why i question > that whether one good solution can have a high performance with a large > vxlan range? if possible, a blueprint is deserved to consider. > > Tanks, > Xurong Yang > > > 2014-05-29 18:12 GMT+08:00 ZZelle : > > Hi, >> >> >> vxlan network are inserted/verified in DB one by one, which could explain >> the time required >> >> >> https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/type_vxlan.py#L138-L172 >> >> Cédric >> >> >> >> On Thu, May 29, 2014 at 12:01 PM, Xurong Yang wrote: >> >>> Hi, Folks, >>> >>> When we configure VXLAN range [1,16M], neutron-server service costs long >>> time and cpu rate is very high(100%) when initiation. One test base on >>> postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. >>> >>> So, any good solution about this performance issue? >>> >>> Thanks, >>> Xurong Yang >>> >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi, Thanks for your response, yes, i get the reason, so, That's why i question that whether one good solution can have a high performance with a large vxlan range? if possible, a blueprint is deserved to consider. Tanks, Xurong Yang 2014-05-29 18:12 GMT+08:00 ZZelle : > Hi, > > > vxlan network are inserted/verified in DB one by one, which could explain > the time required > > > https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/type_vxlan.py#L138-L172 > > Cédric > > > > On Thu, May 29, 2014 at 12:01 PM, Xurong Yang wrote: > >> Hi, Folks, >> >> When we configure VXLAN range [1,16M], neutron-server service costs long >> time and cpu rate is very high(100%) when initiation. One test base on >> postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. >> >> So, any good solution about this performance issue? >> >> Thanks, >> Xurong Yang >> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi, vxlan network are inserted/verified in DB one by one, which could explain the time required https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/type_vxlan.py#L138-L172 Cédric On Thu, May 29, 2014 at 12:01 PM, Xurong Yang wrote: > Hi, Folks, > > When we configure VXLAN range [1,16M], neutron-server service costs long > time and cpu rate is very high(100%) when initiation. One test base on > postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. > > So, any good solution about this performance issue? > > Thanks, > Xurong Yang > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Neutron] One performance issue about VXLAN pool initiation
Hi, Folks, When we configure VXLAN range [1,16M], neutron-server service costs long time and cpu rate is very high(100%) when initiation. One test base on postgresql has been verified: more than 1h when VXLAN range is [1, 1M]. So, any good solution about this performance issue? Thanks, Xurong Yang ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
