Thanks, I'll give it a try.
On Fri, Nov 22, 2013 at 2:35 AM, Carl Baldwin <c...@ecbaldwin.net> wrote: > Hello, > > Please tell me if your experience is similar to what I experienced: > > 1. I would see *at most one* "MySQL server has gone away" error for > each process that was spawned as an API worker. I saw them within a > minute of spawning the workers and then I did not see these errors > anymore until I restarted the server and spawned new processes. > > 2. I noted in patch set 7 the line of code that completely fixed this > for me. Please confirm that you have applied a patch that includes > this fix. > > https://review.openstack.org/#/c/37131/7/neutron/wsgi.py > > 3. I did not change anything with pool_recycle or idle_interval in my > config files. All I did was set api_workers to the number of workers > that I wanted to spawn. The line of code with my comment in it above > was sufficient for me. > > It could be that there is another cause for the errors that you're > seeing. For example, is there a max connections setting in mysql that > might be exceeded when you spawn multiple workers? More detail would > be helpful. > > Cheers, > Carl > > On Wed, Nov 20, 2013 at 7:40 PM, Zhongyue Luo <zhongyue....@intel.com> > wrote: > > Carl, > > > > By 2006 I mean the "MySQL server has gong away" error code. > > > > The error message was still appearing when idle_timeout is set to 1 and > the > > quantum API server did not work in my case. > > > > Could you perhaps share your conf file when applying this patch? > > > > Thanks. > > > > > > > > On Thu, Nov 21, 2013 at 3:34 AM, Carl Baldwin <c...@ecbaldwin.net> > wrote: > >> > >> Hi, sorry for the delay in response. I'm glad to look at it. > >> > >> Can you be more specific about the error? Maybe paste the error your > >> seeing in paste.openstack.org? I don't find any reference to "2006". > >> Maybe I'm missing something. > >> > >> Also, is the patch that you applied the most recent? With the final > >> version of the patch it was no longer necessary for me to set > >> pool_recycle or idle_interval. > >> > >> Thanks, > >> Carl > >> > >> On Tue, Nov 19, 2013 at 7:14 PM, Zhongyue Luo <zhongyue....@intel.com> > >> wrote: > >> > Carl, Yingjun, > >> > > >> > I'm still getting the 2006 error even by configuring idle_interval to > 1. > >> > > >> > I applied the patch to the RDO havana dist on centos 6.4. > >> > > >> > Are there any other options I should be considering such as min/max > pool > >> > size or use_tpool? > >> > > >> > Thanks. > >> > > >> > > >> > > >> > On Sat, Sep 7, 2013 at 3:33 AM, Baldwin, Carl (HPCS Neutron) > >> > <carl.bald...@hp.com> wrote: > >> >> > >> >> This pool_recycle parameter is already configurable using the > >> >> idle_timeout > >> >> configuration variable in neutron.conf. I tested this with a value > of > >> >> 1 > >> >> as suggested and it did get rid of the mysql server gone away > messages. > >> >> > >> >> This is a great clue but I think I would like a long-term solution > that > >> >> allows the end-user to still configure this like they were before. > >> >> > >> >> I'm currently thinking along the lines of calling something like > >> >> pool.dispose() in each child immediately after it is spawned. I > think > >> >> this should invalidate all of the existing connections so that when a > >> >> connection is checked out of the pool a new one will be created > fresh. > >> >> > >> >> Thoughts? I'll be testing. Hopefully, I'll have a fixed patch up > >> >> soon. > >> >> > >> >> Cheers, > >> >> Carl > >> >> > >> >> From: Yingjun Li <liyingjun1...@gmail.com> > >> >> Reply-To: OpenStack Development Mailing List > >> >> <openstack-dev@lists.openstack.org> > >> >> Date: Thursday, September 5, 2013 8:28 PM > >> >> To: OpenStack Development Mailing List > >> >> <openstack-dev@lists.openstack.org> > >> >> Subject: Re: [openstack-dev] [Neutron] The three API server > >> >> multi-worker > >> >> process patches. > >> >> > >> >> > >> >> +1 for Carl's patch, and i have abandoned my patch.. > >> >> > >> >> About the `MySQL server gone away` problem, I fixed it by set > >> >> 'pool_recycle' to 1 in db/api.py. > >> >> > >> >> 在 2013年9月6日星期五,Nachi Ueno 写道: > >> >> > >> >> Hi Folks > >> >> > >> >> We choose https://review.openstack.org/#/c/37131/ <-- This patch to > go > >> >> on. > >> >> We are also discussing in this patch. > >> >> > >> >> Best > >> >> Nachi > >> >> > >> >> > >> >> > >> >> 2013/9/5 Baldwin, Carl (HPCS Neutron) <carl.bald...@hp.com>: > >> >> > Brian, > >> >> > > >> >> > As far as I know, no consensus was reached. > >> >> > > >> >> > A problem was discovered that happens when spawning multiple > >> >> > processes. > >> >> > The mysql connection seems to "go away" after between 10-60 seconds > >> >> > in > >> >> > my > >> >> > testing causing a seemingly random API call to fail. After that, > it > >> >> > is > >> >> > okay. This must be due to some interaction between forking the > >> >> > process > >> >> > and the mysql connection pool. This needs to be solved but I > haven't > >> >> > had > >> >> > the time to look in to it this week. > >> >> > > >> >> > I'm not sure if the other proposal suffers from this problem. > >> >> > > >> >> > Carl > >> >> > > >> >> > On 9/4/13 3:34 PM, "Brian Cline" <bcl...@softlayer.com> wrote: > >> >> > > >> >> >>Was any consensus on this ever reached? It appears both reviews are > >> >> >> still > >> >> >>open. I'm partial to review 37131 as it attacks the problem a more > >> >> >>concisely, and, as mentioned, combined the efforts of the two more > >> >> >>effective patches. I would echo Carl's sentiments that it's an easy > >> >> >>review minus the few minor behaviors discussed on the review thread > >> >> >>today. > >> >> >> > >> >> >>We feel very strongly about these making it into Havana -- being > >> >> >> confined > >> >> >>to a single neutron-server instance per cluster or region is a huge > >> >> >>bottleneck--essentially the only controller process with massive > CPU > >> >> >>churn in environments with constant instance churn, or sudden large > >> >> >>batches of new instance requests. > >> >> >> > >> >> >>In Grizzly, this behavior caused addresses not to be issued to some > >> >> >>instances during boot, due to quantum-server thinking the DHCP > agents > >> >> >>timed out and were no longer available, when in reality they were > >> >> >> just > >> >> >>backlogged (waiting on quantum-server, it seemed). > >> >> >> > >> >> >>Is it realistically looking like this patch will be cut for h3? > >> >> >> > >> >> >>-- > >> >> >>Brian Cline > >> >> >>Software Engineer III, Product Innovation > >> >> >> > >> >> >>SoftLayer, an IBM Company > >> >> >>4849 Alpha Rd, Dallas, TX 75244 > >> >> >>214.782.7876 direct | bcl...@softlayer.com > >> >> >> > >> >> >> > >> >> >>-----Original Message----- > >> >> >>From: Baldwin, Carl (HPCS Neutron) [mailto:carl.bald...@hp.com] > >> >> >>Sent: Wednesday, August 28, 2013 3:04 PM > >> >> >>To: Mark McClain > >> >> >>Cc: OpenStack Development Mailing List > >> >> >>Subject: [openstack-dev] [Neutron] The three API server > multi-worker > >> >> >>process patches. > >> >> >> > >> >> >>All, > >> >> >> > >> >> >>We've known for a while now that some duplication of work happened > >> >> >> with > >> >> >>respect to adding multiple worker processes to the neutron-server. > >> >> >> There > >> >> >>were a few mistakes made which led to three patches being done > >> >> >>independently of each other. > >> >> >> > >> >> >>Can we settle on one and accept it? > >> >> >> > >> >> >>I have changed my patch at the suggestion of one of the other 2 > >> >> >> authors, > >> >> >>Peter Feiner, in attempt to find common ground. It now uses > >> >> >> openstack > >> >> >>common code and therefore it is more concise than any of the > original > >> >> >>three and should be pretty easy to review. I'll admit to some bias > >> >> >>toward > >> >> >>my own implementation but most importantly, I would like for one of > >> >> >> these > >> >> >>implementations to land and start seeing broad usage in the > community > >> >> >>earlier than later. > >> >> >> > >> >> >>Carl Baldwin > >> >> >> > >> >> >>PS Here are the two remaining patches. The third has been > abandoned. > >> >> >> > >> >> >>https://review.openstack.org/#/c/37131/ > >> >> >>https://review.openstack.org/#/c/36487/ > >> >> >> > >> >> >> > >> >> >>_______________________________________________ > >> >> >>OpenStack-dev mailing list > >> >> >>OpenStack-dev@lists.openstack.org > >> >> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > > >> >> > > >> >> > _______________________________________________ > >> >> > OpenStack-dev mailing list > >> >> > OpenStack-dev@lists.openstack.org > >> >> > >> >> > >> >> _______________________________________________ > >> >> OpenStack-dev mailing list > >> >> OpenStack-dev@lists.openstack.org > >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > > >> > > >> > > >> > -- > >> > Intel SSG/STO/DCST/CIT > >> > 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, > Shanghai, > >> > China > >> > +862161166500 > >> > > >> > _______________________________________________ > >> > OpenStack-dev mailing list > >> > OpenStack-dev@lists.openstack.org > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > -- > > Intel SSG/STO/DCST/CIT > > 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai, > > China > > +862161166500 > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- *Intel SSG/STO/DCST/CIT* 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai, China +862161166500
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev