Httplib2 does very little directly with ssl other than a wrap_socket call. Unless requests has special ssl error handling and retry logic, it will be exposed to the same set of underlying errors from the ssl library so a retry that at least catches socket and ssl errors is a good idea. On May 29, 2014 1:17 PM, "Paul Ward" <wpw...@us.ibm.com> wrote:
> Yes, we're still on a code level that uses httplib2. I noticed that as > well, but wasn't sure if that would really > help here as it seems like an ssl thing itself. But... who knows?? I'm > not sure how consistently we can > recreate this, but if we can, I'll try using that patch to use requests > and see if that helps. > > > > "Armando M." <arma...@gmail.com> wrote on 05/29/2014 11:52:34 AM: > > > From: "Armando M." <arma...@gmail.com> > > To: "OpenStack Development Mailing List (not for usage questions)" > > <openstack-dev@lists.openstack.org>, > > Date: 05/29/2014 11:58 AM > > Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > > > Hi Paul, > > > > Just out of curiosity, I am assuming you are using the client that > > still relies on httplib2. Patch [1] replaced httplib2 with requests, > > but I believe that a new client that incorporates this change has not > > yet been published. I wonder if the failures you are referring to > > manifest themselves with the former http library rather than the > > latter. Could you clarify? > > > > Thanks, > > Armando > > > > [1] - https://review.openstack.org/#/c/89879/ > > > > On 29 May 2014 17:25, Paul Ward <wpw...@us.ibm.com> wrote: > > > Well, for my specific error, it was an intermittent ssl handshake error > > > before the request was ever sent to the > > > neutron-server. In our case, we saw that 4 out of 5 resize operations > > > worked, the fifth failed with this ssl > > > handshake error in neutronclient. > > > > > > I certainly think a GET is safe to retry, and I agree with your > statement > > > that PUTs and DELETEs probably > > > are as well. This still leaves a change in nova needing to be made to > > > actually a) specify a conf option and > > > b) pass it to neutronclient where appropriate. > > > > > > > > > Aaron Rosen <aaronoro...@gmail.com> wrote on 05/28/2014 07:38:56 PM: > > > > > >> From: Aaron Rosen <aaronoro...@gmail.com> > > > > > > > > >> To: "OpenStack Development Mailing List (not for usage questions)" > > >> <openstack-dev@lists.openstack.org>, > > >> Date: 05/28/2014 07:44 PM > > > > > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in > neutronclient > > >> > > >> Hi, > > >> > > >> I'm curious if other openstack clients implement this type of retry > > >> thing. I think retrying on GET/DELETES/PUT's should probably be okay. > > >> > > >> What types of errors do you see in the neutron-server when it fails > > >> to respond? I think it would be better to move the retry logic into > > >> the server around the failures rather than the client (or better yet > > >> if we fixed the server :)). Most of the times I've seen this type of > > >> failure is due to deadlock errors caused between (sqlalchemy and > > >> eventlet *i think*) which cause the client to eventually timeout. > > >> > > >> Best, > > >> > > >> Aaron > > >> > > > > > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward <wpw...@us.ibm.com> > wrote: > > >> Would it be feasible to make the retry logic only apply to read-only > > >> operations? This would still require a nova change to specify the > > >> number of retries, but it'd also prevent invokers from shooting > > >> themselves in the foot if they call for a write operation. > > >> > > >> > > >> > > >> Aaron Rosen <aaronoro...@gmail.com> wrote on 05/27/2014 09:40:00 PM: > > >> > > >> > From: Aaron Rosen <aaronoro...@gmail.com> > > >> > > >> > To: "OpenStack Development Mailing List (not for usage questions)" > > >> > <openstack-dev@lists.openstack.org>, > > >> > Date: 05/27/2014 09:44 PM > > >> > > >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in > > >> > neutronclient > > >> > > > >> > Hi, > > >> > > >> > > > >> > Is it possible to detect when the ssl handshaking error occurs on > > >> > the client side (and only retry for that)? If so I think we should > > >> > do that rather than retrying multiple times. The danger here is > > >> > mostly for POST operations (as Eugene pointed out) where it's > > >> > possible for the response to not make it back to the client and for > > >> > the operation to actually succeed. > > >> > > > >> > Having this retry logic nested in the client also prevents things > > >> > like nova from handling these types of failures individually since > > >> > this retry logic is happening inside of the client. I think it would > > >> > be better not to have this internal mechanism in the client and > > >> > instead make the user of the client implement retry so they are > > >> > aware of failures. > > >> > > > >> > Aaron > > >> > > > >> > > >> > On Tue, May 27, 2014 at 10:48 AM, Paul Ward <wpw...@us.ibm.com> > wrote: > > >> > Currently, neutronclient is hardcoded to only try a request once in > > >> > retry_request by virtue of the fact that it uses self.retries as the > > >> > retry count, and that's initialized to 0 and never changed. We've > > >> > seen an issue where we get an ssl handshaking error intermittently > > >> > (seems like more of an ssl bug) and a retry would probably have > > >> > worked. Yet, since neutronclient only tries once and gives up, it > > >> > fails the entire operation. Here is the code in question: > > >> > > > >> > https://github.com/openstack/python-neutronclient/blob/master/ > > >> > neutronclient/v2_0/client.py#L1296 > > >> > > > >> > Does anybody know if there's some explicit reason we don't currently > > >> > allow configuring the number of retries? If not, I'm inclined to > > >> > propose a change for just that. > > >> > > > >> > _______________________________________________ > > >> > OpenStack-dev mailing list > > >> > OpenStack-dev@lists.openstack.org > > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >> > > >> > _______________________________________________ > > >> > OpenStack-dev mailing list > > >> > OpenStack-dev@lists.openstack.org > > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >> > > >> _______________________________________________ > > >> OpenStack-dev mailing list > > >> OpenStack-dev@lists.openstack.org > > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > >> _______________________________________________ > > >> OpenStack-dev mailing list > > >> OpenStack-dev@lists.openstack.org > > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > _______________________________________________ > > > OpenStack-dev mailing list > > > OpenStack-dev@lists.openstack.org > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev