On Wednesday 3 June 2015 at 12:13, Miguel Ángel Ajo wrote:
> Doesn’t this overlap with the work done for the OSProfiler ? > > > More comments inline. > > Miguel Ángel Ajo > > > On Wednesday, 3 de June de 2015 at 11:43, Kekane, Abhishek wrote: > > > Hi Devs, > > > > So for I have got following responses on the proposed solutions: > > > > Solution 1: Return tuple containing headers and body from - 3 +1 > > Solution 2: Use thread local storage to store 'x-openstack-request-id' > > returned from headers - 0 +1 > > Solution 3: Unique request-id across OpenStack Services - 1 +1 > > > > > > > > > I’d vote for Solution 3, without involving keystone (first caller with no > req-id generates one randomly), > the req-id contains a call/hop count, which is incremented on every new > call... > > sorry, that suggestion is naive, simply incrementing the call count won’t work as calls diverge we will have duplicate req-id + hop’s > > > > > > > > > > > > > Requesting community people, cross-project members and PTL's to go through > > this mailing thread [1] and give your suggestions/opinions about the > > solutions proposed so that It will be easy to finalize the solution. > > > > [1] http://lists.openstack.org/pipermail/openstack-dev/2015-May/064842.html > > > > Thanks & Regards, > > > > Abhishek Kekane > > > > -----Original Message----- > > From: Nikhil Komawar [mailto:nik.koma...@gmail.com] > > Sent: 28 May 2015 12:34 > > To: openstack-dev@lists.openstack.org > > (mailto:openstack-dev@lists.openstack.org) > > Subject: Re: [openstack-dev] [all] cross project communication: Return > > request-id to caller > > > > Did you get to talk with anyone in the LogWG ( > > https://wiki.openstack.org/wiki/LogWorkingGroup )? In wonder what kind of > > recommendations, standards we can come up with while adopting a cross > > project solution. If our logs follow certain prefix and or suffix style > > across projects, that would help a long way. > > > > Personally: +1 on Solution 1 > > > > On 5/28/15 2:14 AM, Kekane, Abhishek wrote: > > > > > > Hi Devs, > > > > > > > > > Thank you for your opinions/thoughts. > > > > > > However I would like to suggest that please give +1 against the > > > solution which you will like to propose so that at the end it will be > > > helpful for us to consolidate the voting against each solution and > > > make some decision. > > > > > > > > > Thanks in advance. > > > > > > > > > Abhishek Kekane > > > > > > > > > > > > *From:*Joe Gordon [mailto:joe.gord...@gmail.com] > > > *Sent:* 28 May 2015 00:31 > > > *To:* OpenStack Development Mailing List (not for usage questions) > > > *Subject:* Re: [openstack-dev] [all] cross project communication: > > > Return request-id to caller > > > > > > > > > > > > > > > On Wed, May 27, 2015 at 12:06 AM, Kekane, Abhishek > > > <abhishek.kek...@nttdata.com (mailto:abhishek.kek...@nttdata.com) > > > <mailto:abhishek.kek...@nttdata.com>> wrote: > > > > > > Hi Devs, > > > > > > > > > Each OpenStack service sends a request ID header with HTTP responses. > > > This request ID can be useful for tracking down problems in the logs. > > > However, when operation crosses service boundaries, this tracking can > > > become difficult, as each service has its own request ID. Request ID > > > is not returned to the caller, so it is not easy to track the request. > > > This becomes especially problematic when requests are coming in > > > parallel. For example, glance will call cinder for creating image, but > > > that cinder instance may be handling several other requests at the > > > same time. By using same request ID in the log, user can easily find > > > the cinder request ID that is same as glance request ID in the g-api > > > log. It will help operators/developers to analyse logs effectively. > > > > > > > > > Thank you for writing this up. > > > > > > > > > > > > To address this issue we have come up with following solutions: > > > > > > > > > Solution 1: Return tuple containing headers and body from > > > respective clients (also favoured by Joe Gordon) > > > > > > Reference: > > > > > > https://review.openstack.org/#/c/156508/6/specs/log-request-id-mapping > > > s.rst > > > > > > > > > Pros: > > > > > > 1. Maintains backward compatibility > > > > > > 2. Effective debugging/analysing of the problem as both calling > > > service request-id and called service request-id are logged in > > > same log message > > > > > > 3. Build a full call graph > > > > > > 4. End user will able to know the request-id of the request and > > > can approach service provider to know the cause of failure of > > > particular request. > > > > > > > > > Cons: > > > > > > 1. The changes need to be done first in cross-projects before > > > making changes in clients > > > > > > 2. Applications which are using python-*clients needs to do > > > required changes (check return type of response) > > > > > > > > > Additional cons: > > > > > > > > > 3. Cannot simply search all logs (ala logstash) using the request-id > > > returned to the user without any post processing of the logs. > > > > > > > > > > > > > > > > > > > > > > Solution 2: Use thread local storage to store > > > 'x-openstack-request-id' returned from headers (suggested by Doug > > > Hellmann) > > > > > > Reference: > > > > > > https://review.openstack.org/#/c/156508/9/specs/log-request-id-mapping > > > s.rst > > > > > > > > > Add new method 'get_openstack_request_id' to return this > > > request-id to the caller. > > > > > > > > > Pros: > > > > > > 1. Doesn't break compatibility > > > > > > 2. Minimal changes are required in client > > > > > > 3. Build a full call graph > > > > > > > > > Cons: > > > > > > 1. Malicious user can send long request-id to fill up the > > > disk-space, resulting in potential DoS > > > > > > > > > > > > > > Can’t this be easily mitigated by limiting the request-id size, > and denying any wrong request-id? > > > > > > > 2. Changes need to be done in all python-*clients > > > > > > 3. Last request id should be flushed out in a subsequent call > > > otherwise it will return wrong request id to the caller > > > > > > > > > > > > Solution 3: Unique request-id across OpenStack Services (suggested > > > by Jamie Lennox) > > > > > > Reference: > > > > > > https://review.openstack.org/#/c/156508/10/specs/log-request-id-mappin > > > gs.rst > > > > > > > > > Get 'x-openstack-request-id' from auth plugin and add it to the > > > request headers. If 'x-openstack-request-id' key is present in the > > > request header, then it will use the same one further or else it > > > will generate a new one. > > > > > > > > > Dependencies: > > > > > > https://review.openstack.org/#/c/164582/ - Include request-id in > > > auth plugin and add it to request headers > > > > > > https://review.openstack.org/#/c/166063/ - Add session-object for > > > glance client > > > > > > Add 'UserAuthPlugin' and '_ContextAuthPlugin' same as nova in > > > cinder and neutron > > > > > > > > > > > > Pros: > > > > > > 1. Using same request id for the request crossing multiple service > > > boundaries will help operators/developers identify the problem > > > quickly > > > > > > 2. Required changes only in keystonemiddleware and oslo_middleware > > > libraries. No changes are required in the python client bindings > > > or OpenStack core services > > > > > > > > > > > > > > Why do we need to call keystone to get a request ID?, can’t we just > simply generate a random one if no req-id is provided in headers? > > I believe calling keystone to start a new request is a bottleneck, and that’s > not generally necessary when you have an auth token already... > > > > > > > > > > Cons: > > > > > > 1. As 'x-openstack-request-id' in the request header will be > > > visible to the user, it is possible to send same request id for > > > multiple requests which in turn could create more problems in case > > > of troubleshooting cause of the failure as request_id middleware > > > will not check for its uniqueness in the scope of the running > > > OpenStack service. > > > > > > 2. Having the same request ID for all services for a single user > > > API call means you cannot generate a full call graph. For example > > > if a single user's nova API call produces 2 calls to glance you > > > want to be able to differentiate the two different calls. > > > > > > > Why not tagging the request ID with the call number? > > <req-id>/1 , <req-id>/2, <req-id>/3 … then it’s very easy > to trace order. > > > > > > > > > > > During the Liberty design summit, I had a chance of discussing > > > these designs with some of the core members like Doug, Joe Gordon, > > > Jamie Lennox etc. But not able to came to any conclusion on the > > > final design and know the communities direction by which way they > > > want to use this request-id effectively. > > > > > > > > > However IMO, solution 1 sounds more useful as the debugger can > > > able to build the full call graph which can be helpful for > > > analysing gate failures effectively as well as end user will be > > > able to know his request-id and can track his request. > > > > > > > > > I request all community members to go through these solutions and > > > let us know which is the appropriate way to improve the logs by > > > logging request-id. > > > > > > > > > > > > Thanks & Regards, > > > > > > > > > Abhishek Kekane > > > > > > > > > ______________________________________________________________________ > > > Disclaimer: This email and any attachments are sent in strictest > > > confidence > > > for the sole use of the addressee and may contain legally privileged, > > > confidential, and proprietary data. If you are not the intended > > > recipient, > > > please advise the sender by replying promptly to this email and > > > then delete > > > and destroy this email and any attachments without any further > > > use, copying > > > or forwarding. > > > > > > > > > __________________________________________________________________________ > > > OpenStack Development Mailing List (not for usage questions) > > > Unsubscribe: > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > (mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe) > > > <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > > ______________________________________________________________________ > > > Disclaimer: This email and any attachments are sent in strictest > > > confidence for the sole use of the addressee and may contain legally > > > privileged, confidential, and proprietary data. If you are not the > > > intended recipient, please advise the sender by replying promptly to > > > this email and then delete and destroy this email and any attachments > > > without any further use, copying or forwarding. > > > > > > > > > ______________________________________________________________________ > > > ____ OpenStack Development Mailing List (not for usage questions) > > > Unsubscribe: > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > (mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe) > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > -- > > > > Thanks, > > Nikhil > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > (mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe) > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ______________________________________________________________________ > > Disclaimer: This email and any attachments are sent in strictest confidence > > for the sole use of the addressee and may contain legally privileged, > > confidential, and proprietary data. If you are not the intended recipient, > > please advise the sender by replying promptly to this email and then delete > > and destroy this email and any attachments without any further use, copying > > or forwarding. > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > (mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe) > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > (mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe) > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
