On Dec 6, 2013, at 1:09 AM, John Dickinson <m...@not.mn> wrote: > > On Dec 5, 2013, at 1:36 AM, Maru Newby <ma...@redhat.com> wrote: > >> >> On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gord...@gmail.com> wrote: >> >>> >>> >>> >>> On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <m...@not.mn> wrote: >>> Just to add to the story, Swift uses "X-Trans-Id" and generates it in the >>> outer-most "catch_errors" middleware. >>> >>> Swift's catch errors middleware is responsible for ensuring that the >>> transaction id exists on each request, and that all errors previously >>> uncaught, anywhere in the pipeline, are caught and logged. If there is not >>> a common way to do this, yet, I submit it as a great template for solving >>> this problem. It's simple, scalable, and well-tested (ie tests and running >>> in prod for years). >>> >>> https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py >>> >>> Leaving aside error handling and only focusing on the transaction id (or >>> request id) generation, since OpenStack services are exposed to untrusted >>> clients, how would you propose communicating the appropriate transaction id >>> to a different service? I can see great benefit to having a glance >>> transaction ID carry through to Swift requests (and so on), but how should >>> the transaction id be communicated? It's not sensitive info, but I can >>> imagine a pretty big problem when trying to track down errors if a client >>> application decides to set eg the X-Set-Transaction-Id header on every >>> request to the same thing. >>> >>> -1 to cross service request IDs, for the reasons John mentions above. >>> >>> >>> Thanks for bringing this up, and I'd welcome a patch in Swift that would >>> use a common library to generate the transaction id, if it were installed. >>> I can see that there would be huge advantage to operators to trace requests >>> through multiple systems. >>> >>> Another option would be for each system that calls an another OpenStack >>> system to expect and log the transaction ID for the request that was given. >>> This would be looser coupling and be more forgiving for a heterogeneous >>> cluster. Eg when Glance makes a call to Swift, Glance cloud log the >>> transaction id that Swift used (from the Swift response). Likewise, when >>> Swift makes a call to Keystone, Swift could log the Keystone transaction >>> id. This wouldn't result in a single transaction id across all systems, but >>> it would provide markers so an admin could trace the request. >>> >>> There was a session on this at the summit, and although the notes are a >>> little scarce this was the conclusion we came up with. Every time a cross >>> service call is made, we will log and send a notification for ceilometer to >>> consume, with the request-ids of both request ids. One of the benefits of >>> this approach is that we can easily generate a tree of all the API calls >>> that are made (and clearly show when multiple calls are made to the same >>> service), something that just a cross service request id would have trouble >>> with. >> >> Is wise to trust anything a client provides to ensure traceability? If a >> user receives a request id back from Nova, then submits that request id in >> an unrelated request to Neutron, the traceability would be effectively >> corrupted. If the consensus is that we don't want to securely deliver >> request ids for inter-service calls, how about requiring a service to log >> its request id along with the request id returned from a call to another >> service to achieve the a similar result? > > Yes, this is what I was proposing. I think this is the best path forward.
Ok, great. And as per your suggestion, a middleware-based error handler will soon be proposed for oslo that will secondarily ensure that a request id is added to the request. > > >> The catch is that every call point (or client instantiation?) would have to >> be modified to pass the request id instead of just logging at one place in >> each service. Is that a cost worth paying? > > Perhaps this is my ignorance of how other projects work today, but does this > not already happen? Is it possible to get a response from an API call to an > OpenStack project that doesn't include a request id? We'll have it in Neutron real-soon-now, and then I think the answer will be 'yes'. On reflection, it should be easy enough for a given service to ensure that calls to other services are automatically instrumented to log request id pairs. Again, probably something for oslo. m. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
