Thanks for the response David. That proposal sounds exactly what I'm looking for in terms of middleware.
My motivation for tracing comes from previously working within a complex real time orchestration system with 20+ microservices involved in creating various features per request in the system. Tracing is a must or you end up not being able to determine which microservice is the culprit for high latency requests - and even more granularly - which part of the offending microservice is slow (e.g. feature preprocessing, exception handling, etc.) OpenTelemetry implemented at the c++ level sounds awesome with the caveat that it would be great to have some way to create additional spans and inject headers into external calls from python space (e.g. i might want a span for my feature preprocessing, a span for getting data from another microservice, etc.). Thanks, Eric On Thu, Apr 29, 2021 at 3:16 PM David Li <[email protected]> wrote: > Hey Eric, > > It so happens that I am literally in the middle of drafting a proposal > to the mailing list to integrate with OpenTelemetry on the C++ > level. I would fully support having these capabilities for Flight, and > I think in the past we've discussed having a contrib directory for > such modules, if we don't directly just integrate it into Arrow > itself. While gRPC in theory has some of this built in, either way > we'd need to expose it (and gRPC uses OpenCensus which is being phased > out in favor of OpenTelemetry anyways). > > The usecase here is fairly obvious, I think - but it'd be great if we > could hear more about how OpenTracing is useful for you/your > org. Personally, at my past job, it was a while before people really > grasped the value of distributed tracing & structured logging. > > Also, side note - I've used structlog in Python for JSON logging in > the past and it was fantastic, I would definitely recommend that over > just dumping JSON into a logger yourself. > > Best, > David > > > On 2021/04/29 18:59:30, Eric Henry <[email protected]> wrote: > > Is there currently any interest or effort underway in providing reusable > > Flight middleware for observability and production usage (e.g. logging, > > tracing, metrics)? > > > > For my own usage and experimentation I created a very basic json logging > > and OpenTracing implementation, but I was wondering if there was anything > > else out there (or in the works) that's production grade. > > > > Here's my code for reference: > > https://github.com/ehenry2/xgbatch/blob/main/xgbatch/middleware.py > > > > Thanks, > > Eric > > >
