On Thu, Oct 18, 2018 at 9:24 PM Jeremy Daer <jeremyd...@gmail.com> wrote:
> On Thu, Oct 18, 2018 at 2:52 AM 'Daniel Schierbeck' via Ruby on Rails: > Core <rubyonrails-core@googlegroups.com> wrote: > >> Looks like OpenCensus already has support for development mode UIs, >> currently only for Java and Go though: >> https://opencensus.io/core-concepts/z-pages/ >> <https://opencensus.io/core-concepts/z-pages/> >> > > This is a great starting point. Rails dev can level up from there. > > Have you deployed an OpenCensus integration to production? At least the >> metrics part looks pretty advanced, maybe too much so. >> > > Not in production. We have a branch of Basecamp that exports to > Stackdriver. In production, we'd prefer to use local agents and collectors > rather than export directly to a vendor: > https://github.com/census-instrumentation/opencensus-service > <https://github.com/census-instrumentation/opencensus-service> > Sounds like you are farther ahead than us then – we're pushing stuff directly to Datadog right now. How about this as a starting point: I try to add proper AS::N instrumentation to the places where monkey patches are currently used, e.g. middleware execution. I'll CC you on the PRs. Maybe we can stay in touch regarding your experience with OpenCensus in production, what, if anything, would be needed in order to "natively" support it, and anything else you might think relevant? It sounds like it's too premature to add a dependency on the opencensus gem and push traces from Rails itself, unless you think we're ready? One problem is that we're unlikely to be able to run on Rails master in production, so there's little production feedback I can give. > > >> >> Cheers, >> Daniel >> >> On Thu, Oct 18, 2018 at 11:26 AM Daniel Schierbeck < >> dschierb...@zendesk.com> wrote: >> >>> On Wed, Oct 17, 2018 at 7:26 PM Jeremy Daer <jeremyd...@gmail.com> >>> wrote: >>> >>>> Hey Daniel, >>>> >>>> Absolutely! We're looking at OpenCensus (https://opencensus.io >>>> <https://opencensus.io>) >>>> integration, which seems to be leapfrogging OpenTracing in standardization >>>> and adoption. >>>> >>> >>> So now there are two standards? 😬 >>> Is there clarity on where things are going? The point of a standard >>> would be that we'd only need to support *one*, and not have an extra layer >>> of abstraction. >>> >>> >>>> >>>> Current Ruby integration, including early Rack and Rails support: >>>> https://github.com/census-instrumentation/opencensus-ruby >>>> <https://github.com/census-instrumentation/opencensus-ruby> >>>> >>>> Datadog exporter: >>>> https://github.com/DataDog/opencensus-go-exporter-datadog >>>> <https://github.com/DataDog/opencensus-go-exporter-datadog> >>>> Stackdriver exporter: >>>> https://github.com/census-ecosystem/opencensus-ruby-exporter-stackdriver >>>> <https://github.com/census-ecosystem/opencensus-ruby-exporter-stackdriver> >>>> Zipkin/Jaeger exporter: >>>> https://github.com/census-ecosystem/opencensus-ruby-exporter-zipkin >>>> <https://github.com/census-ecosystem/opencensus-ruby-exporter-zipkin> >>>> Or use a local collector/relay: >>>> https://github.com/census-instrumentation/opencensus-service >>>> <https://github.com/census-instrumentation/opencensus-service> >>>> >>> >>> Compared to OpenTracing, is the ecosystem mature enough to warrant us >>> going all-in on this? I definitely see the theoretical point of a unified >>> stats and trace standard, especially seeing as Statsd has fragmented >>> somewhat, but is it a horse we want to bet on? I'm fine with either, as >>> long as there are working, scalable solutions *today* for getting things >>> working in a variety of languages and without duct tape. For instance, it >>> seems like the Datadog exporter only supports Go? >>> >>> >>>> Production use is fantastic, but I'd particularly love to see a >>>> collector and built-in visualization for local app development and tests. >>>> >>> >>> Me too 😄 but that's probably not going to be my initial focus, I'm >>> meanly looking at the instrumentation side of things. >>> >>> >>> We have an existing ActiveSupport::Notifications API which works much >>>> like typical parent-span instrumentation, but it doesn't propagate or >>>> report trace context. >>>> >>> For deeper Rails integration, we could adapt the AS::N design to more >>>> directly map to OpenCensus, or introduce ActiveSupport::Tracing if there's >>>> too much mismatch or compatibility concern. >>>> >>> >>> I've done a bunch of work on AS::N in the past (I'm the one who created >>> the original ActiveSupport::Subscriber base class) and feel pretty >>> confident that it can form the basis for this work. We'd probably want to >>> instrument more places, e.g. each middleware invocation and maybe filters >>> in controllers, but otherwise it's a good starting point. >>> >>> I do think we need to have a specific mapping from AS::N to the tracing >>> backend, selecting which payload keys should be propagated and maybe >>> formatting some of the values, so it's probably not just a matter of >>> copying everything verbatim. It sounds like you're doing the verbatim thing >>> at Basecamp though – how is that working out? Would you be in favor of that? >>> >>> We've seen issues when tracing gets too granular or too much data is >>> captured, so I'd like to be a bit conservative. >>> >>> >>>> That'd allow these libraries to plug in directly without needing to >>>> carefully instrument Rails on their own. Rails should be able to >>>> participate in distributed tracing out of the box, report stats out of the >>>> box, show traces and stats in development mode, and flip between production >>>> APM vendors without specialized integration. >>>> >>> >>> Yup, that's my goal as well. APM vendors should not compete on their >>> quality of instrumentation, but on the quality of their product. One thing >>> I want to emphasize though is that I think we need to push for >>> standardization *beyond* Rails. AS::N could have been a great standard if >>> it wasn't tied to AS – it hasn't seen widespread adoption because gem >>> authors are unwilling to add a dependency on AS, I think. So we should >>> think holistically about the entire ecosystem and what would make sense for >>> Ruby as a whole. >>> >>> >>>> At Basecamp, we have a home-grown StatsD setup, similar to Datadog, >>>> that hooks Active Support notifications ( >>>> https://signalvnoise.com/posts/3091-pssst-your-rails-application-has-a-secret-to-tell-you >>>> <https://signalvnoise.com/posts/3091-pssst-your-rails-application-has-a-secret-to-tell-you>). >>>> We also >>>> parse logs from Kafka to reconstruct some traces. We'd love to extract this >>>> and rely on Rails to natively export traces and stats. >>>> >>> >>>> I'd love to hear what you're doing at Zendesk, where you're headed, and >>>> whether this sketch aligns well. And anyone else who's working in this >>>> area! >>>> >>> >>> We're currently all-in on Datadog, and I've helped improve their >>> instrumentation. However, I keep running into ad-hoc instrumentation being >>> brittle, which is why I'm interested in first-class support. I think the >>> only sustainable path forward is that gems natively support some form of >>> tracing, either through AS::N (which would need to be extracted) or >>> directly with a standardized tracing gem. >>> >>> How would you feel about extracting AS::N, actually? Then gems could >>> adopt it for pub/sub and it would be a lot simpler to plug in a tracing >>> subscriber. >>> >>> >>>> >>>> Best, >>>> Jeremy >>>> >>>> On Wed, Oct 17, 2018 at 5:58 AM 'Daniel Schierbeck' via Ruby on Rails: >>>> Core <rubyonrails-core@googlegroups.com> wrote: >>>> >>>>> With the advent of OpenTracing (https://opentracing.io >>>>> <https://opentracing.io>) along >>>>> with an official Ruby library ( >>>>> https://github.com/opentracing/opentracing-ruby >>>>> <https://github.com/opentracing/opentracing-ruby>) as well >>>>> as growing industry support (e.g. >>>>> https://www.datadoghq.com/blog/opentracing-datadog-cncf/ >>>>> <https://www.datadoghq.com/blog/opentracing-datadog-cncf/> and >>>>> http://blog.scoutapp.com/articles/2018/01/17/tutorial-distributed-tracing-in-ruby-with-opentracing >>>>> <http://blog.scoutapp.com/articles/2018/01/17/tutorial-distributed-tracing-in-ruby-with-opentracing>), >>>>> would >>>>> it make sense to provide native tracing of Rails using the >>>>> opentracing-ruby >>>>> library? >>>>> >>>>> At Zendesk we're using Datadog's proprietary tracing API, which monkey >>>>> patches Rails and other libraries in order to trace key interactions. I >>>>> think a more sustainable approach would be for libraries to include >>>>> tracing >>>>> support out of the box using the standardized OpenTracing APIs. It is then >>>>> merely a matter of hooking up e.g. the Datadog trace collector in order to >>>>> get a working tracing setup. >>>>> >>>>> If there's interest in this I'd be willing to contribute code. I've >>>>> done a bunch of working in order to trace various aspects of Rails, >>>>> including Rack middleware and before/after filters, but without native >>>>> support, these implementations are brittle and prone to breakage when >>>>> internal Rails APIs change. >>>>> >>>>> I'd love to hear your thoughts on this. >>>>> >>>>> Cheers, >>>>> Daniel Schierbeck >>>>> >>>>> -- >>>>> >>>> You received this message because you are subscribed to the Google >>>>> Groups "Ruby on Rails: Core" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to rubyonrails-core+unsubscr...@googlegroups.com. >>>> >>>> >>>>> To post to this group, send email to rubyonrails-core@googlegroups.com >>>>> . >>>>> Visit this group at https://groups.google.com/group/rubyonrails-core >>>>> <https://groups.google.com/group/rubyonrails-core>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Ruby on Rails: Core" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/rubyonrails-core/qJlL_uVxsnU/unsubscribe >>>> <https://groups.google.com/d/topic/rubyonrails-core/qJlL_uVxsnU/unsubscribe>. >>>> To unsubscribe from this group and all its topics, send an email to >>>> rubyonrails-core+unsubscr...@googlegroups.com. >>>> To post to this group, send email to rubyonrails-core@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/rubyonrails-core >>>> <https://groups.google.com/group/rubyonrails-core>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby on Rails: Core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to rubyonrails-core+unsubscr...@googlegroups.com. >> To post to this group, send email to rubyonrails-core@googlegroups.com. >> Visit this group at https://groups.google.com/group/rubyonrails-core >> <https://groups.google.com/group/rubyonrails-core>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Ruby on Rails: Core" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/rubyonrails-core/qJlL_uVxsnU/unsubscribe > <https://groups.google.com/d/topic/rubyonrails-core/qJlL_uVxsnU/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > rubyonrails-core+unsubscr...@googlegroups.com. > To post to this group, send email to rubyonrails-core@googlegroups.com. > Visit this group at https://groups.google.com/group/rubyonrails-core > <https://groups.google.com/group/rubyonrails-core>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-core+unsubscr...@googlegroups.com. To post to this group, send email to rubyonrails-core@googlegroups.com. Visit this group at https://groups.google.com/group/rubyonrails-core. For more options, visit https://groups.google.com/d/optout.
