On Fri, Oct 19, 2018 at 1:55 AM 'Daniel Schierbeck' via Ruby on Rails: Core <rubyonrails-core@googlegroups.com> wrote:
> On Thu, Oct 18, 2018 at 9:24 PM Jeremy Daer <jeremyd...@gmail.com> wrote: > >> On Thu, Oct 18, 2018 at 2:52 AM 'Daniel Schierbeck' via Ruby on Rails: >> Core <rubyonrails-core@googlegroups.com> wrote: >> >>> Looks like OpenCensus already has support for development mode UIs, >>> currently only for Java and Go though: >>> https://opencensus.io/core-concepts/z-pages/ >>> >> >> This is a great starting point. Rails dev can level up from there. >> >> Have you deployed an OpenCensus integration to production? At least the >>> metrics part looks pretty advanced, maybe too much so. >>> >> >> Not in production. We have a branch of Basecamp that exports to >> Stackdriver. In production, we'd prefer to use local agents and collectors >> rather than export directly to a vendor: >> https://github.com/census-instrumentation/opencensus-service >> > > Sounds like you are farther ahead than us then – we're pushing stuff > directly to Datadog right now. > Your instrumentation is likely further along since Datadog supports tagging and ingests traces :) How about this as a starting point: I try to add proper AS::N > instrumentation to the places where monkey patches are currently used, e.g. > middleware execution. I'll CC you on the PRs. Maybe we can stay in touch > regarding your experience with OpenCensus in production, what, if anything, > would be needed in order to "natively" support it, and anything else you > might think relevant? It sounds like it's too premature to add a dependency > on the opencensus gem and push traces from Rails itself, unless you think > we're ready? > Great path; agreed. One problem is that we're unlikely to be able to run on Rails master in > production, so there's little production feedback I can give. > A "rails-canary" branch could be enough! > > >> >> >>> >>> Cheers, >>> Daniel >>> >>> On Thu, Oct 18, 2018 at 11:26 AM Daniel Schierbeck < >>> dschierb...@zendesk.com> wrote: >>> >>>> On Wed, Oct 17, 2018 at 7:26 PM Jeremy Daer <jeremyd...@gmail.com> >>>> wrote: >>>> >>>>> Hey Daniel, >>>>> >>>>> Absolutely! We're looking at OpenCensus (https://opencensus.io) >>>>> integration, which seems to be leapfrogging OpenTracing in standardization >>>>> and adoption. >>>>> >>>> >>>> So now there are two standards? 😬 >>>> Is there clarity on where things are going? The point of a standard >>>> would be that we'd only need to support *one*, and not have an extra layer >>>> of abstraction. >>>> >>>> >>>>> >>>>> Current Ruby integration, including early Rack and Rails support: >>>>> https://github.com/census-instrumentation/opencensus-ruby >>>>> >>>>> Datadog exporter: >>>>> https://github.com/DataDog/opencensus-go-exporter-datadog >>>>> Stackdriver exporter: >>>>> https://github.com/census-ecosystem/opencensus-ruby-exporter-stackdriver >>>>> Zipkin/Jaeger exporter: >>>>> https://github.com/census-ecosystem/opencensus-ruby-exporter-zipkin >>>>> Or use a local collector/relay: >>>>> https://github.com/census-instrumentation/opencensus-service >>>>> >>>> >>>> Compared to OpenTracing, is the ecosystem mature enough to warrant us >>>> going all-in on this? I definitely see the theoretical point of a unified >>>> stats and trace standard, especially seeing as Statsd has fragmented >>>> somewhat, but is it a horse we want to bet on? I'm fine with either, as >>>> long as there are working, scalable solutions *today* for getting things >>>> working in a variety of languages and without duct tape. For instance, it >>>> seems like the Datadog exporter only supports Go? >>>> >>>> >>>>> Production use is fantastic, but I'd particularly love to see a >>>>> collector and built-in visualization for local app development and tests. >>>>> >>>> >>>> Me too 😄 but that's probably not going to be my initial focus, I'm >>>> meanly looking at the instrumentation side of things. >>>> >>>> >>>> We have an existing ActiveSupport::Notifications API which works much >>>>> like typical parent-span instrumentation, but it doesn't propagate or >>>>> report trace context. >>>>> >>>> For deeper Rails integration, we could adapt the AS::N design to more >>>>> directly map to OpenCensus, or introduce ActiveSupport::Tracing if there's >>>>> too much mismatch or compatibility concern. >>>>> >>>> >>>> I've done a bunch of work on AS::N in the past (I'm the one who created >>>> the original ActiveSupport::Subscriber base class) and feel pretty >>>> confident that it can form the basis for this work. We'd probably want to >>>> instrument more places, e.g. each middleware invocation and maybe filters >>>> in controllers, but otherwise it's a good starting point. >>>> >>>> I do think we need to have a specific mapping from AS::N to the tracing >>>> backend, selecting which payload keys should be propagated and maybe >>>> formatting some of the values, so it's probably not just a matter of >>>> copying everything verbatim. It sounds like you're doing the verbatim thing >>>> at Basecamp though – how is that working out? Would you be in favor of >>>> that? >>>> >>>> We've seen issues when tracing gets too granular or too much data is >>>> captured, so I'd like to be a bit conservative. >>>> >>>> >>>>> That'd allow these libraries to plug in directly without needing to >>>>> carefully instrument Rails on their own. Rails should be able to >>>>> participate in distributed tracing out of the box, report stats out of the >>>>> box, show traces and stats in development mode, and flip between >>>>> production >>>>> APM vendors without specialized integration. >>>>> >>>> >>>> Yup, that's my goal as well. APM vendors should not compete on their >>>> quality of instrumentation, but on the quality of their product. One thing >>>> I want to emphasize though is that I think we need to push for >>>> standardization *beyond* Rails. AS::N could have been a great standard if >>>> it wasn't tied to AS – it hasn't seen widespread adoption because gem >>>> authors are unwilling to add a dependency on AS, I think. So we should >>>> think holistically about the entire ecosystem and what would make sense for >>>> Ruby as a whole. >>>> >>>> >>>>> At Basecamp, we have a home-grown StatsD setup, similar to Datadog, >>>>> that hooks Active Support notifications ( >>>>> https://signalvnoise.com/posts/3091-pssst-your-rails-application-has-a-secret-to-tell-you). >>>>> We >>>>> also parse logs from Kafka to reconstruct some traces. We'd love to >>>>> extract >>>>> this and rely on Rails to natively export traces and stats. >>>>> >>>> >>>>> I'd love to hear what you're doing at Zendesk, where you're headed, >>>>> and whether this sketch aligns well. And anyone else who's working in this >>>>> area! >>>>> >>>> >>>> We're currently all-in on Datadog, and I've helped improve their >>>> instrumentation. However, I keep running into ad-hoc instrumentation being >>>> brittle, which is why I'm interested in first-class support. I think the >>>> only sustainable path forward is that gems natively support some form of >>>> tracing, either through AS::N (which would need to be extracted) or >>>> directly with a standardized tracing gem. >>>> >>>> How would you feel about extracting AS::N, actually? Then gems could >>>> adopt it for pub/sub and it would be a lot simpler to plug in a tracing >>>> subscriber. >>>> >>>> >>>>> >>>>> Best, >>>>> Jeremy >>>>> >>>>> On Wed, Oct 17, 2018 at 5:58 AM 'Daniel Schierbeck' via Ruby on Rails: >>>>> Core <rubyonrails-core@googlegroups.com> wrote: >>>>> >>>>>> With the advent of OpenTracing (https://opentracing.io) along with >>>>>> an official Ruby library ( >>>>>> https://github.com/opentracing/opentracing-ruby) as well as growing >>>>>> industry support (e.g. >>>>>> https://www.datadoghq.com/blog/opentracing-datadog-cncf/ and >>>>>> http://blog.scoutapp.com/articles/2018/01/17/tutorial-distributed-tracing-in-ruby-with-opentracing), >>>>>> would it make sense to provide native tracing of Rails using the >>>>>> opentracing-ruby library? >>>>>> >>>>>> At Zendesk we're using Datadog's proprietary tracing API, which >>>>>> monkey patches Rails and other libraries in order to trace key >>>>>> interactions. I think a more sustainable approach would be for libraries >>>>>> to >>>>>> include tracing support out of the box using the standardized OpenTracing >>>>>> APIs. It is then merely a matter of hooking up e.g. the Datadog trace >>>>>> collector in order to get a working tracing setup. >>>>>> >>>>>> If there's interest in this I'd be willing to contribute code. I've >>>>>> done a bunch of working in order to trace various aspects of Rails, >>>>>> including Rack middleware and before/after filters, but without native >>>>>> support, these implementations are brittle and prone to breakage when >>>>>> internal Rails APIs change. >>>>>> >>>>>> I'd love to hear your thoughts on this. >>>>>> >>>>>> Cheers, >>>>>> Daniel Schierbeck >>>>>> >>>>>> -- >>>>>> >>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Ruby on Rails: Core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to rubyonrails-core+unsubscr...@googlegroups.com. >>>>> >>>>> >>>>>> To post to this group, send email to >>>>>> rubyonrails-core@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/rubyonrails-core. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Ruby on Rails: Core" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/rubyonrails-core/qJlL_uVxsnU/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> rubyonrails-core+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to rubyonrails-core@googlegroups.com >>>>> . >>>>> Visit this group at https://groups.google.com/group/rubyonrails-core. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Ruby on Rails: Core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to rubyonrails-core+unsubscr...@googlegroups.com. >>> To post to this group, send email to rubyonrails-core@googlegroups.com. >>> Visit this group at https://groups.google.com/group/rubyonrails-core. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Ruby on Rails: Core" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/rubyonrails-core/qJlL_uVxsnU/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> rubyonrails-core+unsubscr...@googlegroups.com. >> To post to this group, send email to rubyonrails-core@googlegroups.com. >> Visit this group at https://groups.google.com/group/rubyonrails-core. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to rubyonrails-core+unsubscr...@googlegroups.com. > To post to this group, send email to rubyonrails-core@googlegroups.com. > Visit this group at https://groups.google.com/group/rubyonrails-core. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-core+unsubscr...@googlegroups.com. To post to this group, send email to rubyonrails-core@googlegroups.com. Visit this group at https://groups.google.com/group/rubyonrails-core. For more options, visit https://groups.google.com/d/optout.
