Hi, Just wanted to mention that I updated this document with one detail https://s.apache.org/beam-gcp-debuggability
Date Changes Sept 8, 2020 - Clarified that the InstructionRequest/Control Channel will be used in “Proposal: SDKHs to Report non-bundle metrics.” May 15, 2020 - Completed review with beam dev list. PTAL, and LMK what you think On Fri, May 15, 2020 at 6:02 PM Alex Amato <[email protected]> wrote: > Thanks everyone. I was able to collect a lot of good feedback from > everyone who contributed. I am going to wrap it up for now and label the > design as "Design Finalized (Unimplemented)". > > I really believe we have made a much better design than I initially wrote > up. I couldn't have done it without the help of everyone who offered their > time, energy and viewpoints. :) > > Thanks again, please let me know if you see any major issues with the > design still. I think I have enough information to begin some > implementation as soon as I have some time in the coming weeks. > Alex > > https://s.apache.org/beam-gcp-debuggability > https://s.apache.org/beam-histogram-metrics > > On Thu, May 14, 2020 at 5:22 PM Alex Amato <[email protected]> wrote: > >> Thanks to all who have spent their time on this, there were many great >> suggestions, just another reminder that tomorrow I will be finalizing the >> documents, unless there are any major objections left. Please take a look >> at it if you are interested. >> >> I will still welcome feedback at any time :). >> >> But I believe we have gathered enough information to produce a good >> design, which I will start to work on soon. >> I will begin to build the necessary subset of the new features proposed >> to support the BigQueryIO metrics use case, proposed. >> I will likely start with the python SDK first. >> >> https://s.apache.org/beam-gcp-debuggability >> https://s.apache.org/beam-histogram-metrics >> >> >> On Wed, May 13, 2020 at 3:07 PM Alex Amato <[email protected]> wrote: >> >>> Thanks again for more feedback :). I have iterated on things again. I'll >>> report back at the end of the week. If there are no major disagreements >>> still, I'll close the discussion, believe it to be in a good enough state >>> to start some implementation. But welcome feedback. >>> >>> Latest changes are changing the exponential format to allow denser >>> buckets. Using only two MonitoringInfoSpec now for all of the IOs to use. >>> Requiring some labels, but allowing optional >>> ones for specific IOs to provide more contents. >>> >>> https://s.apache.org/beam-gcp-debuggability >>> https://s.apache.org/beam-histogram-metrics >>> >>> On Mon, May 11, 2020 at 4:24 PM Alex Amato <[email protected]> wrote: >>> >>>> Thanks for the great feedback so far :). I've included many new ideas, >>>> and made some revisions. Both docs have changed a fair bit since the >>>> initial mail out. >>>> >>>> https://s.apache.org/beam-gcp-debuggability >>>> https://s.apache.org/beam-histogram-metrics >>>> >>>> PTAL and let me know what you think, and hopefully we can resolve major >>>> issues by the end of the week. I'll try to finalize things by then, but of >>>> course always stay open to your great ideas. :) >>>> >>>> On Wed, May 6, 2020 at 6:19 PM Alex Amato <[email protected]> wrote: >>>> >>>>> Thanks everyone so far for taking a look so far :). >>>>> >>>>> I am hoping to have this finalize the two reviews by the end of next >>>>> week, May 15th. >>>>> >>>>> I'll continue to follow up on feedback and make changes, and I will >>>>> add some more mentions to the documents to draw attention >>>>> >>>>> https://s.apache.org/beam-gcp-debuggability >>>>> https://s.apache.org/beam-histogram-metrics >>>>> >>>>> On Wed, May 6, 2020 at 10:00 AM Luke Cwik <[email protected]> wrote: >>>>> >>>>>> Thanks, also took a look and left some comments. >>>>>> >>>>>> On Tue, May 5, 2020 at 6:24 PM Alex Amato <[email protected]> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I created another design document. This time for GCP IO >>>>>>> Debuggability Metrics. Which defines some new metrics to collect in the >>>>>>> GCP >>>>>>> IO libraries. This is for monitoring request counts and request >>>>>>> latencies. >>>>>>> >>>>>>> Please take a look and let me know what you think: >>>>>>> https://s.apache.org/beam-gcp-debuggability >>>>>>> >>>>>>> I also sent out a separate design yesterday ( >>>>>>> https://s.apache.org/beam-histogram-metrics) which is related as >>>>>>> this document uses a Histogram style metric :). >>>>>>> >>>>>>> I would love some feedback to make this feature the best possible :D, >>>>>>> Alex >>>>>>> >>>>>>
