Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Hi, Just wanted to mention that I updated this document with one detail https://s.apache.org/beam-gcp-debuggability Date Changes Sept 8, 2020 - Clarified that the InstructionRequest/Control Channel will be used in “Proposal: SDKHs to Report non-bundle metrics.” May 15, 2020 -

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link this time)

wrote the metrics for Java. >>> > >>> > Best >>> > -P. >>> > >>> > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote: >>> >> >>> >> Hello, >>> >> >>> >> I have created a proposal for Ap

Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Thanks everyone. I was able to collect a lot of good feedback from everyone who contributed. I am going to wrap it up for now and label the design as "Design Finalized (Unimplemented)". I really believe we have made a much better design than I initially wrote up. I couldn't have done it without

Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Thanks to all who have spent their time on this, there were many great suggestions, just another reminder that tomorrow I will be finalizing the documents, unless there are any major objections left. Please take a look at it if you are interested. I will still welcome feedback at any time :).

Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Thanks again for more feedback :). I have iterated on things again. I'll report back at the end of the week. If there are no major disagreements still, I'll close the discussion, believe it to be in a good enough state to start some implementation. But welcome feedback. Latest changes are

Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Thanks for the great feedback so far :). I've included many new ideas, and made some revisions. Both docs have changed a fair bit since the initial mail out. https://s.apache.org/beam-gcp-debuggability https://s.apache.org/beam-histogram-metrics PTAL and let me know what you think, and hopefully

Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Thanks everyone so far for taking a look so far :). I am hoping to have this finalize the two reviews by the end of next week, May 15th. I'll continue to follow up on feedback and make changes, and I will add some more mentions to the documents to draw attention

Re: [Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Thanks, also took a look and left some comments. On Tue, May 5, 2020 at 6:24 PM Alex Amato wrote: > Hello, > > I created another design document. This time for GCP IO Debuggability > Metrics. Which defines some new metrics to collect in the GCP IO libraries. > This is for monitoring request

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link this time)

gt; > Best >> > -P. >> > >> > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote: >> >> >> >> Hello, >> >> >> >> I have created a proposal for Apache Beam FN API to support Histogram >> Style Metrics. Which defines a m

[Proposal] Apache Beam Fn API - GCP IO Debuggability Metrics

Hello, I created another design document. This time for GCP IO Debuggability Metrics. Which defines some new metrics to collect in the GCP IO libraries. This is for monitoring request counts and request latencies. Please take a look and let me know what you think:

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link this time)

up getting it merged, but it's worth looking at the work. I also > remember Scott Wegner wrote the metrics for Java. > > > > Best > > -P. > > > > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote: > >> > >> Hello, > >> > >> I have

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link this time)

Best > -P. > > On Mon, May 4, 2020 at 3:33 PM Alex Amato wrote: >> >> Hello, >> >> I have created a proposal for Apache Beam FN API to support Histogram Style >> Metrics. Which defines a method to collect Histogram style metrics and pass >> them

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link this time)

, 2020 at 3:33 PM Alex Amato wrote: > Hello, > > I have created a proposal for Apache Beam FN API to support Histogram > Style Metrics > <https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit#>. > Which defines a method to collect Histogram sty

[Proposal] Apache Beam Fn API - Histogram Style Metrics (Correct link this time)

Hello, I have created a proposal for Apache Beam FN API to support Histogram Style Metrics <https://docs.google.com/document/d/1kiNG2BAR-51pRdBCK4-XFmc0WuIkSuBzeb__Zv8owbU/edit#>. Which defines a method to collect Histogram style metrics and pass them over the FN API. I would love to hea

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics

> Hello, >> >> I have created a proposal for Apache Beam FN API to support Histogram >> Style Metrics >> <https://docs.google.com/document/d/1MtBZYV7NAcfbwyy9Op8STeFNBxtljxgy69FkHMvhTMA/edit#heading=h.c6fjf0g6rsbc>. >> Which defines a method to collect Histogra

Re: [Proposal] Apache Beam Fn API - Histogram Style Metrics

Hi Alex! Thanks for the proposal. I've created https://s.apache.org/beam-histogram-metrics On Mon, May 4, 2020 at 2:44 PM Alex Amato wrote: > Hello, > > I have created a proposal for Apache Beam FN API to support Histogram > Style Metrics > <https://docs.goo

[Proposal] Apache Beam Fn API - Histogram Style Metrics

Hello, I have created a proposal for Apache Beam FN API to support Histogram Style Metrics <https://docs.google.com/document/d/1MtBZYV7NAcfbwyy9Op8STeFNBxtljxgy69FkHMvhTMA/edit#heading=h.c6fjf0g6rsbc>. Which defines a method to collect Histogram style metrics and pass them over the FN API.

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Hello, I have rewritten most of the proposal. Though I think that there is some more research that needs to be done to get the Metric specification perfect. I plan to do more research, and would like to ask you all for more help to make this proposal better. In particular, now that the metrics

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

That sounds like a very reasonable choice -- given the discussion seemed to be focusing on the differences between these two categories, separating them will allow the proposal (and implementation) to address each category in the best way possible without needing to make compromises. Looking

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Hello, I just wanted to give an update . After some discussion, I've realized that its best to break up the two concepts, with two separate way of reporting monitoring data. These two categories are: 1. Metrics - Counters, Gauges, Distributions. These are well defined concepts for

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

I agree that the user/system dichotomy is false, the real question of how counters can be scoped to avoid accidental (or even intentional) interference. A system that entirely controls the interaction between the "user" (from its perspective) and the underlying system can do this by prefixing all

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

One reason I resist the user/system distinction is that Beam is a multi-party system with at least SDK, runner, and pipeline. Often there may be a DSL like SQL or Scio, or similarly someone may be building a platform for their company where there is no user authoring the pipeline. Should Scio,

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

On Fri, Apr 13, 2018 at 4:30 PM Alex Amato wrote: > There are a few more confusing concepts in this thread > *Name* > >- Name can mean a *"string name"* used to refer to a metric in a >metrics system such as stackdriver, i.e. "ElementCount", "ExecutionTime" >-

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

On Fri, Apr 13, 2018 at 3:28 PM Andrea Foegler wrote: > Thanks, Robert! > > I think my lack of clarity is around the MetricSpec. Maybe what's in my > head and what's being proposed are the same thing. When I read that the > MetricSpec describes the proto structure, that

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

t;>>>> what what users want and metrics back end support) it seems >>>>>>>>>>>>>>> like we are >>>>>>>>>>>>>>> doing user metrics right. >>>>>>>>>>>>&g

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

t;>>>>>>>>>>>>> By "type" of metric, I mean both the data types (including >>>>>>>>>>>>>>>> their encoding) and accumulator strategy. So sumint would be a &

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

>>>>>>>>>>>>>>>> their encoding) and accumulator strategy. So sumint would be a >>>>>>>>>>>>>>>> type, as >>>>>>>>>>>>>>>> would double-distribution. >>>>>>>>>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

y, what is the >>>>>>>>>>>>>>>> "type" of sumint, >>>>>>>>>>>>>>>> sumlong, meanlong, etc? >>>>>>>>>>>>>>>> >>>>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

e support custom >>>>>>>>>>>>>>>> metrics of >>>>>>>>>>>>>>>> standard type. >>>>>>>>>>>>>>>> >>&

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

based on the fact they just weren't used enough to >>>>>>>>>>>>>>>> justify support. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is there a reason we are bringing that complexity

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

;>>>>>>>>>>>>>> On Wed, Apr 11, 2018, 5:43 PM Robert Bradshaw < >>>>>>>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>&

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

t;>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. I think this has simplified things.

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

icSpec field be augmented with an additional field >>>>>>>>>>>>>>> "type" which is >>>>>>>>>>>>>>> a urn specifying the type of metric it is (i.e. the contents of >>>>>>>>>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

gt;>>>>>> aggregate and >>>>>>>>>>>>> report/return to the SDK metrics that it did not itself >>>>>>>>>>>>> understand the >>>>>>>>>>>>> semantic meaning of. (It would

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

t;>>>> >>>>>>>>>>> - Robert >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 11, 2018 at 5:12 PM A

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

fied) name. >>>>>>>>> >>>>>>>>> - Robert >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Apr 11, 2018 at 5:12 PM Alex Amato <ajam...@google.com> >

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

> bit. >>>>>>>>> >>>>>>>>> The point of this change was to futureproof the possibility of >>>>>>>>> allowing custom user metrics, with custom aggregation functions for >>>>>>>>> its >>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

h custom aggregation functions for its >>>>>>>> metric updates. >>>>>>>> Now that each metric has an aggregation_entity associated with it >>>>>>>> (e.g. PCollection, PTransform), we can design an approach which >>>>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

at each metric has an aggregation_entity associated with it >>>>>>> (e.g. PCollection, PTransform), we can design an approach which forwards >>>>>>> the opaque bytes metric updates, without deserializing them. These are >>>>>>> forwarded to us

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

me of the URN metric protos, as they >>>>>> do not need to keep track of ptransform names inside themselves now. The >>>>>> result is simpler structures, for the metrics as the entities are pulled >>>>>> outside of the metric. >>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

as they >>>>> do not need to keep track of ptransform names inside themselves now. The >>>>> result is simpler structures, for the metrics as the entities are pulled >>>>> outside of the metric. >>>>> >>>>&

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

e pulled >>>> outside of the metric. >>>> >>>> I have mentioned this in the doc now, and wanted to draw attention to >>>> this particular revision. >>>> >>>> >>>> >>>> On Tue, Apr 10, 2018 at 9:53 AM A

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

t; >>> >>> >>> On Tue, Apr 10, 2018 at 9:53 AM Alex Amato <ajam...@google.com> wrote: >>> >>>> I've gathered a lot of feedback so far and want to make a decision by >>>> Friday, and begin working on related PRs next week. >>&

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

ay, and begin working on related PRs next week. >>> >>> Please make sure that you provide your feedback before then and I will >>> post the final decisions made to this thread Friday afternoon. >>> >>> >>> On Thu, Apr 5, 2018 at 1:38 AM Ismaël M

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

a short link so people can refer to it easily in >> future discussions, website, etc. >> >> https://s.apache.org/beam-fn-api-metrics >> >> Thanks for sharing. >> >> >> On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw <rober...@google.com> >>

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Mejía <ieme...@gmail.com> wrote: > Nice, I created a short link so people can refer to it easily in > future discussions, website, etc. > > https://s.apache.org/beam-fn-api-metrics > > Thanks for sharing. > > > On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw <robe

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Nice, I created a short link so people can refer to it easily in future discussions, website, etc. https://s.apache.org/beam-fn-api-metrics Thanks for sharing. On Wed, Apr 4, 2018 at 11:28 PM, Robert Bradshaw <rober...@google.com> wrote: > Thanks for the nice writeup. I added some

Re: Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Thanks for the nice writeup. I added some comments. On Wed, Apr 4, 2018 at 1:53 PM Alex Amato wrote: > Hello beam community, > > Thank you everyone for your initial feedback on this proposal so far. I > have made some revisions based on the feedback. There were some larger >

Updated [Proposal] Apache Beam Fn API : Defining and adding SDK Metrics

Hello beam community, Thank you everyone for your initial feedback on this proposal so far. I have made some revisions based on the feedback. There were some larger questions asking about alternatives. For each of these I have added a section tagged with [Alternatives] and discussed my

Re: Beam Fn API

t;>> https://s.apache.org/beam-fn-state-api-and-bundle-processing >>> >>> The document does require a strong foundation in the Apache Beam model >>> and >>> a good understanding of the prior shared docs: >>> * How to process a bundle: https://s.apache.

Re: Beam Fn API

te-api-and-bundle-processing The document does require a strong foundation in the Apache Beam model and a good understanding of the prior shared docs: * How to process a bundle: https://s.apache.org/beam-fn-api -processing-a-bundle * How to send and receive data: https://s.apache.org/beam-fn-api -send-and-r

Re: Beam Fn API

ttps://s.apache.org/beam-fn-state-api-and-bundle-processing > > The document does require a strong foundation in the Apache Beam model and > a good understanding of the prior shared docs: > * How to process a bundle: https://s.apache.org/beam-fn-api > -processing-a-bund

Re: Beam Fn API

The document does require a strong foundation in the Apache Beam model and a good understanding of the prior shared docs: * How to process a bundle: https://s.apache.org/beam-fn-api -processing-a-bundle * How to send and receive data: https://s.apache.org/beam-fn-api -send-and-receive-data I could really use

Re: Beam Fn API

e somehow incorrectly formatted > in your mail. > > * How to process a bundle: > https://s.apache.org/beam-fn-api-processing-a-bundle > * How to send and receive data: > https://s.apache.org/beam-fn-api-send-and-receive-data > > By the way, is there a way to find them from the Beam web

Re: Beam Fn API

Thanks Lukasz. The following two links were somehow incorrectly formatted in your mail. * How to process a bundle: https://s.apache.org/beam-fn-api-processing-a-bundle * How to send and receive data: https://s.apache.org/beam-fn-api-send-and-receive-data By the way, is there a way to find them

Re: Beam Fn API

the overview: https://s.apache.org/beam-fn-api And for those of you who have been following this thread and contributors focusing on Runner integration with Apache Beam: * How to process a bundle: https://s.apache.org/beam-fn-api-processing-a- bundle * How to send and receive data: https

Re: Beam Fn API

t; > > > > > > > > On Fri, Jan 20, 2017 at 4:56 AM, Jean-Baptiste Onofré < > > j...@nanthrax.net > > > > > > > > > wrote: > > > > > > > > > >> Hi Luke, > > > > >> > > > > >> that's really great a

Re: Beam Fn API

> Regards >> JB >> >> >> On 01/19/2017 03:56 PM, Lukasz Cwik wrote: >> >>> I have been prototyping several components towards the Beam technical >>> vision of being able to execute an arbitrary language using an arbitrary >>> runner. >

Re: Beam Fn API

he Beam technical >> vision of being able to execute an arbitrary language using an arbitrary >> runner. >> >> I would like to share this overview [1] of what I have been working >> towards. I also share this PR [2] with a proposed API, service definitions >> and partial i

Re: Beam Fn API

I also share this PR [2] with a proposed API, service definitions > and partial implementation. > > 1: https://s.apache.org/beam-fn-api > 2: https://github.com/apache/beam/pull/1801 > > Please comment on the overview within this thread, and any specific code > comments on the PR directly. > > Luke >

Beam Fn API

and partial implementation. 1: https://s.apache.org/beam-fn-api 2: https://github.com/apache/beam/pull/1801 Please comment on the overview within this thread, and any specific code comments on the PR directly. Luke