Re: Beam Metrics questions
Hi Phil,No, spark does not support committed metrics either. Only dataflow supports them, and only in batch mode. All the other runners except Direct Runner and Dataflow use AccumulatedMetricResult that throws an Exception if the user requests committed metrics, see for ex https://github.com/apache/beam/blob/236d0dd89c210e235ef45be9663ec77267d7b451/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java#L413 Also please take a look at the talk I gave at the ApacheCon in September if you have more questions regarding metrics : https://apachecon.dukecon.org/acna/2018/#/scheduledEvent/e22bd89bacbe03a36 . There is a download link for the slides but no audio/video link unfortunately. Best,Etienne Le vendredi 30 novembre 2018 à 10:10 -0600, Phil Franklin a écrit : > Hi again, Etienne! You didn’t mention whether spark is reporting committed > values now, but you also didn’t mention > opening a ticket concerning the spark output. Am I right in inferring that > spark does in fact report committed > values? > Thanks!-Phil > On 2018/11/30 13:57:43, Etienne Chauchot wrote: > Hi Phil,> Thanks for using MetricsPusher and Beam in general ! > - > MetricsHttpSink works that way: it filters out > committed metrics from the json output when committed metrics are not> > supported. I checked, Flink runner still does > not support committed metrics. So there should be no committed metrics> > values in the output json.There might be a > bug. I'll open a ticket: thx for pointing out ! You tested on flink and > spark> right? and both output committed > metrics values right?> - there is no default mechanism to fallback committed > metrics values on attempted ones> - > Apache Flink does no make flink Accumulators available in detached mode, so > indeed, metrics are not available in this> > mode.> CCing dev list.> Etienne> Le lundi 26 novembre 2018 à 15:57 -0600, > Phil Franklin a écrit :> > All of the discussion I’ve seen says that Flink and Spark only provided > attempted metric values, but when I use> > MetricsHttpSink and look at the JSON it has both attempted and committed > values (albeit, both the same for my simple> > testing). Has the metrics processing been updated recently, and I’m just > missing the change updates? Or are the> > committed values being defaulted to the attempted values? > > > Also, I’ve seen it mentioned that Flink doesn’t report metrics when in > detached mode. Is this still the case?> > > Thanks for your help!>
Re: Beam Metrics questions
Hi Phil, Thanks for the update I was checking the code and I was not understanding how the filtering could fail. Etienne Le vendredi 30 novembre 2018 à 10:53 -0600, Phil Franklin a écrit : > Etienne, I’ve just discovered that the code I used for my tests overrides the > command-line arguments, and while I thought I was testing with the > SparkRunner and FlinkRunner, in fact every test used DirectRunner, which > explains why I was seeing the committed values. So there’s no need for a > ticket concerning committed values from the FlinkRunner. Sorry for the > confusion. > > -Phil
Re: Beam Metrics questions
Etienne, I’ve just discovered that the code I used for my tests overrides the command-line arguments, and while I thought I was testing with the SparkRunner and FlinkRunner, in fact every test used DirectRunner, which explains why I was seeing the committed values. So there’s no need for a ticket concerning committed values from the FlinkRunner. Sorry for the confusion. -Phil
Re: Beam Metrics questions
Hi again, Etienne! You didn’t mention whether spark is reporting committed values now, but you also didn’t mention opening a ticket concerning the spark output. Am I right in inferring that spark does in fact report committed values? Thanks! -Phil On 2018/11/30 13:57:43, Etienne Chauchot wrote: > Hi Phil,> > Thanks for using MetricsPusher and Beam in general ! > > - MetricsHttpSink works that way: it filters out committed metrics from the > json output when committed metrics are not> > supported. I checked, Flink runner still does not support committed metrics. > So there should be no committed metrics> > values in the output json.There might be a bug. I'll open a ticket: thx for > pointing out ! You tested on flink and spark> > right? and both output committed metrics values right?> > - there is no default mechanism to fallback committed metrics values on > attempted ones> > - Apache Flink does no make flink Accumulators available in detached mode, so > indeed, metrics are not available in this> > mode.> > CCing dev list.> > Etienne> > Le lundi 26 novembre 2018 à 15:57 -0600, Phil Franklin a écrit :> > > All of the discussion I’ve seen says that Flink and Spark only provided > > attempted metric values, but when I use> > > MetricsHttpSink and look at the JSON it has both attempted and committed > > values (albeit, both the same for my simple> > > testing). Has the metrics processing been updated recently, and I’m just > > missing the change updates? Or are the> > > committed values being defaulted to the attempted values? > > > > > > Also, I’ve seen it mentioned that Flink doesn’t report metrics when in > > detached mode. Is this still the case?> > > > > > > > > Thanks for your help!> >
Re: Beam Metrics questions
Hi, Etienne! Thanks for the response. Yes, I ran the test with both the flink and spark runners, and both showed committed and attempted values. I didn’t actually use MetricsPusher for these tests. I have questions about MetricsPusher, but I’ll put those in another post. -Phil On 2018/11/30 13:57:43, Etienne Chauchot wrote: > Hi Phil,> > Thanks for using MetricsPusher and Beam in general ! > > - MetricsHttpSink works that way: it filters out committed metrics from the > json output when committed metrics are not> > supported. I checked, Flink runner still does not support committed metrics. > So there should be no committed metrics> > values in the output json.There might be a bug. I'll open a ticket: thx for > pointing out ! You tested on flink and spark> > right? and both output committed metrics values right?> > - there is no default mechanism to fallback committed metrics values on > attempted ones> > - Apache Flink does no make flink Accumulators available in detached mode, so > indeed, metrics are not available in this> > mode.> > CCing dev list.> > Etienne> > Le lundi 26 novembre 2018 à 15:57 -0600, Phil Franklin a écrit :> > > All of the discussion I’ve seen says that Flink and Spark only provided > > attempted metric values, but when I use> > > MetricsHttpSink and look at the JSON it has both attempted and committed > > values (albeit, both the same for my simple> > > testing). Has the metrics processing been updated recently, and I’m just > > missing the change updates? Or are the> > > committed values being defaulted to the attempted values? > > > > > > Also, I’ve seen it mentioned that Flink doesn’t report metrics when in > > detached mode. Is this still the case?> > > > > > > > > Thanks for your help!> >
Re: Beam Metrics questions
Hi Phil, Thanks for using MetricsPusher and Beam in general ! - MetricsHttpSink works that way: it filters out committed metrics from the json output when committed metrics are not supported. I checked, Flink runner still does not support committed metrics. So there should be no committed metrics values in the output json.There might be a bug. I'll open a ticket: thx for pointing out ! You tested on flink and spark right? and both output committed metrics values right? - there is no default mechanism to fallback committed metrics values on attempted ones - Apache Flink does no make flink Accumulators available in detached mode, so indeed, metrics are not available in this mode. CCing dev list. Etienne Le lundi 26 novembre 2018 à 15:57 -0600, Phil Franklin a écrit : > All of the discussion I’ve seen says that Flink and Spark only provided > attempted metric values, but when I use > MetricsHttpSink and look at the JSON it has both attempted and committed > values (albeit, both the same for my simple > testing). Has the metrics processing been updated recently, and I’m just > missing the change updates? Or are the > committed values being defaulted to the attempted values? > > Also, I’ve seen it mentioned that Flink doesn’t report metrics when in > detached mode. Is this still the case? > > > Thanks for your help!
Beam Metrics questions
All of the discussion I’ve seen says that Flink and Spark only provided attempted metric values, but when I use MetricsHttpSink and look at the JSON it has both attempted and committed values (albeit, both the same for my simple testing). Has the metrics processing been updated recently, and I’m just missing the change updates? Or are the committed values being defaulted to the attempted values? Also, I’ve seen it mentioned that Flink doesn’t report metrics when in detached mode. Is this still the case? Thanks for your help!
