Re: Custom metrics not showing on Stackdriver
Created this SO question: https://stackoverflow.com/questions/48530496/google-dataflow-custom-metrics-not-showing-on-stackdriver Thanks Andrea! On Tue, Jan 30, 2018 at 9:54 PM Andrea Foegler wrote: > I would suggest Platform Support or StackOverflow as the best places to > request Dataflow-specific support. > > This could be an issue coordinating between your Stackdriver Account(s) > and your Cloud project(s). We can continue to discuss / investigate > through one of the above forums. > > On Tue, Jan 30, 2018 at 12:30 PM, Carlos Alonso > wrote: > >> Thanks Andrea!! >> >> Do you mean using the UserVoice forum? >> >> Aside from that, there's something that could be helpful that is that >> when I navigate https://app.google.stackdriver.com/services/dataflow the >> message I get is this: >> "You do not have any resources of this type being monitored by >> Stackdriver." and that's weird as well. As if our Cloud Dataflow wasn't >> properly connected to Stackdriver, but, on the other hand. Some metrics are >> displayed and can be monitored such as System Lag, Watermark, etc... >> >> Thanks! >> >> On Tue, Jan 30, 2018 at 9:20 PM Andrea Foegler >> wrote: >> >>> Hi Carlos - >>> >>> This sounds like something we should investigate further. Since it >>> appears to be a Dataflow specific question / issue, would you mind posting >>> or following up in a Dataflow-specific forum or through Google Cloud >>> Platform Support: https://cloud.google.com/dataflow/support? Feel free >>> to mention my name in your contact. >>> >>> Cheers >>> Andrea >>> >>> >>> >>> On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso >>> wrote: >>> >>>> Hi Andrea, thank you very much for your response. >>>> >>>> I've followed your directions and only droppedDueToLateness appears. >>>> The way I'm creating those metrics is: >>>> >>>> Metrics.counter("Ingester", "IngestedMessages").inc() >>>> >>>> I can see those counters on the Custom Counters section on the Google >>>> Dataflow UI, but nothing appears on Stackdriver... >>>> >>>> Thanks! >>>> >>>> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler >>>> wrote: >>>> >>>>> Hi Carlos - >>>>> >>>>> Custom metrics can be "listed" by going to the Metric Explorer in >>>>> Stackdriver and entering "custom.googleapis.com/dataflow" in the >>>>> filter. >>>>> If that list contains more than 100 different names, new custom >>>>> metrics will not be created. If this is a case, there should be a message >>>>> in the job log reporting as much. >>>>> (We are working with Stackdriver to improve this experience.) >>>>> >>>>> Also, we do not currently export distribution metrics to Stackdriver >>>>> because we don't yet have a good mechanism to do so. Gauge metrics are >>>>> not >>>>> implemented yet and would not appear in either the Dataflow UI or >>>>> Stackdriver. >>>>> >>>>> These are the only explanations I can think for for these metrics to >>>>> not show up. If neither of these are the case, I'm happy to investigate >>>>> further on a particular instance. >>>>> >>>>> Cheers >>>>> Andrea >>>>> >>>>> >>>>> >>>>> On 2018/01/23 19:59:08, Carlos Alonso wrote: >>>>> > Hi everyone!!> >>>>> > >>>>> > I'm trying to get a deeper view on my dataflow jobs by measuring >>>>> parts of> >>>>> > it using `Metrics.counter|gauge` but I cannot find how to see them >>>>> on> >>>>> > Stackdriver.> >>>>> > >>>>> > I have a premium Stackdriver account and I can see those counters >>>>> under the> >>>>> > Custom Counters section on the Dataflow UI.> >>>>> > >>>>> > I can see droppedDueToLateness 'custom' counter though on >>>>> Stackdriver that> >>>>> > seems to be created via 'Metrics.counter' as well...> >>>>> > >>>>> > What am I missing?> >>>>> > >>>>> > Regards> >>>>> > >>>>> >>>> >>> >
Re: Custom metrics not showing on Stackdriver
I would suggest Platform Support or StackOverflow as the best places to request Dataflow-specific support. This could be an issue coordinating between your Stackdriver Account(s) and your Cloud project(s). We can continue to discuss / investigate through one of the above forums. On Tue, Jan 30, 2018 at 12:30 PM, Carlos Alonso wrote: > Thanks Andrea!! > > Do you mean using the UserVoice forum? > > Aside from that, there's something that could be helpful that is that when > I navigate https://app.google.stackdriver.com/services/dataflow the > message I get is this: > "You do not have any resources of this type being monitored by > Stackdriver." and that's weird as well. As if our Cloud Dataflow wasn't > properly connected to Stackdriver, but, on the other hand. Some metrics are > displayed and can be monitored such as System Lag, Watermark, etc... > > Thanks! > > On Tue, Jan 30, 2018 at 9:20 PM Andrea Foegler wrote: > >> Hi Carlos - >> >> This sounds like something we should investigate further. Since it >> appears to be a Dataflow specific question / issue, would you mind posting >> or following up in a Dataflow-specific forum or through Google Cloud >> Platform Support: https://cloud.google.com/dataflow/support? Feel free >> to mention my name in your contact. >> >> Cheers >> Andrea >> >> >> >> On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso >> wrote: >> >>> Hi Andrea, thank you very much for your response. >>> >>> I've followed your directions and only droppedDueToLateness appears. >>> The way I'm creating those metrics is: >>> >>> Metrics.counter("Ingester", "IngestedMessages").inc() >>> >>> I can see those counters on the Custom Counters section on the Google >>> Dataflow UI, but nothing appears on Stackdriver... >>> >>> Thanks! >>> >>> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler >>> wrote: >>> Hi Carlos - Custom metrics can be "listed" by going to the Metric Explorer in Stackdriver and entering "custom.googleapis.com/dataflow" in the filter. If that list contains more than 100 different names, new custom metrics will not be created. If this is a case, there should be a message in the job log reporting as much. (We are working with Stackdriver to improve this experience.) Also, we do not currently export distribution metrics to Stackdriver because we don't yet have a good mechanism to do so. Gauge metrics are not implemented yet and would not appear in either the Dataflow UI or Stackdriver. These are the only explanations I can think for for these metrics to not show up. If neither of these are the case, I'm happy to investigate further on a particular instance. Cheers Andrea On 2018/01/23 19:59:08, Carlos Alonso wrote: > Hi everyone!!> > > I'm trying to get a deeper view on my dataflow jobs by measuring parts of> > it using `Metrics.counter|gauge` but I cannot find how to see them on> > Stackdriver.> > > I have a premium Stackdriver account and I can see those counters under the> > Custom Counters section on the Dataflow UI.> > > I can see droppedDueToLateness 'custom' counter though on Stackdriver that> > seems to be created via 'Metrics.counter' as well...> > > What am I missing?> > > Regards> > >>> >>
Re: Custom metrics not showing on Stackdriver
Thanks Andrea!! Do you mean using the UserVoice forum? Aside from that, there's something that could be helpful that is that when I navigate https://app.google.stackdriver.com/services/dataflow the message I get is this: "You do not have any resources of this type being monitored by Stackdriver." and that's weird as well. As if our Cloud Dataflow wasn't properly connected to Stackdriver, but, on the other hand. Some metrics are displayed and can be monitored such as System Lag, Watermark, etc... Thanks! On Tue, Jan 30, 2018 at 9:20 PM Andrea Foegler wrote: > Hi Carlos - > > This sounds like something we should investigate further. Since it > appears to be a Dataflow specific question / issue, would you mind posting > or following up in a Dataflow-specific forum or through Google Cloud > Platform Support: https://cloud.google.com/dataflow/support? Feel free > to mention my name in your contact. > > Cheers > Andrea > > > > On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso > wrote: > >> Hi Andrea, thank you very much for your response. >> >> I've followed your directions and only droppedDueToLateness appears. >> The way I'm creating those metrics is: >> >> Metrics.counter("Ingester", "IngestedMessages").inc() >> >> I can see those counters on the Custom Counters section on the Google >> Dataflow UI, but nothing appears on Stackdriver... >> >> Thanks! >> >> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler >> wrote: >> >>> Hi Carlos - >>> >>> Custom metrics can be "listed" by going to the Metric Explorer in >>> Stackdriver and entering "custom.googleapis.com/dataflow" in the filter. >>> If that list contains more than 100 different names, new custom metrics >>> will not be created. If this is a case, there should be a message in the >>> job log reporting as much. >>> (We are working with Stackdriver to improve this experience.) >>> >>> Also, we do not currently export distribution metrics to Stackdriver >>> because we don't yet have a good mechanism to do so. Gauge metrics are not >>> implemented yet and would not appear in either the Dataflow UI or >>> Stackdriver. >>> >>> These are the only explanations I can think for for these metrics to not >>> show up. If neither of these are the case, I'm happy to investigate >>> further on a particular instance. >>> >>> Cheers >>> Andrea >>> >>> >>> >>> On 2018/01/23 19:59:08, Carlos Alonso wrote: >>> > Hi everyone!!> >>> > >>> > I'm trying to get a deeper view on my dataflow jobs by measuring parts >>> of> >>> > it using `Metrics.counter|gauge` but I cannot find how to see them on> >>> > Stackdriver.> >>> > >>> > I have a premium Stackdriver account and I can see those counters >>> under the> >>> > Custom Counters section on the Dataflow UI.> >>> > >>> > I can see droppedDueToLateness 'custom' counter though on Stackdriver >>> that> >>> > seems to be created via 'Metrics.counter' as well...> >>> > >>> > What am I missing?> >>> > >>> > Regards> >>> > >>> >> >
Re: Custom metrics not showing on Stackdriver
Hi Carlos - This sounds like something we should investigate further. Since it appears to be a Dataflow specific question / issue, would you mind posting or following up in a Dataflow-specific forum or through Google Cloud Platform Support: https://cloud.google.com/dataflow/support? Feel free to mention my name in your contact. Cheers Andrea On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso wrote: > Hi Andrea, thank you very much for your response. > > I've followed your directions and only droppedDueToLateness appears. > The way I'm creating those metrics is: > > Metrics.counter("Ingester", "IngestedMessages").inc() > > I can see those counters on the Custom Counters section on the Google > Dataflow UI, but nothing appears on Stackdriver... > > Thanks! > > On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler wrote: > >> Hi Carlos - >> >> Custom metrics can be "listed" by going to the Metric Explorer in >> Stackdriver and entering "custom.googleapis.com/dataflow" in the filter. >> If that list contains more than 100 different names, new custom metrics >> will not be created. If this is a case, there should be a message in the >> job log reporting as much. >> (We are working with Stackdriver to improve this experience.) >> >> Also, we do not currently export distribution metrics to Stackdriver >> because we don't yet have a good mechanism to do so. Gauge metrics are not >> implemented yet and would not appear in either the Dataflow UI or >> Stackdriver. >> >> These are the only explanations I can think for for these metrics to not >> show up. If neither of these are the case, I'm happy to investigate >> further on a particular instance. >> >> Cheers >> Andrea >> >> >> >> On 2018/01/23 19:59:08, Carlos Alonso wrote: >> > Hi everyone!!> >> > >> > I'm trying to get a deeper view on my dataflow jobs by measuring parts >> of> >> > it using `Metrics.counter|gauge` but I cannot find how to see them on> >> > Stackdriver.> >> > >> > I have a premium Stackdriver account and I can see those counters under >> the> >> > Custom Counters section on the Dataflow UI.> >> > >> > I can see droppedDueToLateness 'custom' counter though on Stackdriver >> that> >> > seems to be created via 'Metrics.counter' as well...> >> > >> > What am I missing?> >> > >> > Regards> >> > >> >
Re: Custom metrics not showing on Stackdriver
Hi Andrea, thank you very much for your response. I've followed your directions and only droppedDueToLateness appears. The way I'm creating those metrics is: Metrics.counter("Ingester", "IngestedMessages").inc() I can see those counters on the Custom Counters section on the Google Dataflow UI, but nothing appears on Stackdriver... Thanks! On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler wrote: > Hi Carlos - > > Custom metrics can be "listed" by going to the Metric Explorer in > Stackdriver and entering "custom.googleapis.com/dataflow" in the filter. > If that list contains more than 100 different names, new custom metrics > will not be created. If this is a case, there should be a message in the > job log reporting as much. > (We are working with Stackdriver to improve this experience.) > > Also, we do not currently export distribution metrics to Stackdriver > because we don't yet have a good mechanism to do so. Gauge metrics are not > implemented yet and would not appear in either the Dataflow UI or > Stackdriver. > > These are the only explanations I can think for for these metrics to not > show up. If neither of these are the case, I'm happy to investigate > further on a particular instance. > > Cheers > Andrea > > > > On 2018/01/23 19:59:08, Carlos Alonso wrote: > > Hi everyone!!> > > > > I'm trying to get a deeper view on my dataflow jobs by measuring parts > of> > > it using `Metrics.counter|gauge` but I cannot find how to see them on> > > Stackdriver.> > > > > I have a premium Stackdriver account and I can see those counters under > the> > > Custom Counters section on the Dataflow UI.> > > > > I can see droppedDueToLateness 'custom' counter though on Stackdriver > that> > > seems to be created via 'Metrics.counter' as well...> > > > > What am I missing?> > > > > Regards> > > >
Re: Custom metrics not showing on Stackdriver
Hi Carlos - Custom metrics can be "listed" by going to the Metric Explorer in Stackdriver and entering "custom.googleapis.com/dataflow" in the filter. If that list contains more than 100 different names, new custom metrics will not be created. If this is a case, there should be a message in the job log reporting as much. (We are working with Stackdriver to improve this experience.) Also, we do not currently export distribution metrics to Stackdriver because we don't yet have a good mechanism to do so. Gauge metrics are not implemented yet and would not appear in either the Dataflow UI or Stackdriver. These are the only explanations I can think for for these metrics to not show up. If neither of these are the case, I'm happy to investigate further on a particular instance. Cheers Andrea On 2018/01/23 19:59:08, Carlos Alonso wrote: > Hi everyone!!> > > I'm trying to get a deeper view on my dataflow jobs by measuring parts of> > it using `Metrics.counter|gauge` but I cannot find how to see them on> > Stackdriver.> > > I have a premium Stackdriver account and I can see those counters under the> > Custom Counters section on the Dataflow UI.> > > I can see droppedDueToLateness 'custom' counter though on Stackdriver that> > seems to be created via 'Metrics.counter' as well...> > > What am I missing?> > > Regards> >
Custom metrics not showing on Stackdriver
Hi everyone!! I'm trying to get a deeper view on my dataflow jobs by measuring parts of it using `Metrics.counter|gauge` but I cannot find how to see them on Stackdriver. I have a premium Stackdriver account and I can see those counters under the Custom Counters section on the Dataflow UI. I can see droppedDueToLateness 'custom' counter though on Stackdriver that seems to be created via 'Metrics.counter' as well... What am I missing? Regards