Re: Custom metrics not showing on Stackdriver

[Carlos Alonso]

Created this SO question:
https://stackoverflow.com/questions/48530496/google-dataflow-custom-metrics-not-showing-on-stackdriver

Thanks Andrea!

On Tue, Jan 30, 2018 at 9:54 PM Andrea Foegler <foeg...@google.com> wrote:

> I would suggest Platform Support or StackOverflow as the best places to
> request Dataflow-specific support.
>
> This could be an issue coordinating between your Stackdriver Account(s)
> and your Cloud project(s).  We can continue to discuss / investigate
> through one of the above forums.
>
> On Tue, Jan 30, 2018 at 12:30 PM, Carlos Alonso <car...@mrcalonso.com>
> wrote:
>
>> Thanks Andrea!!
>>
>> Do you mean using the UserVoice forum?
>>
>> Aside from that, there's something that could be helpful that is that
>> when I navigate https://app.google.stackdriver.com/services/dataflow the
>> message I get is this:
>> "You do not have any resources of this type being monitored by
>> Stackdriver." and that's weird as well. As if our Cloud Dataflow wasn't
>> properly connected to Stackdriver, but, on the other hand. Some metrics are
>> displayed and can be monitored such as System Lag, Watermark, etc...
>>
>> Thanks!
>>
>> On Tue, Jan 30, 2018 at 9:20 PM Andrea Foegler <foeg...@google.com>
>> wrote:
>>
>>> Hi Carlos -
>>>
>>> This sounds like something we should investigate further.  Since it
>>> appears to be a Dataflow specific question / issue, would you mind posting
>>> or following up in a Dataflow-specific forum or through Google Cloud
>>> Platform Support: https://cloud.google.com/dataflow/support?  Feel free
>>> to mention my name in your contact.
>>>
>>> Cheers
>>> Andrea
>>>
>>>
>>>
>>> On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso <car...@mrcalonso.com>
>>> wrote:
>>>
>>>> Hi Andrea, thank you very much for your response.
>>>>
>>>> I've followed your directions and only droppedDueToLateness appears.
>>>> The way I'm creating those metrics is:
>>>>
>>>> Metrics.counter("Ingester", "IngestedMessages").inc()
>>>>
>>>> I can see those counters on the Custom Counters section on the Google
>>>> Dataflow UI, but nothing appears on Stackdriver...
>>>>
>>>> Thanks!
>>>>
>>>> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler <foeg...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Carlos -
>>>>>
>>>>> Custom metrics can be "listed" by going to the Metric Explorer in
>>>>> Stackdriver and entering "custom.googleapis.com/dataflow" in the
>>>>> filter.
>>>>> If that list contains more than 100 different names, new custom
>>>>> metrics will not be created.  If this is a case, there should be a message
>>>>> in the job log reporting as much.
>>>>> (We are working with Stackdriver to improve this experience.)
>>>>>
>>>>> Also, we do not currently export distribution metrics to Stackdriver
>>>>> because we don't yet have a good mechanism to do so.  Gauge metrics are 
>>>>> not
>>>>> implemented yet and would not appear in either the Dataflow UI or
>>>>> Stackdriver.
>>>>>
>>>>> These are the only explanations I can think for for these metrics to
>>>>> not show up.  If neither of these are the case, I'm happy to investigate
>>>>> further on a particular instance.
>>>>>
>>>>> Cheers
>>>>> Andrea
>>>>>
>>>>>
>>>>>
>>>>> On 2018/01/23 19:59:08, Carlos Alonso <c...@mrcalonso.com> wrote:
>>>>> > Hi everyone!!>
>>>>> >
>>>>> > I'm trying to get a deeper view on my dataflow jobs by measuring
>>>>> parts of>
>>>>> > it using `Metrics.counter|gauge` but I cannot find how to see them
>>>>> on>
>>>>> > Stackdriver.>
>>>>> >
>>>>> > I have a premium Stackdriver account and I can see those counters
>>>>> under the>
>>>>> > Custom Counters section on the Dataflow UI.>
>>>>> >
>>>>> > I can see droppedDueToLateness 'custom' counter though on
>>>>> Stackdriver that>
>>>>> > seems to be created via 'Metrics.counter' as well...>
>>>>> >
>>>>> > What am I missing?>
>>>>> >
>>>>> > Regards>
>>>>> >
>>>>>
>>>>
>>>
>

Re: Custom metrics not showing on Stackdriver

[Andrea Foegler]

I would suggest Platform Support or StackOverflow as the best places to
request Dataflow-specific support.

This could be an issue coordinating between your Stackdriver Account(s) and
your Cloud project(s).  We can continue to discuss / investigate through
one of the above forums.

On Tue, Jan 30, 2018 at 12:30 PM, Carlos Alonso 
wrote:

> Thanks Andrea!!
>
> Do you mean using the UserVoice forum?
>
> Aside from that, there's something that could be helpful that is that when
> I navigate https://app.google.stackdriver.com/services/dataflow the
> message I get is this:
> "You do not have any resources of this type being monitored by
> Stackdriver." and that's weird as well. As if our Cloud Dataflow wasn't
> properly connected to Stackdriver, but, on the other hand. Some metrics are
> displayed and can be monitored such as System Lag, Watermark, etc...
>
> Thanks!
>
> On Tue, Jan 30, 2018 at 9:20 PM Andrea Foegler  wrote:
>
>> Hi Carlos -
>>
>> This sounds like something we should investigate further.  Since it
>> appears to be a Dataflow specific question / issue, would you mind posting
>> or following up in a Dataflow-specific forum or through Google Cloud
>> Platform Support: https://cloud.google.com/dataflow/support?  Feel free
>> to mention my name in your contact.
>>
>> Cheers
>> Andrea
>>
>>
>>
>> On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso 
>> wrote:
>>
>>> Hi Andrea, thank you very much for your response.
>>>
>>> I've followed your directions and only droppedDueToLateness appears.
>>> The way I'm creating those metrics is:
>>>
>>> Metrics.counter("Ingester", "IngestedMessages").inc()
>>>
>>> I can see those counters on the Custom Counters section on the Google
>>> Dataflow UI, but nothing appears on Stackdriver...
>>>
>>> Thanks!
>>>
>>> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler 
>>> wrote:
>>>
 Hi Carlos -

 Custom metrics can be "listed" by going to the Metric Explorer in
 Stackdriver and entering "custom.googleapis.com/dataflow" in the
 filter.
 If that list contains more than 100 different names, new custom metrics
 will not be created.  If this is a case, there should be a message in the
 job log reporting as much.
 (We are working with Stackdriver to improve this experience.)

 Also, we do not currently export distribution metrics to Stackdriver
 because we don't yet have a good mechanism to do so.  Gauge metrics are not
 implemented yet and would not appear in either the Dataflow UI or
 Stackdriver.

 These are the only explanations I can think for for these metrics to
 not show up.  If neither of these are the case, I'm happy to investigate
 further on a particular instance.

 Cheers
 Andrea



 On 2018/01/23 19:59:08, Carlos Alonso  wrote:
 > Hi everyone!!>
 >
 > I'm trying to get a deeper view on my dataflow jobs by measuring
 parts of>
 > it using `Metrics.counter|gauge` but I cannot find how to see them
 on>
 > Stackdriver.>
 >
 > I have a premium Stackdriver account and I can see those counters
 under the>
 > Custom Counters section on the Dataflow UI.>
 >
 > I can see droppedDueToLateness 'custom' counter though on Stackdriver
 that>
 > seems to be created via 'Metrics.counter' as well...>
 >
 > What am I missing?>
 >
 > Regards>
 >

>>>
>>

Re: Custom metrics not showing on Stackdriver

[Carlos Alonso]

Thanks Andrea!!

Do you mean using the UserVoice forum?

Aside from that, there's something that could be helpful that is that when
I navigate https://app.google.stackdriver.com/services/dataflow the message
I get is this:
"You do not have any resources of this type being monitored by
Stackdriver." and that's weird as well. As if our Cloud Dataflow wasn't
properly connected to Stackdriver, but, on the other hand. Some metrics are
displayed and can be monitored such as System Lag, Watermark, etc...

Thanks!

On Tue, Jan 30, 2018 at 9:20 PM Andrea Foegler  wrote:

> Hi Carlos -
>
> This sounds like something we should investigate further.  Since it
> appears to be a Dataflow specific question / issue, would you mind posting
> or following up in a Dataflow-specific forum or through Google Cloud
> Platform Support: https://cloud.google.com/dataflow/support?  Feel free
> to mention my name in your contact.
>
> Cheers
> Andrea
>
>
>
> On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso 
> wrote:
>
>> Hi Andrea, thank you very much for your response.
>>
>> I've followed your directions and only droppedDueToLateness appears.
>> The way I'm creating those metrics is:
>>
>> Metrics.counter("Ingester", "IngestedMessages").inc()
>>
>> I can see those counters on the Custom Counters section on the Google
>> Dataflow UI, but nothing appears on Stackdriver...
>>
>> Thanks!
>>
>> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler 
>> wrote:
>>
>>> Hi Carlos -
>>>
>>> Custom metrics can be "listed" by going to the Metric Explorer in
>>> Stackdriver and entering "custom.googleapis.com/dataflow" in the filter.
>>> If that list contains more than 100 different names, new custom metrics
>>> will not be created.  If this is a case, there should be a message in the
>>> job log reporting as much.
>>> (We are working with Stackdriver to improve this experience.)
>>>
>>> Also, we do not currently export distribution metrics to Stackdriver
>>> because we don't yet have a good mechanism to do so.  Gauge metrics are not
>>> implemented yet and would not appear in either the Dataflow UI or
>>> Stackdriver.
>>>
>>> These are the only explanations I can think for for these metrics to not
>>> show up.  If neither of these are the case, I'm happy to investigate
>>> further on a particular instance.
>>>
>>> Cheers
>>> Andrea
>>>
>>>
>>>
>>> On 2018/01/23 19:59:08, Carlos Alonso  wrote:
>>> > Hi everyone!!>
>>> >
>>> > I'm trying to get a deeper view on my dataflow jobs by measuring parts
>>> of>
>>> > it using `Metrics.counter|gauge` but I cannot find how to see them on>
>>> > Stackdriver.>
>>> >
>>> > I have a premium Stackdriver account and I can see those counters
>>> under the>
>>> > Custom Counters section on the Dataflow UI.>
>>> >
>>> > I can see droppedDueToLateness 'custom' counter though on Stackdriver
>>> that>
>>> > seems to be created via 'Metrics.counter' as well...>
>>> >
>>> > What am I missing?>
>>> >
>>> > Regards>
>>> >
>>>
>>
>

Re: Custom metrics not showing on Stackdriver

[Andrea Foegler]

Hi Carlos -

This sounds like something we should investigate further.  Since it appears
to be a Dataflow specific question / issue, would you mind posting or
following up in a Dataflow-specific forum or through Google Cloud Platform
Support: https://cloud.google.com/dataflow/support?  Feel free to mention
my name in your contact.

Cheers
Andrea



On Tue, Jan 30, 2018 at 10:27 AM, Carlos Alonso 
wrote:

> Hi Andrea, thank you very much for your response.
>
> I've followed your directions and only droppedDueToLateness appears.
> The way I'm creating those metrics is:
>
> Metrics.counter("Ingester", "IngestedMessages").inc()
>
> I can see those counters on the Custom Counters section on the Google
> Dataflow UI, but nothing appears on Stackdriver...
>
> Thanks!
>
> On Tue, Jan 30, 2018 at 7:22 PM Andrea Foegler  wrote:
>
>> Hi Carlos -
>>
>> Custom metrics can be "listed" by going to the Metric Explorer in
>> Stackdriver and entering "custom.googleapis.com/dataflow" in the filter.
>> If that list contains more than 100 different names, new custom metrics
>> will not be created.  If this is a case, there should be a message in the
>> job log reporting as much.
>> (We are working with Stackdriver to improve this experience.)
>>
>> Also, we do not currently export distribution metrics to Stackdriver
>> because we don't yet have a good mechanism to do so.  Gauge metrics are not
>> implemented yet and would not appear in either the Dataflow UI or
>> Stackdriver.
>>
>> These are the only explanations I can think for for these metrics to not
>> show up.  If neither of these are the case, I'm happy to investigate
>> further on a particular instance.
>>
>> Cheers
>> Andrea
>>
>>
>>
>> On 2018/01/23 19:59:08, Carlos Alonso  wrote:
>> > Hi everyone!!>
>> >
>> > I'm trying to get a deeper view on my dataflow jobs by measuring parts
>> of>
>> > it using `Metrics.counter|gauge` but I cannot find how to see them on>
>> > Stackdriver.>
>> >
>> > I have a premium Stackdriver account and I can see those counters under
>> the>
>> > Custom Counters section on the Dataflow UI.>
>> >
>> > I can see droppedDueToLateness 'custom' counter though on Stackdriver
>> that>
>> > seems to be created via 'Metrics.counter' as well...>
>> >
>> > What am I missing?>
>> >
>> > Regards>
>> >
>>
>

Re: Custom metrics not showing on Stackdriver

[Andrea Foegler]

Hi Carlos -

Custom metrics can be "listed" by going to the Metric Explorer in
Stackdriver and entering "custom.googleapis.com/dataflow" in the filter.
If that list contains more than 100 different names, new custom metrics
will not be created.  If this is a case, there should be a message in the
job log reporting as much.
(We are working with Stackdriver to improve this experience.)

Also, we do not currently export distribution metrics to Stackdriver
because we don't yet have a good mechanism to do so.  Gauge metrics are not
implemented yet and would not appear in either the Dataflow UI or
Stackdriver.

These are the only explanations I can think for for these metrics to not
show up.  If neither of these are the case, I'm happy to investigate
further on a particular instance.

Cheers
Andrea


On 2018/01/23 19:59:08, Carlos Alonso  wrote:
> Hi everyone!!>
>
> I'm trying to get a deeper view on my dataflow jobs by measuring parts
of>
> it using `Metrics.counter|gauge` but I cannot find how to see them on>
> Stackdriver.>
>
> I have a premium Stackdriver account and I can see those counters under
the>
> Custom Counters section on the Dataflow UI.>
>
> I can see droppedDueToLateness 'custom' counter though on Stackdriver
that>
> seems to be created via 'Metrics.counter' as well...>
>
> What am I missing?>
>
> Regards>
>

Custom metrics not showing on Stackdriver

[Carlos Alonso]

Hi everyone!!

I'm trying to get a deeper view on my dataflow jobs by measuring parts of
it using `Metrics.counter|gauge` but I cannot find how to see them on
Stackdriver.

I have a premium Stackdriver account and I can see those counters under the
Custom Counters section on the Dataflow UI.

I can see droppedDueToLateness 'custom' counter though on Stackdriver that
seems to be created via 'Metrics.counter' as well...

What am I missing?

Regards