Re: End to End Latency Tracking in flink

[Oscar Westra van Holthe - Kind]

On Mon, 30 Mar 2020 at 05:08, Lu Niu  wrote:

> $current_processing - $event_time works for event time. How about
> processing time? Is there a good way to measure the latency?
>

To measure latency you'll need some way to determine the time spent between
the start and end of your pipeline.

To measure latency when using processing time, you'll need to partially use
ingestion time. That is, you'll need to add the 'current' processing time
as soon as messages are ingested.

With it, you can then use the $current_processing - $ingest_time solution
that was already mentioned.

Kind regards,
Oscar

-- 
Oscar Westra van Holthe - Kind

Re: About the issue caused by flink's dependency jar package submission method

[Oscar Westra van Holthe - Kind]

On Wed, 21 Nov 2018 at 05:26, clay  wrote:

> hi
>
> I have checked all the dependences, and don't find the jar with different
> version, so ,I double the way to submit jar has some issue? my commend is
> like this:
>

Did you also check the runtime dependencies where the code is run? Because
the only way that two classes with the same name are not the same is if
they are loaded with two different classloaders.

I suspect your runtime loads the LinkedMap via a different classloader than
FlinkKafkaConsumerBase, and that other classloader (probably a parent) also
has the LinkedMap. Hence the problem.

[...]

Issues like these have a similar origin, but now you have two versions of
kafka on your classpath:

> Caused by: org.apache.kafka.common.KafkaException:

org.apache.kafka.common.serialization.ByteArrayDeserializer is not an
> instance of org.apache.kafka.common.serialization.Deserializer
> at
>
> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
> at
>
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:673)
> ... 11 more
>

[...]


>  but in a fat jar, these work all good
>

That's odd.. that means that in that case, your classpath is different. If
you can find out how, solving the exceptions above becomes easy.

Kind regards,
Oscar


-- 
Oscar Westra van Holthe - Kind
Website: http://oscar.westravanholthe.nl/ | Email: os...@westravanholthe.nl