Web UI shows my AssignTImestamp is in high back pressure but in/outPoolUsage are both 0.
Hi, I have a job like 'Source -> assignmentTimestamp -> flatmap -> Window -> Sink' and I get back pressure from 'Source' to the 'FlatMap' operators form the 'BackPressure' tab in the Web UI. When trying to find which operator is the source of back pressure, I use metrics provided by the Web UI, specifically, 'inPoolUsage' and 'outPoolUsage'. Firstly, As far as I know, when both of the metrics are 0, the operator should not be defined as 'back pressured', but when I check the 'AssignmentTimestamp' operator, where 8 subtasks running, I find 1 or 2 of them have 0 value about the back pressure index, and the others have the index higher than 0.80, and all of them are marked in 'HIGH' status. However, the two metrics, 'in/outPoolUsage', are always be 0. So maybe the operator is not back pressured actually? Or is there any problem with my Flink WebUI? Second question is, from my experience, I think the source of the back pressure should be the Window operator because the outPoolUsage of the 'FlatMap' are 1, and the 'Window' is the first downstream operator from the 'Flatmap', but the inPoolUsage and the outPoolUsage are also 0. So the cause of the back pressure should be the network bottleneck between window and flatmap? Am I right? Thanks for your reading, and I'm looking forward for your ideas. Haocheng
Re: NPE when aggregate window.
Hi, I meet the same exception, and find your suggestion here. I'm confused about the word 'grouping key', is that refers to the key of the accumulating hash map, or the key that separate the stream by some information? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Flink PrometheusReporter support for HTTPS
Hi All, Does PrometheusReporter provide support for HTTPS?. I couldn't find any information in flink documentation. Is there any way we can achieve the same? Thanks & Regards, Ashutosh
should i expect POJO serialization warnings when dealing w/ kryo protobuf serialization?
i'm currently using protobufs, and registering the serializers using kryo
protobuf using the following snippet of code:
static void optionalRegisterProtobufSerializer(ExecutionConfig config,
Class clazz) {
if (clazz != null) {
config.registerTypeWithKryoSerializer(clazz, ProtobufSerializer.
class);
}
}
static void configureExecutionConfig(ExecutionConfig config) {
optionalRegisterProtobufSerializer(config, User.class);
optionalRegisterProtobufSerializer(config, View.class);
optionalRegisterProtobufSerializer(config, Request.class);
optionalRegisterProtobufSerializer(config, Insertion.class);
optionalRegisterProtobufSerializer(config, Impression.class);
optionalRegisterProtobufSerializer(config, Action.class);
optionalRegisterProtobufSerializer(config, FlatEvent.class);
optionalRegisterProtobufSerializer(config, LatestImpression.class);
}
// *TODO* - reuse with batch.
void configureStreamExecutionEnvironment(StreamExecutionEnvironment
env) {
configureExecutionConfig(env.getConfig());
if (checkpointInterval > 0) {
env.enableCheckpointing(checkpointInterval);
}
env.getCheckpointConfig().setCheckpointingMode(checkpointingMode);
// *TODO* - evaluate if we want setMinPauseBetweenCheckpoints.
if (minPauseBetweenCheckpoints > 0) {
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(minPauseBetweenCheckpoints);
}
if (unalignedCheckpoints) {
env.getCheckpointConfig().enableUnalignedCheckpoints();
}
if (checkpointTimeout > 0) {
env.getCheckpointConfig().setCheckpointTimeout(checkpointTimeout);
}
env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
}
the concerning thing i have a question on is that i'm seeing these sorts of
info logs in the taskmanager logs:
org.apache.flink.api.java.typeutils.TypeExtractor[] - Class
class ai.promoted.proto.event.FlatEvent cannot be used as a POJO type
because not all fields are valid POJO fields, and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
2021-06-12 17:47:03,230 INFO
org.apache.flink.api.java.typeutils.TypeExtractor[] - class
ai.promoted.proto.event.LatestImpression does not contain a getter for
field impressionId_
2021-06-12 17:47:03,230 INFO
org.apache.flink.api.java.typeutils.TypeExtractor[] - class
ai.promoted.proto.event.LatestImpression does not contain a setter for
field impressionId_
2021-06-12 17:47:03,230 INFO
org.apache.flink.api.java.typeutils.TypeExtractor[] - Class
class ai.promoted.proto.event.LatestImpression cannot be used as a POJO
type because not all fields are valid POJO fields, and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
2021-06-12 17:47:03,230 INFO
org.apache.flink.api.java.typeutils.TypeExtractor[] - class
ai.promoted.proto.event.LatestImpression does not contain a getter for
field impressionId_
2021-06-12 17:47:03,230 INFO
org.apache.flink.api.java.typeutils.TypeExtractor[] - class
ai.promoted.proto.event.LatestImpression does not contain a setter for
field impressionId_
2021-06-12 17:47:03,230 INFO
org.apache.flink.api.java.typeutils.TypeExtractor[] - Class
class ai.promoted.proto.event.LatestImpression cannot be used as a POJO
type because not all fields are valid POJO fields, and must be processed as
GenericType. Please read the Flink documentation on "Data Types &
Serialization" for details of the effect on performance.
can i safely ignore these? is it telling me that it's doing the right
thing since kryo should kick in for GenericType?
Re: Error with extracted type from custom partitioner key
Hi Timo,
Thanks, I’ll give the ResultTypeQueryable interface a try - my previous
experience registering custom Kryo serializers wasn’t so positive.
Though I’m still curious as to whether java.lang.ClassCastException I got was
representative of a bug in Flink, or my doing something wrong.
But with the ongoing deprecation of DataSet support, I imagine that’s a low
priority issue in any case.
Regards,
— Ken
> On Jun 4, 2021, at 7:05 AM, Timo Walther wrote:
>
> Hi Ken,
>
> non-POJOs are serialized with Kryo. This might not give you optimal
> performance. You can register a custom Kryo serializer in ExecutionConfig to
> speed up the serialization.
>
> Alternatively, you can implement `ResultTypeQueryable` provide a custom type
> information with a custom serializer.
>
> I hope this helps. Otherwise can you share a little example how you would
> like to cann partitionCustom()?
>
> Regards,
> Timo
>
> On 04.06.21 15:38, Ken Krugler wrote:
>> Hi all,
>> I'm using Flink 1.12 and a custom partitioner/partitioning key (batch mode,
>> with a DataSet) to do a better job of distributing data to tasks. The
>> classes look like:
>> public class MyPartitioner implements Partitioner
>> {
>> ...
>> }
>> public class MyGroupingKey implements Comparable
>> {
>> ...
>> }
>> This worked fine, but I noticed a warning logged by Flink about
>> MyGroupingKey not having an empty constructor, and thus not being treated as
>> a POJO.
>> I added that empty constructor, and then I got an error because
>> partitionCustom() only works on a single field key.
>> So I changed MyGroupingKey to have a single field (a string), with transient
>> cached values for the pieces of the key that I need while partitioning. Now
>> I get an odd error:
>> java.lang.RuntimeException: Error while calling custom partitioner
>> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
>> MyGroupingKey
>> at MyPartitioner.partition(AdsPinotFilePartitioner.java:11)
>> at
>> org.apache.flink.runtime.operators.shipping.OutputEmitter.customPartition(OutputEmitter.java:235)
>> ... 19 more
>> So I've got two questions…
>> • Should I just get rid of the empty constructor, and have Flink treat it as
>> a non-POJO? This seemed to be working fine.
>> • Is it a bug in Flink that the extracted field from the key is being used
>> as the expected type for partitioning?
>> Thanks!
>> — Ken
>> --
>> Ken Krugler
>> http://www.scaleunlimited.com
>> Custom big data solutions
>> Flink, Pinot, Solr, Elasticsearch
>
--
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch
Re: Flink PrometheusReporter support for HTTPS
Hi Ashutosh, How are you deploying your Flink apps? Would running a reverse proxy like Nginx or Envoy that handles the HTTPS connection work for you? Best, Austin On Sat, Jun 12, 2021 at 1:11 PM Ashutosh Uttam wrote: > Hi All, > > Does PrometheusReporter provide support for HTTPS?. I couldn't find any > information in flink documentation. > > Is there any way we can achieve the same? > > Thanks & Regards, > Ashutosh > > >
Re: Looking for online live training courses
Hi BB, Thanks for your suggestion, I will take a look. I was hoping there are customized training services for flink like Kafka or Spark since out team had some video courses already and would like to dive into some specific topics and ask some questions. Best, *Xia(Nate) Qu* On Fri, Jun 11, 2021 at 9:39 AM B.B. wrote: > There are some beginner courses on Pluralsight. Just look for those with > newer dates. > > BR, > BB > > On Fri, 11 Jun 2021 at 03:33, Xia(Nate) Qu wrote: > >> Hi all, >> >> My team is planning to start our journey of Apache Flink, was wondering >> if there are any professional training courses (online interactive at this >> moment) recommended? Thanks >> >> >> Best, >> >> *Xia(Nate) Qu* >> >> -- > Everybody wants to be a winner > Nobody wants to lose their game > Its insane for me > Its insane for you > Its insane >
Re: Looking for online live training courses
Thanks Jing, I really appreciate your sharing. 多谢老铁！:) Best, *Xia(Nate) Qu* On Thu, Jun 10, 2021 at 10:07 PM JING ZHANG wrote: > Hi XiaQu, > Welcome to Flink community! > I don't know if there are online interactive training courses yet, I would > add it to the email later after I consult offline. > > I would like to list the most popular resources I know as follows, I hope > it helps. > Training Course > 1. https://flink.apache.org/training.html > 2. > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/learn-flink/overview/ > > Training projects in github > 1. https://github.com/apache/flink-playgrounds > 2. https://github.com/apache/flink-training > > History Flink Forward Conference video > 1. https://www.flink-forward.org/ > > If you could understand chinese, the following resource could also help > you, > 1. https://flink-learning.org.cn/ > 2. https://github.com/flink-china/flink-training-course > 3. > https://space.bilibili.com/33807709?spm_id_from=333.788.b_765f7570696e666f.2 > 4. > https://developer.aliyun.com/group/sc?spm=a2c6h.12883283.1377930.10.5f1f201crBKprD#/?_k=3ahyn2 > > Best regards, > JING ZHANG > > Xia(Nate) Qu 于2021年6月11日周五 上午9:33写道： > >> Hi all, >> >> My team is planning to start our journey of Apache Flink, was wondering >> if there are any professional training courses (online interactive at this >> moment) recommended? Thanks >> >> >> Best, >> >> *Xia(Nate) Qu* >> >>
