Flink can't initialize operator state backend when starting from checkpoint

Hi all, When Flink(1.4.2) job starts, it could find checkpoint files at HDFS, but exception occurs during deserializing: [image: image.png] Do you have any insight on this? Thanks, Qingxiang Ma

Re: multiple flink applications on yarn are shown in one application.

Hi, I see from your command that you are using the same jar file twice, so I want to double-check first how you even determine which job should be started? I am also adding Till (in CC) depending on your answer to my first question, he might have some additional thoughts. Best, Stefan > Am 21

Re: Flink can't initialize operator state backend when starting from checkpoint

Hi Qingxiang, Several days ago, Stefan described the causes of this anomaly in a problem similar to this: Typically, these problems have been observed when something was wrong with a serializer or a stateful serializer was used from multiple threads. Thanks, vino. Marvin777 于2018年9月21日周五 下午3:20

Re: S3 connector Hadoop class mismatch

Hi Stefan, Stephan, Yes, the `hadoop.security.group.mapping` option is explicitly set to `org.apache.hadoop.security.LdapGroupsMapping`. Guess that was why the classloader found an unshaded class. I don’t have the permission to change the Hadoop cluster configurations so I modified the `core-

How to join stream and batch data in Flink?

Hi All, Sometimes some “dimension table” need to be joined from the "fact table", if data are not joined before sent to Kafka. So if the data are joined in Flink, does the “dimension table” have to be import as a stream, or there are some other ways can achieve it? Thanks

Re: Flink can't initialize operator state backend when starting from checkpoint

Hi, that is correct. If you are using custom serializers you should double check their correctness, maybe using our test base for type serializers. Another reason could be that you modified the job in a way that silently changed the schema somehow. Concurrent use of serializers across different

Re: How to join stream and batch data in Flink?

Hi Henry, There are three ways I can think of: 1) use DataStream API, implement a flatmap UDF to access dimension table; 2) use table/sql API, implement a UDTF to access dimension table; 3) customize the table/sql join API/statement's implementation (and change the physical plan) Thanks, vino.

How does flink read data from kafka number of TM's are more than topic partitions

Hi All, Let's say a topic in kafka has 5 partitions. If I spawn 10 Task Managers with 1 slot each and parallelism is 10 then how will records be read from the kafka topic if I use the FlinkKafkaConsumer to read. Will 5 TM's read and the rest be ideal in that case? Is over subscribing the

Re: HA failing for 1.6.0 job cluster with docker-compose

Hi Vino and TIll, That's great news. Thank you! Cheers, Tzanko On Thu, Sep 20, 2018 at 11:43 AM vino yang wrote: > Hi all, > > Oh, I took this ticket, will fix it as soon as possible. > > Thanks, vino. > > Till Rohrmann 于2018年9月20日周四 下午4:35写道： > >> Hi Tzanko, >> >> in order to make the cont

Re: ***UNCHECKED*** Error while confirming Checkpoint

Hi, could you provide some logs for this problematic job because I would like to double check the reason why this violated precondition did actually happen? Thanks, Stefan > Am 20.09.2018 um 17:24 schrieb Stefan Richter : > > FYI, here a link to my PR: https://github.com/apache/flink/pull/6723

Re: Data loss when restoring from savepoint

Hi Juho, so it means that the savepoint does not loose at least some dropped records. If it is feasible for your setup, I suggest to insert one more map function after reduce and before sink. The map function should be called right after window is triggered but before flushing to s3. The resul

Re: Flink can't initialize operator state backend when starting from checkpoint

Hi, I do not use custom serializers, and my job contains only source and sink(BucketingSink). What causes this phenomenon in general? I suggest that you also update to a newer version, at least the latest > bugfix release Which version does this sentence refer to? And could you please help l

Re: Data loss when restoring from savepoint

Thanks, Andrey. > so it means that the savepoint does not loose at least some dropped records. I'm not sure what you mean by that? I mean, it was known from the beginning, that not everything is lost before/after restoring a savepoint, just some records around the time of restoration. It's not 10

Re: How does flink read data from kafka number of TM's are more than topic partitions

Hi, Yes, in your case half of the Kafka source tasks wouldn’t read/process any records (you can check that in web UI). This shouldn’t harm you, unless your records will be redistributed after the source. For example: source.keyBy(..).process(new MyVeryHeavyOperator()).print() Should be fine, b

Re: How does flink read data from kafka number of TM's are more than topic partitions

Thanks a lot for the explanation. That was exactly what I thought should happen. However, it is always good to a clear confirmation. Regards, Taher Koitawala GS Lab Pune +91 8407979163 On Fri, Sep 21, 2018 at 6:26 PM Piotr Nowojski wrote: > Hi, > > Yes, in your case half of the Kafka source t

Re: How does flink read data from kafka number of TM's are more than topic partitions

No problem :) Piotrek > On 21 Sep 2018, at 15:04, Taher Koitawala wrote: > > Thanks a lot for the explanation. That was exactly what I thought should > happen. However, it is always good to a clear confirmation. > > > Regards, > Taher Koitawala > GS Lab Pune > +91 8407979163 > > > On Fri,

Re: Running Flink in Google Cloud Platform (GCP) - can Flink be truly elastic?

Hi Alexander, I've redirected your question to user mailing list. The goal of community list is for "Broader community discussions related to meetups, conferences, blog posts and job offers" Quick answer to your question is that dynamic scaling of flink job's is a work in progress. Maybe Gary or

Flink 1.5.4 -- issues w/ TaskManager connecting to ResourceManager

Anybody else seen this? I'm running both the JM and TM on the same host in this setup. This was working fine w/ Flink 1.5.3. On the TaskManager: 00:31:30.268 INFO o.a.f.r.t.TaskExecutor - Could not resolve ResourceManager address akka.tcp://flink@localhost:6123/user/resourcemanager, retrying i

Re: How to join stream and batch data in Flink?

Hi +1 for vino's answer. Also, this kind of join will be supported in FLINK-9712 . You can check more details in the jira. Best, Hequn On Fri, Sep 21, 2018 at 4:51 PM vino yang wrote: > Hi Henry, > > There are three ways I can think of: > > 1)