Re: Using s3 for checkpointing

Hi Arvid, Thanks for the response. I have both the jars under /opt/flink/plugins but I'm still getting the same error message. Also can someone please provide some pointers on how entropy works. How should I setup the directory structure? In the link that you have provided there is a

Re: Issue with committing Kafka offsets

I had to disable auto commit for it to work. I understand auto commit is just for monitoring purpose so I assume it should be safe to run it like that. properties.put("enable.auto.commit", "false"); On Fri, Jan 31, 2020 at 1:09 PM RKandoji wrote: > Hi, > > Thanks for the response. > I'm

Failed to transfer file from TaskExecutor : Vanilla Flink Cluster

Hi I am trying to build a cluster for flink with 1 master and 2 workers. The program is working fine locally. The messages are read from Kafka and just printed on STDOUT. The cluster is successfully created and UI is also shows all config. But the job fails to execute on the cluster. Here are

Re: Issue with committing Kafka offsets

Hi, Thanks for the response. I'm actually using a different consumer group id for each consumer. RK. On Fri, Jan 31, 2020 at 12:20 PM Tzu-Li (Gordon) Tai wrote: > Hi, > > There are no upper limits on the number of Kafka consumers per job. > > For each one of your FlinkKafkaConsumers, are you

Re: Issue with committing Kafka offsets

Hi, There are no upper limits on the number of Kafka consumers per job. For each one of your FlinkKafkaConsumers, are you using the same group.id? That could maybe explain why you are experiencing higher commit times as you are adding more FlinkKafkaConsumers, as AFAIK on the broker side, the

Re: Issue with committing Kafka offsets

Can someone please help me here. Thanks RK On Thu, Jan 30, 2020 at 7:51 PM RKandoji wrote: > Hi Team, > > I'm running into strange issue pasted below: > > Committing offsets to Kafka takes longer than the checkpoint interval. > Skipping commit of previous offsets because newer complete

Re: [ANNOUNCE] Apache Flink 1.9.2 released

Thank you! -- [image: MotaWord] Oytun Tez M O T A W O R D | CTO & Co-Founder oy...@motaword.com On Fri, Jan 31, 2020 at 8:47 AM Till Rohrmann wrote: > Thanks for being our release manager Hequn and thanks for everyone who > made the 1.9.2 release

Re: [ANNOUNCE] Apache Flink 1.9.2 released

Thanks for being our release manager Hequn and thanks for everyone who made the 1.9.2 release possible. Cheers, Till On Fri, Jan 31, 2020 at 1:46 PM Hequn Cheng wrote: > @jincheng sun That's great. Thank you! > > On Fri, Jan 31, 2020 at 7:57 PM jincheng sun > wrote: > >> Thanks for being the

Re: about registering completion function for worker shutdown

Hi, Flink's ProcessFunction has a close() method, which is executed on shutdown of the workers. (You could also use any of the Rich* functions for that purpose). If you add a ProcessFunction with the same parallelism before the KafkaSink, it'll be executed on the same machines as the Kafka

Re: [ANNOUNCE] Apache Flink 1.9.2 released

@jincheng sun That's great. Thank you! On Fri, Jan 31, 2020 at 7:57 PM jincheng sun wrote: > Thanks for being the release manager and the great work Hequn :) > > Also thanks to the community making this release possible! > > BTW: I have add the 1.9.2 release to report. > > Best, > Jincheng > >

Re: [ANNOUNCE] Apache Flink 1.9.2 released

Thanks for being the release manager and the great work Hequn :) Also thanks to the community making this release possible! BTW: I have add the 1.9.2 release to report. Best, Jincheng Hequn Cheng 于2020年1月31日周五 下午6:55写道： > Hi everyone, > > The Apache Flink community is very happy to announce

Read data from Oracle using Flink SQL API

Hi to all, I was looking at the Flink SQL API's and I discovered that only a few drivers are supported [1], i.e. Mysql, Postgres and Derby. You could have problems only on the writing side of the connector (TableSink) because you need to adjust the override statement, but for the read part you

[ANNOUNCE] Apache Flink 1.9.2 released

Hi everyone, The Apache Flink community is very happy to announce the release of Apache Flink 1.9.2, which is the second bugfix release for the Apache Flink 1.9 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate

Re: Flink+YARN HDFS replication factor

The same applies to Flink. Transient data will only be stored on local disks. Cheers, Till On Thu, Jan 30, 2020 at 9:10 PM Piper Piper wrote: > Please disregard my previous email. I found the answer online. > > I thought writing data to local disk automatically meant the data would be >

Re: 回复: Difference between JobManager and JobMaster

They are similar but not quite the same. The JobManager was responsible for managing multiple jobs, whereas the JobMaster only manages a single job. The responsibility for managing multiple jobs (==JobMasters) and accepting job submissions was moved into the Dispatcher. On 31/01/2020 09:39,

回复: Difference between JobManager and JobMaster

Hi Yun, thank you so much for your reply! I have briefly read all links in your reply. So the JobManager.scala was removed and JobMaster was added and functions what JobManager did before? Best Regards Weizheng 发件人: Yun Tang 发送时间: 2020年1月31日 15:20 收件人: Lu