KAFKA SINK ERROR IN Flink-(version 1.14.4)

Dear Team, I am new to pyflink and request for your support in issue I am facing with Pyflink. I am using Pyflink version 1.14.4 & using reference code from pyflink getting started pages. I am getting following error . py4j.protocol.Py4JJavaError: An error occurred while calling

Re: web ui中能查看到job失败的原因吗？

JobManager的log里应该能看到root cause？ On Thu, Apr 21, 2022 at 5:54 PM weishishuo...@163.com wrote: > > 我提交一个postgresql cdc 同步数据到 mysql jdbc sink的job，过了一会儿就失败了，点击job的链接，web > ui界面的状态是FAILED，但是异常信息不明确 > ``` > 2022-04-21 17:30:50 > org.apache.flink.runtime.JobException: Recovery is suppressed by >

Re: Savepoint and cancel questions

Hi, Dan 1. Do you mean put the option into savepoint command? If so, I think it will not work well. This option describe that how checkpoints will be cleaned up in different job status. e.g. FAILED/CANCELED. It cannot be covered in savepoint command. 2. Which flink version you use? I work on 1.14

Flink MySQL CDC 注册 schema registry 问题

Hi, 我想使用 Flink MySQL CDC Connector 以 DataStream 方式消费 MySQL Binlog 输出变更数据到下游kafka topic (1)，同时监听database schema change事件，将最新的schema数据输出到下游另一个kafka topic (2)，又或者直接注册schema到 confluent / apicurio schema registry，查了一下flink cdc官方文档[1]，并没有这方面的信息。请问应该怎么实现呢？有相关文档或例子么？谢谢！ [1]

Savepoint and cancel questions

Hi. 1. Why isn’t –externalizedCheckpointCleanup an option on savepoint (instead of being needed at the start of a job run)? 2. Can we get a confirmation dialog when someone clicks "cancel job" in the UI? Just in case people click on accident. 3. Can we get a way to have Flink clean up the

Re: RocksDB's state size discrepancy with what's seen with state processor API

Hi David, I don't find it troublesome per se, I was rather trying to understand what should be expected (and documented) for my application. Before I restarted the job and changed some configurations, it ran for around 10 days and ended up with a state size of about 1.8GB, so I'm still not

How to use RichAsyncFunction with MiniClusterWithClientResource

Hi, I am writing (functional) unit test for RichAsyncFunction; following thread - https://lists.apache.org/thread/12n9fcj9v50sdpnrqsmxgx2ktc44c32c I am getting following exception. How would I resolve this? Test - @ClassRule val miniCluster = MiniClusterWithClientResource(

RE: Restore Job from CheckPoint in IntelliJ IDE - MiniCluster

Happy to hear that (back-posted to usr list) Thias -Original Message- From: Κωνσταντίνος Αγαπίδης Sent: Friday, April 22, 2022 3:50 PM To: Schwalbe Matthias Subject: Re: Restore Job from CheckPoint in IntelliJ IDE - MiniCluster *** EXTERNAL MESSAGE – CAUTION: Think Before You Click

Re: AvroRowDeserializationSchema

Hi Quynh, I have added an example on how to use AvroRowDeserializationSchema in Python DataStream API in [1]. Please take a look at if that helps for you~ Regards, Dian [1] https://github.com/apache/flink/blob/release-1.15/flink-python/pyflink/examples/datastream/formats/avro_format.py On Fri,

test

Thanks & Regards, Samir Vasani

Re: 关于Flink1.15文档，有一些小疑惑求助

intermediate savepoints指的是非stop-with-savepoint，也就是不停止作业的情况下做savepoint。我的感觉是，这个时候的savepoint不会commit（比如sink写出到外部系统），如果作业失败会从最近一次的checkpoint恢复。如果恰好有一个作业从savepoint恢复，两个作业同时跑，可能会造成结果的重复或者不一致，这个时候最好丢弃掉sink（换uid）。对于只有一个作业运行的情况，比如停了作业再恢复，是不会有问题的。 林影 于2022年4月22日周五 17:05写道： > 在Flink

Re: AvroRowDeserializationSchema

Hi Quynh, Could you show some sample code on how you use it? Regards, Dian On Fri, Apr 22, 2022 at 1:42 PM lan tran wrote: > Wonder if this is a bug or not but if I use > *AvroRowDeserializationSchema,* > > In PyFlink the error still occure ? > > py4j.protocol.Py4JError: An error occurred

Re: JobManager doesn't bring up new TaskManager during failure recovery

I created a JIRA ticket https://issues.apache.org/jira/browse/FLINK-27350 to track this issue. BRs, Chenyu From: "Zheng, Chenyu" Date: Friday, April 22, 2022 at 6:26 PM To: Yang Wang Cc: "u...@flink.apache.org" , "user-zh@flink.apache.org" Subject: Re: JobManager doesn't bring up new

Re: JobManager doesn't bring up new TaskManager during failure recovery

I created a JIRA ticket https://issues.apache.org/jira/browse/FLINK-27350 to track this issue. BRs, Chenyu From: "Zheng, Chenyu" Date: Friday, April 22, 2022 at 6:26 PM To: Yang Wang Cc: "user@flink.apache.org" , "user...@flink.apache.org" Subject: Re: JobManager doesn't bring up new

Re: JobManager doesn't bring up new TaskManager during failure recovery

Thank you, Yang! In fact I have a fine-grained dashboard for Kubernetes cluster health (like apiserver qps/latency etc.), and I didn't find anything unusual… Also, the JobManager container cpu/memory usage is low. Besides, I have a deep dive in these logs and Flink resource manager code, and

Re: JobManager doesn't bring up new TaskManager during failure recovery

Thank you, Yang! In fact I have a fine-grained dashboard for Kubernetes cluster health (like apiserver qps/latency etc.), and I didn't find anything unusual… Also, the JobManager container cpu/memory usage is low. Besides, I have a deep dive in these logs and Flink resource manager code, and

Re: Flink on yarn ，并行度>1的情况下，怎么获取springboot的bean？

@duwenwen 我比较好奇你的算子里做了什么，因为如果你就是要全局初始化唯一一次，那就用一个 parallelism=1 的算子去做就好了。 parallelism=n 你还要确保 once 的话应该得依赖外部系统来做到仅初始化一次。 Best, tison. Paul Lam 于2022年4月22日周五 18:16写道： > 听起来是在 Flink 里启动 springboot? 很有意思的架构，有一点点类似 statefun 了。可以说说这么做的背景吗？ > > 另外请附带上 flink 的部署模式和版本信息，这样大家才好判断问题在哪里。 > > Best, >

Re: Flink on yarn ，并行度>1的情况下，怎么获取springboot的bean？

听起来是在 Flink 里启动 springboot? 很有意思的架构，有一点点类似 statefun 了。可以说说这么做的背景吗？ 另外请附带上 flink 的部署模式和版本信息，这样大家才好判断问题在哪里。 Best, Paul Lam > 2022年4月22日 16:30，duwenwen 写道： > > 您好： >首先很感谢您能在百忙之中看到我的邮件。我是一个写代码的新手，在使用flink框架过程中我遇到了一些问题，希望能得到您的解答。 >

Re: Jobmanager trying to be registered for Zombie Job

FYI: I created FLINK-27354 [1] to cover the issue of retrying to connect to the RM while shutting down the JobMaster. This doesn't explain your issue though, Peter. It's still unclear why the JobMaster is still around as stated in my previous email. Matthias [1]

Re: Jobmanager trying to be registered for Zombie Job

Just by looking through the code, it appears that these logs could be produced while stopping the job. The ResourceManager sends a confirmation of the JobMaster being disconnected at the end back to the JobMaster. If the JobMaster is still around to process the request, it would try to reconnect

Re: Huge number of GAX-Threads in Flink

Hi Shreekant, Since you're mentioning Iceberg, I'm assuming this means you've added the Iceberg runtime to Flink. I'm therefore wondering if this is a Flink issue or an Iceberg issue. I would lean towards the latter. Have you asked them? Best regards, Martijn Visser

关于Flink1.15文档，有一些小疑惑求助

在Flink 官网savepoint的页面中，出现下面一段话 Starting from Flink 1.15 intermediate savepoints (savepoints other than created with stop-with-savepoint ) are not used for recovery and do not commit any

Re: JobManager doesn't bring up new TaskManager during failure recovery

The root cause might be you APIServer is overloaded or not running normally. And then all the pods events of taskmanager-1-9 and taskmanager-1-10 are not delivered to the watch in FlinkResourceManager. So the two taskmanagers are not recognized by ResourceManager and then registration are

Re: JobManager doesn't bring up new TaskManager during failure recovery

The root cause might be you APIServer is overloaded or not running normally. And then all the pods events of taskmanager-1-9 and taskmanager-1-10 are not delivered to the watch in FlinkResourceManager. So the two taskmanagers are not recognized by ResourceManager and then registration are

Flink on yarn ，并行度>1的情况下，怎么获取springboot的bean？

您好： 首先很感谢您能在百忙之中看到我的邮件。我是一个写代码的新手，在使用flink框架过程中我遇到了一些问题，希望能得到您的解答。 由于需求要求，我需要将springboot和flink结合起来使用，我在open方法中获取springboot的上下文来获取bean。当设置parallelism为1时，可以发布到集群正常运行，但是当parallelism>1时，springboot的环境被多次初始化，运行就开始报错，，请问当parallelism>1

Re: Problems with PrometheusReporter

Hi Chesnay, I had a look in my logs, there are not WARNINGS regarding metrics and registering metrics when starting this job. I ran the example jobs - ./examples/table/ChangelogSocketExample.jar (table streaming) - ./examples/streaming/StateMachineExample.jar (streaming) When running those jobs

Re: Jobmanager trying to be registered for Zombie Job

...if possible it would be good to get debug rather than only info logs. Did you encounter anything odd in the TaskManager logs as well. Sharing those might be of value as well. On Fri, Apr 22, 2022 at 8:57 AM Matthias Pohl wrote: > Hi Peter, > thanks for sharing. That doesn't sound right. May

Re: [ANNOUNCE] Apache Kyuubi (Incubating) released 1.5.1-incubating

Hi Paul, It's on purpose, we are building eco on Flink, it's good to share with Flink users and the community. Paul Lam 于2022年4月22日周五 14:57写道： > Hi Fu, > > I think the mail might have been sent to Flink user mail list by mistake? > > Best, > Paul Lam > > 2022年4月22日 11:00，Fu Chen 写道： > > Hi

Re: RocksDB's state size discrepancy with what's seen with state processor API

Alexis, Compaction isn't an all-at-once procedure. RocksDB is organized as a series of levels, each 10x larger than the one below. There are a few different compaction algorithms available, and they are tunable, but what's typically happening during compaction is that one SST file at level n is

Re: Flink batch mode does not sort by event timestamp

The DataStream API's BATCH execution mode first sorts by key, and within each key, it sorts by timestamp. By operating this way, it only needs to keep state for one key at a time, so this keeps the runtime simple and efficient. Regards, David P.S. I see you also asked this question on stack

Unsubscribe

Unsubscribe

Re: Jobmanager trying to be registered for Zombie Job

Hi Peter, thanks for sharing. That doesn't sound right. May you provide the entire jobmanager logs? Best, Matthias On Thu, Apr 21, 2022 at 6:08 PM Peter Schrott wrote: > Hi Flink-Users, > > I am not sure if this does something to my cluster or not. But since > updating to Flink 1.15 (atm rc4)

Re: [ANNOUNCE] Apache Kyuubi (Incubating) released 1.5.1-incubating

Hi Fu, I think the mail might have been sent to Flink user mail list by mistake? Best, Paul Lam > 2022年4月22日 11:00，Fu Chen 写道： > > Hi all, > > The Apache Kyuubi (Incubating) community is pleased to announce that > Apache Kyuubi (Incubating) 1.5.1-incubating has been released! > > Apache