Inquiry about ActiveResourceManager and StandaloneResourceManager in Flink

2023-11-02 Thread Steven Chen
Dear Flink Community, I am currently using Flink for my project and have a question regarding ActiveResourceManager and StandaloneResourceManager. What does "active" mean in ActiveResourceManager and why is StandaloneResourceManager not considered an active resource manager? Thank you

疑似BUG: 在滑动窗口中使用reduce()聚合时数据被多次处理

2023-11-02 Thread tao zhang
reduce()方法的状态在窗口间未被隔离,多个窗口聚合时使用的是同一对象.一个数据进入时,被重复累加 是reduce的特性吗? 还是reduce中的窗口间隔离出现问题? 希望得到回复 测试输入如下: 1001,/home,1000 1002,/home,2000 输出如下: input> test.Event(user=1001, page=/home, ts=1000) input> test.Event(user=1002, page=/home, ts=2000) test.WordCount(word=/home, count=2)

Re: flink-kubernetes-operator cannot handle SPECCHANGE for 100+ FlinkDeployments concurrently

2023-11-02 Thread Tony Chen
One of the operator pods logged the following exception before the container restarted: �[m�[33m2023-11-01 14:24:21,260�[m �[36mo.a.f.s.n.i.n.c.AbstractChannel�[m �[33m[WARN ] Force-closing a channel whose registration task was not accepted by an event loop: [id: 0x1a7718c1]

Re: Issues about removed topics with KafkaSource

2023-11-02 Thread Hector Rios
Hi Emily One workaround that might help is to leverage the state-processor-api[1]. You would have to do some upfront work to create a state-processor job to wipe the state (offsets) of the topic you want to remove and use the newly generated savepoint without the removed state of the topic or

Re: Issues about removed topics with KafkaSource

2023-11-02 Thread Emily Li via user
Hey Martijn Thanks for the clarification. Now it makes sense. I saw this feature FLIP-246 is still a WIP and there's no release date yet, and it actually contains quite some changes in it. We noticed there's a WIP PR for this change, just wondering if there's any plan in releasing this feature?

Fwd: flink-kubernetes-operator cannot handle SPECCHANGE for 100+ FlinkDeployments concurrently

2023-11-02 Thread Tony Chen
Hi Flink Community, I am currently running flink-kubernetes-operator 1.6-patched ( https://github.com/apache/flink-kubernetes-operator/commit/3f0dc2ee5534084bc162e6deaded36e93bb5e384), and I have 3 flink-kubernetes-operator pods running. Recently, I deployed around 110 new FlinkDeployments, and I