[Flink-Kubernetes-Operator] Beam example application getting into "State: RECONCILING" after minikube restart

2023-02-18 Thread Ritesh Nadhani
Hello This is my first setup with Apache Beam and trying to use the flink runner for it. Following the example of https://github.com/apache/flink-kubernetes-operator/tree/main/examples/flink-beam-example I got my app running inside my local minikube. The app is pretty stateless - minimal

Re:[急] flink sql写数据到带主键表时能确保相同主键的记录都被同一个taskmanager处理吗?

2023-02-18 Thread RS
Hi, connector里面配置的主键控制是写入存储的,有些存储在写入数据的时候,能根据主键自动更新去重 所以我感觉,你这里的应该是想在计算的时候shuffle(写入之前),你应该需要先执行一个 group by 主键,然后再执行insert into Thanks 在 2023-02-17 15:56:51,"casel.chen" 写道: >作业场景是kafka cdc数据源关联几张redis维表再和其他流表进行双流regular inner >join,最后将打宽表写入mongodb。使用的是flink 1.13.2 sql模式。开了debug日志。

Re: KeyedProcessFunction within an iteration

2023-02-18 Thread Lorenzo Nicora
Hi Zhipeng IterativeStreams does have keyBy() methods, but they all throw UnsupportedOperationException [1] For some context: the whole thing is to do message enrichment with asyncIO, caching the enrichment info in state (with TTL). I am using an iteration as RichAsyncFunction does not support

Re: KeyedProcessFunction within an iteration

2023-02-18 Thread Zhipeng Zhang
Hi Lorenzo, Could you provide some code example to reproduce your question? As I understand, IterativeStream#keyBy is supported since it is a subclass of DataStream. Moreover, we have implemented an unified iteration module for Flink [1] in Flink ML [2], which relies on Flink 1.15.2. Probably

KeyedProcessFunction within an iteration

2023-02-18 Thread Lorenzo Nicora
Hi all, I am trying to implement an iterative streaming job that processes the loop with a KeyedProcessFunction. I need a KeyedProcessFunction to use keyed state and to emit a side-output (that after further transformations becomes the feedback) Problem is IterativeStream.process() only accepts