Re: flink cdc 3.0 schema变更问题

2024-06-12 Thread Xiqian YU
Zapjone 好， 目前的 Schema Evolution 的实现依赖传递 CDC Event 事件的 Pipeline 连接器和框架。如果您希望插入自定义算子逻辑，建议参考 flink-cdc-composer 模块中的 FlinkPipelineComposer 类构建算子链作业的方式，并在其中插入自定义的 Operator 以实现您的业务逻辑。 另外，对于一些简单的处理逻辑，如果能够使用 YAML 作业的 Route（路由）、Transform（变换）功能表述的话，直接编写对应的 YAML 规则会更简单。 祝好！ Regards, yux De : zapjone

flink cdc 3.0 schema变更问题

2024-06-12 Thread zapjone
大佬们好： 想请假下，在flink cdc3.0中支持schema变更，但看到是pipeline方式的，因业务问题需要使用datastream进行特殊处理，所以想请假下，在flink cdc 3.0中datastream api中怎么使用schema变更呢？或者相关文档呢？

Re:changelogstream删除问题

2024-06-12 Thread Xuyang
Hi, 你可以试一下用statement set[1]，将这个query同时写入到print sink中吗？ 在tm日志里可以查看到print sink的结果，看看里面有没有-D类型的数据。如果没有的话，证明是test_changelog源表可能就没有-D的数据；如果有的话，就需要后续进一步排查sink表在ds和sql上的行为差异。 [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/insert/#insert-into-multiple-tables [2]

Re: Understanding flink-autoscaler behavior

2024-06-12 Thread Chetas Joshi
Got it. Thanks! On Wed, Jun 12, 2024 at 6:49 PM Zhanghao Chen wrote: > > Does this mean it won't trigger a checkpoint before scaling up or > scaling down? > > The in-place rescaling won't do that. > > > Do I need the autotuning to be turned on for exactly once processing? > > It suffices to

Re: Understanding flink-autoscaler behavior

2024-06-12 Thread Zhanghao Chen
> Does this mean it won't trigger a checkpoint before scaling up or scaling > down? The in-place rescaling won't do that. > Do I need the autotuning to be turned on for exactly once processing? It suffices to just go back to the full-restart upgrade mode provided by the operator: disable

Re: Understanding flink-autoscaler behavior

2024-06-12 Thread Chetas Joshi
Hi Zhanghao, If I am using the autoscaler (flink-k8s-operator) without enabling auto-tuning, the documentation says it triggers in-place scaling without going through a full job upgrade lifecycle. Does this mean it won't trigger a checkpoint before scaling up or scaling down? I am observing that

Re: Slack Community Invite?

2024-06-12 Thread Ufuk Celebi
FYI This was recently fixed on the website, too. > On 28. May 2024, at 05:05, Alexandre Lemaire wrote: > > Thank you! > >> On May 27, 2024, at 9:35 PM, Junrui Lee wrote: >> >> Hi Alexandre, >> >> You can try this link: >>

Re: Setting uid hash for non-legacy sinks

2024-06-12 Thread Salva Alcántara
{ "emoji": "♥️", "version": 1 }

Re: Setting uid hash for non-legacy sinks

2024-06-12 Thread Gabor Somogyi
Hey Salva, Good to hear your issue has been resolved one way or another! Thanks for confirming that this operator hash trick is working on V2 as well. G On Wed, Jun 12, 2024 at 5:20 AM Salva Alcántara wrote: > Hey Gabor, > > I didn't finally need to keep compatibility with existing

Re: Changing TTL config without breaking state

2024-06-12 Thread xiangyu feng
Hi Salva, Unfortunately, the state is currently incompatible between enabling TTL and disabling TTL. This issue is tracked by this jira( https://issues.apache.org/jira/browse/FLINK-32955) and not resolved yet. Usually, we need to find a way to reaccumulate the state after enabled the state ttl.