Re: Flink Overwrite parameters in ExecutorUtils

Hi Austin, Thinks for your reply. For example, we want to test if we could improve the throughput by changing the value of `execution.buffer-timeout`. Best, Dan Zou > 2022年2月19日 00:33，Austin Cawley-Edwards 写道： > > Hi Dan, > > I'm not exactly sure why, but could you help me understand the use

Option Types not recognized in Table API

Hi All I am working on building a flink application which reads data from kafka topics, apply some transformations and writes to the Iceberg table. I read the data from kafka topic (which is in json) and use circe to decode that to scala case class with scala Option values in it. All the

关于“批”处理元素导致batch的event timestamp变化，后接window统计错乱的问题。

背景 部分算子需要redis查询，为了性能，需要做批量redis查询，通过redis pipeline机制查询。所以对于输入流A，我会做一个BatchGenerator算子B用于将输入数据转位batch输出，然后算子C查询redis后输出，并且输出时拆分batch为多个单独元素，进入算子D。D算子如果是window统计，...。 问题 算子D部分会发现收到的元素无timestamp、或timestamp错误，导致程序出错，或者窗口划分错误。 原因分析和解决等

Re: RMQSource non-parallel, seems inconsistent with documentation

Hey Daniel, I think you're right that the docs are misleading in this case – anything that extends SourceFunction will always execute at parallelism 1, set parallelism is ignored. Explicitly setting parallelism in the example in the docs is unnecessary and confusing. I personally have only used

Re: No effect from --allowNonRestoredState or "execution.savepoint.ignore-unclaimed-state" in K8S application mode

Hi Austin, Thanks for the reply! Yeah, the docs aren't super explicit about this. But for what it's worth, I'm setting a few options unrelated to kubernetes this way and they all have effect: -Dstate.checkpoints.num-retained=100 \

Re: No effect from --allowNonRestoredState or "execution.savepoint.ignore-unclaimed-state" in K8S application mode

Hi Andrey, It's unclear to me from the docs[1] if the flink native-kubernetes integration supports setting arbitrary config keys via the CLI. I'm cc'ing Yang Wang, who has worked a lot in this area and can hopefully help us out. Best, Austin [1]:

DataStream API: Parquet File Format with Scala Case Classes

Hello, Context: We are working on integrating Hybrid Sources with different Sources and Sinks. I have been working on a Parquet source that allows users to load the FileSource[T] so that the source can be used within Hybrid Sources where the HybridSource is of Type[T]. The environment is Scala

No effect from --allowNonRestoredState or "execution.savepoint.ignore-unclaimed-state" in K8S application mode

Hey all, I'm working on migrating our Flink job away from Hadoop session mode to K8S application mode. It's been going great so far but I'm hitting a wall with this seemingly simple thing. In the first phase of the migration I want to remove some operators (their state can be discarded) and

RMQSource non-parallel, seems inconsistent with documentation

Hello, I’ve noticed that https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/rabbitmq/ suggests that the RabbitMQ Source can be used with parallelization bigger than 1, but one can’t get exactly-once delivery guarantee there. However, the inheritance chain

Re: Fwd: How to get memory specific metrics for tasknodes

As I said, this is not possible. In the custom reporter you can filter out all the metrics you do not want. On 18/02/2022 19:44, Diwakar Jha wrote: Thank you. I understand that filtering metrics is not possible but i can configure it for only the memory metrics listed here? :

Re: Re: [External] : Re: Use TwoPhaseCommitSinkFunction or StatefulSink+TwoPhaseCommittingSink to achieve EXACTLY_ONCE sink?

Hi Yun, Thanks for the reply! This is very helpful. For the Sink interface, I checked ReducingUpsertSink

Re: Fwd: How to get memory specific metrics for tasknodes

Thank you. I understand that filtering metrics is not possible but i can configure it for only the memory metrics listed here? : https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#memory Currently, it's publishing all the container/operator metrics which is what I want to

Re: Flink Overwrite parameters in ExecutorUtils

Hi Dan, I'm not exactly sure why, but could you help me understand the use case for changing these parameters in Flink SQL? Thanks, Austin On Fri, Feb 18, 2022 at 8:01 AM Zou Dan wrote: > Hi, > I am using Flink Batch SQL in version 1.11. I find that Flink will > overwrite some configurations

Re: java.io.IOException: Failed to deserialize consumer record due to/ How to serialize table output for KafkaSink

Hi Yun It works. Thank you On Fri, Feb 18, 2022, 04:17 Yun Gao wrote: > Hi, > > I tried with a simplied version of the attached code, and it shows the > detailed exception is > > > Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to > java.time.Instant > > at

[ANNOUNCE] Apache Flink 1.13.6 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

[ANNOUNCE] Apache Flink 1.13.6 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: Apache Flink - Continuously monitoring directory using filesystem connector - 1.14.3

Thanks FG. On Friday, February 18, 2022, 02:54:44 AM EST, Francesco Guardiani wrote: Hi, Filesystem source directory watching is going to be available from 1.15: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#directory-watching FG On Fri, Feb

Flink Overwrite parameters in ExecutorUtils

Hi, I am using Flink Batch SQL in version 1.11. I find that Flink will overwrite some configurations in ExecutorUtils, which means this parameters bacome un-configurable, such as `pipeline.object-reuse` and `execution.buffer-timeout`, and the default value for this parameters will be not align

Re:退订

退订

Re:flink 不触发checkpoint

1. 图片挂了，看不到，尽量用文字，或者用图床等工具 2. 启动任务有配置checkpoint吗？ 在 2022-02-17 11:40:04，"董少杰" 写道： flink读取csv文件建表，同时消费kafka数据建表，两张表join之后写入hdfs(hudi),读取csv数据的任务已经是finished状态，就会触发不了checkpoint，看有什么办法能让它正常触发checkpoint? flink版本1.12.2。 谢谢！ | | 董少杰 | | eric21...@163.com |