Re: Flink Overwrite parameters in ExecutorUtils

Hi Austin, Thinks for your reply. For example, we want to test if we could improve the throughput by changing the value of `execution.buffer-timeout`. Best, Dan Zou > 2022年2月19日 00:33，Austin Cawley-Edwards 写道： > > Hi Dan, > > I'm not exactly sure why, but could you help me understand the use

Option Types not recognized in Table API

Hi All I am working on building a flink application which reads data from kafka topics, apply some transformations and writes to the Iceberg table. I read the data from kafka topic (which is in json) and use circe to decode that to scala case class with scala Option values in it. All the transfor

Re: RMQSource non-parallel, seems inconsistent with documentation

Hey Daniel, I think you're right that the docs are misleading in this case – anything that extends SourceFunction will always execute at parallelism 1, set parallelism is ignored. Explicitly setting parallelism in the example in the docs is unnecessary and confusing. I personally have only used th

Re: No effect from --allowNonRestoredState or "execution.savepoint.ignore-unclaimed-state" in K8S application mode

Hi Austin, Thanks for the reply! Yeah, the docs aren't super explicit about this. But for what it's worth, I'm setting a few options unrelated to kubernetes this way and they all have effect: -Dstate.checkpoints.num-retained=100 \ -Dfs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdent

Re: No effect from --allowNonRestoredState or "execution.savepoint.ignore-unclaimed-state" in K8S application mode

Hi Andrey, It's unclear to me from the docs[1] if the flink native-kubernetes integration supports setting arbitrary config keys via the CLI. I'm cc'ing Yang Wang, who has worked a lot in this area and can hopefully help us out. Best, Austin [1]: https://nightlies.apache.org/flink/flink-docs-rel

DataStream API: Parquet File Format with Scala Case Classes

Hello, Context: We are working on integrating Hybrid Sources with different Sources and Sinks. I have been working on a Parquet source that allows users to load the FileSource[T] so that the source can be used within Hybrid Sources where the HybridSource is of Type[T]. The environment is Scala 2

No effect from --allowNonRestoredState or "execution.savepoint.ignore-unclaimed-state" in K8S application mode

Hey all, I'm working on migrating our Flink job away from Hadoop session mode to K8S application mode. It's been going great so far but I'm hitting a wall with this seemingly simple thing. In the first phase of the migration I want to remove some operators (their state can be discarded) and focus

RMQSource non-parallel, seems inconsistent with documentation

Hello, I’ve noticed that https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/rabbitmq/ suggests that the RabbitMQ Source can be used with parallelization bigger than 1, but one can’t get exactly-once delivery guarantee there. However, the inheritance chain see

Re: Fwd: How to get memory specific metrics for tasknodes

As I said, this is not possible. In the custom reporter you can filter out all the metrics you do not want. On 18/02/2022 19:44, Diwakar Jha wrote: Thank you. I understand that filtering metrics is not possible but i can configure it for only the memory metrics listed here? : https://nightlies

Re: Re: [External] : Re: Use TwoPhaseCommitSinkFunction or StatefulSink+TwoPhaseCommittingSink to achieve EXACTLY_ONCE sink?

Hi Yun, Thanks for the reply! This is very helpful. For the Sink interface, I checked ReducingUpsertSink implementatio

Re: Fwd: How to get memory specific metrics for tasknodes

Thank you. I understand that filtering metrics is not possible but i can configure it for only the memory metrics listed here? : https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#memory Currently, it's publishing all the container/operator metrics which is what I want to avoid

Re: Flink Overwrite parameters in ExecutorUtils

Hi Dan, I'm not exactly sure why, but could you help me understand the use case for changing these parameters in Flink SQL? Thanks, Austin On Fri, Feb 18, 2022 at 8:01 AM Zou Dan wrote: > Hi, > I am using Flink Batch SQL in version 1.11. I find that Flink will > overwrite some configurations i

Re: java.io.IOException: Failed to deserialize consumer record due to/ How to serialize table output for KafkaSink

Hi Yun It works. Thank you On Fri, Feb 18, 2022, 04:17 Yun Gao wrote: > Hi, > > I tried with a simplied version of the attached code, and it shows the > detailed exception is > > > Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to > java.time.Instant > > at org$apache$f

[ANNOUNCE] Apache Flink 1.13.6 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming a

Re: Apache Flink - Continuously monitoring directory using filesystem connector - 1.14.3

Thanks FG. On Friday, February 18, 2022, 02:54:44 AM EST, Francesco Guardiani wrote: Hi, Filesystem source directory watching is going to be available from 1.15: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#directory-watching FG On Fri, Feb 1

Flink Overwrite parameters in ExecutorUtils

Hi, I am using Flink Batch SQL in version 1.11. I find that Flink will overwrite some configurations in ExecutorUtils, which means this parameters bacome un-configurable, such as `pipeline.object-reuse` and `execution.buffer-timeout`, and the default value for this parameters will be not align