Indefinite hanging when querying batch transactional data from kafka

2024-03-20 Thread Dongwoo Kim
Hello community, I reported a potential bug with the flink kafka connector on Jira a few weeks ago. But I haven't received feedback yet. Could anyone kindly review this issue? Any feedback would be much appreciated. Jira issue: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34470 Than

Re: Table API table2datastream (toChangelogStream)

2023-08-28 Thread Dongwoo Kim
Hi Ravi, Maybe it's due to *execution.batch.adaptive.auto-parallelism.enabled* Option. How about trying to configure *execution.batch.adaptive.auto-parallelism.enabled* to *false* and apply other parallelism. Best, Dongwoo 2023년 8월 27일 (일) 오후 8:00, ravi_suryavanshi.yahoo.com via user < user@flink

Fwd: Flink sql client doesn't work with "partition by" clause

2023-07-31 Thread Dongwoo Kim
-- Forwarded message - 보낸사람: Dongwoo Kim Date: 2023년 7월 31일 (월) 오후 11:36 Subject: Re: Flink sql client doesn't work with "partition by" clause To: liu ron Hi, ron. Actually I'm not receiving any exception message when executing the *partition by* clause in

Re: Flink sql client doesn't work with "partition by" clause

2023-07-28 Thread Dongwoo Kim
= 'filesystem', 'path' = 'hdfs://${out_data_path}/month=202307/day=20230714', 'format' = 'parquet' ); SELECT status FROM source_table_2 GROUP BY status; Best, dongwoo 2023년 7월 28일 (금) 오후 6:19, Dongwoo Kim 님이

Flink sql client doesn't work with "partition by" clause

2023-07-28 Thread Dongwoo Kim
Hello all, I have found that flink sql client doesn't work with "partition by" clause. Is this bug? It's bit weird since when I execute same sql with tableEnv.executeSql(statement) code it works as expected. Has anyone tackled this kind of issue? I have tested in flink 1.16.1 version. Thanks in adv

Cleanup of changelog and materialized state is not working when job restarts.

2023-07-05 Thread Dongwoo Kim
Hi all, While working on a streaming application built with flink I have found some issues and want to ask for advice. First, our application's key configurations are like below. flink version: 1.17.0 state.backend: "rocksdb" state.backend.incremental: "true" state.backend.changelog.enabled: "true

Re: How to set hdfs configuration in flink kubernetes operator?

2023-06-21 Thread Dongwoo Kim
Hi leilinee, I'm not sure whether this is the best practice but I would like to share our experience about configuring HDFS as checkpoint storage while using flink kubernetes operator. There are two steps. *Step 1)* Mount krb5-conf & keytab file to flink kubernetes operator pod You have to creat

Re: Changelog fail leads to job fail regardless of tolerable-failed-checkpoints config

2023-06-21 Thread Dongwoo Kim
il. And only exceptions in > the checkpoint phase will be counted as checkpoint failures. > > Dongwoo Kim 于2023年6月20日周二 18:31写道: > > > > Hello all, I have a question about changelog persist failure. > > When changelog persist process fails due to an S3 time

Changelog fail leads to job fail regardless of tolerable-failed-checkpoints config

2023-06-20 Thread Dongwoo Kim
Hello all, I have a question about changelog persist failure. When changelog persist process fails due to an S3 timeout, it seems to lead to the job failure regardless of our "execution.checkpointing.tolerable-failed-checkpoints" configuration being set to 5 with this log. Caused by: java.io.IOExce

Quick question about flink document.

2023-04-09 Thread Dongwoo Kim
Hi community, I’m new to flink and trying to learn about the concepts of flink to prepare migrating heron application to flink. I have a quick question about this flink document. (https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/concepts/stateful-stream-processing/#snapshotting-ope