Re: Moving from flinkkafkaconsumer to kafkasource issues

2023-04-21 Thread Martijn Visser
Hi, Have you followed the documentation at https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#security ? Best regards, Martijn On Fri, Apr 21, 2023 at 3:00 AM Shammon FY wrote: > Hi Naga > > Could you provide detailed error information? I think it may be us

flink task submits error to k8s

2023-04-21 Thread ????????????
Hello! When we use flink version 1.14 to commit tasks to kubernetes, we get an error using Session mode. The following error is reported some time after the task is submitted . 2023-04-18 17:23:56,818 INFO  org.apache.flink.client.program.rest.RestClusterClient       [] - Submitting job 'Fl

State bootstrapping for Flink SQL / Table API jobs

2023-04-21 Thread Илья Соин
Hi Flink community, We have a quite complex sql job, it unions 5 topics, deduplicates by key and does some daily aggregations. The state TTL is 40 days. We want to be able to bootstrap its state from s3 or clickhouse. We want to have a general solution to this, to use for other SQL jobs as well

Issue with Incremental window aggregation using Aggregate function.

2023-04-21 Thread Sumanta Majumdar
Hi, Currently we have a streaming use case where we have a flink application which runs on a session cluster which is responsible for reading data from Kafka source which is basically table transaction events getting ingested into the upstream kafka topic which is converted to a row and then dedup

Re: flink task submits error to k8s

2023-04-21 Thread Weihua Hu
Hi, It's a DEBUG level log. Does this submission failed? Best, Weihua On Fri, Apr 21, 2023 at 5:44 PM “岁月飞扬” <1639467...@qq.com> wrote: > > Hello! > > When we use flink version 1.14 to commit tasks to kubernetes, we get an > error using Session mode. The following error is reported some time a

Client process in Flink application mode

2023-04-21 Thread Vinay Londhe
Hello I have started to play around with Flink recently and I had a question about the behavior of the Flink application mode. When we start a Flink application using the command flink run-application -t yarn-application my.jar, is it possible for the client process that submits the Flink applica

Kubernetes operator stops responding due to Connection reset by peer

2023-04-21 Thread Alexis Sarda-Espinosa
Hello, Today, we received an alert because the operator appeared to be down. Upon further investigation, we realized the alert was triggered because the endpoint for Prometheus metrics (which we enabled) stopped responding, so it seems the endpoint used for the liveness probe wasn't affected and t

Re: Kubernetes operator stops responding due to Connection reset by peer

2023-04-21 Thread Gyula Fóra
Hi Alexis, We have recently added support for canary deployments which allows the liveness probe to detect general operator problems. https://issues.apache.org/jira/browse/FLINK-31219 It's not completely automatic and you have to deploy the canaries yourself but I think it will be helpful :) Thi