Application Mode Start Using Restful API

Hello, I am wondering if there is any way to start a flink application mode through restful api in k8s? Or other Java APIs? Thank you very much *Xiao Ma*

Re: Using HybridSource

Hi Oscar, As Andrew noted, you could use the approach of overriding the CsvReaderFormat (or whatever you’re using to extract CSV data from S3) to return a Protobuf that’s the same format as what you’re getting from Kafka, though that obviously introduces some inefficiencies. Or you could do the

Re: Application Mode Start Using Restful API

Hi Xiao, Do you mean like sending a REST API request to the K8s API Server to start a new Flink job? If - yes, then it is possible today via Flink Kubernetes Operator. Best regards, Alexey On Thu, Jul 6, 2023 at 5:20 PM Xiao Ma wrote: > Hello, > > I am wondering if there is any way to start a

Re: Multiple Kafka Source for a Data Pipeline

Hi Yogesh, Multiple kafka sources are supported. This warning only indicates that multiple consumers won't be able to register JMX metrics. There are several bugs reported about this, but I believe it should be fixed for consumers in the newer Flink versions (1.14+). On Wed, Jul 5, 2023 at 9:32 P

Re: Multiple Kafka Source for a Data Pipeline

Hello Yaroslavl, I am using flink 1.17.1, so I am assuming that this isn’t fixed. Regards, Yogesh On Thu, 6 Jul 2023 at 10:06 PM, Yaroslav Tkachenko wrote: > Hi Yogesh, > > Multiple kafka sources are supported. This warning only indicates that > multiple consumers won't be able to register JMX

Re: Application Mode Start Using Restful API

Hi Alexey, Yes, sending a REST API request to start an Application mode flink cluster, is it doable for the flink operator right now? Thank you. *Xiao Ma* On Thu, Jul 6, 2023 at 12:19 PM Alexey Novakov wrote: > Hi Xiao, > > Do you mean like sending a REST API request to the K8s API Server to

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congrats to all ! It will be helpful to promote Apache Flink if we can add a page to our website like others[2]. I’ve created an issue to improve this. Best, Leonard [1] https://issues.apache.org/jira/browse/FLINK-32555 [2] https://spark.apache.org/news/sigmod-system-award.html

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Hi, Leonard. I would like to help to add this page. Please assign this issue to me. Thanks. Best, Hang Leonard Xu 于2023年7月7日周五 11:26写道： > Congrats to all ! > > It will be helpful to promote Apache Flink if we can add a page to our > website like others[2]. I’ve created an issue to improve this

Getting rid of getCheckpointLock() method

Hello, I'm trying to upgrade flink from 1.8 to 1.11 and StreamTask's getcheckpointlock method is removed and the new recommendation is to use MailboxExecutor. Currently we're using it like: synchronized(operator.getContainingTask.getCheckpointLock()) { // perform some operation } The pur

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congrats everyone :-) Best Yuan On Fri, Jul 7, 2023 at 11:29 AM Hang Ruan wrote: > Hi, Leonard. > > I would like to help to add this page. Please assign this issue to me. > Thanks. > > Best, > Hang > > Leonard Xu 于2023年7月7日周五 11:26写道： > >> Congrats to all ! >> >> It will be helpful to promote

Re: Multiple Kafka Source for a Data Pipeline

Hi all, This should be fixed. Are you setting a different `client.id.prefix` for each KafkaSource? See this: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/kafka/#additional-properties and https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connec