flink on native k8s里如何使用flink sql gateway

大家好，请教个问题。 用native kubernetes方式在k8s集群上部署好了flink，现在需要在这个flink集群里使用flink sql gateway，大家有什么好的方案吗？ 目前的做法是，进入pod里启动sql gateway，然后在k8s创建flink-sql-gateway service，这样就可以通过这个service来访问sql gateway了，但是这个方法有个问题，部署过程中必需进入pod启服务，这是不利于自动化部署的，具体的操作命令如下，大家帮忙看看有没有好的解决方案来避免这个问题。 1、创建flink集群 ./bin/kubernetes-se

Re: Query around Rocksdb

Hi neha, Due to the limitation of RocksDB, we cannot create a strict-capacity-limit LRUCache which shared among rocksDB instance(s), FLINK-15532[1] is created to track this. BTW, have you set TTL for this job[2], TTL can help control the state size. [1] https://issues.apache.org/jira/browse/FLIN

Re: Checkpointed data size is zero

Hi Kamal, Is the Full Checkpoint Data Size[1] also zero? If not, it may be that no data is processed during this checkpoint. [1] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/monitoring/checkpoint_monitoring/ Shammon FY 于2023年7月4日周二 09:10写道： > > Hi Kamal, > > You can che

CFP for the 2nd Performance Engineering track at Community over Code NA 2023

Hi Apache Flink people - There are only 10 days left to submit a talk proposal (title and abstract only) for Community over Code NA 2023! The 2nd Performance Engineering track is on this year so any Apache project-related performance and scalability talks are welcome, here's the track CFP for mo

Re: SQL-gateway Failed to Run

Hi Xiaolong, >From the exception it seems that the flink session cluster is not running properly. Can you visit the flink web ui and everything is ok? Best, Shammon FY On Mon, Jul 3, 2023 at 2:43 PM Xiaolong Wang wrote: > Hi, > I've tested the Flink SQL-gateway to run some simple Hive queries

Re: Checkpointed data size is zero

Hi Kamal, You can check whether flink job has readed data from source in flink web ui, it will show total record count and size for each operator. Best, Shammon FY On Sat, Jul 1, 2023 at 4:53 PM Kamal Mittal via user wrote: > Hello Community, > > > > I have a requirement to read data coming ov

Re: Query around Rocksdb

Hi neha, Which flink version are you using? We have also encountered the issue of continuous growth of off-heap memory in the TM of the session cluster before, the reason is that the memory fragments cannot be reused like issue [1]. You can check the memory allocator and try to use jemalloc instea

Re: Difference between different values for starting offset

Hi Oscar, You are correct about the OffsetInitializer being only effective when there is no Flink state--in addition, if you have partition discovery on, this initializer will be reused for the new partitions (i.e. splits) discovered. Assuming the job is continuing from the offset in Flink state,

Changing vertex parallelism

Is there any way changing job vertex parallelism during runtime (downtime included)? For example, via REST API?

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congrats everyone! Best, Jark > 2023年7月3日 22:37，Yuval Itzchakov 写道： > > Congrats team! > > On Mon, Jul 3, 2023, 17:28 Jing Ge via user > wrote: >> Congratulations! >> >> Best regards, >> Jing >> >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia >

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congrats team! On Mon, Jul 3, 2023, 17:28 Jing Ge via user wrote: > Congratulations! > > Best regards, > Jing > > > On Mon, Jul 3, 2023 at 3:21 PM yuxia wrote: > >> Congratulations! >> >> Best regards, >> Yuxia >> >> -- >> *发件人: *"Pushpa Ramakrishnan" >> *收件人: *"Xin

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congratulations! Best regards, Jing On Mon, Jul 3, 2023 at 3:21 PM yuxia wrote: > Congratulations! > > Best regards, > Yuxia > > -- > *发件人: *"Pushpa Ramakrishnan" > *收件人: *"Xintong Song" > *抄送: *"dev" , "User" > *发送时间: *星期一, 2023年 7 月 03日 下午 8:36:30 > *主题: *Re: [

Difference between different values for starting offset

Hei, Looking at the flink documentation for kafkasource I see the following values for starting offset: OffsetInitializer.earliest OffsetInitializer.latest OffsetInitializer.commitedOffset(OffsetResetStrategy.EARLIEST) >From what I understand OffsetInitializer.earliest uses earliest offset the f

Using HybridSource

Hei, We want to bootstrap some data from a CSV file before reading from a kafka topic that has a retention period of 7 days. We believe the best tool for that would be the HybridSource but the problem we are facing is that both datasources are of different nature. The KafkaSource returns a protobu

Re: Identifying a flink dashboard

Thanks Alex, Whilst there are good solutions it looks like I am not going to be able to do anything useful. The port changes due to a whole bunch of useful script that create port forwards for various things starting from 3 (that just a number plucked out of the air) Services like mongo p

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congratulations! Best regards, Yuxia 发件人: "Pushpa Ramakrishnan" 收件人: "Xintong Song" 抄送: "dev" , "User" 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30 主题: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award Congratulations \uD83E\uDD73 On 03-Jul-2023, at 3:30 PM, Xintong Song

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Congratulations 🥳 > On 03-Jul-2023, at 3:30 PM, Xintong Song wrote: > >  > Dear Community, > > I'm pleased to share this good news with everyone. As some of you may have > already heard, Apache Flink has won the 2023 SIGMOD Systems Award [1]. > > "Apache Flink greatly expanded the use of s

the new state serializer can not be incompatible

Hi team : restore.from savepoint i meet: the new state serializer can not be incompatible Who can help me? Thanks in advance Kobe24

[ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Dear Community, I'm pleased to share this good news with everyone. As some of you may have already heard, Apache Flink has won the 2023 SIGMOD Systems Award [1]. "Apache Flink greatly expanded the use of stream data-processing." -- SIGMOD Awards Committee SIGMOD is one of the most influential da

Re: Very long launch of the Flink application in BATCH mode

Thanks guys. I tried the 1.17.1 version, but the problem still remains. It seems to be a bug, I created an issue https://issues.apache.org/jira/browse/FLINK-32513. On Thu, 29 Jun 2023 at 10:57, Martijn Visser wrote: > Hi Vladislav, > > I think it might be worthwhile to upgrade to Flink 1.17, giv

Re: Flink TCP socket custom source - savepoint cannot be taken

Hi Kamal, There's no such limitation, so most likely this is related to the implementation of your TCP source connector. Do keep in mind that just by the nature of TCP, I doubt that you will have any guarantees when it comes to this source. E.g. if you roll back to a savepoint of one day ago, how

Query Regarding Optimisation of Timer Management in Flink CEP (Version 1.17)

Hi, I am currently working with Flink CEP version 1.17, and I am in the process of load testing for potential memory leaks related to checkpoint data. While analyzing the CepOperator code, I have come across a particular pattern regarding timer registration and event processing that I believe c