Re: Table API function and expression vs SQL

Thanks a lot Hand and Mate On Saturday, 25 March, 2023 at 06:21:49 pm IST, Mate Czagany wrote: Hi, Please also keep in mind that restoring existing Table API jobs from savepoints when upgrading to a newer minor version of Flink, e.g. 1.16 -> 1.17 is not supported as the topology might

Re: Are metadata columns required to get declared in the table's schema?

Hi, Jie, In Flink, if we want to access a metadata column, we need to declare it in the DDL. More details could be found here[1]. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/create/#columns Jie Han 于2023年3月26日周日 14:58写道： > Hi community, I want

Re: Are metadata columns required to get declared in the table's schema?

ps : DDL I said is the CREATE TABLE statements. Best, Hang Hang Ruan 于2023年3月26日周日 21:33写道： > Hi, Jie, > > In Flink, if we want to access a metadata column, we need to declare it in > the DDL. > More details could be found here[1]. > > Best, > Hang > > [1] > https://nightlies.apache.org/flink/f

Re: Are metadata columns required to get declared in the table's schema?

Thank you for your respond. Actually I noticed that the doc says 'However, declaring a metadata column in a table’s schema is optional’. So, does it mean that we don’t need to declare it when we don't query it rather than we can query it without the declaration? Best, Jay

Re: Are metadata columns required to get declared in the table's schema?

Hi, Jie, If you don't need these metadata columns, you don't need to declare them for the table. Then metadata columns will not be read from sources and will not be written into the sink. You can query a table that is without the metadata column declaration. It depends on your requests. Best, Han

Flink CEP Resource Utilisation Optimisation

Hi Team, *Flink Version:* 1.15.0 *Java Version:* 1.8 *Standalone Cluster* *Task Manager:* AWS EC2 of Instance Type c5n.4xlarge (vCPU 16, Memory 42 Gb, 8 slots per TM) *CEP Scenario:* Kafka Event A followed by Kafka Event B within 10 mins *Throughput:* 20k events per second for Event A, 0 for Kafka

Re: Flink CEP Resource Utilisation Optimisation

退订 发自我的iPhone -- Original -- From: Abhishek Singla

Re: Flink CEP Resource Utilisation Optimisation

Hi Abhishek, Thanks for sharing the experiment! As for the performance question, I believe you could give a try on Flink CEP with version >= 1.16.0, which includes the optimization introduced in FLINK-23890. This optimization will reduce lots

Re: Flink CEP Resource Utilisation Optimisation

Thanks, Geng for the quick and actionable response. I will definitely try this with Flink version >= 1.16.0 and get back with the observations. Regarding the checkpoint size issue, my concern is if there is no more state, shouldn't the checkpoint size be way less than 2 GB? I mean I was expecting

Re: Flink CEP Resource Utilisation Optimisation

I see your point. Are there lots of different keys in your test input? If that is the case, CEP operator in 1.15.0 will not clean some intermediate states(partial matches will be cleaned due to timeout but some computation states are leaked). It is fixed in flink1.16(FLINK-31017) by Juntao Hu. B

Re: Flink CEP Resource Utilisation Optimisation

Thanks, yes there were a lot of keys in the test input. In fact, every event has a unique key which is not repeated in subsequent events. On Sun, Mar 26, 2023 at 10:26 PM Geng Biao wrote: > I see your point. Are there lots of different keys in your test input? If > that is the case, CEP operator