Re: [DISCUSS] Enable scala formatting check

+1, currently the scalastyle does not work well actually, there are a lot of style differences in different files. It would be great if the code can be auto formatted. Best, Wenlong On Wed, 2 Mar 2022 at 16:34, Jingsong Li wrote: > +1. > > Thanks for driving. > > I wrote some scala code, the st

Re: [ANNOUNCE] Flink 1.15 Feature Freeze

Hi,all, I want to merge a pr (https://github.com/apache/flink/pull/18363) belonging to FLIP-190, which was approved yesterday, but not merged before code freeze because of CI queueing and failed by other changes. WDYT? Best, Wenlong On Thu, 17 Feb 2022 at 02:08, Till Rohrmann wrote: > Hi everyo

Re: [VOTE] FLIP-204: Introduce Hash Lookup Join

+1 non-binding! Thanks for driving this, Jing! Best, Wenlong On Mon, 24 Jan 2022 at 16:02, Jingsong Li wrote: > +1 (binding) > > Thanks Jing! > > Best, > Jingsong > > On Mon, Jan 24, 2022 at 3:48 PM Jark Wu wrote: > > > > +1 (binding) > > > > Thanks Jing for driving this! > > > > Best, > > Ja

Re: [DISCUSS] Add the ability of evolution RowData in sql/table job

Hi, Aitozi, thanks for bringing up the discussion on the state evolution of flink sql. It would be a great improvement on flink sql. I think it would be better if you could prepare a document providing more details about the solution, it would be a big story and huge change and we need to discuss i

Re: [Ask for review]

hi, Aitozi, thanks for the contribution first. I checked the issue, it seems that the discussion is blocked and hasn't finished yet. Anyway, I would like to join and help review the solution and pr. Best, Wenlong On Sun, 9 Jan 2022 at 16:53, Aitozi wrote: > Hi community: > I have created

Re: Re: [DISCUSS] Introduce Hash Lookup Join

Hi, Jing, thanks for driving the discussion. Have you made some investigation on the syntax of join hint? Why do you choose USE_HASH from oracle instead of the style of spark SHUFFLE_HASH, they are quite different. People in the big data world may be more familiar with spark/hive, if we need to ch

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

Hi, @Chesnay Schepler I want to forward a feedback from the user mailing list, the jars of the new patch releases are not available at the central maven repo. example: https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java Best, Wenlong On Fri, 17 Dec 2021 at 01:27, Chesnay S

Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs

+1 (non-binding). Thanks for driving this Timo Best, Wenlong On Wed, 15 Dec 2021 at 03:21, Martijn Visser wrote: > +1 (non-binding). Thanks for driving this Timo > > Op di 14 dec. 2021 om 17:45 schreef Timo Walther > > > Hi everyone, > > > > I'd like to start a vote on FLIP-190: Support Versi

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

.e. no classes) and > > is fully declarative. > > > > I will add a remark. > > > > Thanks, > > Timo > > > > > > On 10.12.21 11:43, wenlong.lwl wrote: > > > hi, Timo, thanks for the explanation. I totally agree with what you > said. >

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

date the ExecNode to version 2 and not 3 in > your example. > > Regards, > Timo > > > > On 10.12.21 06:02, wenlong.lwl wrote: > > Hi, Timo, thanks for updating the doc. > > > > I have a comment on plan migration: > > I think we may need to

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

r 于2021年12月8日周三 下午4:57写道： > >>> > >>> Hi Wenlong, > >>> > >>> thanks for the feedback. Great that we reached consensus here. I will > >>> update the entire document with my previous example shortly. > >>> > >&g

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

lanFormat=1.16, > supportedSavepointFormat=1.15) > > > // topology change > > // initial introduction in 1.15 > @ExecNodeMetadata(name=A, version=1, supportedPlanFormat=1.15, > supportedSavepointFormat=1.15) > > // complete new class structure in 1.16 annotated with >

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

Best, Wenlong On Tue, 7 Dec 2021 at 15:08, wenlong.lwl wrote: > hi, Timo, I would prefer to update the version every time we change the > state layer too. > > It could be possible that we change the exec node in 2 steps: > First, we add a newStateLayout because of some improvement in

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

hi, Timo, I would prefer to update the version every time we change the state layer too. It could be possible that we change the exec node in 2 steps: First, we add a newStateLayout because of some improvement in state, in order to keep compatibility we may still keep the old state for the first

Re: [DISCUSS] Deprecate Java 8 support

ion period we won't seen any > improvements on our end. > > Something that I could envision is that the docker images will default to > Java 11 after a while, and that we put a bigger emphasis on performance on > Java 11+ than on Java 8. > > On 01/12/2021 02:31, wenlon

Re: [DISCUSS] Deprecate Java 8 support

hi, @Chesnay Schepler would you explain more about what would happen when deprecating Java 8, but still support it. IMO, if we still generate packages based on Java 8 which seems to be a consensus, we still can not take the advantages you mentioned even if we announce that Java 8 support is depre

Re: [VOTE][FLIP-195] Improve the name and structure of vertex and operator name for job

ang I think your vote can be counted into binding now. > > Best, > Jark > > On Tue, 23 Nov 2021 at 20:19, Jing Zhang wrote: > > +1 (non-binding) > > Best, > Jing Zhang > > Martijn Visser 于2021年11月23日周二 下午7:42写道： > > +1 (non-binding) > > On Tue,

[VOTE][FLIP-195] Improve the name and structure of vertex and operator name for job

Hi everyone, Based on the discussion[1], we seem to have consensus, so I would like to start a vote on FLIP-195 [2]. Thanks for all of your feedback. The vote will last for at least 72 hours (Nov 26th 16:00 GMT) unless there is an objection or insufficient votes. [1] https://lists.apache.org/thr

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

Hi, Timo, thanks for driving the discussion and the preparation on the FLIP. This is a pain point of Flink SQL complaining by our users badly. I have seen many cases where our users suffer while trying to upgrade the flink version in order to take advantage of the bug fixes and performance improv

Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

quot; prefix will not cause too much consume of word > count :) . So I'm + 1 to add "[vertex-x]" prefix to the vertex name. > > Best, > Aitozi > > > 刘建刚 于2021年11月20日周六 下午6:27写道： > > > +1 for the FLIP. We have met the problem that a long name stuck the &

Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

> Yun Tang > > On 2021/11/17 07:42:28 godfrey he wrote: > > Hi Wenlong, I'm fine with the config options. > > > > Best, > > Godfrey > > > > wenlong.lwl 于2021年11月17日周三 下午3:13写道： > > > > > > > > Hi Chesney and Konstantin, > >

Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

f solving this for > DataStream and Table API at the same time, using the same configuration > keys. IMO we shouldn't introduce any additional fragmentation if we can > avoid it. > > Cheers, > > Konstantin > > On Tue, Nov 16, 2021 at 1:50 PM wenlong.lwl > wro

Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

tream API now, maybe take Windowing first as an example? Best, Wenlong On Tue, 16 Nov 2021 at 19:14, Chesnay Schepler wrote: > Why should this be specific to the table API? The datastream API has > similar issues with long operator names (like windowing). > > On 16/11/2021 11:22,

Re: Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

ot;, > another option is "CASCADE" (or "DEFAULT", which is more simple) > > What do you think? > > Best, > Godfrey > > wenlong.lwl 于2021年11月15日周一 下午6:36写道： > > > > > Hi, all, FYI the FLIP doc has been created ： > > > https:/

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

Congratulations Jing！ Best， Wenlong On Tue, 16 Nov 2021 at 09:54, Kurt Young wrote: > Congrats Jing! > > Best, > Kurt > > On Tue, Nov 16, 2021 at 9:52 AM Xintong Song > wrote: > > > Congratulations~! > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Tue, Nov 16, 2021 at 9:50 AM Dian F

Re: Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

Hi, all, FYI the FLIP doc has been created ： https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+sql+job Best, Wenlong On Mon, 15 Nov 2021 at 11:41, wenlong.lwl wrote: > Hi all, > Thanks for the feedback, It seem

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

Congratulations, Leonard! On Mon, 15 Nov 2021 at 10:43, Zhilong Hong wrote: > Congratulations, Leonard! > > On Mon, Nov 15, 2021 at 10:13 AM Qingsheng Ren wrote: > > > Congratulations Leonard! > > > > -- > > Best Regards, > > > > Qingsheng Ren > > Email: renqs...@gmail.com > > On Nov 12, 2021,

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

Congratulations, Yangze! On Mon, 15 Nov 2021 at 10:37, Zhilong Hong wrote: > Congratulations, Yangze! > > On Mon, Nov 15, 2021 at 10:13 AM Qingsheng Ren wrote: > > > Congratulations Yangze! > > > > -- > > Best Regards, > > > > Qingsheng Ren > > Email: renqs...@gmail.com > > On Nov 12, 2021, 10:

Re: Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

riving this, this improvement solves a >> long-complained >> > >> > > > problem, +1 >> > >> > > > >> > >> > > > Best, >> > >> > > > Godfrey >> > >> > > > >> > >>

[DISCUSS] Improve the name and structure of job vertex and operator name for job

Hi, all, I would like to start a discussion about an improvement on name and structure of job vertex name, mainly to improve experience of debugging and analyzing sql job at runtime. the main proposed changes including: 1. separate description and name for operator, so that we can have detailed in

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

Hi Jingsong, thanks for the proposal, providing a built-in storage solution for users will make flink SQL much more easier to use in production. I have some questions which may be missed in the FLIP, but may be important IMO: 1. Is it possible to read historical data from the file store first and

Re: [DISCUSS] FLIP-169: DataStream API for Fine-Grained Resource Requirements

Thanks Yangze for the flip, it is great for users to be able to declare the fine-grained resource requirements for the job. I have one minor suggestion: can we support setting resource requirements by configuration? Currently most of the config options in execution config can be configured by conf

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

Hi，Aljoscha, I would like to share a use case to second setting parallelism of table sink(or limiting parallelism range of table sink): When writing data to databases, there is limitation for number of jdbc connections and query TPS. we would get errors of too many connections or high load for db a

Re: [ANNOUNCE] New Apache Flink PMC Member - Hequn Chen

Congratulations Hequn! Bests, Wenlong On Mon, 20 Apr 2020 at 17:56, Yuan Mei wrote: > Congrats! > > On Fri, Apr 17, 2020 at 2:44 PM Becket Qin wrote: > > > Hi all, > > > > I am glad to announce that Hequn Chen has joined the Flink PMC. > > > > Hequn has contributed to Flink for years. He has w

Re: [DISCUSS] Releasing "fat" and "slim" Flink distributions

Hi all, Regarding slim and fat distributions, I think different kinds of jobs may prefer different type of distribution: For DataStream job, I think we may not like fat distribution containing connectors because user would always need to depend on the connector in user code, it is easy to include

Re: [DISCUSS] FLIP-124: Add open/close and Collector to (De)SerializationSchema

Hi Dawid, thanks for driving this. big +1 for this flip. Currently we have to implement an internal flat map in connectors, which is very inconvenient to support customized parser in DDL. Here is my minor suggestions: 1. Can we support serialize/deserialize from/to a message with a specified type

Re: [DISCUSS]FLIP-113: Support SQL and planner hints

Hi Danny, thanks for the proposal. +1 for adding table hints, it is really a necessary feature for flink sql to integrate with a catalog. For error handling, I think it would be more natural to throw an exception when error table hint provided, because the properties in hint will be merged and use

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

Congrats Jingsong! On Fri, 21 Feb 2020 at 11:41, Dian Fu wrote: > Congrats Jingsong! > > > 在 2020年2月21日，上午11:39，Jark Wu 写道： > > > > Congratulations Jingsong! Well deserved. > > > > Best, > > Jark > > > > On Fri, 21 Feb 2020 at 11:32, zoudan wrote: > > > >> Congratulations! Jingsong > >> > >> >

Re: [ANNOUNCE] Zhu Zhu becomes a Flink committer

Congratulations, Zhu Zhu! On Mon, 16 Dec 2019 at 09:14, Leonard Xu wrote: > Congratulations, Zhu Zhu ! ! > > Best, > Leonard Xu > > > On Dec 16, 2019, at 07:53, Becket Qin wrote: > > > > Congrats, Zhu Zhu! > > > > On Sun, Dec 15, 2019 at 10:26 PM Dian Fu wrote: > > > >> Congrats Zhu Zhu! > >>

Re: [ANNOUNCE] Jark Wu is now part of the Flink PMC

Congratulations Jark, well deserved! Best, Wenlong Lyu On Fri, 8 Nov 2019 at 18:22, tison wrote: > Congrats Jark! > > Best, > tison. > > > Jingsong Li 于2019年11月8日周五 下午6:08写道： > > > Congratulations to Jark. > > Jark has really contributed a lot to the table layer with a long time. > Well > > d

Re: [DISCUSS] FLIP-56: Dynamic Slot Allocation

Hi, Xintong, thanks for the great proposal. big +1 for the feature! It is something like mapreduce-1.0 to mapreduce-2.0. I like the design on the whole. One point may need to be included in the proposal：How we deal with slot share group and dynamic slot allocation? It can be quite different with d

Re: [ANNOUNCE] Kete Young is now part of the Flink PMC

Congratulations Kurt! On Tue, 23 Jul 2019 at 17:24, Robert Metzger wrote: > Hi all, > > On behalf of the Flink PMC, I'm happy to announce that Kete Young is now > part of the Apache Flink Project Management Committee (PMC). > > Kete has been a committer since February 2017, working a lot on Tabl

Re: Re: [ANNOUNCE] Zhijiang Wang has been added as a committer to the Flink project

Congrats, Zhijiang! On Tue, 23 Jul 2019 at 15:59, Becket Qin wrote: > Congrats, Zhijiang! > > On Tue, Jul 23, 2019 at 2:01 PM Jark Wu wrote: > > > Congratulations Zhijiang! > > > > > > On Tue, 23 Jul 2019 at 11:30, vino yang wrote: > > > > > Congratulations Zhijiang! > > > > > > Haibo Sun 于20

Re: [DISCUSS] Flink SQL DDL Design

Hi, Shuyi, thanks for the proposal. I have two concerns about the table ddl: 1. how about remove the source/sink mark from the ddl, because it is not necessary, the framework determine the table referred is a source or a sink according to the context of the query using the table. it will be more

Re: [DISCUSS] FLIP-22: Eager State Declaration

Hi, all, we have jobs which create state according to type of the key and a dynamic configuration: eg: key_type_1's aggregation function is average, while key_type_2's is sum we need to create state dynamically because the aggregation function may change in runtime and different aggregation funct

Re: Jobmanager drops upon submitting a jar

Hi, amir, I think you could check the log of job manager to make sure that job manager [192.168.56.101:6123 ] is running well firstly, you may get what is wrong in the log. On 13 April 2017 at 08:54, amir bahmanyari wrote: > > > > Hi Colleagues,I have a simple

Re: [DISCUSS] FLIP-17 Side Inputs

logic. Windowing can happen upstream and whenever that emits new data > >>>>> then operator will notice because there is new input. Having > windowing > >>>>> inside the side-input of an operator as well would just make the > >>

Re: [DISCUSS] FLIP-17 Side Inputs

this problem fizzled out > because > > > > > we got to deep into discussing special semantics and features. I > think > > > > > the most important thing to agree on right now is the basic API > and the > > > > > implementation plan. What do you think

Re: [DISCUSS] FLIP-17 Side Inputs

Hi Aljoscha, thank you for the proposal, it is great to hear about the progress in side input. Following is my point of view: 1. I think there may be an option to block the processing of the main input instead of buffer the data in state because in production, the through put of the main input is

Re: [DISCUSS] Support Incremental Checkpointing in Flink

Hi, happy to see such a great proposal. Increment Checkpointing is really necessary for jobs with states in large scale, which can save both space and time spent on checkpointing. For checkpoints, I think that it is better to leave the state backend to decide whether a checkpoint is incremental or