Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

2017-06-26 Thread Fabian Hueske
Hi everybody, Shaoxuan, Timo, and I compiled a list of features from the replies to this thread, features that didn't make it into 1.3, and some additional ones. We also graded them by importance, tried to assess the effort, and added links to JIRAs (some existed already others were created) and e

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

2017-06-21 Thread Fabian Hueske
Hi Haohui, thanks for your input! Can you describe the semantics of the join you'd like to see in Flink 1.4? I can think of three types of joins that match your description: 1) `table` is an external table stored in an external database (redis, cassandra, MySQL, etc) and we join with the current

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

2017-06-21 Thread Haohui Mai
Hi, We are interested in building the simplest case of stream-table joins -- essentially calling stream.map(x => (x, table.get(x)). It solves the use cases of augmenting the streams with the information of the database. The operation itself can be batched for better performance. We are happy to c

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

2017-06-16 Thread jincheng sun
Hi Fabian, Thanks for bring up this discuss. In order to enrich Flink's built-in scalar function, friendly user experience, I recommend adding as much scalar functions as possible in version 1.4 release. I have filed the JIRAs( https://issues.apache.org/jira/browse/FLINK-6810), and try my best to w

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

2017-06-16 Thread Fabian Hueske
Thanks for your response Shaoxuan, My "Table-table join with retraction" is probably the same as your "unbounded stream-stream join with retraction". Basically, a join between two dynamic tables with unique keys (either because of an upsert stream->table conversion or an unbounded aggregation). B

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

2017-06-15 Thread Shaoxuan Wang
Nice timing, Fabian! Your checklist aligns our plans very well. Here are the things we are working on & planning to contribute to release 1.4: 1. DDL (with property waterMark config for source-table, and emit config on result-table) 2. unbounded stream-stream joins (with retraction supported) 3. b