[DISCUSS] FLIP-24 - SQL Client
Hey everyone, in the past, the community already asked about having a way to write Flink jobs without extensive programming skills. During the last year we have put a lot of effort in the core of our Table & SQL API. Now it is time to improve the tooling around it as well and make Flink more accessible. I already opened an issue for adding a SQL CLI client [0]. We developed a small protoype at data Artisans that was shown at the last Flink Forward Berlin [1]. For Flink 1.5 it would be great to offer at least a CLI client to play around with Flink and use it for debugging/demo purposes. We created a FLIP-24 [2] that roughly sketches the functionality and architecture. We also show how this architecture can evolve from a CLI client to a SQL gateway/REST server. Most of it is still up for dicussion. The targeted minimum viable product for Flink 1.5 should act as a start for collecting feedback and attracting contributors. So feedback on this FLIP is very welcome. What do you think? Regards, Timo [0] https://issues.apache.org/jira/browse/FLINK-7594 [1] https://berlin-2017.flink-forward.org/kb_sessions/from-streams-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/ [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
Re: [DISCUSS] FLIP-24 - SQL Client
Hi Timo, thanks for putting this FLIP together. I think this will be a great feature for Flink. I think it makes sense to show the long term goals of the SQL client/server architecture, but the current description makes it a bit difficult to figure out what will be part of this FLIP and what will be added later because the information is spread across the FLIP. This could be improved by describing the "Implementation Plan" in more detail, i.e, have subsections for the first version with a detailed list of supported features and another for later extensions. Comments to specific sections: - Proposed changes: Are there any changes necessary on existing code (e.g., job submission with multiple jars) or are all changes just additions to Flink? - Result Retrieval: Extend the table to three columns (Query type, Internal mode, External mode). - The context of the description of the query types "CREATE MATERIALIZED VIEW" and "SELECT" is not clear to me. Add a subsection and explain what is described there? - Implementation plan: Add which result retrieval modes will be supported in the initial version? Which configuration will be available? Best, Fabian 2017-12-19 16:53 GMT+01:00 Timo Walther : > Hey everyone, > > in the past, the community already asked about having a way to write Flink > jobs without extensive programming skills. During the last year we have put > a lot of effort in the core of our Table & SQL API. Now it is time to > improve the tooling around it as well and make Flink more accessible. I > already opened an issue for adding a SQL CLI client [0]. We developed a > small protoype at data Artisans that was shown at the last Flink Forward > Berlin [1]. > > For Flink 1.5 it would be great to offer at least a CLI client to play > around with Flink and use it for debugging/demo purposes. We created a > FLIP-24 [2] that roughly sketches the functionality and architecture. We > also show how this architecture can evolve from a CLI client to a SQL > gateway/REST server. Most of it is still up for dicussion. The targeted > minimum viable product for Flink 1.5 should act as a start for collecting > feedback and attracting contributors. > > So feedback on this FLIP is very welcome. What do you think? > > Regards, > Timo > > > [0] https://issues.apache.org/jira/browse/FLINK-7594 > [1] https://berlin-2017.flink-forward.org/kb_sessions/from-strea > ms-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/ > [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client > >
Re: [DISCUSS] FLIP-24 - SQL Client
Hi Fabian, thank you for your feedback. I extended the "Implementation Plan" by 3 subsections. Regarding the proposed changes, I looked into the code of the ClusterClient and talked to Till Rohrmann about it. By now, I could not identify features that are missing on existing components. It might be necessary to make some internal methods accessible but no big changes. I also extended the "Result Retrieval" section to (hopefully) answer your questions. Feel free to give further comments. Regards, Timo Am 12/21/17 um 3:52 PM schrieb Fabian Hueske: Hi Timo, thanks for putting this FLIP together. I think this will be a great feature for Flink. I think it makes sense to show the long term goals of the SQL client/server architecture, but the current description makes it a bit difficult to figure out what will be part of this FLIP and what will be added later because the information is spread across the FLIP. This could be improved by describing the "Implementation Plan" in more detail, i.e, have subsections for the first version with a detailed list of supported features and another for later extensions. Comments to specific sections: - Proposed changes: Are there any changes necessary on existing code (e.g., job submission with multiple jars) or are all changes just additions to Flink? - Result Retrieval: Extend the table to three columns (Query type, Internal mode, External mode). - The context of the description of the query types "CREATE MATERIALIZED VIEW" and "SELECT" is not clear to me. Add a subsection and explain what is described there? - Implementation plan: Add which result retrieval modes will be supported in the initial version? Which configuration will be available? Best, Fabian 2017-12-19 16:53 GMT+01:00 Timo Walther : Hey everyone, in the past, the community already asked about having a way to write Flink jobs without extensive programming skills. During the last year we have put a lot of effort in the core of our Table & SQL API. Now it is time to improve the tooling around it as well and make Flink more accessible. I already opened an issue for adding a SQL CLI client [0]. We developed a small protoype at data Artisans that was shown at the last Flink Forward Berlin [1]. For Flink 1.5 it would be great to offer at least a CLI client to play around with Flink and use it for debugging/demo purposes. We created a FLIP-24 [2] that roughly sketches the functionality and architecture. We also show how this architecture can evolve from a CLI client to a SQL gateway/REST server. Most of it is still up for dicussion. The targeted minimum viable product for Flink 1.5 should act as a start for collecting feedback and attracting contributors. So feedback on this FLIP is very welcome. What do you think? Regards, Timo [0] https://issues.apache.org/jira/browse/FLINK-7594 [1] https://berlin-2017.flink-forward.org/kb_sessions/from-strea ms-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/ [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client