Thanks for your discussion on google document. Comments addressed and added FileSystem connector chapter, and introduce code prototype for file system connector to unify flink file system and hive connectors.
Looking forward to your feedbacks. Thank you. Best, Jingsong Lee ------------------------------------------------------------------ From:JingsongLee <lzljs3620...@aliyun.com> Send Time:2019年9月18日(星期三) 09:45 To:Kurt Young <ykt...@gmail.com>; dev <dev@flink.apache.org> Subject:Re: [DISCUSS] FLIP-63: Rework table partition support Thanks for your reply and google doc comments. It has been discussed for two weeks now. I will start a vote thread. Best, Jingsong Lee ------------------------------------------------------------------ From:Kurt Young <ykt...@gmail.com> Send Time:2019年9月16日(星期一) 15:55 To:dev <dev@flink.apache.org> Cc:JingsongLee <lzljs3620...@aliyun.com> Subject:Re: [DISCUSS] FLIP-63: Rework table partition support +1 to this feature, I left some comments on google doc. Another comment is I think we should do some reorganize about the content when you converting this to a cwiki page. I will have some offline discussion with you. Since this feature seems to be a fairly big efforts, so I suggest we can settle down the design doc ASAP and start vote process. Best, Kurt On Thu, Sep 12, 2019 at 12:43 PM Biao Liu <mmyy1...@gmail.com> wrote: Hi Jingsong, Thanks for explaining. It looks cool! Thanks, Biao /'bɪ.aʊ/ On Wed, 11 Sep 2019 at 11:37, JingsongLee <lzljs3620...@aliyun.com.invalid> wrote: > Hi biao, thanks for your feedbacks: > > Actually, the runtime source partition of runtime is similar to split, > which concerns data reading, parallelism and fault tolerance, all the > runtime concepts. > While table partition is only a virtual concept. Users are more likely to > choose which partition to read and which partition to write. Users can > manage their partitions. > One is physical implementation correlation, the other is logical concept > correlation. > So I think they are two completely different things. > > About [2], The main problem is that how to write data to a catalog file > system in stream mode, it is a general problem and has little to do with > partition. > > [2] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-notifyOnMaster-for-notifyCheckpointComplete-td32769.html > > Best, > Jingsong Lee > > > ------------------------------------------------------------------ > From:Biao Liu <mmyy1...@gmail.com> > Send Time:2019年9月10日(星期二) 14:57 > To:dev <dev@flink.apache.org>; JingsongLee <lzljs3620...@aliyun.com> > Subject:Re: [DISCUSS] FLIP-63: Rework table partition support > > Hi Jingsong, > > Thank you for bringing this discussion. Since I don't have much experience > of Flink table/SQL, I'll ask some questions from runtime or engine > perspective. > > > ... where we describe how to partition support in flink and how to > integrate to hive partition. > > FLIP-27 [1] introduces "partition" concept officially. The changes of > FLIP-27 are not only about source interface but also about the whole > infrastructure. > Have you ever thought how to integrate your proposal with these changes? > Or you just want to support "partition" in table layer, there will be no > requirement of underlying infrastructure? > > I have seen a discussion [2] that seems be a requirement of infrastructure > to support your proposal. So I have some concerns there might be some > conflicts between this proposal and FLIP-27. > > 1. > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > 2. > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-notifyOnMaster-for-notifyCheckpointComplete-td32769.html > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Fri, 6 Sep 2019 at 13:22, JingsongLee <lzljs3620...@aliyun.com.invalid> > wrote: > Hi everyone, thank you for your comments. Mail name was updated > and streaming-related concepts were added. > > We would like to start a discussion thread on "FLIP-63: Rework table > partition support"(Design doc: [1]), where we describe how to partition > support in flink and how to integrate to hive partition. > > This FLIP addresses: > - Introduce whole story about partition support. > - Introduce and discuss DDL of partition support. > - Introduce static and dynamic partition insert. > - Introduce partition pruning > - Introduce dynamic partition implementation > - Introduce FileFormatSink to deal with streaming exactly-once and > partition-related logic. > > Details can be seen in the design document. > Looking forward to your feedbacks. Thank you. > > [1] > https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing > > Best, > Jingsong Lee > >
