Thanks Shammon, The overall design looks good to me!
## Plan For The Future We should document what is based on FLIP-314. ## AlignedEnumerator and AlignedSourceReader Is the current Source interface sufficient for your functionality? Can we currently achieve the ability to flush all data in a snapshot before snapshot? Best, Jingsong On Mon, Jun 5, 2023 at 7:57 PM Shammon FY <[email protected]> wrote: > > Hi Kelu, > > Thanks for your feedback. In the first stage, we do not want to introduce a > server, but instead store information directly in the Paimon table when > creating and running Flink jobs. A server will be considered when we > encounter more requirements in the future and need a resident service > management. > > Best, > Shammon FY > > On Fri, Jun 2, 2023 at 5:55 PM Kelu Tao <[email protected]> wrote: > > > +1 > > > > cool job ~ > > > > For this PIP, do we need to introduce a new server for the information > > serving? > > > > On 2023/05/31 02:28:21 Shammon FY wrote: > > > Hi devs, > > > > > > We would like to start a discussion about PIP-5: Paimon Table And Data > > > Lineage For Flink[1]. > > > > > > As a streaming lake, users can use Paimon integrated with Flink to > > complete > > > the entire ETL processing. In this process, users need to manage batch & > > > streaming jobs and data streams, including batch & streaming data > > > validation, job debug, and data revision. To support the above ability, > > we > > > introduce table and data lineage for Flink & Paimon. Users can > > conveniently > > > manage the entire ETL processing based on lineage information. > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-5%3A+Paimon+Table+And+Data+Lineage+For+Flink > > > > > > > > > Best, > > > Shammon FY > > > > >
