System Database looks very good~ But perhaps there are some design details here? What API should we use? Paimon Java API? And we should commit every operation?
Best, Jingsong On Mon, Jun 12, 2023 at 4:27 PM Jingsong Li <[email protected]> wrote: > > Thanks Shammon, > > The overall design looks good to me! > > ## Plan For The Future > > We should document what is based on FLIP-314. > > ## AlignedEnumerator and AlignedSourceReader > > Is the current Source interface sufficient for your functionality? > > Can we currently achieve the ability to flush all data in a snapshot > before snapshot? > > Best, > Jingsong > > On Mon, Jun 5, 2023 at 7:57 PM Shammon FY <[email protected]> wrote: > > > > Hi Kelu, > > > > Thanks for your feedback. In the first stage, we do not want to introduce a > > server, but instead store information directly in the Paimon table when > > creating and running Flink jobs. A server will be considered when we > > encounter more requirements in the future and need a resident service > > management. > > > > Best, > > Shammon FY > > > > On Fri, Jun 2, 2023 at 5:55 PM Kelu Tao <[email protected]> wrote: > > > > > +1 > > > > > > cool job ~ > > > > > > For this PIP, do we need to introduce a new server for the information > > > serving? > > > > > > On 2023/05/31 02:28:21 Shammon FY wrote: > > > > Hi devs, > > > > > > > > We would like to start a discussion about PIP-5: Paimon Table And Data > > > > Lineage For Flink[1]. > > > > > > > > As a streaming lake, users can use Paimon integrated with Flink to > > > complete > > > > the entire ETL processing. In this process, users need to manage batch & > > > > streaming jobs and data streams, including batch & streaming data > > > > validation, job debug, and data revision. To support the above ability, > > > we > > > > introduce table and data lineage for Flink & Paimon. Users can > > > conveniently > > > > manage the entire ETL processing based on lineage information. > > > > > > > > Looking forward to hearing from you, thanks. > > > > > > > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-5%3A+Paimon+Table+And+Data+Lineage+For+Flink > > > > > > > > > > > > Best, > > > > Shammon FY > > > > > > >
