Hi Jingsong, It's a good point about the detailed implementation of System Database, I'll update the PIP soon.
Best, Shammon FY On Wed, Jun 14, 2023 at 8:48 AM Shammon FY <[email protected]> wrote: > Hi Jingsong, > > Thanks for your comments. > > > We should document what is based on FLIP-314. > > I have updated the operations supported by FLIP-314 in the future work > > > Is the current Source interface sufficient for your functionality? > > In our design the current Source interface fulfills our requirements. As > described in PIP-5, `AlignedEnumerator` will send checkpoint events to > `AlignedSourceReader`, which will align the checkpoint and snapshot, and > then send split the next operator. More detailed information can be > provided by @liming > > > Can we currently achieve the ability to flush all data in a snapshot > before snapshot? > > Can you provide a more detailed description of this? Do you mean there may > be too much data for a snapshot if the source aligns the checkpoint and > snapshot and causes the snapshot to be too large to flush? > > > Best, > Shammon FY > > > > > > > On Mon, Jun 12, 2023 at 4:30 PM Jingsong Li <[email protected]> > wrote: > >> System Database looks very good~ But perhaps there are some design >> details here? What API should we use? Paimon Java API? And we should >> commit every operation? >> >> Best, >> Jingsong >> >> On Mon, Jun 12, 2023 at 4:27 PM Jingsong Li <[email protected]> >> wrote: >> > >> > Thanks Shammon, >> > >> > The overall design looks good to me! >> > >> > ## Plan For The Future >> > >> > We should document what is based on FLIP-314. >> > >> > ## AlignedEnumerator and AlignedSourceReader >> > >> > Is the current Source interface sufficient for your functionality? >> > >> > Can we currently achieve the ability to flush all data in a snapshot >> > before snapshot? >> > >> > Best, >> > Jingsong >> > >> > On Mon, Jun 5, 2023 at 7:57 PM Shammon FY <[email protected]> wrote: >> > > >> > > Hi Kelu, >> > > >> > > Thanks for your feedback. In the first stage, we do not want to >> introduce a >> > > server, but instead store information directly in the Paimon table >> when >> > > creating and running Flink jobs. A server will be considered when we >> > > encounter more requirements in the future and need a resident service >> > > management. >> > > >> > > Best, >> > > Shammon FY >> > > >> > > On Fri, Jun 2, 2023 at 5:55 PM Kelu Tao <[email protected]> wrote: >> > > >> > > > +1 >> > > > >> > > > cool job ~ >> > > > >> > > > For this PIP, do we need to introduce a new server for the >> information >> > > > serving? >> > > > >> > > > On 2023/05/31 02:28:21 Shammon FY wrote: >> > > > > Hi devs, >> > > > > >> > > > > We would like to start a discussion about PIP-5: Paimon Table And >> Data >> > > > > Lineage For Flink[1]. >> > > > > >> > > > > As a streaming lake, users can use Paimon integrated with Flink to >> > > > complete >> > > > > the entire ETL processing. In this process, users need to manage >> batch & >> > > > > streaming jobs and data streams, including batch & streaming data >> > > > > validation, job debug, and data revision. To support the above >> ability, >> > > > we >> > > > > introduce table and data lineage for Flink & Paimon. Users can >> > > > conveniently >> > > > > manage the entire ETL processing based on lineage information. >> > > > > >> > > > > Looking forward to hearing from you, thanks. >> > > > > >> > > > > >> > > > > [1] >> > > > > >> > > > >> https://cwiki.apache.org/confluence/display/PAIMON/PIP-5%3A+Paimon+Table+And+Data+Lineage+For+Flink >> > > > > >> > > > > >> > > > > Best, >> > > > > Shammon FY >> > > > > >> > > > >> >
