Hi all, Thanks for all the valuable feedback. If there‘s no more comments, I will start a vote for this PIP in the next 2 days.
Best, Shammon FY On Thu, Sep 21, 2023 at 5:19 PM Shammon FY <[email protected]> wrote: > The feature `Replace Main With Branch` is used in duplicate data > correction without modifying jobs. For example: > > 1. We can create branches with the same name for a series of paimon tables > 2. Re-submit all streaming jobs to read and write these branches for tables > 3. After the data in the branch is up to the main, we can stop all the > jobs which read and write main branch > 4. Replace main branch with the created branch, we don't need to do > anything with the jobs read and write the specified branch > > We cannot `Merge Branch To Main` here because the correct jobs will still > read and write the branches which will be completely independent of main. > > Best, > Shammon FY > > > > > On Thu, Sep 21, 2023 at 12:21 AM Jingsong Li <[email protected]> > wrote: > >> Can you explain more about "Replace Main With Branch"? >> >> Does this need to be implemented? >> >> Best, >> Jingsong >> >> On Tue, Sep 19, 2023 at 2:17 PM Shammon FY <[email protected]> wrote: >> > >> > Hi ConradJam, >> > >> > How to handle data conflicts between the main branch and branches is a >> > complex problem. At present, we would like to replace data in main with >> > branch directly. You can think that during merge and replace operations, >> > the data after the specified tag in the main branch will be deleted and >> > then the data after the tag in the branch will be used in the main. >> > >> > We can consider "merge" conflicting data in the future when we meet >> these >> > requirements. >> > >> > Best, >> > Shammon FY >> > >> > On Tue, Sep 19, 2023 at 10:50 AM ConradJam <[email protected]> wrote: >> > >> > > +1 This feature looks a bit like Git’s branch management.If this is >> really >> > > the case, how do we solve the data conflict when merging branches? Do >> we >> > > need the user to specify that a certain branch data shall prevail? >> > > >> > > Shammon FY <[email protected]> 于2023年9月18日周一 20:06写道： >> > > >> > > > Hi Jingsong, >> > > > >> > > > I have updated the PIP-9 to explain that the main `Snapshot`, >> `Schema` >> > > and >> > > > `Tag` will exist in the base directory by default, just as same as >> the >> > > > current directory structure. Thanks >> > > > >> > > > Best, >> > > > Shammon FY >> > > > >> > > > >> > > > On Fri, Sep 15, 2023 at 10:32 AM Shammon FY <[email protected]> >> wrote: >> > > > >> > > > > Hi Jingsong, >> > > > > >> > > > > Thanks for your suggestion, it sounds good to me. Currently I only >> > > > > mentioned it in the `Compatibility` section, I'll update the PIP >> to >> > > > explain >> > > > > this more clearly. >> > > > > >> > > > > Best, >> > > > > Shammon FY >> > > > > >> > > > > On Wed, Sep 13, 2023 at 12:26 PM Jingsong Li < >> [email protected]> >> > > > > wrote: >> > > > > >> > > > >> Thanks Shammon for the proposal! >> > > > >> >> > > > >> It looks very good! >> > > > >> >> > > > >> I don't get the main branch file. >> > > > >> >> > > > >> Can we keep the main branch as it is? Just put snapshot/ tag/ >> schema/ >> > > > >> in the table root directory. >> > > > >> >> > > > >> Best, >> > > > >> Jingsong >> > > > >> >> > > > >> On Tue, Sep 12, 2023 at 3:55 PM Shammon FY <[email protected]> >> wrote: >> > > > >> > >> > > > >> > Hi devs, >> > > > >> > >> > > > >> > I would like to start a discussion about PIP-9: Support Branch >> [1]. >> > > > >> Branch >> > > > >> > in Paimon will help us deal with data correction without >> copying all >> > > > >> data >> > > > >> > from original tables, and it can also enhance Tag for Paimon >> like >> > > > >> > traditional Hive partition tables, providing data correction >> > > > >> capabilities >> > > > >> > on the basis of Tag. >> > > > >> > >> > > > >> > Looking forward to your feedback, thanks! >> > > > >> > >> > > > >> > >> > > > >> > [1] >> > > > >> > >> > > > >> >> > > > >> > > >> https://cwiki.apache.org/confluence/display/PAIMON/PIP-9%3A+Support+Branch >> > > > >> > >> > > > >> > Best, >> > > > >> > Shammon FY >> > > > >> >> > > > > >> > > > >> > > >> > > >> > > -- >> > > Best >> > > >> > > ConradJam >> > > >> >
