Added to PIP-29 https://cwiki.apache.org/confluence/display/PAIMON/PIP-29%3A+Introduce+Table+Multi-Location++Management
On Thu, Dec 19, 2024 at 9:28 AM wj wang <[email protected]> wrote: > > Very thanks for Houliang Qi prepare this doc and PR. > After reading the context, I have some questions. > 1、Could you add the design implementation details of 'Support file > migration between different storage locations' in the doc? > 2、When and how execute the file migration between different storage locations? > 3、What business scenarios need to execute the file migration between > different storage locations? > 4、Can ‘data-file.external-path’ be removed after configuring a table? > For example, at the beginning, the data was written in HDFS and then > written to OSS. After a period of time, if I want to write new data > back to HDFS, can I just remove the ‘data-file.external-path’ > configuration? > > Best, > Weijun Wang > > On Fri, Dec 13, 2024 at 2:56 PM Jingsong Li <[email protected]> wrote: > > > > Hi Houliang, > > > > Thanks for starting this discussion. > > > > Maybe we can just introduce an option: `data-file.external-path`? I > > don't the usage of multi.locations. > > > > In DataFileMeta, yes, we need to add another field: external_path. > > > > About FileIO, I think you can implement an own hybrid FileIO created > > by catalog options. > > > > I think the general idea is fine, but we may need a POC code to > > observe its complexity. > > > > Best, > > Jingsong > > > > On Wed, Dec 11, 2024 at 7:15 PM Houliang Qi <[email protected]> wrote: > > > > > > Hi Paimon devs, > > > > > > > > > I’d like to initiate a discussion: Introduce Table Multi-Location > > > Management[1], currently, the table's data can only be persisted in > > > catalog's warehouse path, which can not be modified once it created, > > > However, users may wish to store data from a table on different storage > > > devices, or even store data from different partitions of a table on > > > different storage devices based on their level of activity. So, the topic > > > of this proposal is how to enable paimon to support multi-location > > > management for a single table. > > > > > > > > > Any opinions are welcome, looking forward to your feedback, thanks. > > > > > > > > > [1] > > > https://docs.google.com/document/d/1NhmOyxM16QmY_rVb3KJtCKRrU_nogIJv532U59qW7EI/edit?tab=t.0#heading=h.xlrl29nlxwpo > > > > > > > > >
