Agreed that a general reconfiguration approach will be ideal for solving the problems
However currently DL is using a metadata store (e.g. ZooKeeper) for storing and managing the metadata part. So the capabilities of metadata reconfiguration is kind of limited by the features offered by the metadata store. For example, zookeeper doesn't support rename operations. A symlink approach here is try to address the metadata reconfiguration problem when the underneath metadata store lacks such ability. Beside that, moving a flat namespace to a hierarchical namespace in general is a long-duration operation. So an approach should exist to allow both co-exist for a while and allow rolling back and forward. - Sijie On Wed, Apr 19, 2017 at 8:14 AM, Dan Benediktson <dbenedikt...@twitter.com> wrote: > It seems to me that the fact that the namespace format needs to be changed > is the specific, immediate problem, but the bigger problem is that there > isn't a mechanism to perform an atomic reconfiguration step for log > metadata. Namespace format change is one example of a reconfiguration (if > we start viewing namespace format as a part of log metadata, which just > happens to currently be unspecified since there is no format other than the > flat one), but there are already other metadata reconfiguration changes > that are desired: https://issues.apache.org/jira/browse/DL-74 and > https://issues.apache.org/jira/browse/DL-77. I think we should try to > solve the general reconfiguration problem correctly first, and then that > should enable us to solve both this problem and those two more naturally. I > expect that a correct reconfiguration solution would obviate the need for > this sub-task: https://issues.apache.org/jira/browse/DL-201. > > On Tue, Apr 18, 2017 at 1:06 PM, Leigh Stewart <lstew...@twitter.com> > wrote: > >> + Dan >> >> On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo <guosi...@gmail.com> wrote: >> >>> Hi all, >>> >>> I created a proposal for supporting symlinks in Dlog, in order to >>> migrate a namespace from flat namespace to a hierarchical namespace. >>> >>> https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log >>> >>> Please take a look and let me know your thoughts. >>> >>> - Sijie >>> >>> >> >
