Another issue is that things like rename go far beyond the dl core library - the eb/dl client refers to backend streams by name, so any config change needs to take the entire stack into account. I agree this begs for better modeling of things like stream name and other configuration, but this work may end up being a fairly large project.
So the capabilities of metadata reconfiguration is kind of limited by the > features offered by the metadata store. For example, zookeeper doesn't > support rename operations. There are things we can do though here, for example copies/renames in transactions. Copying metadata to the same zk server actually might be better than simlink in some ways, although less flexible. Dan, do you have any ideas yet about how general purpose reconfiguration might work? On Wed, Apr 19, 2017 at 10:32 AM, Sijie Guo <guosi...@gmail.com> wrote: > Agreed that a general reconfiguration approach will be ideal for solving > the problems > > However currently DL is using a metadata store (e.g. ZooKeeper) for > storing and managing the metadata part. So the capabilities of metadata > reconfiguration is kind of limited by the features offered by the metadata > store. For example, zookeeper doesn't support rename operations. A symlink > approach here is try to address the metadata reconfiguration problem when > the underneath metadata store lacks such ability. Beside that, moving a > flat namespace to a hierarchical namespace in general is a long-duration > operation. So an approach should exist to allow both co-exist for a while > and allow rolling back and forward. > > - Sijie > > > On Wed, Apr 19, 2017 at 8:14 AM, Dan Benediktson <dbenedikt...@twitter.com > > wrote: > >> It seems to me that the fact that the namespace format needs to be >> changed is the specific, immediate problem, but the bigger problem is that >> there isn't a mechanism to perform an atomic reconfiguration step for log >> metadata. Namespace format change is one example of a reconfiguration (if >> we start viewing namespace format as a part of log metadata, which just >> happens to currently be unspecified since there is no format other than the >> flat one), but there are already other metadata reconfiguration changes >> that are desired: https://issues.apache.org/jira/browse/DL-74 and >> https://issues.apache.org/jira/browse/DL-77. I think we should try to >> solve the general reconfiguration problem correctly first, and then that >> should enable us to solve both this problem and those two more naturally. I >> expect that a correct reconfiguration solution would obviate the need for >> this sub-task: https://issues.apache.org/jira/browse/DL-201. >> >> On Tue, Apr 18, 2017 at 1:06 PM, Leigh Stewart <lstew...@twitter.com> >> wrote: >> >>> + Dan >>> >>> On Tue, Apr 18, 2017 at 1:04 PM, Sijie Guo <guosi...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I created a proposal for supporting symlinks in Dlog, in order to >>>> migrate a namespace from flat namespace to a hierarchical namespace. >>>> >>>> https://cwiki.apache.org/confluence/display/DL/DP-8+-+Symlinked+Log >>>> >>>> Please take a look and let me know your thoughts. >>>> >>>> - Sijie >>>> >>>> >>> >> >
