On Wed, Nov 2, 2011 at 10:12 AM, Selcuk AYA <ayasel...@gmail.com> wrote:
> Hi, > a few points and questions: > *I am planning to use a common ID for all partitions. I checked Hbase > partition and it uses UUID as the ID. Is it ok to use this for all > partitions? > > This great and something we wanted to do for a very long time as Emmanuel stated. > *It seems that I need to move txn and log package implementations to > core-shared because of the recent reorg. Coresession and nexus > implementation is moved there and interceptors might need these > packages. > > +1 > * When preparing txn log edits, I will need to log Modification and > Attribute objects and keep them in memory for a while. Is it necessary > to clone these objects while doing the logging at partition nexus > level? Don't think so. > In other words, will they be modified after reaching nexus > level. I am thinking no but I wanted to check because I looked at > changelog implementation and it does cloning. > > Thanks, Alex > > On Tue, Nov 1, 2011 at 4:09 PM, Emmanuel Lecharny <elecha...@gmail.com> > wrote: > > Hi ! > > > > comments inline... > > > > On 11/1/11 12:19 PM, Selcuk AYA wrote: > >> > >> thanks for the feedback! Please see inline: > > > > <snip/> > >> > >> > >> I guess the xdbm-partition Maven module is now going to be pretty damn > >> thin > >> or non-existent. Was this module destroyed and if not what actually > >> remains? > >> I think some util classes for tree based implementations remain there. > >> Also there some avl related classes. > > > > We can review this project and eventually move the remaining classes > > elsewhere. > > > >> > >>>> The todo is the following: > >>>> *add changes to keep track of dn space changes. > >>>> *test txn manager services > >>>> *move the modification code path in AbstractBTreePartition in xdbm to > >>>> high up to core. Probably modifications should be done in > >>>> partitionNexus and partition nexus should handle preparing txn log > >>>> edits and wal them. The overall flow for modifications will be: > >>>> -DefaultDirectoryService: > >>>> -begintxn > >>>> -execute interceptor chain > >>>> - handle txn abort, or conflict. > >>>> > >>> Moving this into the PartitionNexus might not be a good idea but no > >>> problem > >>> for now we can move it later. Let me explain why: > >>> Eventually we're going to enable a root Partition with Partition > nesting > >>> and > >>> so when this happens the PartitionNexus will just be another nestable > >>> Partition since these will have to handle routing based on DN to other > >>> partitions residing/nested under it. > >>> I see two possible locations for this functionality: > >>> (1) Let the InterceptorChain itself handle this since it can demarcate > >>> the > >>> start and end of calls into the chain with Txn begin and abort/commit > >>> calls. > >>> It does this by making calls against the TxnManager which I guess is > the > >>> entire facade for the transaction subsystem. > >>> ---OR--- > >>> (2) Handle Txn demarcation within the CoreSession. However this might > not > >>> be > >>> optimal due to the need to handle additional logic which might be > >>> required > >>> for handling chain re-entry concerns. > >>> NOTE: I've not actually looked at the code after these major moves so > my > >>> advice might not be very dependable. I will try to setup my environment > >>> to > >>> get a better idea of these matters. > >>> However for the time being do whatever actually makes this thing work. > >>> Let's > >>> follow an agile methodology. This thing is big. So let's get it working > >>> with > >>> solid test coverage then we can actually look at shuffling things > around > >>> to > >>> optimal positions. Not saying what you've chosen is wrong ... it might > >>> just > >>> present the need for some additional refactoring when other features > >>> might > >>> need to be introduced. > >>> These are some of the biggest changes to the architecture to have taken > >>> place in years and you're doing a great job. > >> > >> You are right for txn demarcation. I wrote it wrong in the email. > >> Demarcation has to be done either at defaultcoresession or > >> defaultoperationmanager. > > > > OperationManager is probably the right place. > > > > For inner operations, I'm almost 100% sure they aren't modifiying > anything > > (the only modification done as a inner operation has been removed a while > > ago : it was dealing with the addition of the ModifyTimeStamp and > > ModifiersName attribute, and it' snow done in the main operation). So we > are > > safe if they are done in the current external transaction. > > > >> > >> as for where to handle the change logic, as you mentioned we need a > >> place where all interceptor chain routes end up for modification and > >> ideally we should handle modification logic above partitions using > >> master table and index interfaces so that we have a common place to > >> prepare and apply txn log edits. PartitionNexus seems to fit this > >> requirement for now. If we add another layer above PartitionNexus > >> which can get master and index table from below layers and work with > >> them, we should be able to move the change logic up there. > >> > >>>> *move xdbm-search to core as well. Making search transactional will > >>>> mostly be mostly mechanica after this point I think(hope). It should > >>>> just use the wrappers the txn manager provides for index, master and > >>>> cursors it gets from the partitions. > >>>> > >>>> *handle caches various interceptors keep. I am thinking of handling > >>>> this with a common read-write lock. > >>>> > >>> This was the latest issue for which I see some more threads. Will look > at > >>> that as well. > >> > >> This is the issue I talked you about briefly. These are mostly admin > >> caches that change infrequently and that are read mostly. The > >> difficulty with them is that they are not always entry caches. They > >> might map Dns to some logical property of the entry. A simple and good > >> example is notaliascache (whether we really need this is open to > >> debate but we have it now). > > > > This is a very interesting point. The notAliasCache is definitively > > questionable. The question though is to find a better way to deal with > it. > > An option would be to not support alias (yes, I know, pretty drastic, > but if > > it allows us to speed up the delivery of a production ready server, I > think > > this would be a good tradeoff. Also note that once we have a solid base, > we > > can reintroduce alias handling in a future version). > > > >> This maps a Dn to whether the entry is an > >> alias. If we followed the normal way of merging what is read from > >> partitions with the txn log, we would have to a way of merging what is > >> read from this notaliascache with the log. This is not very difficult > >> but as we have quite a number of these caches, having a separate merge > >> and update logic for each of them is a pain and error prone. > >> > >> Instead, what I thought was to use a single system wide read-write > >> lock for these caches. Since they are read mostly, a txn might update > >> its lock to exclusive when it needs to change one of these caches and > >> might release it when it actually commits/aborts. > > > > We may start a new thread to discuss further those cache issues. We have > > many caches, for may different thinsg, and many different approaches to > deal > > with the (MRU, EhCache, etc...). It's probably a good timing for > clarifying > > this item... > > > > -- > > Regards, > > Cordialement, > > Emmanuel Lécharny > > www.iktek.com > > > > > -- Best Regards, -- Alex
