Hi Josh, responses follow: Using a BatchWriter implies that you'll need some sort of resource > management - both ensuring that the BatchWriter is close()'ed whenever a > compaction/procedure ends and handling rejected mutations. Have you put any > thought into how you would address these? >
Yes, some thoughts: - Accumulo guarantees that iterators at major compaction time are never killed until tablet compaction completes, barring tablet server failure or exception. - We can close the BatchWriter in the finalize() method of the RemoteWriteIterator. This is already done in my code for the RemoteSourceIterator, though I don't think it matters for a Scanner. Not ideal but maybe good enough. - Actually, if we do all computation for a tablet from the seek() method of the branching iterator, then we can create, use and close the BatchWriter in a single method invocation per tablet. This way, we may enclose all BatchWriter use in a try-finally loop, closing it in the finally. - This thread is relevant <https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%3c1398567904594-9433.p...@n5.nabble.com%3E>. Note that our scenario is different in that we're operating from a major compaction. - ACCUMULO-1280 <https://issues.apache.org/jira/browse/ACCUMULO-1280> proposes changing SKVI API to add a close() method, such that iterators know when they are about to be closed and why they are about to be closed. The iterators may then clean themselves up. - Rejected mutations sound like something the computation should handle. I've never encountered them personally. Any chance you could elaborate when we need to worry about them? If it's because the remote table has too heavy a load, then we have bigger problems. This should be rare if the remote table to write to is a new table. One solution I did not include this at first because its complexity may not be necessary: *checkpoint*-and-restart-style. When an iterator dies, Accumulo will recreate it, re-init() it and then seek() it to an appropriate range. This range should start *right after the last entry returned* by the iterator. If we structure the computation in such a way that it can restart in the middle, then we can employ the following strategy: - Suppose the initial computation range from seek() is [a,f]. - The stored procedure table runs as normal. Eventually its RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The RemoteSourceIterator passes c back to the BranchIterator which returns from its seek() and prepares a top entry with row c and a top value of "" (hasTop() is true). The computation continues when next() is called on the BranchIterator - Suppose Accumulo kills the iterator instead of calling next(). Accumulo re-init()s the BranchIterator which sets up the custom computation stack again. Accumulo then seek()s with range (c,f]. - The custom computation stack may now continue with the corresponding entries for range (c,f]. Notice that all we need is a correspondence between seek ranges and entries computed. I still believe the above scheme is a bit too much complexity. Suppose in the worst case we need to restart an entire tablet's worth of computation. Okay, we have some redundant computation but *preserve correctness*, because the result table's VersioningIterator will handle multiple (identical) entries, in most cases. using a wrench as a hammer -- iterators are great for performing some > passing computation, but not really for doing some arbitrary read/writes. > It gets back to how Accumulo/HBase comparisons where people try to compare > Iterators and Coprocessors. They can sometimes do the same thing, but > they're definitely different features. > Yes-- we are talking about different kinds of computation. That's why I distinguish between the types of iterators in the design doc. Traditional iterators are wrenches-- passing computation done at scan or compaction time, not too heavy, done in a sorted, streaming manner, limited to within a single table. They come in three flavors: one-time scan iterators, one-time compaction iterators, and "permanent table property" iterators set on the table at scan or compaction scope. Do correct me if I oversimplify or if I missed a large use case. The iterators I propose in the design doc are hammers-- heavy, one-time, multi-table computation, that may or may not return results in sorted order. That's why I use the phrase "stored procedure." On *HBase Coprocessors*-- I looked up what they are here <https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it seems our "stored procedure" idea bears many similarities to HBase Coprocessors of the "Endpoint" type (the "Observer" type has parallels with the Accumulo Fluo subproject). Maybe we can take some architecture lessons from HBase. Anyone have experience they would like to share? Regards, Dylan Hutchison On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser <josh.el...@gmail.com> wrote: > Thanks for taking the time to write this up, Dylan. > > I'm a little worried about the RemoteWriteIterator. Using a BatchWriter > implies that you'll need some sort of resource management - both ensuring > that the BatchWriter is close()'ed whenever a compaction/procedure ends and > handling rejected mutations. Have you put any thought into how you would > address these? > > I'm not familiar enough with the internals anymore, but I remember that I > had some pains trying to write to another table during compactions when I > was working on replication. I think as long as it's not triggered off of > the metadata table, it wouldn't have any deadlock issues. > > Architecturally, it's a little worrisome, because it feels a bit like > using a wrench as a hammer -- iterators are great for performing some > passing computation, but not really for doing some arbitrary read/writes. > It gets back to how Accumulo/HBase comparisons where people try to compare > Iterators and Coprocessors. They can sometimes do the same thing, but > they're definitely different features. > > Anyways, I need to stew on it some more and give it a few more reads. > Thanks again for sharing! > > Dylan Hutchison wrote: > >> Hello all, >> >> As promised >> <https://mail-archives.apache.org/mod_mbox/accumulo-user/ >> 201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu% >> 2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>, >> here is a design doc open for comments on implementing server-side >> computation in Accumulo. >> >> https://github.com/Accla/accumulo_stored_procedure_design >> >> Would love to hear your opinion, especially if the proposed design >> pattern matches one of /your use cases/. >> >> Regards, >> Dylan Hutchison >> >>