Hi Josh, responses follow:
Using a BatchWriter implies that you'll need some sort of resource
> management - both ensuring that the BatchWriter is close()'ed whenever a
> compaction/procedure ends and handling rejected mutations. Have you put any
> thought into how you would address these?
>
Yes, some thoughts:
- Accumulo guarantees that iterators at major compaction time are never
killed until tablet compaction completes, barring tablet server failure or
exception.
- We can close the BatchWriter in the finalize() method of the
RemoteWriteIterator. This is already done in my code for the
RemoteSourceIterator, though I don't think it matters for a Scanner. Not
ideal but maybe good enough.
- Actually, if we do all computation for a tablet from the seek() method
of the branching iterator, then we can create, use and close the
BatchWriter in a single method invocation per tablet. This way, we may
enclose all BatchWriter use in a try-finally loop, closing it in the
finally.
- This thread is relevant
<https://mail-archives.apache.org/mod_mbox/accumulo-user/201404.mbox/%3c1398567904594-9433.p...@n5.nabble.com%3E>.
Note that our scenario is different in that we're operating from a major
compaction.
- ACCUMULO-1280
<https://issues.apache.org/jira/browse/ACCUMULO-1280> proposes
changing SKVI API to add a close() method, such that iterators know when
they are about to be closed and why they are about to be closed. The
iterators may then clean themselves up.
- Rejected mutations sound like something the computation should
handle. I've never encountered them personally. Any chance you could
elaborate when we need to worry about them? If it's because the remote
table has too heavy a load, then we have bigger problems. This should be
rare if the remote table to write to is a new table.
One solution I did not include this at first because its complexity may not
be necessary: *checkpoint*-and-restart-style. When an iterator dies,
Accumulo will recreate it, re-init() it and then seek() it to an
appropriate range. This range should start *right after the last entry
returned* by the iterator. If we structure the computation in such a way
that it can restart in the middle, then we can employ the following
strategy:
- Suppose the initial computation range from seek() is [a,f].
- The stored procedure table runs as normal. Eventually its
RemoteSourceIterator writes the corresponding entries for, say, [a,c]. The
RemoteSourceIterator passes c back to the BranchIterator which returns from
its seek() and prepares a top entry with row c and a top value of ""
(hasTop() is true). The computation continues when next() is called on the
BranchIterator
- Suppose Accumulo kills the iterator instead of calling next().
Accumulo re-init()s the BranchIterator which sets up the custom computation
stack again. Accumulo then seek()s with range (c,f].
- The custom computation stack may now continue with the corresponding
entries for range (c,f].
Notice that all we need is a correspondence between seek ranges and entries
computed. I still believe the above scheme is a bit too much complexity.
Suppose in the worst case we need to restart an entire tablet's worth of
computation. Okay, we have some redundant computation but *preserve
correctness*, because the result table's VersioningIterator will handle
multiple (identical) entries, in most cases.
using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
Yes-- we are talking about different kinds of computation. That's why I
distinguish between the types of iterators in the design doc.
Traditional iterators are wrenches-- passing computation done at scan or
compaction time, not too heavy, done in a sorted, streaming manner, limited
to within a single table. They come in three flavors: one-time scan
iterators, one-time compaction iterators, and "permanent table property"
iterators set on the table at scan or compaction scope. Do correct me if I
oversimplify or if I missed a large use case.
The iterators I propose in the design doc are hammers-- heavy, one-time,
multi-table computation, that may or may not return results in sorted
order. That's why I use the phrase "stored procedure."
On *HBase Coprocessors*-- I looked up what they are here
<https://blogs.apache.org/hbase/entry/coprocessor_introduction> and it
seems our "stored procedure" idea bears many similarities to HBase
Coprocessors of the "Endpoint" type (the "Observer" type has parallels with
the Accumulo Fluo subproject). Maybe we can take some architecture lessons
from HBase. Anyone have experience they would like to share?
Regards,
Dylan Hutchison
On Thu, Feb 26, 2015 at 3:43 PM, Josh Elser <josh.el...@gmail.com> wrote:
> Thanks for taking the time to write this up, Dylan.
>
> I'm a little worried about the RemoteWriteIterator. Using a BatchWriter
> implies that you'll need some sort of resource management - both ensuring
> that the BatchWriter is close()'ed whenever a compaction/procedure ends and
> handling rejected mutations. Have you put any thought into how you would
> address these?
>
> I'm not familiar enough with the internals anymore, but I remember that I
> had some pains trying to write to another table during compactions when I
> was working on replication. I think as long as it's not triggered off of
> the metadata table, it wouldn't have any deadlock issues.
>
> Architecturally, it's a little worrisome, because it feels a bit like
> using a wrench as a hammer -- iterators are great for performing some
> passing computation, but not really for doing some arbitrary read/writes.
> It gets back to how Accumulo/HBase comparisons where people try to compare
> Iterators and Coprocessors. They can sometimes do the same thing, but
> they're definitely different features.
>
> Anyways, I need to stew on it some more and give it a few more reads.
> Thanks again for sharing!
>
> Dylan Hutchison wrote:
>
>> Hello all,
>>
>> As promised
>> <https://mail-archives.apache.org/mod_mbox/accumulo-user/
>> 201502.mbox/%3CCAPx%3DJkakO3ice7vbH%2BeUo%2B6AP1JPebVbTDu%
>> 2Bg71KV8SvQ4J9WA%40mail.gmail.com%3E>,
>> here is a design doc open for comments on implementing server-side
>> computation in Accumulo.
>>
>> https://github.com/Accla/accumulo_stored_procedure_design
>>
>> Would love to hear your opinion, especially if the proposed design
>> pattern matches one of /your use cases/.
>>
>> Regards,
>> Dylan Hutchison
>>
>>
