Hi Nicolas,
A quick check of the code suggests that AbstractWriter is a Json-serialized
description of the physical plan. It represents the information sent from the
planner to the execution engine, and is interpreted by the scan operator. That
is, it is the "physical plan."
The question is, how does the execution engine translate create the actual
writer based on the physical plan? The only good example seems to be for the
FileSystemPlugin. That particular storage plugin is complicated by the
additional layer of the format plugins.
There is a bit of magic here. Briefly, Drill uses a BatchCreator to create your
writer. It does so via some Java introspection magic. Drill looks for all
subclases of BatchCreator, the uses the type of the second argument to the
getBatch() method to find the correct class. This may mean that you need to
create one with MapRDBFormatPluginConfig as the type of the second argument.
The getBatch() method then creates the CloseableRecordBatch implementation.
This is a full Drill operator, meaning it must handle the Volcano iterator
protocol. Looks like you can perhaps use WriterRecordBatch as the writer
operator itself. (See EasyWriterBatchCreator and follow the code to understand
the plumbing.)
You create a RecordWriter to do the actual work. AFAIK, MapRDB supports JSON
data model (at least in some form). If this is the version you are working on,
the fastest development path might just be to copy the JsonRecordWriter, and
replace the writes to JSON with writes to MapRDB. At least this gives you a
place to start looking.
A more general solution would be to build the writer using some of the recent
additions to Drill such as the row set mechanisms for reading a record batch.
But, since copying the JSON approach provides a quick & dirty solution, perhaps
that is good enough for this particular use case.
In our book, we recommend building each step one-by-one and doing a quick test
to verify that each step works as you expect. If you create your BatchCreator,
but not the writer, things won't actually work, but you can set a breakpoint in
the getBatch() method to verify the Drill did find your class. And so on.
Thanks,
- Paul
On Thursday, May 30, 2019, 3:05:39 AM PDT, Nicolas A Perez
<[email protected]> wrote:
Can anyone give me an overview of how to implement AbstractRecordWriter?
What are the mechanics it follows, what should I do and so on? It will very
helpful.
Best Regards,
Nicolas A Perez
--
--------------------------------------------------------------------------------------------
Sent by Nicolas A Perez from my GMAIL account.
--------------------------------------------------------------------------------------------
