Hi Vlad,
Would be great to get insight from the original authors. Here ismy two cents as
a late comer who made extensive use of the classes in question.
Many of your questions are at the implementation level. It is worth looking at
the question from two other perspectives: history and design.
Historically, Drill adopted Netty for networking, and wisely looked for ways of
using the same buffers for both network transfer and internal operations to
avoid copies. Some overview is in [1]. In this view, a Drill vector is a
network buffer. Network buffers use the ByteBuffer protocol to serialize binary
values. DrillBuf follows that model for the most part. Because a ByteBuffer is
a low-level abstraction over a buffer, each operation must perform bounds
checks to ensure safe operation.
DrillBuf provides the ability to present a "view" of a slice of a larger underlying
buffer. For example, when reading data from a spill file, all data for all internal vectors is read
into a single buffer. For a nullable VarChar, for example, the buffer contains the bit vectors, the
offset vectors and the data vectors. The value vectors point to DrillBufs which point to a slice of
the underlying buffers. It is this layout (there are at least three different layouts) that makes
our "record batch sizer" so complex: the size of memory used is NOT the sum of the
DrillBufs.
Drill is a columnar system. So, the team introduced a typed "vector"
abstraction.Value vectors provide an abstraction that sweeps away the ByteBuffer heritage
and replaces it with a strongly typed, accessor/mutator structure that works in terms of
Drill data types and record counts. Vectors also understand the relationship between bit
vectors and data vectors, between offset vectors and data vectors, and so on.
Your question implies a desire to think about the future direction. Two things
to say. First, vectors themselves do not provide sufficient abstraction for the
needs of operators. As a result, operators become very complex, we must
generate large amounts of boiler-plate code, and we fix the same bugs over and
over. These issues are discussed at length in [2]. This is the motivation for
the result set reader and loader.
The row set abstractions encapsulate not just knowledge of a vector, but of the
entire batch. As a result, these abstractions know the number of records, know
the vector and batch size targets, and track vectors as they fill. One key
result is that these abstractions ensure that data is read or written within
the bounds of each buffer, eliminating the need for bounds checks on every
access.
The other consideration is memory management. Drill has a very complex, but surprisingly
robust, memory management system. However, it is based on a "malloc" model of
memory with operators negotiating among themselves (via the OUT_OF_MEMORY iterator
status) about who needs memory and who should release it. [2] discusses the limitations
of this system. As a result, we've been moving to a budget-based system in which each
fragment and operator is given a budget based on total available memory, and operators
use spilling to stay within the budget.
Memory fragmentation is a classic problem in malloc-based systems which strive
to operate at high memory utilization rates and which do not include memory
compaction. Drill is such a system. So, if this issue ever prevents Drill from
achieving maximum performance, we can consider the classic system used by
databases to solve this problem: fixed-size memory blocks.
If we were to move to fixed-size buffers, we'd want the row set and vector
abstractions to remain unchanged. We'd only want to replace DrillBuf with a new
block-based abstraction, perhaps with chaining (a vector may consist of a chain
of, say, 1 MB blocks.) The buffer slicing mechanism would become unnecessary,
as would the existing malloc-based allocator. Instead, data would be read,
written and held in buffers allocated from and returned to a buffer pool.
We may or may not ever make such a change. But, by considering this
possibility, we readily see that DrillBuf should be an implementation detail of
the higher-level abstractions and that operators should only use those
higher-level abstractions because doing so isolates operators from the details
of memory layout. This argument applies even more so to the abstractions below
DrillBuf: UDLE, Netty ByteBuf, ledgers and so on.
Said another way, even with the current system, we should be free to improve
DrillBuf on down with no impact to operator code because vectors and the row
set abstractions should be the only clients of DrillBuf.
In short, by understanding the history of the code, and agreeing upon the right
design abstractions, we can then make informed decisions about how best to
improve our low-level abstractions, including DrillBuf.
Thanks,
- Paul
[1] http://drill.apache.org/docs/value-vectors/
[2] https://github.com/paul-rogers/drill/wiki/Batch-Handling-Upgrades
On Wednesday, April 4, 2018, 10:34:18 AM PDT, Vlad Rozov
<[email protected]> wrote:
I have several questions and concerns regarding DrillBuf usage, design
and implementation. There is a limited documentation available for the
subject (Java doc,
https://github.com/apache/drill/blob/master/exec/memory/base/src/main/java/org/apache/drill/exec/memory/README.md
and https://github.com/paul-rogers/drill/wiki/Memory-Management) and I
hope that a few members of the community may have more information.
What are the design goals behind DrillBuf? It seems like it is supposed
to be Drill access gate for direct byte buffers. How is it different
(for that goal) from UnsafeDirectLittleEndian? Both use
wrapper/delegation pattern, with DrillBuf delegating to
UnsafeDirectLittleEndian (not always) and UnsafeDirectLittleEndian
delegating to ByteBuf it wraps. Is it necessary to have both? Are there
any out of the box netty classes that already provide required
functionality? I guess that answer to the last question was "no" back
when DrillBuf and UnsafeDirectLittleEndian were introduced into Drill.
Is it still "no" for the latest netty release? What extra functionality
DrillBuf (and UnsafeDirectLittleEndian) provides on top of existing
netty classes?
As far as I can see from the source code, DrillBuf changes validation
(boundary and reference count checks) mechanism, making it optional
(compared to always enabled boundary checks inside netty) for get/set
Byte/Char/Short/Long/Float/Double. Is this a proper place to make
validation optional or the validation (or portion of the validation)
must be always on or off (there are different opinions, see
https://issues.apache.org/jira/browse/DRILL-6004,
https://issues.apache.org/jira/browse/DRILL-6202,
https://github.com/apache/drill/pull/1060 and
https://github.com/apache/drill/pull/1144)? Are there any performance
benchmark that justify or explain such behavior (if such benchmark does
not exist, are there any volunteer to do the benchmark)? My experience
is that the reference count check is significantly more expensive
compared to boundary checking and boundary checking adds tens of percent
to direct memory read when reading just a few bytes, so my vote is to
keep validation as optional with the ability to enable it for debug
purposes at run-time. What is the reason the same approach do not apply
to get/set Bytes and those methods are delegated to
UnsafeDirectLittleEndian that delegates it further?
Why DrillBuf reverses how AbstractByteBuf calls _get from get (and _set
from set), making _get to call get (_set to call set)? Why not to follow
a base class design patter?
Another question is usage of netty "io.netty.buffer" package for Drill
classes. Is this absolutely necessary? I don't think that netty
developers expect this and support semantic version compatibility for
package private classes/members.
Thank you,
Vlad
