[ https://issues.apache.org/jira/browse/DRILL-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957987#comment-15957987 ]
Paul Rogers commented on DRILL-5416: ------------------------------------ The original design for serialization is that each vector serializes to a buffer. This is simple for single-buffer vectors (required int, say). For composite vectors (nullable int, Varchar), the serialization process results in all buffers being combined into a single write buffer, and the corresponding read buffer being sliced into the individual composite vectors. For a Varchar: {code} Data: [FredBarneyWilma_] Offsets: [01041015] Output buffer: [01041015FredBarneyWilma_] Input buffer: [01041015FredBarneyWilma_] New Offsets: [^^^^^^^^] New Data [^^^^^^^^^^^^^^^] {code} > Vectors read from disk report incorrect memory sizes > ---------------------------------------------------- > > Key: DRILL-5416 > URL: https://issues.apache.org/jira/browse/DRILL-5416 > Project: Apache Drill > Issue Type: Bug > Affects Versions: 1.8.0 > Reporter: Paul Rogers > Assignee: Paul Rogers > Priority: Minor > Fix For: 1.11.0 > > > The external sort and revised hash agg operators spill to disk using a vector > serialization mechanism. This mechanism serializes each vector as a (length, > bytes) pair. > Before spilling, if we check the memory used for a vector (using the new > {{RecordBatchSizer}} class), we learn of the actual memory consumed by the > vector, including any unused space in the vector. > If we spill the vector, then reread it, the reported storage size is wrong. > On reading, the code allocates a buffer, based on the saved length, rounded > up to the next power of two. Then, when building the vector, we "slice" the > read buffer, setting the memory size to the data size. > For example, suppose we save 20 1-byte fields. The size on disk is 20. The > read buffer is rounded to 32 bytes (the size of the original, pre-spill > buffer.) We read the 20 bytes and create a vector. Creating the vector > reports the memory size as 20, "hiding" the extra, unused 12 bytes. > As a result, when computing memory sizes, we receive incorrect numbers. > Working with false numbers means that the code cannot safely operate within a > memory budget, causing the user to receive an unexpected OOM error. > As it turns out, the code path that does the slicing is used only for reads > from disk. This ticket asks to remove the slicing step: just use the > allocated buffer directly so that the after-read vector reports the correct > memory usage; same as the before-spill vector. -- This message was sent by Atlassian JIRA (v6.3.15#6346)