Thanks Salim. Can you add this to the JIRA/design doc. Also, I would venture to suggest that the section on predicate pushdown can be made clearer. Also, Since you're proposing the average batch size approach with overflow handling, some detail on the proposed changes to the framework would be useful in the design doc. (Perhaps pseudo code and affected classes.) Essentially some guarantees provided by the framework will change and this may affect (or not) the existing usage. These should be enumerated in the design doc.
On Fri, Feb 9, 2018 at 11:52 PM, salim achouche <sachouc...@gmail.com> wrote: > Thank you Parth for providing feedback; please find my answers below: > > I have created Apache JIRA DRILL-6147 > <https://issues.apache.org/jira/browse/DRILL-6147?filter=-1> for tracking > this improvement. > > > 2) Not sure where you were going with the predicate pushdown section and > how it pertains to your proposed batch sizing. > > Predicate push down was part of the Design Considerations section; the > intent is that the design should be able to handle future use-cases such as > push down. Notice how the Page based Statistical Approach will not work > well with predicate push down as one single batch can span many pages per > column. > > > 3) Assuming that you go with the average batch size calculation approach, > are you proposing to have a Parquet scan specific overflow implementation? > Or are you planning to leverage the ResultSet loader mechanism? If you plan > to use the latter, it will need to be enhanced to handle a bulk chunk as > opposed to a single value at a time. If not using the ResultSet loader > mechanism, why not (you would be reinventing the wheel) ? > > Padma Penumarthy and I are currently working on the batch sizing > functionality and selected few TPCH queries to show case end-to-end use > cases. Immediately after this task, I'll be working on enhancing the new > framework to support columnar processing and as such retrofit DRILL-6147 > implementation as part of the new framework. So essentially we want to make > progress in both fronts so that a) OOM conditions are minimized as soon as > possible and b) the new Reader framework is applied to all readers and > operators is rolled out in the next few releases. > > > Also note that memory allocations by Netty greater than the 16MB chunk > size > are returned to the OS when the memory is free'd. Both this document and > the original document on memory fragmentation state incorrectly that such > memory is not released back to the OS. A quick thought experiment - where > does this memory go if it is not released back to the OS? > > I have the same understanding as you: > - I think Paul meant that 16 MB blocks are not released to the OS (cached > within Netty) > - Many memory allocators exhibit the same behavior as the release mechanism > is slow (heuristics used to decide when to release so to balance between > performance and resource usage) > - Basically, if Drill holds a large count of 16 MB blocks, than a 32 MB, 64 > MB , etc memory allocation might fail since > * none of the Netty allocated blocks can satisfy the new request > * a new OS allocation will take Drill beyond the maximum direct memory > > > On Fri, Feb 9, 2018 at 4:08 AM, Parth Chandra <par...@apache.org> wrote: > > > Is there a JIRA for this? Would be useful to capture the comments in the > > JIRA. Note that the document itself is not comment-able as it is shared > > with view-only permissions. > > > > Some thoughts in no particular order- > > 1) The Page based statistical approach is likely to run into trouble with > > the encoding used for Parquet fields especially RLE which drastically > > changes the size of the field. So pageSize/numValues is going to be > wildly > > inaccurate with RLE. > > 2) Not sure where you were going with the predicate pushdown section and > > how it pertains to your proposed batch sizing. > > 3) Assuming that you go with the average batch size calculation approach, > > are you proposing to have a Parquet scan specific overflow > implementation? > > Or are you planning to leverage the ResultSet loader mechanism? If you > plan > > to use the latter, it will need to be enhanced to handle a bulk chunk as > > opposed to a single value at a time. If not using the ResultSet loader > > mechanism, why not (you would be reinventing the wheel) ? > > 4) Parquet page level stats are probably not reliable. You can assume > page > > size (compressed/uncompressed) and value count are accurate, but nothing > > else. > > > > Also note that memory allocations by Netty greater than the 16MB chunk > size > > are returned to the OS when the memory is free'd. Both this document and > > the original document on memory fragmentation state incorrectly that such > > memory is not released back to the OS. A quick thought experiment - where > > does this memory go if it is not released back to the OS? > > > > > > > > On Fri, Feb 9, 2018 at 7:12 AM, salim achouche <sachouc...@gmail.com> > > wrote: > > > > > The following document > > > <https://docs.google.com/document/d/1A6zFkjxnC_- > > > 9RwG4h0sI81KI5ZEvJ7HzgClCUFpB5WE/edit?ts=5a793606#> > > > describes > > > a proposal for enforcing batch sizing constraints (count and memory) > > within > > > the Parquet Reader (Flat Schema). Please feel free to take a look and > > > provide feedback. > > > > > > Thanks! > > > > > > Regards, > > > Salim > > > > > >
