Steve Loughran created HADOOP-19101:
---------------------------------------
Summary: Vectored Read into off-heap buffer broken
Key: HADOOP-19101
URL: https://issues.apache.org/jira/browse/HADOOP-19101
Project: Hadoop Common
Issue Type: Sub-task
Components: fs, fs/azure
Affects Versions: 3.3.6, 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
{{VectoredReadUtils.readInDirectBuffer()}} always starts off reading at
position zero even when the range is at a different offset. As a result: you
can get incorrect information.
Thanks for this is straightforward: we pass in a FileRange and use its offset
as the starting position.
However, this does mean that all shipping releases 3.3.5-3.4.0 cannot safely
read vectorIO into direct buffers through HDFS, ABFS or Azure. Note that we
have never seen this in production because the parquet and ORC libraries both
read into on-heap storage.
Those libraries needs to be audited to make sure that they never attempt to
read into off-heap DirectBuffers. This is a bit trickier than you would think
because an allocator is passed in. For PARQUET-2171 we will
* only invoke the API on streams which explicitly declare their support for the
API (so fallback in parquet itself)
* not invoke when direct buffer allocation is in use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
