zratkai commented on code in PR #1431:
URL: https://github.com/apache/orc/pull/1431#discussion_r1129103949
##########
java/core/src/java/org/apache/orc/impl/TreeReaderFactory.java:
##########
@@ -2292,10 +2292,11 @@ private void readDictionaryStream(InStream in) throws
IOException {
int dictionaryBufferSize =
dictionaryOffsets[dictionaryOffsets.length - 1];
dictionaryBuffer = new byte[dictionaryBufferSize];
int pos = 0;
- int chunkSize = in.available();
- byte[] chunkBytes = new byte[chunkSize];
+ //check if dictionary size is smaller than available stream size to
avoid ArrayIndexOutOfBoundsException
+ int readSize = Math.min(in.available(), dictionaryBufferSize);
+ byte[] chunkBytes = new byte[readSize];
Review Comment:
In Hive the ORC's Reader classes (e.g. StringTreeReader) have sublclasses
and ORC file is read with LLAP. LLAP uses a default 4kB blocks to read, and
this causes that the in.available() can be bigger than dictionary size. (If
dictionary is smaller than 4kB.) This fix will not cause any issue if ORC file
is read without LLAP, only with ORC classes, since in System.arraycopy() the
maximum amount which can be copied can not be bigger than destination or source
size. The previous reading mode created a bigger destination array in Hive than
necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
