[
https://issues.apache.org/jira/browse/DRILL-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450079#comment-16450079
]
ASF GitHub Bot commented on DRILL-6094:
---------------------------------------
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1232#discussion_r183350828
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/ParquetFixedWidthDictionaryReaders.java
---
@@ -248,27 +227,61 @@ protected void readField(long
recordsToReadInThisPass) {
}
}
- static class DictionaryDecimal18Reader extends
FixedByteAlignedReader<Decimal18Vector> {
- DictionaryDecimal18Reader(ParquetRecordReader parentReader, int
allocateSize, ColumnDescriptor descriptor,
- ColumnChunkMetaData columnChunkMetaData,
boolean fixedLength, Decimal18Vector v,
- SchemaElement schemaElement) throws
ExecutionSetupException {
+ static class DictionaryVarDecimalReader extends
FixedByteAlignedReader<VarDecimalVector> {
+
+ DictionaryVarDecimalReader(ParquetRecordReader parentReader, int
allocateSize, ColumnDescriptor descriptor,
+ ColumnChunkMetaData columnChunkMetaData, boolean fixedLength,
VarDecimalVector v,
+ SchemaElement schemaElement) throws ExecutionSetupException {
super(parentReader, allocateSize, descriptor, columnChunkMetaData,
fixedLength, v, schemaElement);
}
// this method is called by its superclass during a read loop
@Override
protected void readField(long recordsToReadInThisPass) {
+ recordsReadInThisIteration =
+ Math.min(pageReader.currentPageCount - pageReader.valuesRead,
+ recordsToReadInThisPass - valuesReadInCurrentPass);
+
+ switch (columnDescriptor.getType()) {
+ case INT32:
+ if (usingDictionary) {
+ for (int i = 0; i < recordsReadInThisIteration; i++) {
+ byte[] bytes =
Ints.toByteArray(pageReader.dictionaryValueReader.readInteger());
+ setValueBytes(i, bytes);
+ }
+ setWriteIndex();
+ } else {
+ super.readField(recordsToReadInThisPass);
+ }
+ break;
+ case INT64:
+ if (usingDictionary) {
+ for (int i = 0; i < recordsReadInThisIteration; i++) {
+ byte[] bytes =
Longs.toByteArray(pageReader.dictionaryValueReader.readLong());
+ setValueBytes(i, bytes);
+ }
+ setWriteIndex();
+ } else {
+ super.readField(recordsToReadInThisPass);
+ }
+ break;
+ }
+ }
- recordsReadInThisIteration = Math.min(pageReader.currentPageCount
- - pageReader.valuesRead, recordsToReadInThisPass -
valuesReadInCurrentPass);
+ /**
+ * Set the write Index. The next page that gets read might be a page
that does not use dictionary encoding
+ * and we will go into the else condition below. The readField method
of the parent class requires the
+ * writer index to be set correctly.
+ */
+ private void setWriteIndex() {
+ readLengthInBits = recordsReadInThisIteration * dataTypeLengthInBits;
+ readLength = (int) Math.ceil(readLengthInBits / 8.0);
--- End diff --
Do you know how this magic number was chosen?
> Decimal data type enhancements
> ------------------------------
>
> Key: DRILL-6094
> URL: https://issues.apache.org/jira/browse/DRILL-6094
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.12.0
> Reporter: Volodymyr Vysotskyi
> Assignee: Volodymyr Vysotskyi
> Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> Currently, Decimal types are disabled by default since existing Decimal
> implementation has a lot of flaws and performance problems. The goal of this
> Jira to describe majority of them and possible ways of improving existing
> implementation to be able to enable Decimal data types by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
