----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29723/ -----------------------------------------------------------
Review request for drill and Jacques Nadeau. Repository: drill-git Description ------- When reaching the end of a buffer, the current way drill handles this is for setSafe() to return false, and then whatever operator is doing the write will send the batch, and the redo the last record. This creates a lot of difficulty, because it sometimes requires being able to "rewind" the input stream to replay the last record. The proposal is to move the handling of buffer sizing and allocation into the value vectors themselves, making it transparent to the user of the value vector. The operators will now no longer have to worry about the possibility that writing into a vector may fail due to lack of space. Diffs ----- contrib/storage-hbase/src/main/java/org/apache/drill/exec/store/hbase/HBaseRecordReader.java 33bf376 contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java 82e038c contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveRecordReader.java 625a7b2 contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveTextRecordReader.java 5406048 contrib/storage-mongo/src/main/java/org/apache/drill/exec/store/mongo/MongoRecordReader.java 79abe60 exec/java-exec/src/main/codegen/templates/ComplexReaders.java 027f61d exec/java-exec/src/main/codegen/templates/ComplexWriters.java 5ba1c64 exec/java-exec/src/main/codegen/templates/FixedValueVectors.java 58e6ccc exec/java-exec/src/main/codegen/templates/MapWriters.java b8bd73e exec/java-exec/src/main/codegen/templates/NullableValueVectors.java b222024 exec/java-exec/src/main/codegen/templates/RepeatedValueVectors.java d261050 exec/java-exec/src/main/codegen/templates/TypeHelper.java c83c301 exec/java-exec/src/main/codegen/templates/VariableLengthVectors.java b8ffe5d exec/java-exec/src/main/java/org/apache/drill/exec/expr/EvaluationVisitor.java 5cf4a35 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillAggFuncHolder.java 6e0b282 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillComplexWriterFuncHolder.java df56174 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggBatch.java 113e883 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/HashAggTemplate.java d7cf904 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggBatch.java ef85a36 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/aggregate/StreamingAggTemplate.java 556b260 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/ChainedHashTable.java 0502f7e exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTable.java e8ccd62 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java 5b56f8e exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java e82dd29 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinBatch.java 5deb67f exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinProbe.java 7599f9e exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/HashJoinProbeTemplate.java c58f9a3 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/JoinTemplate.java c1dffc1 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/join/MergeJoinBatch.java d0f9d7d exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingReceiverGeneratorBase.java 2885c52 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingReceiverTemplate.java c29ef75 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java acbb755 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionProjectorTemplate.java f5068b4 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 20f6195 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectorTemplate.java 49ad390 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/CopierTemplate2.java 5cc308a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/svremover/CopierTemplate4.java c42332d exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameRecordBatch.java a3e7940 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/window/StreamingWindowFrameTemplate.java b4e3fed exec/java-exec/src/main/java/org/apache/drill/exec/record/TransferPair.java 9645be9 exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/FixedWidthRepeatedReader.java 0c4437a exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/NullableBitReader.java b9b808b exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthColumnReaders.java 83f9bde exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/VarLengthValuesColumn.java 7c4b33b exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoRecordReader.java 241fa95 exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/PojoWriter.java 0ffa55c exec/java-exec/src/main/java/org/apache/drill/exec/store/pojo/Writers.java fee011a exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b exec/java-exec/src/main/java/org/apache/drill/exec/vector/AllocationHelper.java 51726a3 exec/java-exec/src/main/java/org/apache/drill/exec/vector/BitVector.java d2523c5 exec/java-exec/src/main/java/org/apache/drill/exec/vector/CopyUtil.java 1ed7f37 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedFixedWidthVector.java aadc563 exec/java-exec/src/main/java/org/apache/drill/exec/vector/RepeatedMutator.java ad2ba1b exec/java-exec/src/main/java/org/apache/drill/exec/vector/VariableWidthVector.java ff3ee63 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/MapVector.java cc3d24c exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedListVector.java 362d806 exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/RepeatedMapVector.java e140c8b exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedListReaderImpl.java c60730c exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/RepeatedMapReaderImpl.java 15f8a2b exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/impl/SingleMapReaderImpl.java 3ec66ff exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/window/TestWindowFrame.java 7c04477 exec/java-exec/src/test/java/org/apache/drill/exec/vector/TestAdaptiveAllocation.java ebc4df7 Diff: https://reviews.apache.org/r/29723/diff/ Testing ------- Thanks, Steven Phillips
