Indhumathi27 commented on a change in pull request #3584: [CARBONDATA-3718]
Support SegmentLevel MinMax for better Pruning and less driver memory usage for
cache
URL: https://github.com/apache/carbondata/pull/3584#discussion_r389656759
##########
File path:
core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMapFactory.java
##########
@@ -186,6 +217,110 @@ private void
getTableBlockUniqueIdentifierWrappers(List<PartitionSpec> partition
}
}
+ /**
+ * Using blockLevel minmax values, identify if segment has to be added for
further pruning and to
+ * load segment index info to cache
+ * @param segment to be identified if needed for loading block datamaps
+ * @param segmentMinMaxList list of block level min max values
+ * @param filter filter expression
+ * @param identifiers tableBlockIndexUniqueIdentifiers
+ * @param tableBlockIndexUniqueIdentifierWrappers to add
tableBlockIndexUniqueIdentifiers
+ */
+ private void getTableBlockIndexUniqueIdentifierUsingSegmentMinMax(Segment
segment,
+ List<SegmentMinMax> segmentMinMaxList, DataMapFilter filter,
+ Set<TableBlockIndexUniqueIdentifier> identifiers,
+ List<TableBlockIndexUniqueIdentifierWrapper>
tableBlockIndexUniqueIdentifierWrappers) {
+ boolean isScanRequired = false;
+ for (SegmentMinMax segmentMinMax : segmentMinMaxList) {
+ Map<String, SegmentBlockMinMaxInfo> segmentBlockMinMaxInfoMap =
+ segmentMinMax.getSegmentBlockMinMaxInfo();
+ int length = segmentBlockMinMaxInfoMap.size();
+ // Add columnSchemas based on the columns present in segment
+ List<ColumnSchema> columnSchemas = new ArrayList<>();
+ byte[][] min = new byte[length][];
+ byte[][] max = new byte[length][];
+ boolean[] minMaxFlag = new boolean[length];
+ int i = 0;
+
+ // get current columnSchema list for the table
+ Map<String, ColumnSchema> tableColumnSchemas =
+
this.getCarbonTable().getTableInfo().getFactTable().getListOfColumns().stream()
+ .collect(Collectors.toMap(ColumnSchema::getColumnUniqueId,
ColumnSchema::clone));
Review comment:
it will modify the current carbonTable columnSchemas list, if we use the
same object list
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
