tarun11Mavani commented on code in PR #19093:
URL: https://github.com/apache/pinot/pull/19093#discussion_r3975873563
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/openstruct/OpenStructColumnSplitter.java:
##########
@@ -575,14 +572,20 @@ private void writeSparseJsonColumn(List<String>
sparseKeys)
try {
String json = JsonUtils.objectToString(sparseEntries);
jsonPerDoc[docId] = json;
- maxLen = Math.max(maxLen,
json.getBytes(StandardCharsets.UTF_8).length);
- nonNullCount++;
+ maxLen = Math.max(maxLen, Utf8.encodedLength(json));
} catch (IOException e) {
throw new RuntimeException("Failed to serialize sparse entries for
docId " + docId, e);
}
}
}
+ // Absent docs store "" in the raw forward index (see loop below) and are
flagged in the null vector, so feed
+ // the same placeholder through the stats collector and record it as the
default null value. Collected inside
+ // the write loop rather than in a pass of its own: it needs the exact
same per-doc branch.
+ DimensionFieldSpec sparseFieldSpec = new DimensionFieldSpec(sparseCol,
DataType.STRING, true);
+ String defaultValue = "";
+ AbstractColumnStatisticsCollector statsCollector =
StatsCollectorUtil.createStatsCollector(sparseFieldSpec, null);
Review Comment:
Went a different way than the minimal-`ColumnStatistics` suggestion:
`NoDictColumnStatisticsCollector` already exists for exactly this (exact
cardinality via a bounded set, HLL beyond that, no sort at `seal()`) but only
had the schema-based constructor. Added a `(FieldSpec, FieldConfig,
PartitionFunction)` constructor mirroring the pattern the other collectors
already expose for OPEN_STRUCT's synthetic columns, and pointed the sparse
column at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
