hehuiyuan commented on code in PR #10234:
URL: https://github.com/apache/hudi/pull/10234#discussion_r1413595424
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##########
@@ -94,7 +94,7 @@ private boolean evaluate(String partition) {
Map<String, ColumnStats> partStats = new LinkedHashMap<>();
for (int idx = 0; idx < partitionKeys.length; idx++) {
String partKey = partitionKeys[idx];
- Object partVal = partKey.equals(defaultParName)
+ Object partVal = partStrArray[idx].equals(defaultParName)
Review Comment:
If part filed value is null , use `defaultParName` instead of when writing
data.
When reading:
1. If partVal equals `defaultParName`, the value of part field is null
and the code return null.
```
Object partVal = partStrArray[idx].equals(defaultParName)
? null : DataTypeUtils.resolvePartition(partStrArray[idx],
partitionTypes.get(idx));
```
ColumnStats columnStats = new ColumnStats(null, null, 1);
2. if parVal equals`defaultParName` and parKey is not equal to
`defaultParName`, it will return `defaultParName`
```
Object partVal = partKey.equals(defaultParName)
? null : DataTypeUtils.resolvePartition(partStrArray[idx],
partitionTypes.get(idx));
```
ColumnStats columnStats = new ColumnStats(`defaultParName`,`defaultParName`,
0);
3. Example for LeafEvaluator evaluator:
```
public static class IsNotNull extends LeafEvaluator {
private static final long serialVersionUID = 1L;
public static IsNotNull getInstance() {
return new IsNotNull();
}
@Override
public boolean eval(Map<String, ColumnStats> columnStatsMap) {
ColumnStats columnStats = getColumnStats(columnStatsMap);
// should consider FLOAT/DOUBLE & NAN
return columnStats.getMinVal() != null || columnStats.getNullCnt() <=
0;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
