Re: [PR] Comparison between defaultParName and partValue [hudi]
hehuiyuan commented on code in PR #10234:
URL: https://github.com/apache/hudi/pull/10234#discussion_r1413595424
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##
@@ -94,7 +94,7 @@ private boolean evaluate(String partition) {
Map partStats = new LinkedHashMap<>();
for (int idx = 0; idx < partitionKeys.length; idx++) {
String partKey = partitionKeys[idx];
-Object partVal = partKey.equals(defaultParName)
+Object partVal = partStrArray[idx].equals(defaultParName)
Review Comment:
1. If partVal equals `defaultParName`, the value of part field is null
and the code return null.
```
Object partVal = partStrArray[idx].equals(defaultParName)
? null : DataTypeUtils.resolvePartition(partStrArray[idx],
partitionTypes.get(idx));
```
ColumnStats columnStats = new ColumnStats(null, null, 1);
2. if parVal equals`defaultParName` and parKey is not equal to
`defaultParName`, it will return `defaultParName`
```
Object partVal = partKey.equals(defaultParName)
? null : DataTypeUtils.resolvePartition(partStrArray[idx],
partitionTypes.get(idx));
```
ColumnStats columnStats = new ColumnStats(`defaultParName`,`defaultParName`,
0);
3. Example for LeafEvaluator evaluator:
```
public static class IsNotNull extends LeafEvaluator {
private static final long serialVersionUID = 1L;
public static IsNotNull getInstance() {
return new IsNotNull();
}
@Override
public boolean eval(Map columnStatsMap) {
ColumnStats columnStats = getColumnStats(columnStatsMap);
// should consider FLOAT/DOUBLE & NAN
return columnStats.getMinVal() != null || columnStats.getNullCnt() <=
0;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Comparison between defaultParName and partValue [hudi]
hudi-bot commented on PR #10234: URL: https://github.com/apache/hudi/pull/10234#issuecomment-1838118983 ## CI report: * a21ab0df2ee9e6a24219a6973624784a37017d00 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21286) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Comparison between defaultParName and partValue [hudi]
danny0405 commented on code in PR #10234:
URL: https://github.com/apache/hudi/pull/10234#discussion_r1413431893
##
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/prune/PartitionPruners.java:
##
@@ -94,7 +94,7 @@ private boolean evaluate(String partition) {
Map partStats = new LinkedHashMap<>();
for (int idx = 0; idx < partitionKeys.length; idx++) {
String partKey = partitionKeys[idx];
-Object partVal = partKey.equals(defaultParName)
+Object partVal = partStrArray[idx].equals(defaultParName)
Review Comment:
The partKey is never used, maybe just switch it to ` String partValStr=
partStrArray[idx];` and use this variable then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Comparison between defaultParName and partValue [hudi]
hudi-bot commented on PR #10234: URL: https://github.com/apache/hudi/pull/10234#issuecomment-1837908541 ## CI report: * a21ab0df2ee9e6a24219a6973624784a37017d00 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21286) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Comparison between defaultParName and partValue [hudi]
hudi-bot commented on PR #10234: URL: https://github.com/apache/hudi/pull/10234#issuecomment-1837901855 ## CI report: * a21ab0df2ee9e6a24219a6973624784a37017d00 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Comparison between defaultParName and partValue [hudi]
hehuiyuan opened a new pull request, #10234: URL: https://github.com/apache/hudi/pull/10234 ### Change Logs Comparison between defaultParName and partValue ### Impact ### Risk level (write none, low medium or high below) ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
