voonhous commented on code in PR #19463:
URL: https://github.com/apache/hudi/pull/19463#discussion_r3765218370
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java:
##########
@@ -279,9 +279,13 @@ public static List<String> orderFields(String
fieldNameCsv, String fieldOrderCsv
List<String> fieldNames = fieldNameCsv.isEmpty() ? new ArrayList<>() :
Arrays.stream(fieldNameCsv.split(",")).collect(Collectors.toList());
Set<String> fieldNamesSet = new LinkedHashSet<>(fieldNames);
if (fieldNamesSet.size() != fieldOrders.length) {
+ // Report the de-duplicated counts, since those are what were compared:
quoting the raw name count
+ // can produce a message whose two numbers are equal even though the
mismatch is real. The inputs are
+ // included because the projection lists come from Hive and are the only
way to diagnose the mismatch.
throw new HoodieException(String
- .format("Error ordering fields for storage read. #fieldNames: %d,
#fieldPositions: %d",
- fieldNames.size(), fieldOrders.length));
+ .format("Error ordering fields for storage read. #fieldNames: %d,
#fieldPositions: %d, "
+ + "read column names: [%s], read column ids: [%s]",
+ fieldNamesSet.size(), fieldOrders.length, fieldNameCsv,
fieldOrderCsv));
}
TreeMap<Integer, String> orderedFieldMap = new TreeMap<>();
String[] fieldNamesArray = fieldNamesSet.toArray(new String[0]);
Review Comment:
Following up in this thread rather than opening a new one: the blank-token
filter interacts with what is filed as #19506.
The count check was incidentally catching some of this divergence. With the
filter, inputs that used to throw now resolve to the mis-paired order instead.
Worked example, every step a real Hive/Hudi call on one JobConf:
```
Hudi addProjectionField x2 ids "2,0" names
"_hoodie_record_key,_hoodie_commit_time"
Hive appendReadColumns([7],["g"]) ids "7,2,0" names
"_hoodie_record_key,_hoodie_commit_time,g"
^ ids prepend, names append -- the
divergence in this thread
Hive appendReadColumns([],[]) ids ",7,2,0"
Hive appendReadColumns([],[]) ids ",,7,2,0"
cleanProjectionColumnIds ids ",7,2,0" (strips one comma only)
orderFields("_hoodie_record_key,_hoodie_commit_time,g", ",7,2,0")
master: throws #fieldNames: 3, #fieldPositions: 4
patched: returns [g, _hoodie_commit_time, _hoodie_record_key]
correct pairing is [_hoodie_commit_time, _hoodie_record_key, g]
```
The wrongness is #19506 and not this PR, and you noted downstream resolves
by name so it is likely masked today. Worth recording on the issue that the
count check no longer catches it, so whoever picks up #19506 does not assume
the loud failure is still there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
