Gabriel39 commented on code in PR #68027:
URL: https://github.com/apache/doris/pull/68027#discussion_r4021871055
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,20 +161,21 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ getCsvCharacter(params, tableParams, SEPARATOR_CHAR, ','));
+ // OpenCSVSerde does not use table-level line.delim to frame records.
Preserve the existing
+ // SerDe-only override; applying a table property here can merge
otherwise valid newline records.
+ String lineDelimiter = params == null ? null : params.get(LINE_DELIM);
+ result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
+ lineDelimiter == null ? DEFAULT_LINE_DELIM : lineDelimiter);
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
result.put(ScanNodePropertyKeys.TEXT_ENCLOSE, quoteChar);
- // #65501: BE strips the wrapping quotes only when the enclose char is
exactly the double-quote '"'.
- // The connector owns this CSV serde semantics, so decide here and
pass an explicit flag; the generic
- // PluginDrivenScanNode then just applies it instead of trimming for
any enclose char. Compare the
- // first byte, matching how the node sets enclose
(enclose.getBytes()[0]) and BE's getEnclose() == '"'.
- boolean trimDoubleQuotes = !quoteChar.isEmpty() &&
quoteChar.getBytes()[0] == (byte) '"';
- result.put(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES,
String.valueOf(trimDoubleQuotes));
- String escapeChar = getParamOrDefault(params, ESCAPE_CHAR, "\\");
- result.put(ScanNodePropertyKeys.TEXT_ESCAPE, escapeChar);
+ // BE's extra double-quote trimming is valid only for the effective
double-quote enclosure.
+ result.put(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES,
String.valueOf("\"".equals(quoteChar)));
+ result.put(ScanNodePropertyKeys.TEXT_ESCAPE, getCsvCharacter(params,
tableParams, ESCAPE_CHAR, '\\'));
result.put(ScanNodePropertyKeys.TEXT_NULL_FORMAT, "");
Review Comment:
Addressed in 81d106e4a9. The resolver now uses Hive's writer-default escape
and maps a normalized double quote to the reader-effective backslash before
validation and scan-property emission. Table/SerDe precedence still applies
before interpreting the sentinel.
Added property tests for both sources, multi-character sentinel values, and
precedence in both directions; FE tests verify the backslash after a real
scan-node/Thrift round trip. A test using Hive 3.1.3's actual OpenCSVSerde
checks backslash-escaped embedded quotes with both standard and custom quote
characters. The Hive regression suite writes with an explicit backslash,
switches only the metadata to the sentinel in each property source, and
compares source, Hive CSV, and Doris results.
Validation: all 479 Hive connector tests (1,249 including upstream modules)
and eight FE Core wire tests pass. The new property and Hive-reader tests
reproduce the pre-fix failures. The HiveServer2 end-to-end suite compiles but
remains pending because the local service is unavailable.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,20 +161,21 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ getCsvCharacter(params, tableParams, SEPARATOR_CHAR, ','));
+ // OpenCSVSerde does not use table-level line.delim to frame records.
Preserve the existing
Review Comment:
Addressed in 81d106e4a9. Validation now operates on the complete
parser-effective tuple after first-character normalization, property
precedence, and default-escape sentinel conversion. A NUL separator or any
equal active pair fails during planning. NUL quote/escape remain valid disabled
settings, and a raw double-quote quote/escape pair stays valid because the
reader-effective escape is backslash.
Added pairwise conflict tests from table, SerDe, and mixed sources,
normalization-induced conflicts, sentinel-induced conflicts, and NUL
acceptance/rejection tests. The new compatibility test uses Hive 3.1.3's actual
reader as the oracle for 343 tuples across those three property sources. FE
tests cover rejection before Thrift construction and zero-valued quote/escape
bytes after serialization. The Hive regression suite also includes all three
pairwise conflicts and a sentinel-induced conflict.
All 479 Hive connector tests (1,249 with upstream modules), including the
actual Hive-reader comparisons, and all eight FE Core wire tests pass.
End-to-end HiveServer2 execution remains pending locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
