Re: [PR] [Feature](routine_load) auto fill missing columns [doris]
github-actions[bot] commented on code in PR #64071:
URL: https://github.com/apache/doris/pull/64071#discussion_r3374434556
##
fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsLoadScanProvider.java:
##
@@ -195,9 +196,16 @@ private void
fillContextExprMap(List columnDescList, Ne
// If user does not specify the file field names, generate it by using
base schema of table.
// So that the following process can be unified
boolean specifyFileFieldNames = copiedColumnExprs.stream().anyMatch(p
-> p.isColumn());
-if (!specifyFileFieldNames) {
+if (!specifyFileFieldNames || isFillMissingColumns(fileGroup)) {
Review Comment:
For routine load this condition will still be false even when the job
property was set. `KafkaRoutineLoadJob.toNereidsRoutineLoadTaskInfo()` copies
`jobProperties` into `NereidsRoutineLoadTaskInfo`, but
`NereidsDataDescription(NereidsLoadTaskInfo)` only copies JSON props like
`strip_outer_array`, `jsonpaths`, `json_root`, `fuzzy_parse`,
`read_json_by_line`, and `num_as_string` into `analysisMap`. It never copies
`fill_missing_columns`, so `analyzeFileFormatProperties()` builds a
`JsonFileFormatProperties` with the default `false`, and this new branch is not
taken for the routine-load execution path. Please add the property to the
Nereids task-info/data-description propagation path and cover it with a test
that reaches `NereidsLoadScanProvider`.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsLoadScanProvider.java:
##
@@ -420,15 +428,37 @@ private void
fillContextExprMap(List columnDescList, Ne
}
/**
- * if not set sequence column and column size is null or only have deleted
sign ,return true
+ * Returns true when the sequence column should be auto-added, i.e.,
+ * if not set sequence column and column size is null or only have deleted
sign,
+ * or fill_missing_columns is enabled, meaning schema will be auto-filled.
*/
-private boolean shouldAddSequenceColumn(List
columnDescList) {
+private boolean shouldAddSequenceColumn(List
columnDescList,
+NereidsBrokerFileGroup fileGroup) {
+if (isFillMissingColumns(fileGroup)) {
+return true;
+}
if (columnDescList.isEmpty()) {
return true;
}
return columnDescList.size() == 1 &&
columnDescList.get(0).getColumnName().equalsIgnoreCase(Column.DELETE_SIGN);
}
+/**
+ * Returns true if the file format is JSON and fill_missing_columns is
enabled. Only meaningful for JSON.
+ */
+private boolean isFillMissingColumns(NereidsBrokerFileGroup fileGroup) {
+return fileGroup.getFileFormatProperties() instanceof
JsonFileFormatProperties
+&& ((JsonFileFormatProperties)
fileGroup.getFileFormatProperties()).isFillMissingColumns();
+}
+
+/**
+ * Returns true if the file format is JSON and fill_missing_columns is
enabled. Only meaningful for JSON.
+ */
+private boolean isFillMissingColumns(NereidsBrokerFileGroup fileGroup) {
Review Comment:
This duplicates the `isFillMissingColumns(NereidsBrokerFileGroup)` method
declared just above at lines 449-452, so `NereidsLoadScanProvider` will not
compile (`method isFillMissingColumns(...) is already defined`). Please remove
one of the duplicate declarations before this can pass FE compilation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [Feature](routine_load) auto fill missing columns [doris]
liaoxin01 commented on PR #64071: URL: https://github.com/apache/doris/pull/64071#issuecomment-4649832885 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Feature](routine_load) auto fill missing columns [doris]
billryan commented on PR #64071: URL: https://github.com/apache/doris/pull/64071#issuecomment-4618814359 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Feature](routine_load) auto fill missing columns [doris]
hello-stephen commented on PR #64071: URL: https://github.com/apache/doris/pull/64071#issuecomment-4611008632 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[PR] [Feature](routine_load) auto fill missing columns [doris]
billryan opened a new pull request, #64071: URL: https://github.com/apache/doris/pull/64071 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: For routine load jobs, if we have thousands of columns to import, we can use json format let doris automatically match the column name with table schema. But for some scenarios, the upstream data does not have the column name(can be derived from current columns) and we can not add new column name easily, so we consider if we can just specify the new derived column only and let doris fill the missing columns with table schema. e.g. The MQ data have `stat_time_day` field but does not have `p_date` column, we can add new column like this `p_date = stat_time_day`, fill missing columns from table schema. ```sql CREATE ROUTINE LOAD db.job_name ON table_a COLUMNS(p_date = stat_time_day) PROPERTIES ( "format" = "json", "fill_missing_columns" = "true" ) FROM KAFKA (...); ``` ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
