[ https://issues.apache.org/jira/browse/SPARK-30546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kent Yao updated SPARK-30546: ----------------------------- Description: Before 3.0 we may make some efforts for the current interval type to make it more future-proofing. e.g. 1. add unstable annotation to the CalendarInterval class. People already use it as UDF inputs so it’s better to make it clear it’s unstable. 2. Add a schema checker to prohibit create v2 custom catalog table with intervals, as same as what we do for the builtin catalog 3. Add a schema checker for DataFrameWriterV2 too 4. Make the interval type incomparable as version 2.4 for disambiguation of comparison between year-month and day-time fields 5. The 3.0 newly added to_csv should not support output intervals as same as using CSV file format 6. The function to_json should not allow using interval as a key field as same as the value field and JSON datasource, with a legacy config to restore. 7. Revert interval ISO/ANSI SQL Standard output since we decide not to follow ANSI, so there is no round trip. was: Before 3.0 we maymake some efforts for the current interval type to make it more future-proofing. e.g. 1. add unstable annotation to the CalendarInterval class. People already use it as UDF inputs so it’s better to make it clear it’s unstable. 2. Add a schema checker to prohibit create v2 custom catalog table with intervals, as same as what we do for the builtin catalog 3. Add a schema checker for DataFrameWriterV2 too 4. Make the interval type incomparable as version 2.4 for disambiguation of comparison between year-month and day-time fields 5. The 3.0 newly added to_csv should not support output intervals as same as using CSV file format 6. The function to_json should not allow using interval as a key field as same as the value field and JSON datasource, with a legacy config to restore. 7. Revert interval ISO/ANSI SQL Standard output since we decide not to follow ANSI, so there is no round trip. > Make interval type more future-proofing > --------------------------------------- > > Key: SPARK-30546 > URL: https://issues.apache.org/jira/browse/SPARK-30546 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Kent Yao > Priority: Major > > Before 3.0 we may make some efforts for the current interval type to make it > more future-proofing. e.g. > 1. add unstable annotation to the CalendarInterval class. People already use > it as UDF inputs so it’s better to make it clear it’s unstable. > 2. Add a schema checker to prohibit create v2 custom catalog table with > intervals, as same as what we do for the builtin catalog > 3. Add a schema checker for DataFrameWriterV2 too > 4. Make the interval type incomparable as version 2.4 for disambiguation of > comparison between year-month and day-time fields > 5. The 3.0 newly added to_csv should not support output intervals as same as > using CSV file format > 6. The function to_json should not allow using interval as a key field as > same as the value field and JSON datasource, with a legacy config to > restore. > 7. Revert interval ISO/ANSI SQL Standard output since we decide not to > follow ANSI, so there is no round trip. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org