[ 
https://issues.apache.org/jira/browse/SPARK-61442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61442.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 510
[https://github.com/apache/spark-connect-rust/pull/510]

> Parse the atomic and parameterized type strings of JSON and of DDL with one 
> parser in Spark Connect Rust
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61442
>                 URL: https://issues.apache.org/jira/browse/SPARK-61442
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> In Spark Connect Rust, DataType::from_json and DataType::from_ddl 
> (crates/spark-connect/src/types.rs) each have their own table of atomic type 
> names and their own chain of prefix checks for decimal(p,s), char(n), 
> varchar(n), time(p), timestamp_ntz(p), intervals and collated strings, and 
> each reads "string collate <name>" with separate code. The leaf functions 
> that they do share take no account of which grammar is being read, so a 
> change made for one grammar has changed the other without that being decided, 
> and both differ from PySpark in places:
> * JSON reads decimal(5), which PySpark's _FIXED_DECIMAL does not (it came 
> with the DDL form).
> * JSON does not read "interval second to day", the json() of 
> DayTimeIntervalType(3, 0), which PySpark's _parse_datatype_json_value reads, 
> so the JSON of such a type does not read back.
> * DDL reads a negative scale, decimal(10,-2), which is not an INTEGER_VALUE 
> of Spark's grammar.
> * Both read a signed argument such as char(-3), time(+6) or decimal(+5,2); 
> PySpark's patterns are \d+.
> * "string collate" without a name and "string collateX" are read by both, by 
> DDL as a string of the empty collation; JSON does not read "string  collate  
> X", which PySpark's \s+ does; neither requires the name to be \w+.
> * DDL does not read "string collate system.builtin.<name>" as 
> CollationFactory.resolveFullyQualifiedName does.
> Both parsers should go through one parser of the atomic names and one of the 
> parameterized forms, which is told the grammar (JSON or DDL) for the places 
> where the two grammars of PySpark really differ: letter case and whitespace, 
> whether a decimal needs its scale and may have a negative one, the order of 
> the fields of an interval range, and whether the name of a collation is kept 
> as written (JSON, as StringType(name)) or given in the canonical form of the 
> server (DDL). A table-driven test should feed each form through both parsers 
> and assert the same type, the same error, or the listed difference.
> Neither parser checks the range of the precision or scale of a decimal, as 
> PySpark's does not; that stays as SPARK-61423 left it (checked at the 
> conversion to Arrow).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to