L. C. Hsieh created SPARK-61442:
-----------------------------------
Summary: Parse the atomic and parameterized type strings of JSON
and of DDL with one parser in Spark Connect Rust
Key: SPARK-61442
URL: https://issues.apache.org/jira/browse/SPARK-61442
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
In Spark Connect Rust, DataType::from_json and DataType::from_ddl
(crates/spark-connect/src/types.rs) each have their own table of atomic type
names and their own chain of prefix checks for decimal(p,s), char(n),
varchar(n), time(p), timestamp_ntz(p), intervals and collated strings, and each
reads "string collate <name>" with separate code. The leaf functions that they
do share take no account of which grammar is being read, so a change made for
one grammar has changed the other without that being decided, and both differ
from PySpark in places:
* JSON reads decimal(5), which PySpark's _FIXED_DECIMAL does not (it came with
the DDL form).
* JSON does not read "interval second to day", the json() of
DayTimeIntervalType(3, 0), which PySpark's _parse_datatype_json_value reads, so
the JSON of such a type does not read back.
* DDL reads a negative scale, decimal(10,-2), which is not an INTEGER_VALUE of
Spark's grammar.
* Both read a signed argument such as char(-3), time(+6) or decimal(+5,2);
PySpark's patterns are \d+.
* "string collate" without a name and "string collateX" are read by both, by
DDL as a string of the empty collation; JSON does not read "string collate
X", which PySpark's \s+ does; neither requires the name to be \w+.
* DDL does not read "string collate system.builtin.<name>" as
CollationFactory.resolveFullyQualifiedName does.
Both parsers should go through one parser of the atomic names and one of the
parameterized forms, which is told the grammar (JSON or DDL) for the places
where the two grammars of PySpark really differ: letter case and whitespace,
whether a decimal needs its scale and may have a negative one, the order of the
fields of an interval range, and whether the name of a collation is kept as
written (JSON, as StringType(name)) or given in the canonical form of the
server (DDL). A table-driven test should feed each form through both parsers
and assert the same type, the same error, or the listed difference.
Neither parser checks the range of the precision or scale of a decimal, as
PySpark's does not; that stays as SPARK-61423 left it (checked at the
conversion to Arrow).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]