Sandip Agarwala created SPARK-61677:
---------------------------------------

             Summary: Add variantPrimitivesAsString option to keep primitive 
values as strings when CSV, JSON and XML are read into VARIANT
                 Key: SPARK-61677
                 URL: https://issues.apache.org/jira/browse/SPARK-61677
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Sandip Agarwala


h3. Problem

When CSV, JSON or XML data is read into a VARIANT (via the 
{{singleVariantColumn}} option or a {{VARIANT}} column in a user-specified 
schema), the parser infers a type for each primitive leaf value. This can 
change or lose data:

* CSV / XML: {{"0001"}} becomes the integer {{1}}, {{"1e5"}} becomes a number, 
{{"true"}} becomes a boolean.
* JSON: numbers that do not fit {{DECIMAL(38)}} or use exponent notation are 
stored as {{DOUBLE}}, losing precision (e.g. a 40-digit integer).

The only existing ways to turn this off are inconsistent and tied to 
{{inferSchema}}, whose default differs per format:

* CSV: read option {{variantRespectInferSchema=true}} plus 
{{inferSchema=false}} (SPARK-58917, unreleased, branch-4.x only).
* XML: SQL conf {{spark.sql.xml.variant.respectInferSchema=true}} plus 
{{inferSchema=false}} (SPARK-56554).
* JSON: no option.

h3. Proposal

Add a read option {{variantPrimitivesAsString}} (default {{false}}) to CSV, 
JSON and XML. When {{true}}, every non-null primitive leaf value is stored as 
its raw text as a variant string, with no type inference:

* CSV: the field token.
* XML: element text, attribute values and value tags (after 
{{ignoreSurroundingSpaces}} trimming).
* JSON: number and boolean tokens, exactly as written.

Nulls and {{nullValue}} matches remain variant null. Objects, arrays and XML 
nesting are preserved. The option applies to {{singleVariantColumn}} reads, 
{{VARIANT}} columns in a user-specified schema, and {{from_csv}} / 
{{from_json}} / {{from_xml}} with a variant schema. {{parse_json}} is not 
affected.

The name mirrors the JSON {{primitivesAsString}} option, which does the same 
for struct reads. The two options stay independent, so existing JSON reads do 
not change.

h3. Compatibility

* The default ({{false}}) keeps today's behavior.
* When {{variantPrimitivesAsString}} is not set, the existing knobs keep 
working unchanged.
* When it is set, it takes precedence over them.
* CSV {{variantRespectInferSchema}} and the XML conf 
{{spark.sql.xml.variant.respectInferSchema}} are deprecated in favor of the new 
option.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to