[ 
https://issues.apache.org/jira/browse/SPARK-61677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61677:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add variantPrimitivesAsString option to keep primitive values as strings when 
> CSV, JSON and XML are read into VARIANT
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61677
>                 URL: https://issues.apache.org/jira/browse/SPARK-61677
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Sandip Agarwala
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> When CSV, JSON or XML data is read into a VARIANT (via the 
> {{singleVariantColumn}} option or a {{VARIANT}} column in a user-specified 
> schema), the parser infers a type for each primitive leaf value. This can 
> change or lose data:
> * CSV / XML: {{"0001"}} becomes the integer {{1}}, {{"1e5"}} becomes a 
> number, {{"true"}} becomes a boolean.
> * JSON: numbers that do not fit {{DECIMAL(38)}} or use exponent notation are 
> stored as {{DOUBLE}}, losing precision (e.g. a 40-digit integer).
> The only existing ways to turn this off are inconsistent and tied to 
> {{inferSchema}}, whose default differs per format:
> * CSV: read option {{variantRespectInferSchema=true}} plus 
> {{inferSchema=false}} (SPARK-58917, unreleased, branch-4.x only).
> * XML: SQL conf {{spark.sql.xml.variant.respectInferSchema=true}} plus 
> {{inferSchema=false}} (SPARK-56554).
> * JSON: no option.
> h3. Proposal
> Add a read option {{variantPrimitivesAsString}} (default {{false}}) to CSV, 
> JSON and XML. When {{true}}, every non-null primitive leaf value is stored as 
> its raw text as a variant string, with no type inference:
> * CSV: the field token.
> * XML: element text, attribute values and value tags (after 
> {{ignoreSurroundingSpaces}} trimming).
> * JSON: number and boolean tokens, exactly as written.
> Nulls and {{nullValue}} matches remain variant null. Objects, arrays and XML 
> nesting are preserved. The option applies to {{singleVariantColumn}} reads, 
> {{VARIANT}} columns in a user-specified schema, and {{from_csv}} / 
> {{from_json}} / {{from_xml}} with a variant schema. {{parse_json}} is not 
> affected.
> The name mirrors the JSON {{primitivesAsString}} option, which does the same 
> for struct reads. The two options stay independent, so existing JSON reads do 
> not change.
> h3. Compatibility
> * The default ({{false}}) keeps today's behavior.
> * When {{variantPrimitivesAsString}} is not set, the existing knobs keep 
> working unchanged.
> * When it is set, it takes precedence over them.
> * CSV {{variantRespectInferSchema}} and the XML conf 
> {{spark.sql.xml.variant.respectInferSchema}} are deprecated in favor of the 
> new option.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to