[
https://issues.apache.org/jira/browse/SPARK-61677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61677:
-----------------------------------
Labels: pull-request-available (was: )
> Add variantPrimitivesAsString option to keep primitive values as strings when
> CSV, JSON and XML are read into VARIANT
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61677
> URL: https://issues.apache.org/jira/browse/SPARK-61677
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Sandip Agarwala
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> When CSV, JSON or XML data is read into a VARIANT (via the
> {{singleVariantColumn}} option or a {{VARIANT}} column in a user-specified
> schema), the parser infers a type for each primitive leaf value. This can
> change or lose data:
> * CSV / XML: {{"0001"}} becomes the integer {{1}}, {{"1e5"}} becomes a
> number, {{"true"}} becomes a boolean.
> * JSON: numbers that do not fit {{DECIMAL(38)}} or use exponent notation are
> stored as {{DOUBLE}}, losing precision (e.g. a 40-digit integer).
> The only existing ways to turn this off are inconsistent and tied to
> {{inferSchema}}, whose default differs per format:
> * CSV: read option {{variantRespectInferSchema=true}} plus
> {{inferSchema=false}} (SPARK-58917, unreleased, branch-4.x only).
> * XML: SQL conf {{spark.sql.xml.variant.respectInferSchema=true}} plus
> {{inferSchema=false}} (SPARK-56554).
> * JSON: no option.
> h3. Proposal
> Add a read option {{variantPrimitivesAsString}} (default {{false}}) to CSV,
> JSON and XML. When {{true}}, every non-null primitive leaf value is stored as
> its raw text as a variant string, with no type inference:
> * CSV: the field token.
> * XML: element text, attribute values and value tags (after
> {{ignoreSurroundingSpaces}} trimming).
> * JSON: number and boolean tokens, exactly as written.
> Nulls and {{nullValue}} matches remain variant null. Objects, arrays and XML
> nesting are preserved. The option applies to {{singleVariantColumn}} reads,
> {{VARIANT}} columns in a user-specified schema, and {{from_csv}} /
> {{from_json}} / {{from_xml}} with a variant schema. {{parse_json}} is not
> affected.
> The name mirrors the JSON {{primitivesAsString}} option, which does the same
> for struct reads. The two options stay independent, so existing JSON reads do
> not change.
> h3. Compatibility
> * The default ({{false}}) keeps today's behavior.
> * When {{variantPrimitivesAsString}} is not set, the existing knobs keep
> working unchanged.
> * When it is set, it takes precedence over them.
> * CSV {{variantRespectInferSchema}} and the XML conf
> {{spark.sql.xml.variant.respectInferSchema}} are deprecated in favor of the
> new option.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]