Sandip Agarwala created SPARK-61677:
---------------------------------------
Summary: Add variantPrimitivesAsString option to keep primitive
values as strings when CSV, JSON and XML are read into VARIANT
Key: SPARK-61677
URL: https://issues.apache.org/jira/browse/SPARK-61677
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Sandip Agarwala
h3. Problem
When CSV, JSON or XML data is read into a VARIANT (via the
{{singleVariantColumn}} option or a {{VARIANT}} column in a user-specified
schema), the parser infers a type for each primitive leaf value. This can
change or lose data:
* CSV / XML: {{"0001"}} becomes the integer {{1}}, {{"1e5"}} becomes a number,
{{"true"}} becomes a boolean.
* JSON: numbers that do not fit {{DECIMAL(38)}} or use exponent notation are
stored as {{DOUBLE}}, losing precision (e.g. a 40-digit integer).
The only existing ways to turn this off are inconsistent and tied to
{{inferSchema}}, whose default differs per format:
* CSV: read option {{variantRespectInferSchema=true}} plus
{{inferSchema=false}} (SPARK-58917, unreleased, branch-4.x only).
* XML: SQL conf {{spark.sql.xml.variant.respectInferSchema=true}} plus
{{inferSchema=false}} (SPARK-56554).
* JSON: no option.
h3. Proposal
Add a read option {{variantPrimitivesAsString}} (default {{false}}) to CSV,
JSON and XML. When {{true}}, every non-null primitive leaf value is stored as
its raw text as a variant string, with no type inference:
* CSV: the field token.
* XML: element text, attribute values and value tags (after
{{ignoreSurroundingSpaces}} trimming).
* JSON: number and boolean tokens, exactly as written.
Nulls and {{nullValue}} matches remain variant null. Objects, arrays and XML
nesting are preserved. The option applies to {{singleVariantColumn}} reads,
{{VARIANT}} columns in a user-specified schema, and {{from_csv}} /
{{from_json}} / {{from_xml}} with a variant schema. {{parse_json}} is not
affected.
The name mirrors the JSON {{primitivesAsString}} option, which does the same
for struct reads. The two options stay independent, so existing JSON reads do
not change.
h3. Compatibility
* The default ({{false}}) keeps today's behavior.
* When {{variantPrimitivesAsString}} is not set, the existing knobs keep
working unchanged.
* When it is set, it takes precedence over them.
* CSV {{variantRespectInferSchema}} and the XML conf
{{spark.sql.xml.variant.respectInferSchema}} are deprecated in favor of the new
option.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]