gene-db commented on code in PR #58175:
URL: https://github.com/apache/spark/pull/58175#discussion_r3905770590
##########
docs/sql-data-sources-csv.md:
##########
@@ -225,7 +225,13 @@ Data source options of CSV can be set via:
<tr>
<td><code>singleVariantColumn</code></td>
<td>(none)</td>
- <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields.</td>
+ <td>If specified, the entire CSV record is parsed and stored as a single
column of <code>VariantType</code> with the given column name, instead of being
split into individual fields. By default, scalar values are type-inferred
inside the variant (for example, <code>"0001"</code> becomes the integer
<code>1</code>). To keep every scalar as a string instead, set
<code>variantRespectInferSchema</code> to <code>true</code> and
<code>inferSchema</code> to <code>false</code>.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>variantRespectInferSchema</code></td>
Review Comment:
What does the xml config look like? I'm not familiar with it.
If we go this route, `variantScalarStrings` would ignore `inferSchema`,
because I think the other variant config also ignores `inferSchema`.
Although, I don't know if there is already a convention of these string
behaviors in xml or not. If so, maybe we should just stay consistent between
xml and csv.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
