szehon-ho commented on code in PR #57658:
URL: https://github.com/apache/spark/pull/57658#discussion_r3715174164
##########
docs/sql-data-sources-csv.md:
##########
@@ -209,7 +209,7 @@ Data source options of CSV can be set via:
<td>PERMISSIVE</td>
<td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default).<br>
<ul>
- <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To keep corrupt records, an user can set a string type field
named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a
schema does not have the field, it drops corrupt records during parsing. A
record with less/more tokens than schema is not a corrupted record to CSV. When
it meets a record having fewer tokens than the length of the schema, sets
<code>null</code> to extra fields. When the record has more tokens than the
length of the schema, it drops extra tokens.</li>
+ <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To capture the malformed string, a user can set a string
type field named <code>columnNameOfCorruptRecord</code> in a user-defined
schema. If a schema does not have the field, the corrupt record is still
retained with its malformed fields set to <code>null</code>, but the malformed
string is not available. A record with a different number of tokens than the
schema <em>is</em> a corrupted record, and is handled by this <code>mode</code>
like any other; it is still parsed as far as it can be, with <code>null</code>
set for tokens the record does not have and extra tokens dropped.</li>
Review Comment:
do we need the <em> in the doc? seems a bit unnecessary here and no
precedent in this doc
##########
docs/sql-data-sources-csv.md:
##########
@@ -209,7 +209,7 @@ Data source options of CSV can be set via:
<td>PERMISSIVE</td>
<td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default).<br>
<ul>
- <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To keep corrupt records, an user can set a string type field
named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a
schema does not have the field, it drops corrupt records during parsing. A
record with less/more tokens than schema is not a corrupted record to CSV. When
it meets a record having fewer tokens than the length of the schema, sets
<code>null</code> to extra fields. When the record has more tokens than the
length of the schema, it drops extra tokens.</li>
+ <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To capture the malformed string, a user can set a string
type field named <code>columnNameOfCorruptRecord</code> in a user-defined
schema. If a schema does not have the field, the corrupt record is still
retained with its malformed fields set to <code>null</code>, but the malformed
string is not available. A record with a different number of tokens than the
schema <em>is</em> a corrupted record, and is handled by this <code>mode</code>
like any other; it is still parsed as far as it can be, with <code>null</code>
set for tokens the record does not have and extra tokens dropped.</li>
Review Comment:
do we need the `<em>` in the doc? seems a bit unnecessary here and no
precedent in this doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
