Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20666#discussion_r170498519
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -209,13 +209,15 @@ def json(self, path, schema=None,
primitivesAsString=None, prefersDecimal=None,
:param mode: allows a mode for dealing with corrupt records during
parsing. If None is
set, it uses the default value, ``PERMISSIVE``.
- * ``PERMISSIVE`` : sets other fields to ``null`` when it
meets a corrupted \
- record, and puts the malformed string into a field
configured by \
- ``columnNameOfCorruptRecord``. To keep corrupt records,
an user can set \
- a string type field named ``columnNameOfCorruptRecord``
in an user-defined \
- schema. If a schema does not have the field, it drops
corrupt records during \
- parsing. When inferring a schema, it implicitly adds a \
- ``columnNameOfCorruptRecord`` field in an output schema.
+ * ``PERMISSIVE`` : when it meets a corrupted record, puts
the malformed string \
+ into a field configured by
``columnNameOfCorruptRecord``, and sets other \
+ fields to ``null``. To keep corrupt records, an user can
set a string type \
+ field named ``columnNameOfCorruptRecord`` in an
user-defined schema. If a \
+ schema does not have the field, it drops corrupt records
during parsing. \
+ When inferring a schema, it implicitly adds a
``columnNameOfCorruptRecord`` \
--- End diff --
When users set a string type field named `columnNameOfCorruptRecord` in an
user-defined schema, even no corrupted record, I think the field is still
added. Or I misread this sentence?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
