Hello Denis,
The from_json function supports only the fail fast mode, see:
https://github.com/apache/spark/blob/e2ab7deae76d3b6f41b9ad4d0ece14ea28db40ce/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L568
Your settings "mode" -> "PERMISSIVE" will be overwritten
On Wed, Aug 15, 2018 at 4:52 PM dbolshak <bolshakov.de...@gmail.com> wrote:
> Hello community,
>
> I can not manage to run from_json method with "columnNameOfCorruptRecord"
> option.
> ```
> import org.apache.spark.sql.functions._
>
> val data = Seq(
> "{'number': 1}",
> "{'number': }"
> )
>
> val schema = new StructType()
> .add($"number".int)
> .add($"_corrupt_record".string)
>
> val sourceDf = data.toDF("column")
>
> val jsonedDf = sourceDf
> .select(from_json(
> $"column",
> schema,
> Map("mode" -> "PERMISSIVE", "columnNameOfCorruptRecord" ->
> "_corrupt_record")
> ) as "data").selectExpr("data.number", "data._corrupt_record")
>
> jsonedDf.show()
> ```
> Does anybody can help me get `_corrupt_record` non empty?
>
> Thanks in advance.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
--
Maxim Gekk
Technical Solutions Lead
Databricks Inc.
maxim.g...@databricks.com
databricks.com
<http://databricks.com/>
