You can sort of hack this by reading it as an RDD[String] and trying to
implement a custom parser i.e.
Val rddRows = rdd.map parseMyCols
Def parseMyCols(rawVal: String) : Row = {
parse(rawVal) match {
case Success(parsedRowValues) = > Row(parsedRowValues :+ “”: _*)
case Failure(exception) =>
There's a column which captures the corrupted record. However, the
exception isn't captured. If the exception is captured in another column
it'll be very useful.
On Mon, 17 Jun, 2019, 10:56 AM Gourav Sengupta,
wrote:
> Hi,
>
> it already does, I think, you just have to add the column in the
Hi,
it already does, I think, you just have to add the column in the schema
that you are using to read.
Regards,
Gourav
On Sun, Jun 16, 2019 at 2:48 PM wrote:
> Hi Team,
>
>
>
> Can we have another column which gives the corrupted record reason in
> permissive mode while reading csv.
>
>
>
>
Hi Team,
Can we have another column which gives the corrupted record reason in
permissive mode while reading csv.
Thanks,
Ajay
