Re: Issue with rogue data in csv file used in Spark application

[Adrian Bridgett]

We use the spark-csv (a successor of which is built in to spark 2.0) for 
this.  It doesn't cause crashes, failed parsing is logged.   We run on 
Mesos so I have to pull back all the logs from all the executors and 
search for failed lines (so that we can ensure that the failure rate 
isn't too high).

Hope this helps.
Adrian


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org