Re: Spark SQL Exception handling

2014-11-20 Thread Daniel Haviv
Thanks I'll try it out! On Thu, Nov 20, 2014 at 8:39 PM, Michael Armbrust wrote: > If you run master or the 1.2 preview release then it should automatically > skip lines that fail to parse. The corrupted text will be in the column > _corrupted_record and the other columns will be null. > > On T

Re: Spark SQL Exception handling

2014-11-20 Thread Michael Armbrust
If you run master or the 1.2 preview release then it should automatically skip lines that fail to parse. The corrupted text will be in the column _corrupted_record and the other columns will be null. On Thu, Nov 20, 2014 at 7:34 AM, Daniel Haviv wrote: > Hi Guys, > I really need your help with

Spark SQL Exception handling

2014-11-20 Thread Daniel Haviv
Hi Guys, I really need your help with this: I'm loading a bunch of json files uploaded via webhdfs, some of them have some incosistencies (the json ends mid-line for example) and that causes my whole application to fail. How can I continue processing valid json records without failing the applicat