Re: Skip Corrupted Parquet blocks / footer.

In Spark 2.1, set spark.sql.files.ignoreCorruptFiles to true. On Sun, Jan 1, 2017 at 1:11 PM, khyati wrote: > Hi, > > I am trying to read the multiple parquet files in sparksql. In one dir > there > are two files, of which one is corrupted. While trying to read these

Re: Skip Corrupted Parquet blocks / footer.

You will have to change the metadata file under _spark_metadata folder to remove the listing of corrupt files. Thanks, Shobhit G Sent from my iPhone > On Dec 31, 2016, at 8:11 PM, khyati [via Apache Spark Developers List] > wrote: > > Hi, > > I am

Re: context.runJob() was suspended in getPreferredLocations() function

Hi, Simply said, you submit another Job in the event thread which will be blocked and can't receive the this job submission event. So your second job submission is never processed, and the getPreferredLocations method is never returned. Fei Hu wrote > Dear all, > > I tried to customize my