In Spark 2.1, set spark.sql.files.ignoreCorruptFiles to true.
On Sun, Jan 1, 2017 at 1:11 PM, khyati <khyati.s...@guavus.com> wrote:
> Hi,
>
> I am trying to read the multiple parquet files in sparksql. In one dir
> there
> are two files, of which one is corrupted. While trying to read these files,
> sparksql throws Exception for the corrupted file.
>
> val newDataDF =
> sqlContext.read.parquet("/data/testdir/data1.parquet","/
> data/testdir/corruptblock.0")
> newDataDF.show
>
> throws Exception.
>
> Is there any way to just skip the file having corrupted block/footer and
> just read the file/files which are proper?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/Skip-Corrupted-
> Parquet-blocks-footer-tp20418.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
