Forget to say, another option is we can replace readAllFootersInParallel
with our parallel reading logic, so we can ignore corrupt files.
Liang-Chi Hsieh wrote
> Hi,
>
> The method readAllFootersInParallel is implemented in Parquet's
> ParquetFileReader. So the spark config
> "spark.sql.files.ignoreCorruptFiles" doesn't work for it.
>
> Reading all footers in parallel can speed up the task. However, we can't
> control if ignoring corrupt files or not.
>
> Of course we can read this footers in sequence and ignore the corrupt
> ones. But it might be inefficient. Since this is a relatively corner use
> case, I don't expect we can have this.
>
> Maybe Parquet can implement an option to ignore corrupt files. However,
> even so, it can't be expected to have this updated Parquet implementation
> available to Spark very soon.
>
> khyati wrote
>> Hi Reynold Xin,
>>
>> In spark 2.1.0,
>> I tried setting spark.sql.files.ignoreCorruptFiles = true by using
>> commands,
>>
>> val sqlContext =new org.apache.spark.sql.hive.HiveContext(sc)
>>
>> sqlContext.setConf("spark.sql.files.ignoreCorruptFiles","true") /
>> sqlContext.sql("set spark.sql.files.ignoreCorruptFiles=true")
>>
>> but still getting error while reading parquet files using
>> val newDataDF =
>> sqlContext.read.parquet("/data/tempparquetdata/corruptblock.0","/data/tempparquetdata/data1.parquet")
>>
>> Error: ERROR executor.Executor: Exception in task 0.0 in stage 4.0 (TID
>> 4)
>> java.io.IOException: Could not read footer: java.lang.RuntimeException:
>> hdfs://192.168.1.53:9000/data/tempparquetdata/corruptblock.0 is not a
>> Parquet file. expected magic number at tail [80, 65, 82, 49] but found
>> [65, 82, 49, 10]
>> at
>> org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:248)
>>
>>
>> Please let me know if I am missing anything.
-----
Liang-Chi Hsieh | @viirya
Spark Technology Center
http://www.spark.tc/
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Skip-Corrupted-Parquet-blocks-footer-tp20418p20451.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
