The relevant error lines are:
Caused by: parquet.io.ParquetDecodingException: Can't read value in
column [roll_key] BINARY at value 19600 out of 4814, 19600 out of
19600 in currentPage. repetition level: 0, definition level: 1
Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 131 in stage 0.0 failed 4 times, most recent failure:
Lost task 131.3 in stage 0.0 (TID 198, dap.changed.com):
parquet.io.ParquetDecodingException: Can not read value at 19600 in
block 0 in file
hdfs://dap.changed.com:8020/data/part-r-00177-51654832-053d-4074-b906-b97ac173807a.gz.parquet
But when I am using spark client and reading it, I am not getting any error.
sqlContext.read.load("/data/").select("roll_key")
Kindly let me know how to debug it.
--
-Kapil Rajak <http://cse.iitkgp.ac.in/~kdkr/>
