I don't have experience with Cascading, but we saw similar issue for importing the data generated in Spark into Hive.
Did you try this setting "spark.sql.parquet.writeLegacyFormat" to true? https://stackoverflow.com/questions/44279870/why-cant-impala-read-parquet-files-after-spark-sqls-write [https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/44279870/why-cant-impala-read-parquet-files-after-spark-sqls-write> java - Why can't Impala read parquet files after Spark SQL ...<https://stackoverflow.com/questions/44279870/why-cant-impala-read-parquet-files-after-spark-sqls-write> stackoverflow.com Having some issues with the way that Spark is interpreting columns for parquet. I have an Oracle source with confirmed schema (df.schema() method): root |-- LM_PERSON ... ________________________________ From: Vikas Gandham <g.73vi...@gmail.com> Sent: Wednesday, November 15, 2017 2:30 PM To: user@spark.apache.org Subject: Parquet files from spark not readable in Cascading Hi, When I tried reading parquet data that was generated by spark in cascading it throws following error Caused by: org.apache.parquet.io<http://org.apache.parquet.io/>.ParquetDecodingException: Can not read value at 0 in block -1 in file "" at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:228) at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:201) at org.apache.parquet.hadoop.mapred.DeprecatedParquetInputFormat$RecordReaderWrapper.<init>(DeprecatedParquetInputFormat.java:103) at org.apache.parquet.hadoop.mapred.DeprecatedParquetInputFormat.getRecordReader(DeprecatedParquetInputFormat.java:47) at cascading.tap.hadoop.io<http://cascading.tap.hadoop.io/>.MultiInputFormat$1.operate(MultiInputFormat.java:253) at cascading.tap.hadoop.io<http://cascading.tap.hadoop.io/>.MultiInputFormat$1.operate(MultiInputFormat.java:248) at cascading.util.Util.retry(Util.java:1044) at cascading.tap.hadoop.io<http://cascading.tap.hadoop.io/>.MultiInputFormat.getRecordReader(MultiInputFormat.java:247) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:268) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 at java.util.ArrayList.elementData(ArrayList.java:418) at java.util.ArrayList.get(ArrayList.java:431) at org.apache.parquet.io<http://org.apache.parquet.io/>.GroupColumnIO.getLast(GroupColumnIO.java:98) at org.apache.parquet.io<http://org.apache.parquet.io/>.GroupColumnIO.getLast(GroupColumnIO.java:98) at org.apache.parquet.io<http://org.apache.parquet.io/>.PrimitiveColumnIO.getLast(PrimitiveColumnIO.java:83) at org.apache.parquet.io<http://org.apache.parquet.io/>.PrimitiveColumnIO.isLast(PrimitiveColumnIO.java:77) at org.apache.parquet.io<http://org.apache.parquet.io/>.RecordReaderImplementation.<init>(RecordReaderImplementation.java:293) at org.apache.parquet.io<http://org.apache.parquet.io/>.MessageColumnIO$1.visit(MessageColumnIO.java:134) at org.apache.parquet.io<http://org.apache.parquet.io/>.MessageColumnIO$1.visit(MessageColumnIO.java:99) at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154) at org.apache.parquet.io<http://org.apache.parquet.io/>.MessageColumnIO.getRecordReader(MessageColumnIO.java:99) at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:137) at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:208) This is mostly seen when parquet has nested structures. I didnt find any solution to this. I see some JIRA issues like this https://issues.apache.org/jira/browse/SPARK-10434 (parquet compatability /interoperabilityissues) where reading parquet files in Spark 1.4 where the files [SPARK-10434] Parquet compatibility with 1.4 is broken ...<https://issues.apache.org/jira/browse/SPARK-10434> issues.apache.org This behavior is a hybrid of parquet-avro and parquet-hive: the 3-level structure and repeated group name "bag" are borrowed from parquet-hive, while the innermost ... were generated by Spark 1.5 .This was fixed in later versions but was it fixed in Cascading? Not sure if this is something to do with Parquet version or Cascading has a bug or Spark is doing something with Parquet files which cascading is not accepting Note : I am trying to read Parquet with avro schema in Cascading I have posted in Cascading mailing list too. -- Thanks Vikas Gandham
