Re: task getting stuck

2014-09-24 Thread Debasish Das
First test would be if you can write parquet files fine on HDFS from your Spark job fine...If that also get's stuck then there is something with the logic...If parquet files are dumped fine and u can load them on HBase then there is something going on with Spark-HBase interaction On Wed, Sep 24, 2

Re: task getting stuck

2014-09-24 Thread Debasish Das
spark SQL reads parquet file fine...did you follow one of these to read/write parquet from spark ? http://zenfractal.com/2013/08/21/a-powerful-big-data-trio/ On Wed, Sep 24, 2014 at 9:29 AM, Ted Yu wrote: > Adding a subject. > > bq. at parquet.hadoop.ParquetFileReader$ > ConsecutiveChunkL

Re: task getting stuck

2014-09-24 Thread Ted Yu
Adding a subject. bq. at parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll( ParquetFileReader.java:599) Looks like there might be some issue reading the Parquet file. Cheers On Wed, Sep 24, 2014 at 9:10 AM, Jianshi Huang wrote: > Hi Ted, > > See my previous reply to Debasish