@spark.apache.orgmailto:user@spark.apache.org
Subject: Re: SparkSQL IndexOutOfBoundsException when reading from Parquet
Hello Terry,
I guess you hit this bughttps://issues.apache.org/jira/browse/SPARK-3559. The
list of needed column ids was messed up. Can you try the master branch or apply
the code
To: Terry Siu terry@smartfocus.com
Cc: Michael Armbrust mich...@databricks.com, user@spark.apache.org
user@spark.apache.org
Subject: Re: SparkSQL IndexOutOfBoundsException when reading from Parquet
Hello Terry,
How many columns does pqt_rdt_snappy have?
Thanks,
Yin
On Tue, Oct
...@databricks.com,
user@spark.apache.orgmailto:user@spark.apache.org
user@spark.apache.orgmailto:user@spark.apache.org
Subject: Re: SparkSQL IndexOutOfBoundsException when reading from Parquet
Hello Terry,
How many columns does pqt_rdt_snappy have?
Thanks,
Yin
On Tue, Oct 14, 2014 at 11:52 AM
@spark.apache.orgmailto:user@spark.apache.org
user@spark.apache.orgmailto:user@spark.apache.org
Subject: Re: SparkSQL IndexOutOfBoundsException when reading from Parquet
There are some known bug with the parquet serde and spark 1.1.
You can try setting spark.sql.hive.convertMetastoreParquet=true
...@databricks.com
Date: Monday, October 13, 2014 at 5:05 PM
To: Terry Siu terry@smartfocus.com
Cc: user@spark.apache.org user@spark.apache.org
Subject: Re: SparkSQL IndexOutOfBoundsException when reading from Parquet
There are some known bug with the parquet serde and spark 1.1.
You can try
I am currently using Spark 1.1.0 that has been compiled against Hadoop 2.3. Our
cluster is CDH5.1.2 which is runs Hive 0.12. I have two external Hive tables
that point to Parquet (compressed with Snappy), which were converted over from
Avro if that matters.
I am trying to perform a join with
There are some known bug with the parquet serde and spark 1.1.
You can try setting spark.sql.hive.convertMetastoreParquet=true to cause
spark sql to use built in parquet support when the serde looks like parquet.
On Mon, Oct 13, 2014 at 2:57 PM, Terry Siu terry@smartfocus.com wrote:
I am
