Could you please provide the full stack trace of the exception? And
what's the Git commit hash of the version you were using?
Cheng
On 7/24/15 6:37 AM, Jerry Lam wrote:
Hi Cheng,
I ran into issues related to ENUM when I tried to use Filter push
down. I'm using Spark 1.5.0 (which contains fixes for parquet filter
push down). The exception is the following:
java.lang.IllegalArgumentException: FilterPredicate column: item's
declared type (org.apache.parquet.io.api.Binary) does not match the
schema found in file metadata. Column item is of type:
FullTypeDescriptor(PrimitiveType: BINARY, OriginalType: ENUM)
Valid types for this column are: null
Is it because Spark does not recognize ENUM type in parquet?
Best Regards,
Jerry
On Wed, Jul 22, 2015 at 12:21 AM, Cheng Lian <lian.cs....@gmail.com
<mailto:lian.cs....@gmail.com>> wrote:
On 7/22/15 9:03 AM, Ankit wrote:
Thanks a lot Cheng. So it seems even in spark 1.3 and 1.4,
parquet ENUMs were treated as Strings in Spark SQL right? So does
this mean partitioning for enums already works in previous
versions too since they are just treated as strings?
It’s a little bit complicated. A Thrift/Avro/ProtoBuf |ENUM| value
is represented as a |BINARY| annotated with original type |ENUM|
in Parquet. For example, an optional |ENUM| field |e| is
translated to something like |optional BINARY e (ENUM)| in
Parquet. And the underlying data is always a UTF8 string of the
|ENUM| name. However, the Parquet original type |ENUM| is not
documented, thus Spark 1.3 and 1.4 doesn’t recognize the |ENUM|
annotation and just see it as a normal |BINARY|. (I didn’t even
notice the existence of |ENUM| in Parquet before PR #7048…)
On the other hand, Spark SQL has a boolean option named
|spark.sql.parquet.binaryAsString|. When this option is set to
|true|, all Parquet |BINARY| values are considered and converted
to UTF8 strings. The original purpose of this option is used to
work around a bug of Hive, which writes strings as plain Parquet
|BINARY| values without a proper |UTF8| annotation.
That said, by using
|sqlContext.setConf("spark.sql.parquet.binaryAsString", "true")|
in Scala/Java/Python, or |SET
spark.sql.parquet.binaryAsString=true| in SQL, you may read those
|ENUM| values as plain UTF8 strings.
Also, is there a good way to verify that the partitioning is
being used? I tried "explain" like (where data is partitioned by
"type" column)
scala> ev.filter("type = 'NON'").explain
== Physical Plan ==
Filter (type#4849 = NON)
PhysicalRDD [...cols..], MapPartitionsRDD[103] at map at
newParquet.scala:573
but that is the same even with non partitioned data.
Do you mean how to verify whether partition pruning is effective?
You should be able to see log lines like this:
15/07/22 11:14:35 INFO DataSourceStrategy: Selected 1
partitions out of 3, pruned 66.66666666666667% partitions.
On Tue, Jul 21, 2015 at 4:35 PM, Cheng Lian
<lian.cs....@gmail.com <mailto:lian.cs....@gmail.com>> wrote:
Parquet support for Thrift/Avro/ProtoBuf ENUM types are just
added to the master branch.
https://github.com/apache/spark/pull/7048
ENUM types are actually not in the Parquet format spec,
that's why we didn't have it at the first place. Basically,
ENUMs are always treated as UTF8 strings in Spark SQL now.
Cheng
On 7/22/15 3:41 AM, ankits wrote:
Hi, I am using a custom build of spark 1.4 with the
parquet dependency
upgraded to 1.7. I have thrift data encoded with parquet
that i want to
partition by a column of type ENUM. Spark programming
guide says partition
discovery is only supported for string and numeric
columns, so it seems
partition discovery won't work out of the box here.
Is there any workaround that will allow me to partition
by ENUMs? Will hive
partitioning help here? I am unfamiliar with Hive, and
how it plays into
parquet, thrift and spark so I would appreciate any
pointers in the right
direction. Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Partition-parquet-data-by-ENUM-column-tp23939.html
Sent from the Apache Spark User List mailing list archive
at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail:
user-h...@spark.apache.org
<mailto:user-h...@spark.apache.org>
