Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Michael Armbrust
Do you have a full stack trace? On Thu, Apr 2, 2015 at 11:45 AM, ogoh oke...@gmail.com wrote: Hello, My ETL uses sparksql to generate parquet files which are served through Thriftserver using hive ql. It especially defines a schema programmatically since the schema can be only known at

Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Michael Armbrust
This looks to me like you have incompatible versions of scala on your classpath? On Thu, Apr 2, 2015 at 4:28 PM, Okehee Goh oke...@gmail.com wrote: yes, below is the stacktrace. Thanks, Okehee java.lang.NoSuchMethodError:

Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Okehee Goh
Michael, You are right. The build brought org.scala-lang:scala-library:2.10.1 from other package (as below). It works fine after excluding the old scala version. Thanks a lot, Okehee == dependency: |+--- org.apache.kafka:kafka_2.10:0.8.1.1 ||+---

Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Okehee Goh
yes, below is the stacktrace. Thanks, Okehee java.lang.NoSuchMethodError: scala.reflect.NameTransformer$.LOCAL_SUFFIX_STRING()Ljava/lang/String; at scala.reflect.internal.StdNames$CommonNames.init(StdNames.scala:97) at

Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread ogoh
Hello, My ETL uses sparksql to generate parquet files which are served through Thriftserver using hive ql. It especially defines a schema programmatically since the schema can be only known at runtime. With spark 1.2.1, it worked fine (followed