Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Okehee Goh
Michael, You are right. The build brought " org.scala-lang:scala-library:2.10.1" from other package (as below). It works fine after excluding the old scala version. Thanks a lot, Okehee == dependency: |+--- org.apache.kafka:kafka_2.10:0.8.1.1 ||+--- com.yammer.metrics:metrics-core:2

Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Michael Armbrust
This looks to me like you have incompatible versions of scala on your classpath? On Thu, Apr 2, 2015 at 4:28 PM, Okehee Goh wrote: > yes, below is the stacktrace. > Thanks, > Okehee > > java.lang.NoSuchMethodError: > scala.reflect.NameTransformer$.LOCAL_SUFFIX_STRING()Ljava/lang/String; >

Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Okehee Goh
yes, below is the stacktrace. Thanks, Okehee java.lang.NoSuchMethodError: scala.reflect.NameTransformer$.LOCAL_SUFFIX_STRING()Ljava/lang/String; at scala.reflect.internal.StdNames$CommonNames.(StdNames.scala:97) at scala.reflect.internal.StdNames$Keywords.(StdNames.scala:203)

Re: Generating a schema in Spark 1.3 failed while using DataTypes.

2015-04-02 Thread Michael Armbrust
Do you have a full stack trace? On Thu, Apr 2, 2015 at 11:45 AM, ogoh wrote: > > Hello, > My ETL uses sparksql to generate parquet files which are served through > Thriftserver using hive ql. > It especially defines a schema programmatically since the schema can be > only > known at runtime. > W