Michael, You are right. The build brought " org.scala-lang:scala-library:2.10.1" from other package (as below). It works fine after excluding the old scala version. Thanks a lot, Okehee
== dependency: | +--- org.apache.kafka:kafka_2.10:0.8.1.1 | | +--- com.yammer.metrics:metrics-core:2.2.0 | | | \--- org.slf4j:slf4j-api:1.7.2 -> 1.7.7 | | +--- org.xerial.snappy:snappy-java:1.0.5 | | +--- org.apache.zookeeper:zookeeper:3.3.4 -> 3.4.5 | | | +--- org.slf4j:slf4j-api:1.6.1 -> 1.7.7 | | | +--- log4j:log4j:1.2.15 | | | +--- jline:jline:0.9.94 | | | \--- org.jboss.netty:netty:3.2.2.Final | | +--- net.sf.jopt-simple:jopt-simple:3.2 -> 4.6 | | +--- org.scala-lang:scala-library:2.10.1 On Thu, Apr 2, 2015 at 4:45 PM, Michael Armbrust <mich...@databricks.com> wrote: > This looks to me like you have incompatible versions of scala on your > classpath? > > On Thu, Apr 2, 2015 at 4:28 PM, Okehee Goh <oke...@gmail.com> wrote: > >> yes, below is the stacktrace. >> Thanks, >> Okehee >> >> java.lang.NoSuchMethodError: >> scala.reflect.NameTransformer$.LOCAL_SUFFIX_STRING()Ljava/lang/String; >> at scala.reflect.internal.StdNames$CommonNames.<init>(StdNames.scala:97) >> at scala.reflect.internal.StdNames$Keywords.<init>(StdNames.scala:203) >> at scala.reflect.internal.StdNames$TermNames.<init>(StdNames.scala:288) >> at scala.reflect.internal.StdNames$nme$.<init>(StdNames.scala:1045) >> at >> scala.reflect.internal.SymbolTable.nme$lzycompute(SymbolTable.scala:16) >> at scala.reflect.internal.SymbolTable.nme(SymbolTable.scala:16) >> at scala.reflect.internal.StdNames$class.$init$(StdNames.scala:1041) >> at scala.reflect.internal.SymbolTable.<init>(SymbolTable.scala:16) >> at scala.reflect.runtime.JavaUniverse.<init>(JavaUniverse.scala:16) >> at scala.reflect.runtime.package$.universe$lzycompute(package.scala:17) >> at scala.reflect.runtime.package$.universe(package.scala:17) >> at org.apache.spark.sql.types.NativeType.<init>(dataTypes.scala:337) >> at org.apache.spark.sql.types.StringType.<init>(dataTypes.scala:351) >> at org.apache.spark.sql.types.StringType$.<init>(dataTypes.scala:367) >> at org.apache.spark.sql.types.StringType$.<clinit>(dataTypes.scala) >> at org.apache.spark.sql.types.DataTypes.<clinit>(DataTypes.java:30) >> at >> com.quixey.dataengine.dataprocess.parser.ToTableRecord.generateTableSchemaForSchemaRDD(ToTableRecord.java:282) >> at >> com.quixey.dataengine.dataprocess.parser.ToUDMTest.generateTableSchemaTest(ToUDMTest.java:132) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:483) >> at >> org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85) >> at org.testng.internal.Invoker.invokeMethod(Invoker.java:696) >> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:882) >> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1189) >> at >> org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124) >> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108) >> at org.testng.TestRunner.privateRun(TestRunner.java:767) >> at org.testng.TestRunner.run(TestRunner.java:617) >> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348) >> at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343) >> at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305) >> at org.testng.SuiteRunner.run(SuiteRunner.java:254) >> at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) >> at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) >> at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224) >> at org.testng.TestNG.runSuitesLocally(TestNG.java:1149) >> at org.testng.TestNG.run(TestNG.java:1057) >> at >> org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.stop(TestNGTestClassProcessor.java:115) >> at >> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:57) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:483) >> at >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) >> at >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) >> at >> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) >> at >> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) >> at com.sun.proxy.$Proxy2.stop(Unknown Source) >> at >> org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:115) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:483) >> at >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) >> at >> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) >> at >> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355) >> at >> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> >> On Thu, Apr 2, 2015 at 2:51 PM, Michael Armbrust <mich...@databricks.com> >> wrote: >> >>> Do you have a full stack trace? >>> >>> On Thu, Apr 2, 2015 at 11:45 AM, ogoh <oke...@gmail.com> wrote: >>> >>>> >>>> Hello, >>>> My ETL uses sparksql to generate parquet files which are served through >>>> Thriftserver using hive ql. >>>> It especially defines a schema programmatically since the schema can be >>>> only >>>> known at runtime. >>>> With spark 1.2.1, it worked fine (followed >>>> >>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema >>>> ). >>>> >>>> I am trying to migrate into spark 1.3.0, but the API are confusing. >>>> I am not sure if the example of >>>> >>>> https://spark.apache.org/docs/latest/sql-programming-guide.html#programmatically-specifying-the-schema >>>> is still valid on Spark1.3.0? >>>> For example, DataType.StringType is not there any more. >>>> Instead, I found DataTypes.StringType etc. So, I migrated as below and >>>> it >>>> builds fine. >>>> But at runtime, it throws Exception. >>>> >>>> I appreciate any help. >>>> Thanks, >>>> Okehee >>>> >>>> == Exception thrown >>>> java.lang.reflect.InvocationTargetException >>>> scala.reflect.NameTransformer$.LOCAL_SUFFIX_STRING()Ljava/lang/String; >>>> java.lang.NoSuchMethodError: >>>> scala.reflect.NameTransformer$.LOCAL_SUFFIX_STRING()Ljava/lang/String; >>>> >>>> ==== my code's snippet >>>> import org.apache.spark.sql.types.DataTypes; >>>> DataTypes.createStructField(property, DataTypes.IntegerType, true) >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Generating-a-schema-in-Spark-1-3-failed-while-using-DataTypes-tp22362.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >
