The problem should be "toMap", as I tested that "val maps2=maps.collect"
runs ok. When I run spark-shell, I run with "--master
mesos://cluster-1:5050" parameter which is the same with "spark-submit".
Confused here.
2015-07-22 20:01 GMT-05:00 Yana Kadiyska <yana.kadiy...@gmail.com>:
> Is it complaining about "collect" or "toMap"? In either case this error is
> indicative of an old version usually -- any chance you have an old
> installation of Spark somehow? Or scala? You can try running spark-submit
> with --verbose. Also, when you say it runs with spark-shell do you run
> spark shell in local mode or with --master? I'd try with --master <whatever
> master you use for spark-submit>
>
> Also, if you're using standalone mode I believe the worker log contains
> the launch command for the executor -- you probably want to examine that
> classpath carefully
>
> On Wed, Jul 22, 2015 at 5:25 PM, Dan Dong <dongda...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a simple test spark program as below, the strange thing is that
>> it runs well under a spark-shell, but will get a runtime error of
>>
>> java.lang.NoSuchMethodError:
>>
>> in spark-submit, which indicate the line of:
>>
>> val maps2=maps.collect.toMap
>>
>> has problem. But why the compilation has no problem and it works well
>> under spark-shell(==>maps2: scala.collection.immutable.Map[Int,String] =
>> Map(269953 -> once, 97 -> a, 451002 -> upon, 117481 -> was, 226916 ->
>> there, 414413 -> time, 146327 -> king) )? Thanks!
>>
>> import org.apache.spark.SparkContext._
>> import org.apache.spark.SparkConf
>> import org.apache.spark.mllib.feature.HashingTF
>> import org.apache.spark.mllib.linalg.Vector
>> import org.apache.spark.rdd.RDD
>> import org.apache.spark.SparkContext
>> import org.apache.spark._
>> import SparkContext._
>>
>>
>> val docs=sc.parallelize(Array(Array("once" ,"upon", "a", "time"),
>> Array("there", "was", "a", "king")))
>>
>> val hashingTF = new HashingTF()
>>
>> val maps=docs.flatMap{term=>term.map(ele=>(hashingTF.indexOf(ele),ele))}
>>
>> val maps2=maps.collect.toMap
>>
>>
>> Cheers,
>>
>> Dan
>>
>>
>
