[jira] [Commented] (SPARK-5798) Spark shell issue
[
https://issues.apache.org/jira/browse/SPARK-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320763#comment-14320763
]
DeepakVohra commented on SPARK-5798:
Re-tested on local OS Oracle Linux 6.5 and did not get the Spark shell issue.
The earlier test, which generated the Spark shell error, was on Amazon EC2.
Issue may be closed.
> Spark shell issue
> -
>
> Key: SPARK-5798
> URL: https://issues.apache.org/jira/browse/SPARK-5798
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
>Affects Versions: 1.2.0
> Environment: Spark 1.2
> Scala 2.10.4
>Reporter: DeepakVohra
>
> The Spark shell terminates when Spark code is run indicating an issue with
> Spark shell.
> The error is coming from the spark shell file
>
> /apachespark/spark-1.2.0-bin-cdh4/bin/spark-shell: line 48
>
> "$FWDIR"/bin/spark-submit --class org.apache.spark.repl.Main
> "${SUBMISSION_OPTS[@]}" spark-shell "${APPLICATION_OPTS[@]}"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5798) Spark shell issue
[
https://issues.apache.org/jira/browse/SPARK-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320533#comment-14320533
]
DeepakVohra commented on SPARK-5798:
Thanks Sean for testing.
Not all Spark/Scala code generates an error in Spark Shell.
For example, run all pre-requisite import, var, and method code and
subsequently run the following code to test:
model(sc, rawUserArtistData, rawArtistData, rawArtistAlias)
from:
https://github.com/sryza/aas/blob/master/ch03-recommender/src/main/scala/com/cloudera/datascience/recommender/RunRecommender.scala
Data files are local to Spark/Scala and not in HDFS.
Environment is different: Oracle Linux 6.5, but should't be a factor.
If the preceding test also does not generate an error would agree it is some
other factor and not a bug.
> Spark shell issue
> -
>
> Key: SPARK-5798
> URL: https://issues.apache.org/jira/browse/SPARK-5798
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
>Affects Versions: 1.2.0
> Environment: Spark 1.2
> Scala 2.10.4
>Reporter: DeepakVohra
>
> The Spark shell terminates when Spark code is run indicating an issue with
> Spark shell.
> The error is coming from the spark shell file
>
> /apachespark/spark-1.2.0-bin-cdh4/bin/spark-shell: line 48
>
> "$FWDIR"/bin/spark-submit --class org.apache.spark.repl.Main
> "${SUBMISSION_OPTS[@]}" spark-shell "${APPLICATION_OPTS[@]}"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5798) Spark shell issue
DeepakVohra created SPARK-5798:
--
Summary: Spark shell issue
Key: SPARK-5798
URL: https://issues.apache.org/jira/browse/SPARK-5798
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 1.2.0
Environment: Spark 1.2
Scala 2.10.4
Reporter: DeepakVohra
The Spark shell terminates when Spark code is run indicating an issue with
Spark shell.
The error is coming from the spark shell file
/apachespark/spark-1.2.0-bin-cdh4/bin/spark-shell: line 48
"$FWDIR"/bin/spark-submit --class org.apache.spark.repl.Main
"${SUBMISSION_OPTS[@]}" spark-shell "${APPLICATION_OPTS[@]}"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
[ https://issues.apache.org/jira/browse/SPARK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311597#comment-14311597 ] DeepakVohra edited comment on SPARK-5631 at 2/8/15 10:38 PM: - Thanks for the clarification. The error gets removed. was (Author: dvohra): Thanks for the clarification. > Server IPC version 7 cannot communicate with client version 4 > -- > > Key: SPARK-5631 > URL: https://issues.apache.org/jira/browse/SPARK-5631 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Scala 2.10.4 > Spark 1.2 > CDH4.2 >Reporter: DeepakVohra > > A Spark application generates the error > Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
[ https://issues.apache.org/jira/browse/SPARK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311597#comment-14311597 ] DeepakVohra commented on SPARK-5631: Thanks for the clarification. > Server IPC version 7 cannot communicate with client version 4 > -- > > Key: SPARK-5631 > URL: https://issues.apache.org/jira/browse/SPARK-5631 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Scala 2.10.4 > Spark 1.2 > CDH4.2 >Reporter: DeepakVohra > > A Spark application generates the error > Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311596#comment-14311596 ] DeepakVohra commented on SPARK-5625: Thanks Sean. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
[ https://issues.apache.org/jira/browse/SPARK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311506#comment-14311506 ] DeepakVohra commented on SPARK-5631: "This means you have mismatched Hadoop versions, either between your Spark and Hadoop deployment, " Hadoop version is hadoop-2.0.0-cdh4.2.0.tar.gz. Spark binaries are compiled with the same version: spark-1.2.0-bin-cdh4.tgz "or because you included Hadoop code in your app." The Spark application is the RunRecommender application. > Server IPC version 7 cannot communicate with client version 4 > -- > > Key: SPARK-5631 > URL: https://issues.apache.org/jira/browse/SPARK-5631 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Scala 2.10.4 > Spark 1.2 > CDH4.2 >Reporter: DeepakVohra > > A Spark application generates the error > Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311500#comment-14311500 ] DeepakVohra commented on SPARK-5625: On re-test Spark classes get found in Spark application. But the following error is still generated with RunRecommender. Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 at org.apache.hadoop.ipc.Client.call(Client.java:1113) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422) at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:281) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:245) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1351) at org.apache.spark.rdd.RDD.reduce(RDD.scala:867) at org.apache.spark.rdd.DoubleRDDFunctions.stats(DoubleRDDFunctions.scala:43) at com.cloudera.datascience.recommender.RunRecommender$.preparation(RunRecommender.scala:63) at com.cloudera.datascience.recommender.RunRecommender$.main(RunRecommender.scala:29) at com.cloudera.datascience.recommender.RunRecommender.main(RunRecommender.scala) > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311447#comment-14311447 ] DeepakVohra edited comment on SPARK-5625 at 2/8/15 5:56 PM: The jar tf does list the Spark classes, which verifies the Binaries include the Spark artifact classes. The issue subject should be modified to: Is the Spark Assembly a Valid Archive? Extracting/opening with WinZip is only to verify the archive is valid. The following indicate that the spark assembly jar is not a valid archive. 1. Even though the assembly jar is in the classpath, a Spark application does not find the classes in the assembly jar. 2. The assembly jar does not get opened/extracted with WinZip which generates the error: http://s763.photobucket.com/user/dvohra10/media/SparkAssembly_zps4319294c.jpg.html?o=0 All indicators suggest the assembly jar is not a valid archive. Adding a Spark core artifact jar to the same directory, the lib directory of Spark binaries, adds the classes from the Spark Core to the classpath. Could it be verified: 1. The assembly jar gets extracted and is a valid archive? 2. Adding the jar in the classpath adds the classes to classpath? was (Author: dvohra): Extracting/opening with WinZip is only to verify the archive is valid. The following indicate that the spark assembly jar is not a valid archive. 1. Even though the assembly jar is in the classpath, a Spark application does not find the classes in the assembly jar. 2. The assembly jar does not get opened/extracted with WinZip which generates the error: http://s763.photobucket.com/user/dvohra10/media/SparkAssembly_zps4319294c.jpg.html?o=0 All indicators suggest the assembly jar is not a valid archive. Adding a Spark core artifact jar to the same directory, the lib directory of Spark binaries, adds the classes from the Spark Core to the classpath. Could it be verified: 1. The assembly jar gets extracted and is a valid archive? 2. Adding the jar in the classpath adds the classes to classpath? > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311447#comment-14311447 ] DeepakVohra commented on SPARK-5625: Extracting/opening with WinZip is only to verify the archive is valid. The following indicate that the spark assembly jar is not a valid archive. 1. Even though the assembly jar is in the classpath, a Spark application does not find the classes in the assembly jar. 2. The assembly jar does not get opened/extracted with WinZip which generates the error: http://s763.photobucket.com/user/dvohra10/media/SparkAssembly_zps4319294c.jpg.html?o=0 All indicators suggest the assembly jar is not a valid archive. Adding a Spark core artifact jar to the same directory, the lib directory of Spark binaries, adds the classes from the Spark Core to the classpath. Could it be verified: 1. The assembly jar gets extracted and is a valid archive? 2. Adding the jar in the classpath adds the classes to classpath? > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311348#comment-14311348 ] DeepakVohra edited comment on SPARK-5625 at 2/8/15 3:21 PM: The error is not too many files. The error is the archive is not valid as in the screenshot. http://s763.photobucket.com/user/dvohra10/media/SparkAssembly_zps4319294c.jpg.html?o=0 The other jars in the Spark binaries lib directory get opened/extracted except the assembly jar. Could it be verified that the assembly jar gets extracted? And which extraction tool is used? was (Author: dvohra): The other jars in the Spark binaries lib directory get opened/extracted except the assembly jar. Could it be verified that the assembly jar gets extracted? And which extraction tool is used? > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311348#comment-14311348 ] DeepakVohra commented on SPARK-5625: The other jars in the Spark binaries lib directory get opened/extracted except the assembly jar. Could it be verified that the assembly jar gets extracted? And which extraction tool is used? > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311347#comment-14311347 ] DeepakVohra commented on SPARK-5625: WinZIp version is the latest 18.5. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311076#comment-14311076 ] DeepakVohra commented on SPARK-5625: The spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar is not a valid archive. http://s763.photobucket.com/user/dvohra10/media/SparkAssembly_zps4319294c.jpg.html?o=0 The spark-1.2.0-bin-cdh4.tgz is downloaded from http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-cdh4.tgz > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311073#comment-14311073 ] DeepakVohra commented on SPARK-5625: The spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar has too many classes, which may be causing classloading issue. The classes do not even get extracted with WinZip and generate the following error. Error: too many entries in central directory according to end of central directory info. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5670) Spark artifacts compiled with Hadoop 1.x
[ https://issues.apache.org/jira/browse/SPARK-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311031#comment-14311031 ] DeepakVohra commented on SPARK-5670: Not using Maven to run the Spark application to be able to set "provided" scope. Running Spark application as "local" master URL. > Spark artifacts compiled with Hadoop 1.x > > > Key: SPARK-5670 > URL: https://issues.apache.org/jira/browse/SPARK-5670 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Spark 1.2 >Reporter: DeepakVohra > > Why are Spark artifacts available from Maven compiled with Hadoop 1.x while > the Spark binaries for Hadoop 1.x are not available? Also CDH is not > available for Hadoop 1.x. > Using Hadoop 2.0.0 or Hadoop 2.3 with Spark artifacts generates error such as > the following. > Server IPC version 7 cannot communicate with client version 4 > Server IPC version 9 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311029#comment-14311029 ] DeepakVohra commented on SPARK-5625: Thanks, yes the assembly jar has the Spark artifact classes. Shall re-test as to why the Spark classes are not getting found when a .scala file is compiled even though the spark-1.2.0-bin-cdh4/lib/* is in the classpath. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5670) Spark artifacts compiled with Hadoop 1.x
DeepakVohra created SPARK-5670: -- Summary: Spark artifacts compiled with Hadoop 1.x Key: SPARK-5670 URL: https://issues.apache.org/jira/browse/SPARK-5670 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 1.2.0 Environment: Spark 1.2 Reporter: DeepakVohra Why are Spark artifacts available from Maven compiled with Hadoop 1.x while the Spark binaries for Hadoop 1.x are not available? Also CDH is not available for Hadoop 1.x. Using Hadoop 2.0.0 or Hadoop 2.3 with Spark artifacts generates error such as the following. Server IPC version 7 cannot communicate with client version 4 Server IPC version 9 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311012#comment-14311012 ] DeepakVohra commented on SPARK-5625: Spark artifacts are not in the Spark binaries/assembly. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5531) Spark download .tgz file does not get unpacked
[ https://issues.apache.org/jira/browse/SPARK-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310882#comment-14310882 ] DeepakVohra commented on SPARK-5531: Thanks for the clarification. > Spark download .tgz file does not get unpacked > -- > > Key: SPARK-5531 > URL: https://issues.apache.org/jira/browse/SPARK-5531 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: Linux >Reporter: DeepakVohra > > The spark-1.2.0-bin-cdh4.tgz file downloaded from > http://spark.apache.org/downloads.html does not get unpacked. > tar xvf spark-1.2.0-bin-cdh4.tgz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error is not recoverable: exiting now -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5531) Spark download .tgz file does not get unpacked
[ https://issues.apache.org/jira/browse/SPARK-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310796#comment-14310796 ] DeepakVohra commented on SPARK-5531: Earlier the following link was getting listed with Direct Download. http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-cdh4.tgz Seems to have been updated to the tgz file link, which is fine. > Spark download .tgz file does not get unpacked > -- > > Key: SPARK-5531 > URL: https://issues.apache.org/jira/browse/SPARK-5531 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: Linux >Reporter: DeepakVohra > > The spark-1.2.0-bin-cdh4.tgz file downloaded from > http://spark.apache.org/downloads.html does not get unpacked. > tar xvf spark-1.2.0-bin-cdh4.tgz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error is not recoverable: exiting now -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5531) Spark download .tgz file does not get unpacked
[ https://issues.apache.org/jira/browse/SPARK-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310147#comment-14310147 ] DeepakVohra commented on SPARK-5531: Thanks for updating the download links. > Spark download .tgz file does not get unpacked > -- > > Key: SPARK-5531 > URL: https://issues.apache.org/jira/browse/SPARK-5531 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: Linux >Reporter: DeepakVohra > > The spark-1.2.0-bin-cdh4.tgz file downloaded from > http://spark.apache.org/downloads.html does not get unpacked. > tar xvf spark-1.2.0-bin-cdh4.tgz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error is not recoverable: exiting now -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5531) Spark download .tgz file does not get unpacked
[ https://issues.apache.org/jira/browse/SPARK-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309938#comment-14309938 ] DeepakVohra commented on SPARK-5531: Why two options, Direct Download and Select Apache Mirror, if both direct to the same HTML page? > Spark download .tgz file does not get unpacked > -- > > Key: SPARK-5531 > URL: https://issues.apache.org/jira/browse/SPARK-5531 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: Linux >Reporter: DeepakVohra > > The spark-1.2.0-bin-cdh4.tgz file downloaded from > http://spark.apache.org/downloads.html does not get unpacked. > tar xvf spark-1.2.0-bin-cdh4.tgz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error is not recoverable: exiting now -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309935#comment-14309935 ] DeepakVohra commented on SPARK-5625: Not clear if the assembly jar is to be extracted. Is the assembly jar to be extracted? Because if added as such to classpath the core classes are not found. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5625) Spark binaries do not incude Spark Core
[ https://issues.apache.org/jira/browse/SPARK-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309636#comment-14309636 ] DeepakVohra commented on SPARK-5625: Thanks for the clarification. Would be better if the main artifacts such as the core and mllib are included as these have to be downloaded separately and included in the lib directory. > Spark binaries do not incude Spark Core > --- > > Key: SPARK-5625 > URL: https://issues.apache.org/jira/browse/SPARK-5625 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: CDH4 >Reporter: DeepakVohra > > Spark binaries for CDH 4 do not include the Spark Core Jar. > http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5531) Spark download .tgz file does not get unpacked
[ https://issues.apache.org/jira/browse/SPARK-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309462#comment-14309462 ] DeepakVohra commented on SPARK-5531: If the Select A Mirror option is selected the download is fine. The Direct Download does not download a .tgz file and the error is usually generated if a HTML page to which the link is supposed to redirect to is getting downloaded instead of the .tgz file. > Spark download .tgz file does not get unpacked > -- > > Key: SPARK-5531 > URL: https://issues.apache.org/jira/browse/SPARK-5531 > Project: Spark > Issue Type: Bug >Affects Versions: 1.2.0 > Environment: Linux >Reporter: DeepakVohra > > The spark-1.2.0-bin-cdh4.tgz file downloaded from > http://spark.apache.org/downloads.html does not get unpacked. > tar xvf spark-1.2.0-bin-cdh4.tgz > gzip: stdin: not in gzip format > tar: Child returned status 1 > tar: Error is not recoverable: exiting now -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
[ https://issues.apache.org/jira/browse/SPARK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308475#comment-14308475 ] DeepakVohra commented on SPARK-5631: The Spark is built with Hadoop 2.0 spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar The Hadoop server is also Hadoop 2.0 hadoop-2.0.0-cdh4.6.0 Still getting the error org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4 The same version hadoop2.0.0-mr1-cdh4.2.0 as server also generates the error Server IPC version 7 cannot communicate with client version 4 > Server IPC version 7 cannot communicate with client version 4 > -- > > Key: SPARK-5631 > URL: https://issues.apache.org/jira/browse/SPARK-5631 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Scala 2.10.4 > Spark 1.2 > CDH4.2 >Reporter: DeepakVohra > > A Spark application generates the error > Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3185) SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting JOURNAL_FOLDER
[
https://issues.apache.org/jira/browse/SPARK-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308471#comment-14308471
]
DeepakVohra commented on SPARK-3185:
The Spark is built with Hadoop 2.0
spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar
The Hadoop server is also Hadoop 2.0
hadoop-2.0.0-cdh4.6.0
Still getting the error
org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate
with
client version 4
The same version hadoop2.0.0-mr1-cdh4.2.0 as server also generates the error
Server IPC version 7 cannot communicate with
client version 4
> SPARK launch on Hadoop 2 in EC2 throws Tachyon exception when Formatting
> JOURNAL_FOLDER
> ---
>
> Key: SPARK-3185
> URL: https://issues.apache.org/jira/browse/SPARK-3185
> Project: Spark
> Issue Type: Bug
> Components: EC2
>Affects Versions: 1.0.2
> Environment: Amazon Linux AMI
> [ec2-user@ip-172-30-1-145 ~]$ uname -a
> Linux ip-172-30-1-145 3.10.42-52.145.amzn1.x86_64 #1 SMP Tue Jun 10 23:46:43
> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> https://aws.amazon.com/amazon-linux-ami/2014.03-release-notes/
> The build I used (and MD5 verified):
> [ec2-user@ip-172-30-1-145 ~]$ wget
> http://supergsego.com/apache/spark/spark-1.0.2/spark-1.0.2-bin-hadoop2.tgz
>Reporter: Jeremy Chambers
>
> {code}
> org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
> communicate with client version 4
> {code}
> When I launch SPARK 1.0.2 on Hadoop 2 in a new EC2 cluster, the above tachyon
> exception is thrown when "Formatting JOURNAL_FOLDER".
> No exception occurs when I launch on Hadoop 1.
> Launch used:
> {code}
> ./spark-ec2 -k spark_cluster -i /home/ec2-user/kagi/spark_cluster.ppk
> --zone=us-east-1a --hadoop-major-version=2 --spot-price=0.0165 -s 3 launch
> sparkProd
> {code}
> {code}
> log snippet
> Formatting Tachyon Master @ ec2-54-80-49-244.compute-1.amazonaws.com
> Formatting JOURNAL_FOLDER: /root/tachyon/libexec/../journal/
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
> communicate with client version 4
> at tachyon.util.CommonUtils.runtimeException(CommonUtils.java:246)
> at tachyon.UnderFileSystemHdfs.(UnderFileSystemHdfs.java:73)
> at tachyon.UnderFileSystemHdfs.getClient(UnderFileSystemHdfs.java:53)
> at tachyon.UnderFileSystem.get(UnderFileSystem.java:53)
> at tachyon.Format.main(Format.java:54)
> Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot
> communicate with client version 4
> at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:238)
> at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:203)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
> at tachyon.UnderFileSystemHdfs.(UnderFileSystemHdfs.java:69)
> ... 3 more
> Killed 0 processes
> Killed 0 processes
> ec2-54-167-219-159.compute-1.amazonaws.com: Killed 0 processes
> ec2-54-198-198-17.compute-1.amazonaws.com: Killed 0 processes
> ec2-54-166-36-0.compute-1.amazonaws.com: Killed 0 processes
> ---end snippet---
> {code}
> *I don't have this problem when I launch without the
> "--hadoop-major-version=2" (which defaults to Hadoop 1.x).*
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
[ https://issues.apache.org/jira/browse/SPARK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308011#comment-14308011 ] DeepakVohra commented on SPARK-5631: Which CDH4 version and MR version is used for spark-1.2.0-bin-cdh4.tgz? > Server IPC version 7 cannot communicate with client version 4 > -- > > Key: SPARK-5631 > URL: https://issues.apache.org/jira/browse/SPARK-5631 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Scala 2.10.4 > Spark 1.2 > CDH4.2 >Reporter: DeepakVohra > > A Spark application generates the error > Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
[ https://issues.apache.org/jira/browse/SPARK-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308005#comment-14308005 ] DeepakVohra commented on SPARK-5631: The error is due to a different version of CDH4 and Hadoop used for spark-1.2.0-bin-cdh4.tgz than in the Hadoop client, which is CDH4.2 MR1. The download does not mention which version of CDH4 and Hadoop 1.0/2.0 is used. If the version is not mentioned should imply any CDH4 version may be used, but CDH 4.2 generates an error. http://spark.apache.org/downloads.html > Server IPC version 7 cannot communicate with client version 4 > -- > > Key: SPARK-5631 > URL: https://issues.apache.org/jira/browse/SPARK-5631 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 1.2.0 > Environment: Scala 2.10.4 > Spark 1.2 > CDH4.2 >Reporter: DeepakVohra > > A Spark application generates the error > Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5631) Server IPC version 7 cannot communicate with client version 4
DeepakVohra created SPARK-5631: -- Summary: Server IPC version 7 cannot communicate with client version 4 Key: SPARK-5631 URL: https://issues.apache.org/jira/browse/SPARK-5631 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 1.2.0 Environment: Scala 2.10.4 Spark 1.2 CDH4.2 Reporter: DeepakVohra A Spark application generates the error Server IPC version 7 cannot communicate with client version 4 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5625) Spark binaries do not incude Spark Core
DeepakVohra created SPARK-5625: -- Summary: Spark binaries do not incude Spark Core Key: SPARK-5625 URL: https://issues.apache.org/jira/browse/SPARK-5625 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 1.2.0 Environment: CDH4 Reporter: DeepakVohra Spark binaries for CDH 4 do not include the Spark Core Jar. http://spark.apache.org/downloads.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5531) Spark download .tgz file does not get unpacked
[
https://issues.apache.org/jira/browse/SPARK-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302670#comment-14302670
]
DeepakVohra commented on SPARK-5531:
The following url does not download a valid ("not in gzip format") tgz file.
http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-cdh4.tgz
> Spark download .tgz file does not get unpacked
> --
>
> Key: SPARK-5531
> URL: https://issues.apache.org/jira/browse/SPARK-5531
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.2.0
> Environment: Linux
>Reporter: DeepakVohra
>
> The spark-1.2.0-bin-cdh4.tgz file downloaded from
> http://spark.apache.org/downloads.html does not get unpacked.
> tar xvf spark-1.2.0-bin-cdh4.tgz
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5531) Spark download .tgz file does not get unpacked
DeepakVohra created SPARK-5531: -- Summary: Spark download .tgz file does not get unpacked Key: SPARK-5531 URL: https://issues.apache.org/jira/browse/SPARK-5531 Project: Spark Issue Type: Bug Affects Versions: 1.2.0 Environment: Linux Reporter: DeepakVohra The spark-1.2.0-bin-cdh4.tgz file downloaded from http://spark.apache.org/downloads.html does not get unpacked. tar xvf spark-1.2.0-bin-cdh4.tgz gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[
https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300243#comment-14300243
]
DeepakVohra commented on SPARK-2356:
Thanks Sean.
HADOOP_CONF_DIR shouldn't be required to be set if Hadoop is not used.
Hadoop doesn't even get installed on Windows.
> Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
> ---
>
> Key: SPARK-2356
> URL: https://issues.apache.org/jira/browse/SPARK-2356
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.0.0
>Reporter: Kostiantyn Kudriavtsev
>Priority: Critical
>
> I'm trying to run some transformation on Spark, it works fine on cluster
> (YARN, linux machines). However, when I'm trying to run it on local machine
> (Windows 7) under unit test, I got errors (I don't use Hadoop, I'm read file
> from local filesystem):
> {code}
> 14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the
> hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
> at org.apache.hadoop.util.Shell.(Shell.java:326)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
> at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
> at org.apache.hadoop.security.Groups.(Groups.java:77)
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
> at
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
> at
> org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:36)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:109)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala)
> at org.apache.spark.SparkContext.(SparkContext.scala:228)
> at org.apache.spark.SparkContext.(SparkContext.scala:97)
> {code}
> It's happened because Hadoop config is initialized each time when spark
> context is created regardless is hadoop required or not.
> I propose to add some special flag to indicate if hadoop config is required
> (or start this configuration manually)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-2356) Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
[
https://issues.apache.org/jira/browse/SPARK-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299971#comment-14299971
]
DeepakVohra commented on SPARK-2356:
Following error gets generated on Windows with master url as "local" for KMeans
clustering. But the application completes without any other error.
java.io.IOException: Could not locate executable null\bin\winutils.exe in the
Hadoop
binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.(Shell.java:293)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths
(FileInputFormat.java:362)
at
org.apache.spark.SparkContext$$anonfun$26.apply(SparkContext.scala:696)
at
org.apache.spark.SparkContext$$anonfun$26.apply(SparkContext.scala:696)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply
(HadoopRDD.scala:170)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply
(HadoopRDD.scala:170)
at scala.Option.map(Option.scala:145)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:170)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.rdd.ZippedPartitionsBaseRDD.getPartitions
(ZippedPartitionsRDD.scala:55)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328)
at org.apache.spark.rdd.RDD.count(RDD.scala:910)
at org.apache.spark.rdd.RDD.takeSample(RDD.scala:403)
at org.apache.spark.mllib.clustering.KMeans.initKMeansParallel
(KMeans.scala:277)
at
org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155)
at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132)
at org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352)
at org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362)
at org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala)
at kmeans.KMeansClusterer.main(KMeansClusterer.java:40)
> Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
> ---
>
> Key: SPARK-2356
> URL: https://issues.apache.org/jira/browse/SPARK-2356
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.0.0
>Reporter: Kostiantyn Kudriavtsev
>Priority: Critical
>
> I'm trying to run some transformation on Spark, it works fine on cluster
> (YARN, linux machines). However, when I'm trying to run it on local machine
> (Windows 7) under unit test, I got errors (I don't use Hadoop, I'm read file
> from local filesystem):
> {code}
> 14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the
> hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in the
> Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
>
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299889#comment-14299889 ] DeepakVohra edited comment on SPARK-5483 at 1/31/15 4:54 PM: - Tests fine on Linux and Windows with mvn compile. Did not import, but created a new Maven project in Eclipse IDE. Is a different repository used on Linux? And the compiler error gets removed by modifying "2.10" to "2.11". Not a Spark issue, and may be closed if no further feedback. was (Author: dvohra): Tests fine on Windows with mvn compile. Did not import, but created a new Maven project. Is a different repository used on Linux? And the compiler error gets removed by modifying "2.10" to "2.11". Not a Spark issue, and may be closed if no further feedback. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > Attachments: test.tar.gz > > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55)
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299889#comment-14299889 ] DeepakVohra commented on SPARK-5483: Tests fine on Windows. Did not import, but created a new Maven project. Is a different repository used on Linux? And the compiler error gets removed by modifying "2.10" to "2.11". Not a Spark issue, and may be closed if no further feedback. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > Attachments: test.tar.gz > > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299889#comment-14299889 ] DeepakVohra edited comment on SPARK-5483 at 1/31/15 4:44 PM: - Tests fine on Windows with mvn compile. Did not import, but created a new Maven project. Is a different repository used on Linux? And the compiler error gets removed by modifying "2.10" to "2.11". Not a Spark issue, and may be closed if no further feedback. was (Author: dvohra): Tests fine on Windows. Did not import, but created a new Maven project. Is a different repository used on Linux? And the compiler error gets removed by modifying "2.10" to "2.11". Not a Spark issue, and may be closed if no further feedback. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > Attachments: test.tar.gz > > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299880#comment-14299880 ] DeepakVohra commented on SPARK-5483: Sean, Thanks for testing. Which OS is used? The error is generated in Linux(Eclipse Luna). Screenshots pom.xml http://s763.photobucket.com/user/dvohra10/media/pom_zpsd5ef4491.jpg.html?filters[user]=137762904&filters[recent]=1&sort=1&o=1 Test.java http://s763.photobucket.com/user/dvohra10/media/Test_zps3970c465.jpg.html?filters[user]=137762904&filters[recent]=1&sort=1&o=0 OS Shouldn't make a difference. Tested in Windows (Eclipse Luna) and no compiler error is generated. thanks, Deepak > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > Attachments: test.tar.gz > > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classif
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[
https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299571#comment-14299571
]
DeepakVohra commented on SPARK-5483:
To test:
1. Create a Maven project in Eclipse IDE.
2. Add the Spark MLLib dependency 2.10.
org.apache.spark
spark-mllib_2.10
1.2.0
3. Add a Java class to the Maven project.
public class Test{}
4. Add the following import statements to the Java class Test.
import org.apache.spark.mllib.clustering.KMeans;
import org.apache.spark.mllib.clustering.KMeansModel;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
> java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
> ---
>
> Key: SPARK-5483
> URL: https://issues.apache.org/jira/browse/SPARK-5483
> Project: Spark
> Issue Type: Bug
> Components: MLlib
>Affects Versions: 1.2.0
> Environment: Maven
> Spark 1.2
>Reporter: DeepakVohra
>
> Naive Bayes classifier generates following error.
> ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
> at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188)
> at
> breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303)
> at
> breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87)
> at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303)
> at
> breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38)
> at
> breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22)
> at breeze.linalg.DenseVector$.(DenseVector.scala:225)
> at breeze.linalg.DenseVector$.(DenseVector.scala)
> at breeze.linalg.DenseVector.(DenseVector.scala:63)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55)
> at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329)
> at
> org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112)
> at
> org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199)
> at
> org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142)
> at
> org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
> at
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in
> thread Thread[Executor task launch worker-0,5,main]
> java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
> at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188)
> at
> breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303)
> at
> breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87)
> at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303)
> at
> breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38)
> at
> breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22)
> at breeze.linalg.DenseVector$.(DenseVector.scala:225)
> at breeze.linalg.DenseVector$.(DenseVector.scala)
> at breeze.linalg.DenseVector.(DenseVector.scala:63)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55)
> at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329)
> at
> org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:11
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299009#comment-14299009 ] DeepakVohra edited comment on SPARK-5483 at 1/30/15 7:09 PM: - Thanks for the clarification about why the master URL is not set. Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue and the org.apache.spark.mllib.* packages get found, but introduces the Scala version issue. was (Author: dvohra): Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue and the org.apache.spark.mllib.* packages get found, but introduces the Scala version issue. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299009#comment-14299009 ] DeepakVohra edited comment on SPARK-5483 at 1/30/15 6:31 PM: - Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue and the org.apache.spark.mllib.* packages get found, but introduces the Scala version issue. was (Author: dvohra): Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue of the org.apache.spark.mllib.* packages being found. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) >
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299009#comment-14299009 ] DeepakVohra edited comment on SPARK-5483 at 1/30/15 6:30 PM: - Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue of the org.apache.spark.mllib.* packages being found. was (Author: dvohra): Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299009#comment-14299009 ] DeepakVohra commented on SPARK-5483: Is the Maven dependency >spark-mllib_2.10 not a Spark issue? org.apache.spark spark-mllib_2.10 1.2.0 There must be some issue in how you are adding the classes to the classpath. Modifying "2.10" to "2.11" fixes the issue. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter
[jira] [Commented] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299001#comment-14299001 ] DeepakVohra commented on SPARK-5489: Already did before posting the previous message and the jar does have the classes, but are indicated as not found with the Maven dependency. Gets fixed with MLLib 2.11. The Maven dependency MLlib 2.10 has some issue. > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298823#comment-14298823 ] DeepakVohra edited comment on SPARK-5483 at 1/30/15 4:41 PM: - The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.* packages, which it should according to http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.classification.NaiveBayes; import org.apache.spark.mllib.classification.NaiveBayesModel; import org.apache.spark.mllib.linalg.Vectors; import org.apache.spark.mllib.regression.LabeledPoint; import org.apache.spark.mllib.util.MLUtils; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. was (Author: dvohra): The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.clustering packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.classification.NaiveBayes; import org.apache.spark.mllib.classification.NaiveBayesModel; import org.apache.spark.mllib.linalg.Vectors; import org.apache.spark.mllib.regression.LabeledPoint; import org.apache.spark.mllib.util.MLUtils; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExce
[jira] [Comment Edited] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298824#comment-14298824 ] DeepakVohra edited comment on SPARK-5489 at 1/30/15 4:40 PM: - The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.* packages, which it should according to http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. was (Author: dvohra): The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.* packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298824#comment-14298824 ] DeepakVohra edited comment on SPARK-5489 at 1/30/15 4:40 PM: - The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.* packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. was (Author: dvohra): The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.clustering packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298823#comment-14298823 ] DeepakVohra edited comment on SPARK-5483 at 1/30/15 4:39 PM: - The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.clustering packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.classification.NaiveBayes; import org.apache.spark.mllib.classification.NaiveBayesModel; import org.apache.spark.mllib.linalg.Vectors; import org.apache.spark.mllib.regression.LabeledPoint; import org.apache.spark.mllib.util.MLUtils; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. was (Author: dvohra): The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.clustering packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread
[jira] [Commented] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298824#comment-14298824 ] DeepakVohra commented on SPARK-5489: The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.clustering packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298823#comment-14298823 ] DeepakVohra commented on SPARK-5483: The issue is with the Maven dependency org.apache.spark spark-mllib_2.10 1.2.0 spark-mllib_2.10 does not include the org.apache.spark.mllib.clustering packages, which it should according to the Maven http://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10/1.2.0 Generates error at import statements: import org.apache.spark.mllib.clustering.KMeans; import org.apache.spark.mllib.clustering.KMeansModel; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.mllib.linalg.Vectors; "The import org.apache.spark.mllib cannot be resolved". The 2.11 version spark-mllib_2.11 fixes the error but seems to be referring Scala 2.11. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(De
[jira] [Commented] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298063#comment-14298063 ] DeepakVohra commented on SPARK-5489: If Scala 2.11.1 is used the scala.Cloneable is not found, which is available in Scala 2.10.4, but not not Scala 2.11.1. > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297951#comment-14297951 ] DeepakVohra commented on SPARK-5489: Sean, Some dependency is making use of scala.runtime.IntRef.create, which was introduced in Scala 2.11. https://github.com/scala/scala/blob/v2.11.0/src/library/scala/runtime/IntRef.java Scala 2.10.4, which is included with Spark 1.2, does not include the scala.runtime.IntRef.create method. https://github.com/scala/scala/blob/v2.10.4/src/library/scala/runtime/IntRef.java thanks, Deepak > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
[ https://issues.apache.org/jira/browse/SPARK-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297937#comment-14297937 ] DeepakVohra commented on SPARK-5489: Sean, Made the Scala version the same, but still getting the error. "For the Scala API, Spark 1.2.0 uses Scala 2.10. " http://spark.apache.org/docs/1.2.0/ Made Maven dependencies Scala version also 2.10. org.apache.spark spark-core_2.10 1.2.0 org.scala-lang scala-library org.scala-lang scala-compiler org.apache.spark spark-mllib_2.11 1.2.0 org.scala-lang scala-library org.scala-lang scala-compiler org.scala-lang scala-library 2.10.0 org.scala-lang scala-compiler 2.10.0 thanks, Deepak > KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > - > > Key: SPARK-5489 > URL: https://issues.apache.org/jira/browse/SPARK-5489 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Spark 1.2 > Maven >Reporter: DeepakVohra > > The KMeans clustering generates following error, which also seems to be due > version mismatch between Scala used for compiling Spark and Scala in Spark > 1.2 Maven dependency. > Exception in thread "main" java.lang.NoSuchMethodError: > scala.runtime.IntRef.create > (I)Lscala/runtime/IntRef; > at > org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) > at > org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) > at > org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) > at > org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) > at > org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) > at > clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297936#comment-14297936 ] DeepakVohra commented on SPARK-5483: Sean, Made the Scala version the same, but still getting the error. "For the Scala API, Spark 1.2.0 uses Scala 2.10. " http://spark.apache.org/docs/1.2.0/ Made Maven dependencies Scala version also 2.10. org.apache.spark spark-core_2.10 1.2.0 org.scala-lang scala-library org.scala-lang scala-compiler org.apache.spark spark-mllib_2.11 1.2.0 org.scala-lang scala-library org.scala-lang scala-compiler org.scala-lang scala-library 2.10.0 org.scala-lang scala-compiler 2.10.0 thanks, Deepak > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Th
[jira] [Created] (SPARK-5489) KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef;
DeepakVohra created SPARK-5489: -- Summary: KMeans clustering java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef; Key: SPARK-5489 URL: https://issues.apache.org/jira/browse/SPARK-5489 Project: Spark Issue Type: Bug Components: MLlib Affects Versions: 1.2.0 Environment: Spark 1.2 Maven Reporter: DeepakVohra The KMeans clustering generates following error, which also seems to be due version mismatch between Scala used for compiling Spark and Scala in Spark 1.2 Maven dependency. Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.IntRef.create (I)Lscala/runtime/IntRef; at org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:282) at org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:155) at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:132) at org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:352) at org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:362) at org.apache.spark.mllib.clustering.KMeans.train(KMeans.scala) at clusterer.kmeans.KMeansClusterer.main(KMeansClusterer.java:35) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297886#comment-14297886 ] DeepakVohra commented on SPARK-5483: Sean, As indicated Spark is compiled with Scala 2.10, but the Scala version packaged in Maven Spark 1.2 is 2.10.4, which seems to be causing version mismatch and the error. Spark 1.2 should be packaged with Scala 2.10 instead of 2.10.4. thanks, Deepak > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at
[jira] [Comment Edited] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297202#comment-14297202 ] DeepakVohra edited comment on SPARK-5483 at 1/29/15 5:37 PM: - Run as Java application. The scala version in build path added from dependencies is 2.10.4. was (Author: dvohra): Run a s Java application. The scala version in build path added from dependencies is 2.10.4. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297202#comment-14297202 ] DeepakVohra commented on SPARK-5483: Run a s Java application. The scala version in build path added from dependencies is 2.10.4. > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[
https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297143#comment-14297143
]
DeepakVohra commented on SPARK-5483:
Just using the following dependencies with no compiler error.
org.apache.spark
spark-core_2.10
1.2.0
org.apache.spark
spark-mllib_2.11
1.2.0
Spark Master URL : "local".
On a related issue the Classification examples don't set a master url.
SparkConf conf = new SparkConf().setAppName("SVM Classifier Example");
should be replaced with:
SparkConf conf = new SparkConf().setAppName("SVM Classifier
Example").setMaster("master url");
http://spark.apache.org/docs/1.2.0/mllib-linear-methods.html#binary-classification
> java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
> ---
>
> Key: SPARK-5483
> URL: https://issues.apache.org/jira/browse/SPARK-5483
> Project: Spark
> Issue Type: Bug
> Components: MLlib
>Affects Versions: 1.2.0
> Environment: Maven
> Spark 1.2
>Reporter: DeepakVohra
>
> Naive Bayes classifier generates following error.
> ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
> at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188)
> at
> breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303)
> at
> breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87)
> at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303)
> at
> breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38)
> at
> breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22)
> at breeze.linalg.DenseVector$.(DenseVector.scala:225)
> at breeze.linalg.DenseVector$.(DenseVector.scala)
> at breeze.linalg.DenseVector.(DenseVector.scala:63)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55)
> at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329)
> at
> org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112)
> at
> org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200)
> at
> org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199)
> at
> org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142)
> at
> org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
> at
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205)
> at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:56)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in
> thread Thread[Executor task launch worker-0,5,main]
> java.lang.NoSuchMethodError:
> scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
> at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188)
> at
> breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303)
> at
> breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87)
> at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303)
> at
> breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38)
> at
> breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22)
> at breeze.linalg.DenseVector$.(DenseVector.scala:225)
> at breeze.linalg.DenseVector$.(DenseVector.scala)
> at breeze.linalg.DenseVector.(DenseVector.scala:63)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50)
> at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55)
> at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329)
>
[jira] [Commented] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
[ https://issues.apache.org/jira/browse/SPARK-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297099#comment-14297099 ] DeepakVohra commented on SPARK-5483: The following statement generates the error. final NaiveBayesModel model = NaiveBayes.train(training.rdd(), 1.0); > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > --- > > Key: SPARK-5483 > URL: https://issues.apache.org/jira/browse/SPARK-5483 > Project: Spark > Issue Type: Bug > Components: MLlib >Affects Versions: 1.2.0 > Environment: Maven > Spark 1.2 >Reporter: DeepakVohra > > Naive Bayes classifier generates following error. > ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) > at > org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) > at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) > at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in > thread Thread[Executor task launch worker-0,5,main] > java.lang.NoSuchMethodError: > scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; > at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) > at > breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) > at > breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) > at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) > at > breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) > at > breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) > at breeze.linalg.DenseVector$.(DenseVector.scala:225) > at breeze.linalg.DenseVector$.(DenseVector.scala) > at breeze.linalg.DenseVector.(DenseVector.scala:63) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) > at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) > at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) > at > org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) > at > org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) > at > org.apache.spark.util.collection.Append
[jira] [Created] (SPARK-5483) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
DeepakVohra created SPARK-5483: -- Summary: java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; Key: SPARK-5483 URL: https://issues.apache.org/jira/browse/SPARK-5483 Project: Spark Issue Type: Bug Components: MLlib Affects Versions: 1.2.0 Environment: Maven Spark 1.2 Reporter: DeepakVohra Naive Bayes classifier generates following error. ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) at breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) at breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) at breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) at breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) at breeze.linalg.DenseVector$.(DenseVector.scala:225) at breeze.linalg.DenseVector$.(DenseVector.scala) at breeze.linalg.DenseVector.(DenseVector.scala:63) at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) at org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) at org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) at org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) at org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) at org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) at org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/01/28 21:50:06 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main] java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object; at breeze.generic.MMRegistry2$class.register(Multimethod.scala:188) at breeze.linalg.VectorOps$$anon$1.breeze$linalg$operators$BinaryRegistry$$super$register(Vector.scala:303) at breeze.linalg.operators.BinaryRegistry$class.register(BinaryOp.scala:87) at breeze.linalg.VectorOps$$anon$1.register(Vector.scala:303) at breeze.linalg.operators.DenseVectorOps$$anon$1.(DenseVectorOps.scala:38) at breeze.linalg.operators.DenseVectorOps$class.$init$(DenseVectorOps.scala:22) at breeze.linalg.DenseVector$.(DenseVector.scala:225) at breeze.linalg.DenseVector$.(DenseVector.scala) at breeze.linalg.DenseVector.(DenseVector.scala:63) at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:50) at breeze.linalg.DenseVector$mcD$sp.(DenseVector.scala:55) at org.apache.spark.mllib.linalg.DenseVector.toBreeze(Vectors.scala:329) at org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:112) at org.apache.spark.mllib.classification.NaiveBayes$$anonfun$3.apply(NaiveBayes.scala:110) at org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:200) at org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:199) at org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:142) at org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205) at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:58) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) a
[jira] [Commented] (SPARK-5471) java.lang.NumberFormatException: For input string:
[ https://issues.apache.org/jira/browse/SPARK-5471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296307#comment-14296307 ] DeepakVohra commented on SPARK-5471: Not a bug. The sample data has to be split at the ,. > java.lang.NumberFormatException: For input string: > > > Key: SPARK-5471 > URL: https://issues.apache.org/jira/browse/SPARK-5471 > Project: Spark > Issue Type: New Feature >Affects Versions: 1.2.0 > Environment: Spark 1.2.0 Maven >Reporter: DeepakVohra > > Naive Bayes Classifier generates exception with sample_naive_bayes_data.txt > java.lang.NumberFormatException: For input string: "0,1" > at > sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) > at java.lang.Double.parseDouble(Double.java:540) > at > scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232) > at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31) > at > org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:79) > at > org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:77) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249) > at > org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:163) > at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) > at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:13:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, > localhost): java.lang.NumberFormatException: For input string: "0,1" > at > sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) > at java.lang.Double.parseDouble(Double.java:540) > at > scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232) > at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31) > at > org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:79) > at > org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:77) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) > at > org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249) > at > org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:163) > at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) > at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/01/28 21:13:57 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; > aborting job > 15/01/28 21:13:57 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks > have all completed, from pool > 15/01/28 21:13:57 INFO TaskSchedulerImpl: Cancelling stage 0 > 15/01/28 21:13:57 INFO DAGScheduler: Job 0 failed: reduce at > MLUtils.scala:96, took 1.180869 s > Exception in thread "main" org.apache.spark.SparkException: Job aborted due > to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: > Lost task 0.0 in stage 0.0 (TID 0, localhost): > java.lang.NumberFormatException: For input string: "0,1" > at > sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) > at java.lang.Double.parseDouble(Double.java:540) > at > scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232) > at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31) > at > org.apache.spa
[jira] [Created] (SPARK-5471) java.lang.NumberFormatException: For input string:
DeepakVohra created SPARK-5471: -- Summary: java.lang.NumberFormatException: For input string: Key: SPARK-5471 URL: https://issues.apache.org/jira/browse/SPARK-5471 Project: Spark Issue Type: New Feature Affects Versions: 1.2.0 Environment: Spark 1.2.0 Maven Reporter: DeepakVohra Naive Bayes Classifier generates exception with sample_naive_bayes_data.txt java.lang.NumberFormatException: For input string: "0,1" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232) at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31) at org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:79) at org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:77) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:163) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/01/28 21:13:57 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NumberFormatException: For input string: "0,1" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232) at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31) at org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:79) at org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:77) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249) at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:163) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) at org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15/01/28 21:13:57 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job 15/01/28 21:13:57 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 15/01/28 21:13:57 INFO TaskSchedulerImpl: Cancelling stage 0 15/01/28 21:13:57 INFO DAGScheduler: Job 0 failed: reduce at MLUtils.scala:96, took 1.180869 s Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NumberFormatException: For input string: "0,1" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike$class.toDouble(StringLike.scala:232) at scala.collection.immutable.StringOps.toDouble(StringOps.scala:31) at org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:79) at org.apache.spark.mllib.util.MLUtils$$anonfun$4.apply(MLUtils.scala:77) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249) at org.apache.spark.CacheManager.putInBl
