Partitioning a libsvm format file
Hi, I am using spark-scala system to train distributed svm. For training svm I am using the files in LIBSVM format. I want to partition a file into fixed number of partititions, with each partition having equal number of datapoints(assume that the number of datapoints in the file is exactly divisible by number of partitions). How could I do that using spark-scala. Please help. Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Partitioning-a-libsvm-format-file-tp11852.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Problem in running mosek in spark cluster - java.lang.UnsatisfiedLinkError: no mosekjava7_0 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)
We have a small apache spark cluster of 6 computers. We are trying to solve a distributed problem which requires solving a optimization problem at each machine during a spark map operation. We decided to use mosek as the solver and I collected an academic license to this end. We observed that mosek works fine in a single system. However, when we prepare a jar file, include the mosek.jar into the library and try to run the jar in the cluster as a spark job it gives errors. java.lang.UnsatisfiedLinkError: no mosekjava7_0 in java.library.path Does this problem has any thing to do with the license? We have set the necessary path variables i n the profile of the user in the master machine but we are not sure about what changes needs to be made to the other machines in the cluster. We shall be greatly obliged if you please suggest the necessary solution and help us out. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-in-running-mosek-in-spark-cluster-java-lang-UnsatisfiedLinkError-no-mosekjava7-0-in-java-lib-tp13799.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
How to set java.library.path in a spark cluster
Hi,
I am working on a 3 machine cloudera cluster. Whenever I submit a spark job
as a jar file with native dependency on mosek it shows the following error.
java.lang.UnsatisfiedLinkError: no mosekjava7_0 in java.library.path
How should I set the java.library.path. I printed the environment variable
and it shows;
-Djava.library.path= -Xms512m -Xmx512m,
I added the following lines to the spark-env.sh file but it was of no help.
The path contains both the mosek.jar and the libmosek_7.0.so files.
export
SPARK_LIBRARY_PATH=${SPARK_HOME}/lib:/home/chanda/mosek/7/tools/platform/linux64x86/bin
export
SPARK_MASTER_OPTS='-Djava.library.path="/home/chanda/mosek/7/tools/platform/linux64x86/bin'
export
SPARK_WORKER_OPTS='/home/chanda/mosek/7/tools/platform/linux64x86/bin'
export
SPARK_HISTORY_OPTS='/home/chanda/mosek/7/tools/platform/linux64x86/bin'
export
SPARK_DAEMON_JAVA_OPTS='/home/chanda/mosek/7/tools/platform/linux64x86/bin'
Please help
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-set-java-library-path-in-a-spark-cluster-tp13854.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Kyro deserialisation error
Hi, I am also facing the same problem. Has any one found out the solution yet? It just returns a vague set of characters. Please help.. Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Exception while deserializing and fetching task: com.esotericsoftware.kryo.KryoException: Unable to find class: "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv; "$&(*,.02468:<>@BDFHJLNPRTVXZ^`bdfhlnprtvD^bjlnpv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv: "$&(*,.02468:<>@BDFHJNPRTVXZ\`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv8@p= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtv= "$&(*,.02468:<>@BDFHJLNPRTVXZ\^`bdfhjlnprtvxz|~ Serialization trace: -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kyro-deserialisation-error-tp6798p14071.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
java.lang.OutOfMemoryError: Java heap space during reduce operation
Hi,
*In a reduce operation I am trying to accumulate a list of SparseVectors.
The code is given below;*
val WNode = trainingData.reduce{(node1:Node,node2:Node) =>
val wNode = new Node(num1,num2)
wNode.WhatList ++= (node1.WList)
wNode.WList ++= (node2.WList)
wNode
}
where Whatlist is a list of SparseVectors. The average size of a
SparseVector is 21000 and the approximate number of
elements in the final list at the end of the reduce operation varies between
20 to 100.
*However, at run time I am getting the following error messages from some of
the executor machines.*
14/10/20 22:38:41 INFO BlockManagerInfo: Added taskresult_30 in memory on
cse-hadoop-113:34602 (size: 789.0 MB, free: 22.2 GB)
14/10/20 22:38:41 INFO TaskSetManager: Starting task 1.0:12 as TID 34 on
executor 6: cse-hadoop-113 (PROCESS_LOCAL)
14/10/20 22:38:41 INFO TaskSetManager: Serialized task 1.0:12 as 2170 bytes
in 2 ms
14/10/20 22:38:41 INFO SendingConnection: Initiating connection to
[cse-hadoop-113/192.168.0.113:34602]
14/10/20 22:38:41 INFO SendingConnection: Connected to
[cse-hadoop-113/192.168.0.113:34602], 1 messages pending
14/10/20 22:38:41 INFO ConnectionManager: Accepted connection from
[cse-hadoop-113/192.168.0.113]
Exception in thread "pool-5-thread-3" java.lang.OutOfMemoryError: Java heap
space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at org.apache.spark.network.Message$.create(Message.scala:88)
at
org.apache.spark.network.ReceivingConnection$Inbox.org$apache$spark$network$ReceivingConnection$Inbox$$createNewMessage$1(Connection.scala:438)
at
org.apache.spark.network.ReceivingConnection$Inbox$$anonfun$1.apply(Connection.scala:448)
at
org.apache.spark.network.ReceivingConnection$Inbox$$anonfun$1.apply(Connection.scala:448)
at
scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)
at
org.apache.spark.network.ReceivingConnection$Inbox.getChunk(Connection.scala:448)
at
org.apache.spark.network.ReceivingConnection.read(Connection.scala:525)
at
org.apache.spark.network.ConnectionManager$$anon$6.run(ConnectionManager.scala:176)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
*Please help.*
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-Java-heap-space-during-reduce-operation-tp16835.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
