Re: java.io.NotSerializableException

2014-02-24 Thread leosand...@gmail.com 
Which class is not Serializable?

I run shark0.9 has a similarity  exception:
java.io.NotSerializableException (java.io.NotSerializableException: 
shark.execution.ReduceKeyReduceSide)

java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:28)
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:176)
org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:191)
org.apache.spark.util.collection.ExternalAppendOnlyMap.insert(ExternalAppendOnlyMap.scala:141)
org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:59)
org.apache.hadoop.hive.ql.exec.GroupByPostShuffleOperator$$anonfun$7.apply(GroupByPostShuffleOperator.scala:225)
org.apache.hadoop.hive.ql.exec.GroupByPostShuffleOperator$$anonfun$7.apply(GroupByPostShuffleOperator.scala:225)
org.apache.spark.rdd.RDD$$anonfun$3.apply(RDD.scala:471)
org.apache.spark.rdd.RDD$$anonfun$3.apply(RDD.scala:471)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:109)
org.apache.spark.scheduler.Task.run(Task.scala:53)
org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:49)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)




leosand...@gmail.com

From: yaoxin
Date: 2014-02-24 19:18
To: user
Subject: java.io.NotSerializableException
I got a error 
 org.apache.spark.SparkException: Job aborted: Task not serializable:
java.io.NotSerializableException:
But the class it complains is a java lib class that I dependents on, that I
can't change it to Serializable.
Is there any method to work this around?

I am using Spark 0.9, spark master using local[2] mode.




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-io-NotSerializableException-tp1973.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

How could I set spark.scheduler.pool in the shark cli ?

2014-02-13 Thread leosand...@gmail.com 
Hi all
How could I set the shark cli use different pools ?
I set spark.scheduler.mode = fair and scheduler.file=xxx .
and I see my scheduler pools in the appilication .
but when I set spark.scheduler.pool=mypool in the shark cli ,
these stage still run in the default pool .

THX 




leosand...@gmail.com

Can you help me ?

2014-01-13 Thread leosand...@gmail.com 
HI,
I run a hql in hive set these params :
set hive.exec.parallel=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
set mapred.max.split.size=1;
set mapred.min.split.size.per.node=1;
set mapred.min.split.size.per.rack=1;
set tcl.name=cr_24hourdM.tcl;
set mapred.queue.name=tcl2;
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set mapred.min.split.size=536870912;
set hive.exec.reducers.max=239;
set hive.exec.reducers.bytes.per.reducer=8000;

Could I also set the same params in the cli when I run the hql in shark ?
or I should set shark.exec.mode=hive ?
What's the diff between the two mode ?

If I just run a hql without catch table, could I change 
spark.storage.memoryFraction=0.1 or smaller ?

THX ! 




leosand...@gmail.com

Re: squestion on using spark parallelism vs using num partitions in spark api

2014-01-13 Thread leosand...@gmail.com 
I think the parallelism param just control how many tasks could be run together 
in each work.
it could't control how many tasks should be split .




leosand...@gmail.com

From: hussam_jar...@dell.com
Date: 2014-01-14 09:17
To: user@spark.incubator.apache.org
Subject: squestion on using spark parallelism vs using num partitions in spark 
api
Hi,
 
Using spark 0.8.1 … jave code running on 8 CPU with 16GRAM single node
 
It’s looks like upon setting spark parallelism using 
System.setProperty(spark.default.parallelism, 24) before creating my spark 
context as described in 
http://spark.incubator.apache.org/docs/latest/tuning.html#level-of-parallelism 
has no effect on the default number of partitions that spark uses in its api’s 
like saveAsTextFile() .
 
For example if I set spark.default.parallelism to 24, I was expecting 24 tasks 
to be invoked upon calling saveAsTextFile() but it’s not the case as I am 
seeing only 1 task get invoked
 
If I set my RDD parallelize() to 2 as 
dataSetRDD = SparkDriver.getSparkContext().parallelize(mydata,2);
then invoke 
dataSetRDD.saveAsTextFile(JavaRddFilePath);
 
I am seeing 2 tasks get invoked even my spark.default.parallelism was set to 24
 
Can someone explain the above behavior?
 
Thanks,
Hussam

转发: some problems about shark on spark

2014-01-10 Thread leosand...@gmail.com 





leosand...@gmail.com

发件人： leosand...@gmail.com
发送时间： 2014-01-10 22:29
收件人： user; shark-users
主题： some problems about shark on spark
HI ALL,
How could I set the param MEMORY_ONLY_SER 、Spark.kryoserializer.buffer.mb 、 
Spark.default.parallelism and Spark.worker.timeout
when I run a shark query ? 
May I set other params in spark-env.sh or hive-site.xml instead ?
or set name=value in the shark cli ?

I have a shark query test :
table a 38b ; table b 23b ;
sql: select a.* , b.* from a join b on a.id = b.id ;
it build three stages :
stage1 has tow tasks:
task1: rdd.HadoopRDD : input split table a 0+19 ;
task2: rdd.HadoopRDD : input split table a 19+19;
stage2 has two tasks: 
task1: rdd.HadoopRDD : input split table b 0+11 ;
task2: rdd.HadoopRDD : input split table b 11+12;
stage3 has one task:
task1: just fetch map outputs for shuffle and write to hdfs path .

Why these tables so small , but build two tasks to read it ?
How could I control the reduce task nums in shark ? It seems compute by the 
biggest father RDD's partitions ?

THX !




leosand...@gmail.com

some problems

2014-01-09 Thread leosand...@gmail.com 
Hi,
I'm runing a shark sql, but I don't know How spark build these stages and what 
work it does in each stage .
I waited the job 1 hour , it seems that the task 504 has some problem.

there are some logs in the machine which runs task 504 :

14/01/09 20:22:55 INFO executor.Executor: Finished task ID 287
14/01/09 20:23:00 INFO executor.StandaloneExecutorBackend: Got assigned task 504
14/01/09 20:23:00 INFO executor.Executor: Running task ID 504
14/01/09 20:23:00 INFO executor.Executor: Its epoch is 2
14/01/09 20:23:00 INFO spark.MapOutputTracker: Updating epoch to 2 and clearing 
cache
14/01/09 20:23:00 INFO spark.MapOutputTracker: Don't have map outputs for 
shuffle 2, fetching them
14/01/09 20:23:00 INFO spark.MapOutputTracker: Doing the fetch; tracker actor = 
Actor[akka://spark@OCDC-DD-002:53976/user/MapOutputTracker]
14/01/09 20:23:00 INFO spark.MapOutputTracker: Got the output locations
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: 
maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: 
maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: 
maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: 
maxBytesInFlight: 50331648, minRequest: 10066329
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: 
Getting 24 non-zero-bytes blocks out of 264 blocks
14/01/09 20:23:00 INFO storage.BlockFetcherIterator$BasicBlockFetcherIterator: 
Started 1 remote gets in  74 ms
14/01/09 20:23:00 INFO network.SendingConnection: Initiating connection to 
[OCDC-DATA-013/172.17.254.33:54930]
14/01/09 20:23:00 INFO network.SendingConnection: Connected to 
[OCDC-DATA-013/172.17.254.33:54930], 1 messages pending
14/01/09 20:23:00 INFO network.ConnectionManager: Accepted connection from 
[OCDC-DATA-013/172.17.254.33]
14/01/09 20:23:18 INFO network.SendingConnection: Initiating connection to 
[OCDC-DATA-012/172.17.254.32:45718]
14/01/09 20:23:18 INFO network.SendingConnection: Connected to 
[OCDC-DATA-012/172.17.254.32:45718], 1 messages pending
14/01/09 20:23:18 INFO network.ConnectionManager: Accepted connection from 
[OCDC-DATA-012/172.17.254.32]
14/01/09 20:23:46 INFO network.SendingConnection: Initiating connection to 
[OCDC-DATA-011/172.17.254.31:51067]
14/01/09 20:23:46 INFO network.SendingConnection: Connected to 
[OCDC-DATA-011/172.17.254.31:51067], 1 messages pending
14/01/09 20:23:46 INFO network.ConnectionManager: Accepted connection from 
[OCDC-DATA-011/172.17.254.31]
14/01/09 20:56:48 INFO storage.BlockManager: BlockManager reregistering with 
master
14/01/09 20:56:48 INFO storage.BlockManagerMaster: Trying to register 
BlockManager
14/01/09 20:56:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 20:56:48 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:04:43 INFO storage.BlockManager: BlockManager reregistering with 
master
14/01/09 21:04:43 INFO storage.BlockManagerMaster: Trying to register 
BlockManager
14/01/09 21:04:43 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:04:43 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:06:43 INFO storage.BlockManager: BlockManager reregistering with 
master
14/01/09 21:06:43 INFO storage.BlockManagerMaster: Trying to register 
BlockManager
14/01/09 21:06:43 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:06:43 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:11:48 INFO storage.BlockManager: BlockManager reregistering with 
master
14/01/09 21:11:48 INFO storage.BlockManagerMaster: Trying to register 
BlockManager
14/01/09 21:11:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:11:48 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:12:48 INFO storage.BlockManager: BlockManager reregistering with 
master
14/01/09 21:12:48 INFO storage.BlockManagerMaster: Trying to register 
BlockManager
14/01/09 21:12:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/01/09 21:12:48 INFO storage.BlockManager: Reporting 117 blocks to the master.
14/01/09 21:13:51 INFO storage.BlockManager: BlockManager reregistering with 
master
14/01/09 21:13:51 INFO storage.BlockManagerMaster: Trying to register 
BlockManager

can anyone give me the hint

thank you !
 




leosand...@gmail.com

native-lzo / gpl lib

2014-01-08 Thread leosand...@gmail.com 
HI,
I do a query from shark , it read a compress data from hdfs . but spark 
could't find the native-lzo lib .

14/01/08 22:58:21 ERROR executor.Executor: Exception in task ID 286
java.lang.RuntimeException: native-lzo library not available
at com.hadoop.compression.lzo.LzoCodec.getDecompressorType(LzoCodec.java:175)
at org.apache.hadoop.hive.ql.io.CodecPool.getDecompressor(CodecPool.java:122)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1299)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1139)
at org.apache.hadoop.hive.ql.io.RCFile$Reader.init(RCFile.java:1118)
at 
org.apache.hadoop.hive.ql.io.RCFileRecordReader.init(RCFileRecordReader.java:52)
at 
org.apache.hadoop.hive.ql.io.RCFileInputFormat.getRecordReader(RCFileInputFormat.java:57)
at org.apache.spark.rdd.HadoopRDD$$anon$1.init(HadoopRDD.scala:93)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:83)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:51)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:29)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:36)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at org.apache.spark.rdd.UnionPartition.iterator(UnionRDD.scala:29)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:69)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at 
org.apache.spark.rdd.MapPartitionsWithIndexRDD.compute(MapPartitionsWithIndexRDD.scala:40)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at 
org.apache.spark.rdd.MapPartitionsWithIndexRDD.compute(MapPartitionsWithIndexRDD.scala:40)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
at org.apache.spark.scheduler.ResultTask.run(ResultTask.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)

can anyone give me the hint

thank you !




leosand...@gmail.com

Shark compile

2014-01-03 Thread leosand...@gmail.com 
Hi
I want to compile Shark , Could I just run $ sbt/sbt package under shark_home 
,without compile spark and install hive ?

tharks !




leosand...@gmail.com

Re: RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0

2013-12-29 Thread leosand...@gmail.com 
What is your classpath ?
Had you builded your spark when you changed the new version and with yarn?
Have you find your jar under the $SPARK_HOME/assembly/target/scala-2.9.3  ? or 
there is not just only one ?




leosand...@gmail.com

From: Liu, Raymond
Date: 2013-12-30 08:36
To: user@spark.incubator.apache.org
Subject: RE: Errors with spark-0.8.1 hadoop-yarn 2.2.0
Hi Izhar

Is that the exact command you are running? Say with 0.8.0 instead of 0.8.1 in 
the cmd?

Raymond Liu

From: Izhar ul Hassan [mailto:ezh...@gmail.com] 
Sent: Friday, December 27, 2013 9:40 PM
To: user@spark.incubator.apache.org
Subject: Errors with spark-0.8.1 hadoop-yarn 2.2.0

Hi,
I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed 
spark-0.8.1 with support for spark enabled. I get the following errors when 
trying to run the examples:
SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar
 \
./spark-class org.apache.spark.deploy.yarn.Client \
  --jar 
examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
  --class org.apache.spark.examples.SparkPi \
  --args yarn-standalone \
  --num-workers 3 \
  --master-memory 4g \
  --worker-memory 2g \
  --worker-cores 1

Exception in thread main java.lang.NoClassDefFoundError: 
org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program 
will exit.
spark-0.8.0 with hadooop 2.0.5-alpha works fine.
-- 
/Izhar

Re: Re: endless job and slant tasks

2013-12-25 Thread leosand...@gmail.com 
No , just standalone cluster 




leosand...@gmail.com

From: Azuryy Yu
Date: 2013-12-25 19:21
To: user@spark.incubator.apache.org
Subject: Re: endless job and slant tasks
Hi Leo,
Did you run Spark on Yarn or mesos?







On Wed, Dec 25, 2013 at 6:58 PM, leosand...@gmail.com leosand...@gmail.com 
wrote:

hi all :
I run an example three times , it just read data from hdfs then do map and 
reduce then write to hdfs .
the first time and second time it works well ,  read almost 7G data and 
finished in 15 minutes , but there have a problem when I run it the third time .
one machine in my cluster lack of hard disk . 
The job begin at 17:11:15 , but it has been unable to end . I wait it for 1 
hour then kill it . there are the logs :

LogA (from the sick machine):
.
13/12/25 17:13:56 INFO Executor: Its epoch is 0
13/12/25 17:13:56 ERROR Executor: Exception in task ID 26
java.lang.NullPointerException
at 
org.apache.spark.storage.DiskStore$DiskBlockObjectWriter.revertPartialWrites(DiskStore.scala:99)
at 
org.apache.spark.scheduler.ShuffleMapTask$$anonfun$run$2.apply(ShuffleMapTask.scala:175)
at 
org.apache.spark.scheduler.ShuffleMapTask$$anonfun$run$2.apply(ShuffleMapTask.scala:175)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at 
org.apache.spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:175)
at 
org.apache.spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
END

LogB (a healthy machine):
..
13/12/25 17:12:35 INFO Executor: Serialized size of result for 54 is 817
13/12/25 17:12:35 INFO Executor: Finished task ID 54
[GC 4007686K-7782K(15073280K), 0.0287070 secs]
END

LogC(the master and a worker):
...
[GC 4907439K-110393K(15457280K), 0.0533800 secs]
13/12/25 17:13:23 INFO Executor: Serialized size of result for 203 is 817
13/12/25 17:13:23 INFO Executor: Finished task ID 203
13/12/25 17:13:24 INFO Executor: Serialized size of result for 202 is 817
13/12/25 17:13:24 INFO Executor: Finished task ID 202
END

I don't know why the job doesn't shut down ?  the log message doesn't been 
writen when the job runs 2 minuts .
why one machine assigned tasks so many more than others ?  
How could I get the job and task' status when I run a big job ? it looks like a 
black box ...





leosand...@gmail.com

Re: Re: endless job and slant tasks

2013-12-25 Thread leosand...@gmail.com 
Yes , disk space is full of the whole machine .




leosand...@gmail.com

From: Matei Zaharia
Date: 2013-12-26 01:50
To: user
Subject: Re: endless job and slant tasks
Does that machine maybe have a full disk drive, or no space in /tmp (where 
Spark stores local files by default)?


On Dec 25, 2013, at 7:50 AM, leosand...@gmail.com wrote:


No , just standalone cluster




leosand...@gmail.com

From: Azuryy Yu
Date: 2013-12-25 19:21
To: user@spark.incubator.apache.org
Subject: Re: endless job and slant tasks
Hi Leo,
Did you run Spark on Yarn or mesos?







On Wed, Dec 25, 2013 at 6:58 PM, leosand...@gmail.com leosand...@gmail.com 
wrote:

hi all :
I run an example three times , it just read data from hdfs then do map and 
reduce then write to hdfs .
the first time and second time it works well ,  read almost 7G data and 
finished in 15 minutes , but there have a problem when I run it the third time .
one machine in my cluster lack of hard disk .
The job begin at 17:11:15 , but it has been unable to end . I wait it for 1 
hour then kill it . there are the logs :

LogA (from the sick machine):
.
13/12/25 17:13:56 INFO Executor: Its epoch is 0
13/12/25 17:13:56 ERROR Executor: Exception in task ID 26
java.lang.NullPointerException
at 
org.apache.spark.storage.DiskStore$DiskBlockObjectWriter.revertPartialWrites(DiskStore.scala:99)
at 
org.apache.spark.scheduler.ShuffleMapTask$$anonfun$run$2.apply(ShuffleMapTask.scala:175)
at 
org.apache.spark.scheduler.ShuffleMapTask$$anonfun$run$2.apply(ShuffleMapTask.scala:175)
at 
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at 
org.apache.spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:175)
at 
org.apache.spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:158)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
END

LogB (a healthy machine):
..
13/12/25 17:12:35 INFO Executor: Serialized size of result for 54 is 817
13/12/25 17:12:35 INFO Executor: Finished task ID 54
[GC 4007686K-7782K(15073280K), 0.0287070 secs]
END

LogC(the master and a worker):
...
[GC 4907439K-110393K(15457280K), 0.0533800 secs]
13/12/25 17:13:23 INFO Executor: Serialized size of result for 203 is 817
13/12/25 17:13:23 INFO Executor: Finished task ID 203
13/12/25 17:13:24 INFO Executor: Serialized size of result for 202 is 817
13/12/25 17:13:24 INFO Executor: Finished task ID 202
END

I don't know why the job doesn't shut down ?  the log message doesn't been 
writen when the job runs 2 minuts .
why one machine assigned tasks so many more than others ? 
How could I get the job and task' status when I run a big job ? it looks like a 
black box ...





leosand...@gmail.com

How to set Akka frame size

2013-12-24 Thread leosand...@gmail.com 
Hi, everyone

I have a question about the arg spark.akka.frameSize , it default value is 10m .
I execute the JavaWordCount read data from hdfs , there is a 7G file .
there is a oom error caused by 
some task result exceeded Akka frame size .
but when I modify the arg 1G ，2G ， 10G , it show me 
ERROR ClusterScheduler: Lost executor 1 on ocnosql84: remote Akka client 
shutdown 
13/12/24 19:41:14 ERROR StandaloneExecutorBackend: Driver terminated or 
disconnected! Shutting down. 

Sometimes it show me different error info :
[lh1@ocnosql84 src]$ java MyWordCount spark://ocnosql84:7077 
hdfs://ocnosql76:8030/user/lh1/cdr_ismp_20130218 15000 1g 120
13/12/24 19:20:33 ERROR Client$ClientActor: Failed to connect to master
org.jboss.netty.channel.ChannelPipelineException: Failed to initialize a 
pipeline.
at 
org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:209)
at 
org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at 
akka.remote.netty.ActiveRemoteClient$$anonfun$connect$1.apply$mcV$sp(Client.scala:173)
at akka.util.Switch.liftedTree1$1(LockUtil.scala:33)
at akka.util.Switch.transcend(LockUtil.scala:32)
at akka.util.Switch.switchOn(LockUtil.scala:55)
at akka.remote.netty.ActiveRemoteClient.connect(Client.scala:158)
at 
akka.remote.netty.NettyRemoteTransport.send(NettyRemoteSupport.scala:153)
at akka.remote.RemoteActorRef.$bang(RemoteActorRefProvider.scala:247)
at 
org.apache.spark.deploy.client.Client$ClientActor.preStart(Client.scala:61)
at akka.actor.ActorCell.create$1(ActorCell.scala:508)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:600)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:209)
at akka.dispatch.Mailbox.run(Mailbox.scala:178)
at 
akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Caused by: java.lang.IllegalArgumentException: maxFrameLength must be a 
positive integer: -1451229184
at 
org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.init(LengthFieldBasedFrameDecoder.java:270)
at 
org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.init(LengthFieldBasedFrameDecoder.java:236)
at 
akka.remote.netty.ActiveRemoteClientPipelineFactory.getPipeline(Client.scala:340)
at 
org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:207)
... 18 more
13/12/24 19:20:33 ERROR SparkDeploySchedulerBackend: Disconnected from Spark 
cluster!
13/12/24 19:20:33 ERROR ClusterScheduler: Exiting due to error from cluster 
scheduler: Disconnected from Spark cluster

It seems caused by 
LengthFieldBasedFrameDecoder lenDec = new 
LengthFieldBasedFrameDecoder(this.client.netty().settings().MessageFrameSize(), 
0, 4, 0, 4);
I don't know what's the value of  
this.client.netty().settings().MessageFrameSize() and how  to calculate  this 
value .

my spark args :
export SPARK_DAEMON_MEMORY=4000m 
export SPARK_MEM=1000m 
export SPARK_WORKER_MEMORY=8g 
spark.akka.frameSize = 1000 / 2000 / 5000 / 1 / 15000
spark.executor.memory  = 1g
spark.akka.askTimeout = 120

Any help or reply is very appriciated !  Thanks very much




leosand...@gmail.com

AbstractMethodError

2013-12-23 Thread leosand...@gmail.com 
I write a example MyWordCount , just set spark.akka.frameSize larger than 
default . but when I run this jar , there is a problem :

13/12/19 18:53:48 INFO ClusterTaskSetManager: Lost TID 0 (task 0.0:0)
13/12/19 18:53:48 INFO ClusterTaskSetManager: Loss was due to 
java.lang.AbstractMethodError
java.lang.AbstractMethodError: 
org.apache.spark.api.java.function.WrappedFunction1.call(Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.spark.api.java.function.WrappedFunction1.apply(WrappedFunction1.scala:31)
at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:90)
at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:90)
at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.Iterator$$anon$21.foreach(Iterator.scala:437)
at 
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at 
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
at 
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:250)
at scala.collection.Iterator$$anon$21.toBuffer(Iterator.scala:437)
at 
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:237)
at scala.collection.Iterator$$anon$21.toArray(Iterator.scala:437)
at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:560)
at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:560)
at 
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:758)
at 
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:758)

it caused by  this code :
JavaRDDString words = lines.flatMap(new FlatMapFunctionString, String() {
public IterableString call(String s) {
return Arrays.asList(s.split( ));
} });

there is the parent class:

private[spark] abstract class WrappedFunction1[T, R] extends 
AbstractFunction1[T, R] {
  @throws(classOf[Exception])
  def call(t: T): R

  final def apply(t: T): R = call(t)
}
 
my code is same as the JavaWordCount , I don't know what's the error .

Thanks 

Leo




leosand...@gmail.com

AbstractMethodError

2013-12-19 Thread leosand...@gmail.com 
I write a example MyWordCount , just set spark.akka.frameSize larger than 
default . but when I run this jar , there is a problem :

13/12/19 18:53:48 INFO ClusterTaskSetManager: Lost TID 0 (task 0.0:0)
13/12/19 18:53:48 INFO ClusterTaskSetManager: Loss was due to 
java.lang.AbstractMethodError
java.lang.AbstractMethodError: 
org.apache.spark.api.java.function.WrappedFunction1.call(Ljava/lang/Object;)Ljava/lang/Object;
at 
org.apache.spark.api.java.function.WrappedFunction1.apply(WrappedFunction1.scala:31)
at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:90)
at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:90)
at scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.Iterator$$anon$21.foreach(Iterator.scala:437)
at 
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at 
scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
at 
scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:250)
at scala.collection.Iterator$$anon$21.toBuffer(Iterator.scala:437)
at 
scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:237)
at scala.collection.Iterator$$anon$21.toArray(Iterator.scala:437)
at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:560)
at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:560)
at 
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:758)
at 
org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:758)

it caused by  this code :
JavaRDDString words = lines.flatMap(new FlatMapFunctionString, String() {
public IterableString call(String s) {
return Arrays.asList(s.split( ));
} });

there is the parent class:

private[spark] abstract class WrappedFunction1[T, R] extends 
AbstractFunction1[T, R] {
  @throws(classOf[Exception])
  def call(t: T): R

  final def apply(t: T): R = call(t)
}
 
the code is same as the JavaWordCount , I don't know what's the error .

Thanks 

Leo




leosand...@gmail.com

resultset exceed Akka frame size

2013-12-18 Thread leosand...@gmail.com 
Hi, everyone

I have a problem when I run the WordCount example. I read 6G data from hdfs , 
when I run collect(), the executer had died .
there is the exception :
13/12/18 13:19:39 INFO ClusterTaskSetManager: Lost TID 55 (task 0.0:3)
13/12/18 13:19:39 INFO ClusterTaskSetManager: Loss was due to task 55 result 
exceeding Akka frame size; aborting job
13/12/18 13:19:39 INFO ClusterScheduler: Remove TaskSet 0.0 from pool
13/12/18 13:19:39 INFO DAGScheduler: Failed to run collect at 
JavaWordCount.java:60
Exception in thread main org.apache.spark.SparkException: Job failed: Task 55 
result exceeded Akka frame size
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)
at 
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)
at 
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)

I saw there are some issues about this question in the github , it seems that 
if the middle resultset is larger than Akka frame size , the job will fail . 
I want to know if I can change some params to solve the problem ?

Thanks 

Leo




leosand...@gmail.com

Re: Re: resultset exceed Akka frame size

2013-12-18 Thread leosand...@gmail.com 
Thank you !




leosand...@gmail.com

From: Azuryy Yu
Date: 2013-12-18 17:29
To: user
Subject: Re: resultset exceed Akka frame size
Hi Leo,


Akka is used to transfer the data back to the master, and there is a setting in 
Akka for the max message size, which is default to 10 MB here, you can find it 
at: core/src/main/scala/org/apache/spark/util/AkkaUtils.scala



So just increase spark.akka.frameSize to a larger number.







On Wed, Dec 18, 2013 at 4:49 PM, leosand...@gmail.com leosand...@gmail.com 
wrote:

Hi, everyone

I have a problem when I run the WordCount example. I read 6G data from hdfs , 
when I run collect(), the executer had died .
there is the exception :
13/12/18 13:19:39 INFO ClusterTaskSetManager: Lost TID 55 (task 0.0:3)
13/12/18 13:19:39 INFO ClusterTaskSetManager: Loss was due to task 55 result 
exceeding Akka frame size; aborting job
13/12/18 13:19:39 INFO ClusterScheduler: Remove TaskSet 0.0 from pool
13/12/18 13:19:39 INFO DAGScheduler: Failed to run collect at 
JavaWordCount.java:60
Exception in thread main org.apache.spark.SparkException: Job failed: Task 55 
result exceeded Akka frame size
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)
at 
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)
at 
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)

I saw there are some issues about this question in the github , it seems that 
if the middle resultset is larger than Akka frame size , the job will fail . 
I want to know if I can change some params to solve the problem ?

Thanks 

Leo




leosand...@gmail.com

转发: OOM

2013-12-16 Thread leosand...@gmail.com 





leosand...@gmail.com

发件人： leosand...@gmail.com
发送时间： 2013-12-16 20:01
收件人： user-subscribe
主题： OOM
hello everyone,
I have a problem when I run the wordcount example. I read data from hdfs , its 
almost 7G.
I haven't seen the info from the web ui or sparkhome/work . This is the console 
info : 
.
13/12/16 19:48:02 INFO LocalTaskSetManager: Size of task 52 is 1834 bytes
13/12/16 19:48:02 INFO LocalScheduler: Running 52
13/12/16 19:48:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 
52 non-zero-bytes blocks out of 52 blocks
13/12/16 19:48:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 
0 remote gets in  7 ms
13/12/16 19:48:09 INFO LocalTaskSetManager: Loss was due to 
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at 
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at 
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1857)
at 
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1766)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at 
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:27)
at 
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:47)
at 
org.apache.spark.scheduler.local.LocalScheduler.runTask(LocalScheduler.scala:204)
at 
org.apache.spark.scheduler.local.LocalActor$$anonfun$launchTask$1$$anon$1.run(LocalScheduler.scala:68)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
13/12/16 19:48:09 INFO LocalScheduler: Remove TaskSet 0.0 from pool
13/12/16 19:48:09 INFO DAGScheduler: Failed to run collect at console:17
org.apache.spark.SparkException: Job failed: Task 0.0:0 failed more than 4 
times; aborting job java.lang.OutOfMemoryError: Java heap space
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)
at 
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)
at 
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)

this is my spark-env.sh :

export SPARK_HOME=/home/lh1/spark_hadoopapp/spark-0.8.0-hadoop2.0.0-cdh4.2.1
export JAVA_HOME=/home/lh1/app/jdk1.7.0
export SCALA_HOME=/home/lh1/sparkapp/scala-2.9.3
export SPARK_WORKER_CORES=2  
export SPARK_WORKER_MEMORY=1024m
export SPARK_WORKER_INSTANCES=2 
export SPARK_DAEMON_JAVA_OPTS=9000m 

I just started to use Spark , so  can you give me some suggestions ?

Thanks .

Leo



leosand...@gmail.com

OOM, help

2013-12-16 Thread leosand...@gmail.com 
hello everyone,
I have a problem when I run the wordcount example. I read data from hdfs , its 
almost 7G.
I haven't seen the info from the web ui or sparkhome/work . This is the console 
info : 
.
13/12/16 19:48:02 INFO LocalTaskSetManager: Size of task 52 is 1834 bytes
13/12/16 19:48:02 INFO LocalScheduler: Running 52
13/12/16 19:48:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting 
52 non-zero-bytes blocks out of 52 blocks
13/12/16 19:48:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started 
0 remote gets in  7 ms
13/12/16 19:48:09 INFO LocalTaskSetManager: Loss was due to 
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at 
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at 
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1857)
at 
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1766)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at 
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:27)
at 
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:47)
at 
org.apache.spark.scheduler.local.LocalScheduler.runTask(LocalScheduler.scala:204)
at 
org.apache.spark.scheduler.local.LocalActor$$anonfun$launchTask$1$$anon$1.run(LocalScheduler.scala:68)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
13/12/16 19:48:09 INFO LocalScheduler: Remove TaskSet 0.0 from pool
13/12/16 19:48:09 INFO DAGScheduler: Failed to run collect at console:17
org.apache.spark.SparkException: Job failed: Task 0.0:0 failed more than 4 
times; aborting job java.lang.OutOfMemoryError: Java heap space
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)
at 
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)
at 
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)

this is my spark-env.sh :

export SPARK_HOME=/home/lh1/spark_hadoopapp/spark-0.8.0-hadoop2.0.0-cdh4.2.1
export JAVA_HOME=/home/lh1/app/jdk1.7.0
export SCALA_HOME=/home/lh1/sparkapp/scala-2.9.3
export SPARK_WORKER_CORES=2  
export SPARK_WORKER_MEMORY=1024m
export SPARK_WORKER_INSTANCES=2 
export SPARK_DAEMON_JAVA_OPTS=9000m 

I just started to use Spark , so  can you give me some suggestions ?

Thanks .

Leo







leosand...@gmail.com

Re: Re: I need some help

2013-12-11 Thread leosand...@gmail.com 
What will be cleaned if I compile Spark with sbt/sbt clean assembly?
Actually I find there is a problem in my product's url 
sparkhome/assembly/target/scala-2.9.3 , there are two jars named 
spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.2.1.jar  and 
spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar , and the 
compute-classpath.sh make the classpath with ASSEMBLY_JAR=`ls 
$FWDIR/assembly/target/scala-$SCALA_VERSION/spark-assembly*hadoop*.jar`  , so 
the two jars splited with a space in the classpath .
Somebody encounter the same problem ?




leosand...@gmail.com

From: Paco Nathan
Date: 2013-12-12 04:25
To: user
Subject: Re: I need some help
did you try using:


   sbt/sbt clean assembly



On Tue, Dec 10, 2013 at 10:23 PM, leosand...@gmail.com leosand...@gmail.com 
wrote:

I  have deployed two Spark clusters .
The first is a simple standalone cluster which is working well . ( sbt/sbt 
assembly )
But I built Spark against Hadoop2.0.0-cdh4.2.1 in the second cluster, there  
seems to be a problem when I start the master ! ( 
SPARK_HADOOP_VERSION=2.0.0-cdh4.2.1  sbt/sbt assembly  )

[lh1@ocnosql84 bin]$ start-master.sh
starting org.apache.spark.deploy.master.Master, logging to 
/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/bin/../logs/spark-lh1-org.apache.spark.deploy.master.Master-1-ocnosql84.out
failed to launch org.apache.spark.deploy.master.Master:
  [Loaded java.lang.Shutdown from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
  [Loaded java.lang.Shutdown$Lock from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
full log in 
/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/bin/../logs/spark-lh1-org.apache.spark.deploy.master.Master-1-ocnosql84.out

It seems that I failed to load the Master , but the class 
org.apache.spark.deploy.master.Master exists 
spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.2.1.jar
 . I set the parameter SPARK_DAEMON_JAVA_OPTS = -verbose:class , there are some 
 logs :

Spark Command: /home/lh1/app/jdk1.7.0/bin/java -cp 
:/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/conf:/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.2.1.jar
/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar
 -verbose:class -Djava.library.path= -Xms512m -Xmx512m 
org.apache.spark.deploy.master.Master --ip ocnosql84 --port 7077 --webui-port 
8080


..

[Loaded java.text.Format$Field from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
[Loaded java.text.MessageFormat$Field from 
/home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
Error: Could not find or load main class org.apache.spark.deploy.master.Master
[Loaded java.lang.Shutdown from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
[Loaded java.lang.Shutdown$Lock from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]

Thanks !






leosand...@gmail.com

Re: RE: I need some help

2013-12-11 Thread leosand...@gmail.com 
NO, I build with sbt




leosand...@gmail.com

From: Liu, Raymond
Date: 2013-12-12 14:12
To: user@spark.incubator.apache.org
Subject: RE: Re: I need some help
The latter one sound to me like been built by mvn?

Best Regards,
Raymond Liu

From: leosand...@gmail.com [mailto:leosand...@gmail.com] 
Sent: Thursday, December 12, 2013 2:02 PM
To: user
Subject: Re: Re: I need some help

What will be cleaned if I compile Spark with sbt/sbt clean assembly?
Actually I find there is a problem in my product's url 
sparkhome/assembly/target/scala-2.9.3 , there are two jars named 
spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.2.1.jar  and 
spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar , and the 
compute-classpath.sh make the classpath with ASSEMBLY_JAR=`ls 
$FWDIR/assembly/target/scala-$SCALA_VERSION/spark-assembly*hadoop*.jar`  , so 
the two jars splited with a space in the classpath .
Somebody encounter the same problem ?


leosand...@gmail.com

From: Paco Nathan
Date: 2013-12-12 04:25
To: user
Subject: Re: I need some help
did you try using: 

   sbt/sbt clean assembly

On Tue, Dec 10, 2013 at 10:23 PM, leosand...@gmail.com leosand...@gmail.com 
wrote:
I  have deployed two Spark clusters .
The first is a simple standalone cluster which is working well . ( sbt/sbt 
assembly )
But I built Spark against Hadoop2.0.0-cdh4.2.1 in the second cluster, there  
seems to be a problem when I start the master ! ( 
SPARK_HADOOP_VERSION=2.0.0-cdh4.2.1  sbt/sbt assembly  )

[lh1@ocnosql84 bin]$ start-master.sh
starting org.apache.spark.deploy.master.Master, logging to 
/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/bin/../logs/spark-lh1-org.apache.spark.deploy.master.Master-1-ocnosql84.out
failed to launch org.apache.spark.deploy.master.Master:
  [Loaded java.lang.Shutdown from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
  [Loaded java.lang.Shutdown$Lock from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
full log in 
/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/bin/../logs/spark-lh1-org.apache.spark.deploy.master.Master-1-ocnosql84.out

It seems that I failed to load the Master , but the class 
org.apache.spark.deploy.master.Master exists 
spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.2.1.jar
 . I set the parameter SPARK_DAEMON_JAVA_OPTS = -verbose:class , there are some 
 logs :

Spark Command: /home/lh1/app/jdk1.7.0/bin/java -cp 
:/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/conf:/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.2.1.jar
/home/lh1/spark_hadoopapp/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3/spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar
 -verbose:class -Djava.library.path= -Xms512m -Xmx512m 
org.apache.spark.deploy.master.Master --ip ocnosql84 --port 7077 --webui-port 
8080


..

[Loaded java.text.Format$Field from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
[Loaded java.text.MessageFormat$Field from 
/home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
Error: Could not find or load main class org.apache.spark.deploy.master.Master
[Loaded java.lang.Shutdown from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]
[Loaded java.lang.Shutdown$Lock from /home/lh1/app/jdk1.7.0/jre/lib/rt.jar]

Thanks !




leosand...@gmail.com