Re: Compiling Spark master (284771ef) with sbt/sbt assembly fails on EC2
I also ran into same issue. What is the solution? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Compiling-Spark-master-284771ef-with-sbt-sbt-assembly-fails-on-EC2-tp11155p11189.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Installing Spark 0.9.1 on EMR Cluster
Have you tried flag --spark-version of spark-ec2 ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Installing-Spark-0-9-1-on-EMR-Cluster-tp11084p11096.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
spark-ec2 script with Tachyon
Hi, It seems that spark-ec2 script deploys Tachyon module along with other setup. I am trying to use .persist(OFF_HEAP) for RDD persistence, but on worker I see this error -- Failed to connect (2) to master localhost/127.0.0.1:19998 : java.net.ConnectException: Connection refused -- From netstat I see that worker is connected to master node on port 19998 -- Proto Recv-Q Send-Q Local Address Foreign Address State tcp0 0 ip-10-16-132-190.ec2.:49239 ip-10-158-45-248.ec2.:19998 ESTABLISHED -- Does Tachyon on EC work out of the box? or does it requite further configuration ? Am I supposed to set spark.tachyonStore.url to Masters IP ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-ec2-script-with-Tachyon-tp9996.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Yay for 1.0.0! EC2 Still has problems.
I am also running into modules/mod_authn_alias.so issue on r3.8xlarge when launched cluster with ./spark-ec2; so ganglia is not accessible. From the posts it seems that Patrick suggested using Ubuntu 12.04. Can you please provide name of AMI that can be used with -a flag that will not have this issue? - I am running script with --spark-git-repo=https://github.com/apache/spark;, which I assume should deploy the latest code. - I have been able to launch cluster on m2.4xlarge, where ganglia works. - From what I understand we are not supposed to use any random AMI??; it will be helpful to publish list of AMIS that people use with different instances. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Yay-for-1-0-0-EC2-Still-has-problems-tp6578p9306.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Yay for 1.0.0! EC2 Still has problems.
I am also running into modules/mod_authn_alias.so issue on r3.8xlarge when launched cluster with ./spark-ec2; so ganglia is not accessible. From the posts it seems that Patrick suggested using Ubuntu 12.04. Can you please provide name of AMI that can be used with -a flag that will not have this issue? - I am running script with --spark-git-repo=https://github.com/apache/spark;, which I assume should deploy the latest code. - I have been able to launch cluster on m2.4xlarge, where ganglia works. - From what I understand we are not supposed to use any random AMI??; it will be helpful to publish list of AMIS that people use with different instances. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Yay-for-1-0-0-EC2-Still-has-problems-tp6578p9307.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
java.io.FileNotFoundException: shuffle
Hi, I am running my spark job on Yarn; using latest code from master branch..synced few days back. I see this IO Exception during shuffle(in resource manager logs). What could be wrong and how to debug it? I have seen this few times before; I was suspecting that this could side effect of memory pressure..but I could never figure out the root cause. -- 14/07/02 07:34:45 WARN TaskSetManager: Loss was due to java.io.FileNotFoundException java.io.FileNotFoundException: /var/storage/sda3/nm-local/usercache/nit/appcache/application_1403208801430_0183/spark-local-20140702065054-388d/0e/shuffle_3_193_787 (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:221) at org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:116) at org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:177) at org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:161) at org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:158) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.Task.run(Task.scala:51) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) -- -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-io-FileNotFoundException-shuffle-tp8644.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
