Re: Can Spark 1.0.2 run on CDH-4.3.0 with yarn? And Will Spark 1.2.0 support CDH5.1.2 with yarn?
now we have spark 1.3.0 on chd 5.1.0 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-Spark-1-0-2-run-on-CDH-4-3-0-with-yarn-And-Will-Spark-1-2-0-support-CDH5-1-2-with-yarn-tp20760p22509.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Can Spark 1.0.2 run on CDH-4.3.0 with yarn? And Will Spark 1.2.0 support CDH5.1.2 with yarn?
I did not compile spark 1.1.0 source code on CDH4.3.0 with yarn successfully. Does it support CDH4.3.0 with yarn ? And will spark 1.2.0 support CDH5.1.2? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-Spark-1-0-2-run-on-CDH-4-3-0-with-yarn-And-Will-Spark-1-2-0-support-CDH5-1-2-with-yarn-tp20760.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Can Spark 1.0.2 run on CDH-4.3.0 with yarn? And Will Spark 1.2.0 support CDH5.1.2 with yarn?
The question is really: will Spark 1.1 work with a particular version of YARN? many, but not all versions of YARN are supported. The stable versions are (2.2.x+). Before that, support is patchier, and in fact has been removed in Spark 1.3. The yarn profile supports YARN stable which is about 2.2.x and onwards. The yarn-alpha profile should work for YARN about 0.23.x. 2.0.x and 2.1.x were a sort of beta period and I recall that yarn-alpha works with some of it, but not all, and there is no yarn-beta profile. I believe early CDH 4.x has basically YARN beta. Later 4.x has stable. I think I'd try the yarn-alpha profile and see if it compiles. But the version of YARN in that release may well be among those that fall in the gap between alpha and stable support. Thankfully things got a lot more stable past Hadoop / YARN 2.2 or so, so it far more just works without version issues. And CDH 5 is based on Hadoop 2.3 and then 2.5, so you should be much more able to build whatever versions together that you want. CDH 5.1.x ships Spark 1.0.x. There should be no problem using 1.1.x, 1.2.x, etc. with it; you just need to make and support your own binaries. 5.2.x has 1.1.x; 5.3.x will have 1.2.x. On Thu, Dec 18, 2014 at 9:18 AM, Canoe canoe...@gmail.com wrote: I did not compile spark 1.1.0 source code on CDH4.3.0 with yarn successfully. Does it support CDH4.3.0 with yarn ? And will spark 1.2.0 support CDH5.1.2? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-Spark-1-0-2-run-on-CDH-4-3-0-with-yarn-And-Will-Spark-1-2-0-support-CDH5-1-2-with-yarn-tp20760.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Can Spark 1.0.2 run on CDH-4.3.0 with yarn? And Will Spark 1.2.0 support CDH5.1.2 with yarn?
Hi, Sean Thank you for your reply. I will try to use Spark 1.1 and 1.2 on CHD5.X. :) 2014-12-18 17:38 GMT+08:00 Sean Owen so...@cloudera.com: The question is really: will Spark 1.1 work with a particular version of YARN? many, but not all versions of YARN are supported. The stable versions are (2.2.x+). Before that, support is patchier, and in fact has been removed in Spark 1.3. The yarn profile supports YARN stable which is about 2.2.x and onwards. The yarn-alpha profile should work for YARN about 0.23.x. 2.0.x and 2.1.x were a sort of beta period and I recall that yarn-alpha works with some of it, but not all, and there is no yarn-beta profile. I believe early CDH 4.x has basically YARN beta. Later 4.x has stable. I think I'd try the yarn-alpha profile and see if it compiles. But the version of YARN in that release may well be among those that fall in the gap between alpha and stable support. Thankfully things got a lot more stable past Hadoop / YARN 2.2 or so, so it far more just works without version issues. And CDH 5 is based on Hadoop 2.3 and then 2.5, so you should be much more able to build whatever versions together that you want. CDH 5.1.x ships Spark 1.0.x. There should be no problem using 1.1.x, 1.2.x, etc. with it; you just need to make and support your own binaries. 5.2.x has 1.1.x; 5.3.x will have 1.2.x. On Thu, Dec 18, 2014 at 9:18 AM, Canoe canoe...@gmail.com wrote: I did not compile spark 1.1.0 source code on CDH4.3.0 with yarn successfully. Does it support CDH4.3.0 with yarn ? And will spark 1.2.0 support CDH5.1.2? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-Spark-1-0-2-run-on-CDH-4-3-0-with-yarn-And-Will-Spark-1-2-0-support-CDH5-1-2-with-yarn-tp20760.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- 谁谓河广，一苇航之
