[ https://issues.apache.org/jira/browse/SPARK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148071#comment-14148071 ]
Xuefu Zhang edited comment on SPARK-3691 at 9/25/14 6:21 PM: ------------------------------------------------------------- cc [~sandyr], [~rxin] was (Author: xuefuz): cc [~sandyr] > Provide a mini cluster for testing system built on Spark > -------------------------------------------------------- > > Key: SPARK-3691 > URL: https://issues.apache.org/jira/browse/SPARK-3691 > Project: Spark > Issue Type: Test > Components: Spark Core > Affects Versions: 1.1.0 > Reporter: Xuefu Zhang > > Most Hadoop components such MR, DFS, Tez, and Yarn provide a mini cluster > that can be used to test the external systems that rely on those frameworks, > such as Pig and Hive. While Spark's local mode can be used to do such testing > and is friendly for debugging, it's too far from a real Spark cluster and a > lot of problems cannot be discovered. Thus, an equivalent of Hadoop MR mini > cluster in Spark would be very helpful in testing system such as Hive/Pig on > Spark. > Spark's local-cluster is considered for this purpose but it doesn't fit well > because it requires a Spark installation on the box where the tests run. > Also, local-cluster isn't exposed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org