[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420390#comment-13420390
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537535/MAPREDUCE-987_branch-1.0_rev2.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2646//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Affects Versions: 1.2.0, 2.2.0-alpha
>Reporter: Philip Zeyliger
>Assignee: Ahmed Radwan
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_branch-1.0_rev2.patch, MAPREDUCE-987_rev2.patch,
> MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419869#comment-13419869
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
Thanks Ahmed. Committed to trunk and branch-2, I'll leave it open until we have
the patch with docs for branch-1 and we commit it there.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419847#comment-13419847
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Mapreduce-trunk #1143 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1143/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419823#comment-13419823
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Hdfs-trunk # (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk//])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419702#comment-13419702
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2531 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2531/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419679#comment-13419679
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2575 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2575/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419681#comment-13419681
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Common-trunk-Commit #2510 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2510/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419669#comment-13419669
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
+1
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418617#comment-13418617
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537227/MAPREDUCE-987_rev4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The applied patch generated 2067 javac compiler warnings (more
than the trunk's current 2066 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418405#comment-13418405
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
Ahmed, a couple of Nits in the docs:
*1*
The sentence:
+ The CLI MiniCLuster can
+ start a full Hadoop cluster including the <<>> <<>>,
+ <<>>, and <<>> <<>> and <<>>, in
+ addition to the <<>> <<>>.
it is a bit too complex, why not just replace it with:
The CLI MiniCluster starts both a YARN/MapReduce & HDFS clusters.
*2*
Instead using VERSION and a paragraph explaining what VERSION has to be
replaced with, you can use ${project.version}, within *.apt.vm files you can
use Maven variables.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418154#comment-13418154
]
Ahmed Radwan commented on MAPREDUCE-987:
The Jenkins reported test failure doesn't seem to be related to this patch.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418146#comment-13418146
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537142/MAPREDUCE-987_rev3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The applied patch generated 2067 javac compiler warnings (more
than the trunk's current 2066 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417850#comment-13417850
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
+1. Ahmed, please add documentation for it (in the yarn-site submodule, in APT
format), and we are good to go.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415760#comment-13415760
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536742/MAPREDUCE-987_rev2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The patch appears to cause the build to fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2605//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
> MAPREDUCE-987_rev2.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410155#comment-13410155
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535777/MAPREDUCE-987_branch-1.0.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2561//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408552#comment-13408552
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535470/MAPREDUCE-987.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The applied patch generated 2071 javac compiler warnings (more
than the trunk's current 2070 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//console
This message is automatically generated.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408451#comment-13408451
]
Ahmed Radwan commented on MAPREDUCE-987:
Please note that the patch requires also the patches from the blocker tickets
MAPREDUCE-4406 and MAPREDUCE-4407.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408450#comment-13408450
]
Ahmed Radwan commented on MAPREDUCE-987:
Here is an updated patch for trunk, various changes were done due to MR2
changes.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
> MAPREDUCE-987.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923157#action_12923157
]
Konstantin Boudnik commented on MAPREDUCE-987:
--
Phillip, is there any intention to address last comments from Chris, so this
can be committed into trunk?
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838055#action_12838055
]
Allen Wittenauer commented on MAPREDUCE-987:
There is a patch for this in the Cloudera distribution but it doesn't appear to
have been committed upstream. What is the status?
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767135#action_12767135
]
Chris Douglas commented on MAPREDUCE-987:
-
This seems appropriate for the test jar. Small notes:
* This picks up \-D params like the generic parser; would it make sense to also
accept \-conf? The other params make less sense in this context, though it may
be worth considering Tool/ToolRunner
* It'd be better if sleepForever monitored the Mini\*Cluster, rather than
waking up every minute for no reason. Not sure if it makes sense to include a
poison pill (Path?) + configurable polling interval that might signal an
orderly shutdown.
* If this is intended for tests, should {{start}} wait for the TT/DNs to come
up before returning?
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756349#action_12756349
]
Todd Lipcon commented on MAPREDUCE-987:
---
+1 - I code reviewed this internally before posting, and have also been using
it for a couple of weeks, finding it very useful. Happy to post a copy of our
internal review discussion if necessary.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755777#action_12755777
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-987:
--
> Agreed that circular dependencies are to be avoided. I've moved this issue
> into MAPREDUCE, and spun up a new patch.
Thanks, Philip.
> Do we anticipate a world where MR doesn't depend statically on HDFS (i.e., it
> only depends on the FileSystem interfaces)?
Theoretically, mapreduce does not depend on hdfs. However, the performance of
mapreduce would be bad without hdfs. I guess there is no serious applications
using mapreduce without hdfs, except for testing.
> Exposing MiniDFS and MiniMR clusters as a single process command-line
> -
>
> Key: MAPREDUCE-987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: build, test
>Reporter: Philip Zeyliger
>Assignee: Philip Zeyliger
>Priority: Minor
> Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
>
>
> It's hard to test non-Java programs that rely on significant mapreduce
> functionality. The patch I'm proposing shortly will let you just type
> "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a
> cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
> of daemons, etc. A test that checks how some external process interacts with
> Hadoop might start minicluster as a subprocess, run through its thing, and
> then simply kill the java subprocess.
> I've been using just such a system for a couple of weeks, and I like it.
> It's significantly easier than developing a lot of scripts to start a
> pseudo-distributed cluster, and then clean up after it. I figure others
> might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
> have all the required libraries, so I've put it there. I could conceivably
> split this into "minimr" and "minihdfs", but it's specifically the fact that
> they're configured to talk to each other that I like about having them
> together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
