[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420390#comment-13420390
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537535/MAPREDUCE-987_branch-1.0_rev2.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2646//console
This message is automatically generated.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Affects Versions: 1.2.0, 2.2.0-alpha
Reporter: Philip Zeyliger
Assignee: Ahmed Radwan
Priority: Minor
Fix For: 2.2.0-alpha
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_branch-1.0_rev2.patch, MAPREDUCE-987_rev2.patch,
MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419823#comment-13419823
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Hdfs-trunk # (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk//])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419847#comment-13419847
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Mapreduce-trunk #1143 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1143/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419869#comment-13419869
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
Thanks Ahmed. Committed to trunk and branch-2, I'll leave it open until we have
the patch with docs for branch-1 and we commit it there.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419669#comment-13419669
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
+1
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419679#comment-13419679
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2575 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2575/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419681#comment-13419681
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Common-trunk-Commit #2510 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2510/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419702#comment-13419702
]
Hudson commented on MAPREDUCE-987:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2531 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2531/])
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
command-line. (ahmed via tucu) (Revision 1364020)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1364020
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418146#comment-13418146
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537142/MAPREDUCE-987_rev3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The applied patch generated 2067 javac compiler warnings (more
than the trunk's current 2066 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//console
This message is automatically generated.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418154#comment-13418154
]
Ahmed Radwan commented on MAPREDUCE-987:
The Jenkins reported test failure doesn't seem to be related to this patch.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418405#comment-13418405
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
Ahmed, a couple of Nits in the docs:
*1*
The sentence:
+ The CLI MiniCLuster can
+ start a full Hadoop cluster including the Yarn ResourceManager,
+ NodeManager(s), and HDFS NameNode and DataNode(s), in
+ addition to the MapReduce JobHistoryServer.
it is a bit too complex, why not just replace it with:
The CLI MiniCluster starts both a YARN/MapReduce HDFS clusters.
*2*
Instead using VERSION and a paragraph explaining what VERSION has to be
replaced with, you can use ${project.version}, within *.apt.vm files you can
use Maven variables.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13418617#comment-13418617
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537227/MAPREDUCE-987_rev4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The applied patch generated 2067 javac compiler warnings (more
than the trunk's current 2066 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//console
This message is automatically generated.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch, MAPREDUCE-987_rev3.patch, MAPREDUCE-987_rev4.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13417850#comment-13417850
]
Alejandro Abdelnur commented on MAPREDUCE-987:
--
+1. Ahmed, please add documentation for it (in the yarn-site submodule, in APT
format), and we are good to go.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13415760#comment-13415760
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536742/MAPREDUCE-987_rev2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The patch appears to cause the build to fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2605//console
This message is automatically generated.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch,
MAPREDUCE-987_rev2.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13410155#comment-13410155
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535777/MAPREDUCE-987_branch-1.0.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2561//console
This message is automatically generated.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch, MAPREDUCE-987_branch-1.0.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408450#comment-13408450
]
Ahmed Radwan commented on MAPREDUCE-987:
Here is an updated patch for trunk, various changes were done due to MR2
changes.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408451#comment-13408451
]
Ahmed Radwan commented on MAPREDUCE-987:
Please note that the patch requires also the patches from the blocker tickets
MAPREDUCE-4406 and MAPREDUCE-4407.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13408552#comment-13408552
]
Hadoop QA commented on MAPREDUCE-987:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535470/MAPREDUCE-987.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
-1 javac. The applied patch generated 2071 javac compiler warnings (more
than the trunk's current 2070 warnings).
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//console
This message is automatically generated.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch,
MAPREDUCE-987.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12923157#action_12923157
]
Konstantin Boudnik commented on MAPREDUCE-987:
--
Phillip, is there any intention to address last comments from Chris, so this
can be committed into trunk?
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12838055#action_12838055
]
Allen Wittenauer commented on MAPREDUCE-987:
There is a patch for this in the Cloudera distribution but it doesn't appear to
have been committed upstream. What is the status?
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12767135#action_12767135
]
Chris Douglas commented on MAPREDUCE-987:
-
This seems appropriate for the test jar. Small notes:
* This picks up \-D params like the generic parser; would it make sense to also
accept \-conf? The other params make less sense in this context, though it may
be worth considering Tool/ToolRunner
* It'd be better if sleepForever monitored the Mini\*Cluster, rather than
waking up every minute for no reason. Not sure if it makes sense to include a
poison pill (Path?) + configurable polling interval that might signal an
orderly shutdown.
* If this is intended for tests, should {{start}} wait for the TT/DNs to come
up before returning?
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12756349#action_12756349
]
Todd Lipcon commented on MAPREDUCE-987:
---
+1 - I code reviewed this internally before posting, and have also been using
it for a couple of weeks, finding it very useful. Happy to post a copy of our
internal review discussion if necessary.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
[
https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12755777#action_12755777
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-987:
--
Agreed that circular dependencies are to be avoided. I've moved this issue
into MAPREDUCE, and spun up a new patch.
Thanks, Philip.
Do we anticipate a world where MR doesn't depend statically on HDFS (i.e., it
only depends on the FileSystem interfaces)?
Theoretically, mapreduce does not depend on hdfs. However, the performance of
mapreduce would be bad without hdfs. I guess there is no serious applications
using mapreduce without hdfs, except for testing.
Exposing MiniDFS and MiniMR clusters as a single process command-line
-
Key: MAPREDUCE-987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: build, test
Reporter: Philip Zeyliger
Assignee: Philip Zeyliger
Priority: Minor
Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
It's hard to test non-Java programs that rely on significant mapreduce
functionality. The patch I'm proposing shortly will let you just type
bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster to start a
cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified number
of daemons, etc. A test that checks how some external process interacts with
Hadoop might start minicluster as a subprocess, run through its thing, and
then simply kill the java subprocess.
I've been using just such a system for a couple of weeks, and I like it.
It's significantly easier than developing a lot of scripts to start a
pseudo-distributed cluster, and then clean up after it. I figure others
might find it useful as well.
I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests
have all the required libraries, so I've put it there. I could conceivably
split this into minimr and minihdfs, but it's specifically the fact that
they're configured to talk to each other that I like about having them
together. And one JVM is better than two for my test programs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
