[jira] Commented: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749082#action_12749082 ] Hadoop QA commented on MAPREDUCE-885: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418034/MAPREDUCE-885.5.patch against trunk revision 808730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/536/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/536/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/536/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/536/console This message is automatically generated. > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.5.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749074#action_12749074 ] Aaron Kimball commented on MAPREDUCE-885: - Failures are unrelated. > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.5.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749059#action_12749059 ] Hadoop QA commented on MAPREDUCE-885: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418034/MAPREDUCE-885.5.patch against trunk revision 808730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/535/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/535/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/535/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/535/console This message is automatically generated. > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.5.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[
https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749057#action_12749057
]
Hong Tang commented on MAPREDUCE-839:
-
test and test-patch passed on my local machine:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
> unit test TestMiniMRChildTask fails on mac os-x
> ---
>
> Key: MAPREDUCE-839
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-839
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.21.0
>Reporter: Hong Tang
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: mapreduce-839-20090828.patch
>
>
> The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[ https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-839: Fix Version/s: 0.21.0 Assignee: Hong Tang Status: Patch Available (was: Open) > unit test TestMiniMRChildTask fails on mac os-x > --- > > Key: MAPREDUCE-839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-839 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Hong Tang >Assignee: Hong Tang >Priority: Minor > Fix For: 0.21.0 > > Attachments: mapreduce-839-20090828.patch > > > The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749048#action_12749048
]
Chris Douglas commented on MAPREDUCE-679:
-
bq. Chris, to what API are you referring?
o.a.h.metrics and o.a.h.mapred.JobTrackerInstrumentation
(JobTrackerMetricsInst). Ganglia, Chukwa, etc. push metrics through this
interface. Adding a new interface isn't a small change. For example, the
current patch calls {{JobTracker::runningJobs}} rather than
{{JobTracker::getRunningJobs}}; the former is unsynchronized and accesses
shared data structures. There have been similar issues caused by metrics
frameworks attempting to pull information out of the JobTracker without an
audit of the (regrettable) lock hierarchy. If this receives its data through
the metrics API, then neither you nor maintainers need to consider how this
servlet affects the shared JT data.
bq. No API reports job history in a universally-readable manner (e.g., REST +
XML).
That's fair. Again, I like machine-readable formats and wouldn't object to this
even if it were an interim solution, but I want to be clear about how it gets
its data and what it supports, since we'll be committing ourselves to
maintaining both.
Would JobHistory be a better home for this? Much of the data are not currently
available in the web UI, let alone in a reasonable format.
> XML-based metrics as JSP servlet for JobTracker
> ---
>
> Key: MAPREDUCE-679
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-679
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobtracker
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: example-jobtracker-completed-job.xml,
> example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch,
> MAPREDUCE-679.3.patch, MAPREDUCE-679.4.patch, MAPREDUCE-679.patch
>
>
> In HADOOP-4559, a general REST API for reporting metrics was proposed but
> work seems to have stalled. In the interim, we have a simple XML translation
> of the existing JobTracker status page which provides the same metrics
> (including the tables of running/completed/failed jobs) as the human-readable
> page. This is a relatively lightweight addition to provide some
> machine-understandable metrics reporting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[
https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Tang updated MAPREDUCE-839:
Attachment: mapreduce-839-20090828.patch
Attached is the patch that incorporate the idea - using environment variables
TEMP and TMP to override the default absolute and relative path.
To run unit test on OS-X, one needs to invoke the command like the following:
{noformat}
env TEMP=`pwd`/build/tmp TMP=build/tmp ant test-core
-Dhadoop.tmp.dir=`pwd`/build/tmp
{noformat}
hadoop.tmp.dir needs to be modified because it defaults to /tmp/hadoop-${user}/.
> unit test TestMiniMRChildTask fails on mac os-x
> ---
>
> Key: MAPREDUCE-839
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-839
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.21.0
>Reporter: Hong Tang
>Priority: Minor
> Attachments: mapreduce-839-20090828.patch
>
>
> The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-937) Allow comments in mapred.hosts and mapred.hosts.exclude files
[ https://issues.apache.org/jira/browse/MAPREDUCE-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated MAPREDUCE-937: Tags: fb > Allow comments in mapred.hosts and mapred.hosts.exclude files > - > > Key: MAPREDUCE-937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-937 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Trivial > > It will be useful to have comments in the mapred.hosts and > mapred.hosts.exclude files that will give description to hosts in there and > to the file itself so different people maintaining these files will be able > to share certain information about the process and hosts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[ https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749043#action_12749043 ] Hong Tang commented on MAPREDUCE-839: - I did some further investigation, and it seems that both errors were caused by "/tmp" being symlinked to "/private/tmp" on os-x. I propose a solution to allow people to customize the absolute and relative tmp directories being tested. > unit test TestMiniMRChildTask fails on mac os-x > --- > > Key: MAPREDUCE-839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-839 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Hong Tang >Priority: Minor > > The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-936) Allow a load difference in fairshare scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-936: --- Tags: fb > Allow a load difference in fairshare scheduler > -- > > Key: MAPREDUCE-936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-936 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Zheng Shao > Attachments: MAPREDUCE-936.1.patch > > > The problem we are facing: It takes a long time for all tasks of a job to get > scheduled on the cluster, even if the cluster is almost empty. > There are two reasons that together lead to this situation: > 1. The load factor makes sure each TT runs the same number of tasks. (This is > the part that this patch tries to change). > 2. The scheduler tries to schedule map tasks locally (first node-local, then > rack-local). There is a wait time (mapred.fairscheduler.localitywait.node and > mapred.fairscheduler.localitywait.rack, both are around 10 sec in our conf), > and accumulated wait time (JobInfo.localityWait). The accumulated wait time > is reset to 0 whenever a non-local map task is scheduled. That means it takes > N * wait_time to schedule N non-local map tasks. > Because of 1, a lot of TT will not be able to take more tasks, even if they > have free slots. As a result, a lot of the map tasks cannot be scheduled > locally. > Because of 2, it's really hard to schedule a non-local task. > As a result, sometimes we are seeing that it takes more than 2 minutes to > schedule all the mappers of a job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Attachment: MAPREDUCE-679.4.patch Attaching new patch. Fixes formatting issue. Also makes JobTrackerJspHelper non-static to avoid thread-safety issue with DecimalFormat. > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.4.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-679: Status: Patch Available (was: Open) > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.4.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-918) Test hsqldb server should be memory-only.
[
https://issues.apache.org/jira/browse/MAPREDUCE-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749026#action_12749026
]
Aaron Kimball commented on MAPREDUCE-918:
-
Test failures are unrelated.
> Test hsqldb server should be memory-only.
> -
>
> Key: MAPREDUCE-918
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-918
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/sqoop
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-918.patch
>
>
> Sqoop launches a standalone hsqldb server for unit tests, but it currently
> writes its database to disk and uses a connect string of {{//localhost}}. If
> multiple test instances are running concurrently, one test server may serve
> to the other instance of the unit tests, causing race conditions.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-938) Postgresql support for Sqoop
[
https://issues.apache.org/jira/browse/MAPREDUCE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-938:
Attachment: MAPREDUCE-938.patch
Attaching patch which provides support and testcases for Postgresql imports via
Sqoop. Provides both a JDBC implementation as well as {{\-\-direct}} mode
support via a psql {{COPY TO STDOUT}} command.
Includes test cases enabled in ThirdPartyTests; I've run these successfully.
Also tested via importing a large (multi-GB) dataset to HDFS.
> Postgresql support for Sqoop
>
>
> Key: MAPREDUCE-938
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-938
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/sqoop
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-938.patch
>
>
> Sqoop should be able to import from postgresql databases.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-938) Postgresql support for Sqoop
[ https://issues.apache.org/jira/browse/MAPREDUCE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-938: Release Note: Postgresql support for Sqoop Status: Patch Available (was: Open) > Postgresql support for Sqoop > > > Key: MAPREDUCE-938 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-938 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/sqoop >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-938.patch > > > Sqoop should be able to import from postgresql databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-938) Postgresql support for Sqoop
Postgresql support for Sqoop Key: MAPREDUCE-938 URL: https://issues.apache.org/jira/browse/MAPREDUCE-938 Project: Hadoop Map/Reduce Issue Type: New Feature Components: contrib/sqoop Reporter: Aaron Kimball Assignee: Aaron Kimball Sqoop should be able to import from postgresql databases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-936) Allow a load difference in fairshare scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749020#action_12749020 ] Matei Zaharia commented on MAPREDUCE-936: - Hi Zheng, For issue 1, the provided patch looks good. It might be nice to add a unit test for it though. For issue 2, I believe the implementation of locality waits in MAPREDUCE-706 has solved the issue. In that implementation, once a job has launched a non-local task, it can keep launching non-local tasks right away without further waits. However, if it ever manages to launch a local task again, it needs to wait to start launching non-local tasks. The reasoning for this is that maybe the job had just been unlucky earlier and still has lots of tasks left to launch, and we don't want it to stay stuck at the non-local level. I think the locality wait code you guys are running at Facebook is much older than the one in MAPREDUCE-706, so it would be nice if you could upgrade to MAPREDUCE-706 when you upgrade Hadoop in general. I believe it would not be too difficult to port the trunk version of the fair scheduler to 0.20 and get all the architectural changes and improvements in 706 with that. Matei > Allow a load difference in fairshare scheduler > -- > > Key: MAPREDUCE-936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-936 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Zheng Shao > Attachments: MAPREDUCE-936.1.patch > > > The problem we are facing: It takes a long time for all tasks of a job to get > scheduled on the cluster, even if the cluster is almost empty. > There are two reasons that together lead to this situation: > 1. The load factor makes sure each TT runs the same number of tasks. (This is > the part that this patch tries to change). > 2. The scheduler tries to schedule map tasks locally (first node-local, then > rack-local). There is a wait time (mapred.fairscheduler.localitywait.node and > mapred.fairscheduler.localitywait.rack, both are around 10 sec in our conf), > and accumulated wait time (JobInfo.localityWait). The accumulated wait time > is reset to 0 whenever a non-local map task is scheduled. That means it takes > N * wait_time to schedule N non-local map tasks. > Because of 1, a lot of TT will not be able to take more tasks, even if they > have free slots. As a result, a lot of the map tasks cannot be scheduled > locally. > Because of 2, it's really hard to schedule a non-local task. > As a result, sometimes we are seeing that it takes more than 2 minutes to > schedule all the mappers of a job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749019#action_12749019 ] Aaron Kimball commented on MAPREDUCE-679: - Chris, to what API are you referring? No API reports job history in a universally-readable manner (e.g., REST + XML). > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-885:
Attachment: MAPREDUCE-885.5.patch
Just realized that the previous patches forgot to include changes to
{{src/test/}} -- adding now.
> More efficient SQL queries for DBInputFormat
>
>
> Key: MAPREDUCE-885
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-885
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
> Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch,
> MAPREDUCE-885.4.patch, MAPREDUCE-885.5.patch, MAPREDUCE-885.patch
>
>
> DBInputFormat generates InputSplits by counting the available rows in a
> table, and selecting subsections of the table via the "LIMIT" and "OFFSET"
> SQL keywords. These are only meaningful in an ordered context, so the query
> also includes an "ORDER BY" clause on an index column. The resulting queries
> are often inefficient and require full table scans. Actually using multiple
> mappers with these queries can lead to O(n^2) behavior in the database, where
> n is the number of splits. Attempting to use parallelism with these queries
> is counter-productive.
> A better mechanism is to organize splits based on data values themselves,
> which can be performed in the WHERE clause, allowing for index range scans of
> tables, and can better exploit parallelism in the database.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-885: Status: Patch Available (was: Open) > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.5.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball reassigned MAPREDUCE-885: --- Assignee: Aaron Kimball (was: zhuweimin) > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.5.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-885: Assignee: zhuweimin (was: Aaron Kimball) Status: Patch Available (was: Open) > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: zhuweimin > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-885: Status: Open (was: Patch Available) > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: zhuweimin > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-885: Status: Open (was: Patch Available) > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-679) XML-based metrics as JSP servlet for JobTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749016#action_12749016 ] Chris Douglas commented on MAPREDUCE-679: - Scraping HTML is obviously a poor choice, but there's already a metrics API and several implementations. This would exist outside of that framework and report (IIRC) a subset of the available metrics. Is there some aspect of that API that makes it unsuitable? > XML-based metrics as JSP servlet for JobTracker > --- > > Key: MAPREDUCE-679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-679 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: example-jobtracker-completed-job.xml, > example-jobtracker-running-job.xml, MAPREDUCE-679.2.patch, > MAPREDUCE-679.3.patch, MAPREDUCE-679.patch > > > In HADOOP-4559, a general REST API for reporting metrics was proposed but > work seems to have stalled. In the interim, we have a simple XML translation > of the existing JobTracker status page which provides the same metrics > (including the tables of running/completed/failed jobs) as the human-readable > page. This is a relatively lightweight addition to provide some > machine-understandable metrics reporting. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-885) More efficient SQL queries for DBInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Kimball updated MAPREDUCE-885: Attachment: MAPREDUCE-885.4.patch new patch resync'd with trunk. > More efficient SQL queries for DBInputFormat > > > Key: MAPREDUCE-885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-885 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Aaron Kimball >Assignee: Aaron Kimball > Attachments: MAPREDUCE-885.2.patch, MAPREDUCE-885.3.patch, > MAPREDUCE-885.4.patch, MAPREDUCE-885.patch > > > DBInputFormat generates InputSplits by counting the available rows in a > table, and selecting subsections of the table via the "LIMIT" and "OFFSET" > SQL keywords. These are only meaningful in an ordered context, so the query > also includes an "ORDER BY" clause on an index column. The resulting queries > are often inefficient and require full table scans. Actually using multiple > mappers with these queries can lead to O(n^2) behavior in the database, where > n is the number of splits. Attempting to use parallelism with these queries > is counter-productive. > A better mechanism is to organize splits based on data values themselves, > which can be performed in the WHERE clause, allowing for index range scans of > tables, and can better exploit parallelism in the database. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-937) Allow comments in mapred.hosts and mapred.hosts.exclude files
[ https://issues.apache.org/jira/browse/MAPREDUCE-937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated MAPREDUCE-937: Description: It will be useful to have comments in the mapred.hosts and mapred.hosts.exclude files that will give description to hosts in there and to the file itself so different people maintaining these files will be able to share certain information about the process and hosts. (was: Add the ability to have inline comments in the mapred.hosts and mapred.hosts.exclude files) > Allow comments in mapred.hosts and mapred.hosts.exclude files > - > > Key: MAPREDUCE-937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-937 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Trivial > > It will be useful to have comments in the mapred.hosts and > mapred.hosts.exclude files that will give description to hosts in there and > to the file itself so different people maintaining these files will be able > to share certain information about the process and hosts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-937) Allow comments in mapred.hosts and mapred.hosts.exclude files
Allow comments in mapred.hosts and mapred.hosts.exclude files - Key: MAPREDUCE-937 URL: https://issues.apache.org/jira/browse/MAPREDUCE-937 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Reporter: Dmytro Molkov Assignee: Dmytro Molkov Priority: Trivial Add the ability to have inline comments in the mapred.hosts and mapred.hosts.exclude files -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-936) Allow a load difference in fairshare scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Shao updated MAPREDUCE-936: - Attachment: MAPREDUCE-936.1.patch > Allow a load difference in fairshare scheduler > -- > > Key: MAPREDUCE-936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-936 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/fair-share >Reporter: Zheng Shao > Attachments: MAPREDUCE-936.1.patch > > > The problem we are facing: It takes a long time for all tasks of a job to get > scheduled on the cluster, even if the cluster is almost empty. > There are two reasons that together lead to this situation: > 1. The load factor makes sure each TT runs the same number of tasks. (This is > the part that this patch tries to change). > 2. The scheduler tries to schedule map tasks locally (first node-local, then > rack-local). There is a wait time (mapred.fairscheduler.localitywait.node and > mapred.fairscheduler.localitywait.rack, both are around 10 sec in our conf), > and accumulated wait time (JobInfo.localityWait). The accumulated wait time > is reset to 0 whenever a non-local map task is scheduled. That means it takes > N * wait_time to schedule N non-local map tasks. > Because of 1, a lot of TT will not be able to take more tasks, even if they > have free slots. As a result, a lot of the map tasks cannot be scheduled > locally. > Because of 2, it's really hard to schedule a non-local task. > As a result, sometimes we are seeing that it takes more than 2 minutes to > schedule all the mappers of a job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-936) Allow a load difference in fairshare scheduler
Allow a load difference in fairshare scheduler -- Key: MAPREDUCE-936 URL: https://issues.apache.org/jira/browse/MAPREDUCE-936 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Reporter: Zheng Shao The problem we are facing: It takes a long time for all tasks of a job to get scheduled on the cluster, even if the cluster is almost empty. There are two reasons that together lead to this situation: 1. The load factor makes sure each TT runs the same number of tasks. (This is the part that this patch tries to change). 2. The scheduler tries to schedule map tasks locally (first node-local, then rack-local). There is a wait time (mapred.fairscheduler.localitywait.node and mapred.fairscheduler.localitywait.rack, both are around 10 sec in our conf), and accumulated wait time (JobInfo.localityWait). The accumulated wait time is reset to 0 whenever a non-local map task is scheduled. That means it takes N * wait_time to schedule N non-local map tasks. Because of 1, a lot of TT will not be able to take more tasks, even if they have free slots. As a result, a lot of the map tasks cannot be scheduled locally. Because of 2, it's really hard to schedule a non-local task. As a result, sometimes we are seeing that it takes more than 2 minutes to schedule all the mappers of a job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-935) There's little to be gained by putting a host into the penaltybox at reduce time, if its the only host you have
[
https://issues.apache.org/jira/browse/MAPREDUCE-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748989#action_12748989
]
Steve Loughran commented on MAPREDUCE-935:
--
looking closer the client backoff thread is not related to the problem, but the
penalty box is possibly causing needless suffering. Though I suspect that local
task tracker has stopped listening for RPC calls, which is the root cause of
trouble.
> There's little to be gained by putting a host into the penaltybox at reduce
> time, if its the only host you have
> ---
>
> Key: MAPREDUCE-935
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-935
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.21.0
>Reporter: Steve Loughran
>
> Exponential backoff may be good for dealing with troublesome hosts, but not
> if you only have one host in the entire system. From the log of
> {{TestNodeRefresh}}, which for some reason is blocking in the reduce phase, I
> can see it doesn't take much for the backoff to kick in so rapidly that the
> reducer is waiting for longer than the test
> {code}
> 2009-08-28 21:39:16,788 WARN mapred.ReduceTask
> (ReduceTask.java:fetchOutputs(2192)) -
> attempt_20090828213826033_0001_r_00_0 adding host localhost to penalty
> box, next contact in 150 seconds
> {code}
> The result of this backoff process is that the reduce process ends up
> appearing to hang, getting killed from above.
> Note that this isn't the root cause of the problem, but it certainly
> amplifies things.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-935) There's little to be gained by putting a host into the penaltybox at reduce time, if its the only host you have
[
https://issues.apache.org/jira/browse/MAPREDUCE-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748984#action_12748984
]
Steve Loughran commented on MAPREDUCE-935:
--
Looking at the thread dump, it also looks like the exponential backoff feature
in the {{Client.handleConnectFailure()}} is interfering with heartbeats. A
failure to connect to the server is triggering backoff, stopping progress from
being reported.
> There's little to be gained by putting a host into the penaltybox at reduce
> time, if its the only host you have
> ---
>
> Key: MAPREDUCE-935
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-935
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.21.0
>Reporter: Steve Loughran
>
> Exponential backoff may be good for dealing with troublesome hosts, but not
> if you only have one host in the entire system. From the log of
> {{TestNodeRefresh}}, which for some reason is blocking in the reduce phase, I
> can see it doesn't take much for the backoff to kick in so rapidly that the
> reducer is waiting for longer than the test
> {code}
> 2009-08-28 21:39:16,788 WARN mapred.ReduceTask
> (ReduceTask.java:fetchOutputs(2192)) -
> attempt_20090828213826033_0001_r_00_0 adding host localhost to penalty
> box, next contact in 150 seconds
> {code}
> The result of this backoff process is that the reduce process ends up
> appearing to hang, getting killed from above.
> Note that this isn't the root cause of the problem, but it certainly
> amplifies things.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-935) There's little to be gained by putting a host into the penaltybox at reduce time, if its the only host you have
[
https://issues.apache.org/jira/browse/MAPREDUCE-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748983#action_12748983
]
Steve Loughran commented on MAPREDUCE-935:
--
Thread dump of the killed process
{code}
2009-08-28 21:40:39,572 INFO mapred.Task
(ReflectionUtils.java:logThreadInfo(204)) - Process Thread Dump: Communication
exception
14 active threads
Thread 26 (Thread for polling Map Completion Events):
State: TIMED_WAITING
Blocked count: 25
Waited count: 117
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:392)
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:334)
org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:197)
org.apache.hadoop.ipc.Client.getConnection(Client.java:886)
org.apache.hadoop.ipc.Client.call(Client.java:746)
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
org.apache.hadoop.mapred.$Proxy0.getMapCompletionEvents(Unknown Source)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2725)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2692)
Thread 25 (Thread for merging in memory files):
State: WAITING
Blocked count: 12
Waited count: 13
Waiting on java.lang.obj...@500ab58d
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.waitForDataToMerge(ReduceTask.java:1092)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2590)
Thread 24 (Thread for merging on-disk files):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.tree...@659c2931
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2493)
Thread 23 (MapOutputCopier attempt_20090828213826033_0001_r_00_0.4):
State: WAITING
Blocked count: 267
Waited count: 266
Waiting on java.util.arrayl...@15ab7626
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 22 (MapOutputCopier attempt_20090828213826033_0001_r_00_0.3):
State: WAITING
Blocked count: 273
Waited count: 270
Waiting on java.util.arrayl...@15ab7626
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 21 (MapOutputCopier attempt_20090828213826033_0001_r_00_0.2):
State: WAITING
Blocked count: 274
Waited count: 275
Waiting on java.util.arrayl...@15ab7626
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 20 (MapOutputCopier attempt_20090828213826033_0001_r_00_0.1):
State: WAITING
Blocked count: 257
Waited count: 257
Waiting on java.util.arrayl...@15ab7626
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 19 (MapOutputCopier attempt_20090828213826033_0001_r_00_0.0):
State: WAITING
Blocked count: 77
Waited count: 72
Waiting on java.util.arrayl...@15ab7626
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 18 (communication thread):
State: RUNNABLE
Blocked count: 16
Waited count: 34
Stack:
sun.management.ThreadImpl.getThreadInfo0(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:147)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:123)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:532)
java.lang.Thread.run(Thread.java:637)
Thread 15 (Thread for syncLogs):
State: TIMED_WAITING
Blocked count: 0
Waited count: 24
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.mapred.Child$2.run(Child.java:88)
Thread 5 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 5
Waited count: 6
Waiting on java.lang.ref.referencequeue$l...@21e30857
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(R
[jira] Created: (MAPREDUCE-935) There's little to be gained by putting a host into the penaltybox at reduce time, if its the only host you have
There's little to be gained by putting a host into the penaltybox at reduce
time, if its the only host you have
---
Key: MAPREDUCE-935
URL: https://issues.apache.org/jira/browse/MAPREDUCE-935
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.21.0
Reporter: Steve Loughran
Exponential backoff may be good for dealing with troublesome hosts, but not if
you only have one host in the entire system. From the log of
{{TestNodeRefresh}}, which for some reason is blocking in the reduce phase, I
can see it doesn't take much for the backoff to kick in so rapidly that the
reducer is waiting for longer than the test
{code}
2009-08-28 21:39:16,788 WARN mapred.ReduceTask
(ReduceTask.java:fetchOutputs(2192)) -
attempt_20090828213826033_0001_r_00_0 adding host localhost to penalty box,
next contact in 150 seconds
{code}
The result of this backoff process is that the reduce process ends up appearing
to hang, getting killed from above.
Note that this isn't the root cause of the problem, but it certainly amplifies
things.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-934) rumen should be able to filter and fold job traces
rumen should be able to filter and fold job traces -- Key: MAPREDUCE-934 URL: https://issues.apache.org/jira/browse/MAPREDUCE-934 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Dick King The rumen command line creates job traces from job history logs. These job traces are read by the rumen runtime system at the behest of other applications. We should add a command line tool to rumen to manipulate a job trace in various ways to modify a workload: * overlap job trace segments to increase the workload * extract a short segment from a long job trace * stretch out job traces to dilute the workload -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-933) rumen needs a unit test where its input job history logs are created as part of the test
rumen needs a unit test where its input job history logs are created as part of the test Key: MAPREDUCE-933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-933 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Dick King Currently, rumen's test cases use job history logs that are stored as part of the test case. We should fix this because rumen will break if job history log formats change incompatibly, but the unit tests will still pass. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-932) Rumen needs a job trace sorter
Rumen needs a job trace sorter -- Key: MAPREDUCE-932 URL: https://issues.apache.org/jira/browse/MAPREDUCE-932 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Dick King Assignee: Dick King Rumen reads job history logs and produces job traces. The jobs in a job trace do not occur in any promised order. Certain tools need the jobs to be ordered by job submission time. We should include, in Rumen, a tool to sort job traces. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-931) rumen should use its own interpolation classes to create runtimes for simulated tasks
rumen should use its own interpolation classes to create runtimes for simulated tasks - Key: MAPREDUCE-931 URL: https://issues.apache.org/jira/browse/MAPREDUCE-931 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Dick King Assignee: Dick King Priority: Minor Currently, when a simulator or benchmark is running and simulating hadoop jobs using rumen data, and rumen's runtime system is used to get execution times for the tasks in the simulated jobs, rumen would use some ad hoc code, despite the fact that rumen has a perfectly good interpolation framework to generate random variables that fit discrete CDFs. We should use the interpolation framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748925#action_12748925 ] Arun C Murthy commented on MAPREDUCE-777: - bq. # Instead of get(Map,Reduce,SetupAndCleanup)TaskReports, should that just be a getTaskReport(TaskType)? The number of task types is likely to increase. +1 > A method for finding and tracking jobs from the new API > --- > > Key: MAPREDUCE-777 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-777 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Reporter: Owen O'Malley >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: m-777.patch, patch-777-1.txt, patch-777-2.txt, > patch-777.txt > > > We need to create a replacement interface for the JobClient API in the new > interface. In particular, the user needs to be able to query and track jobs > that were launched by other processes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-930) rumen should interpret job history log input paths with respect to default FS, not local FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748919#action_12748919
]
Dick King commented on MAPREDUCE-930:
-
The solution is a one-line change.
{{wd = FileSystem.getLocal(getConf()).getWorkingDirectory();}}
should become
{{wd = FileSystem.get(getConf()).getWorkingDirectory();}}
.
> rumen should interpret job history log input paths with respect to default
> FS, not local FS
> ---
>
> Key: MAPREDUCE-930
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-930
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Dick King
>Assignee: Dick King
>Priority: Minor
>
> This allows job history log file/directory names that don't specify a file
> system to use the configured default FS instead of the local FS [when the
> configured default is not the local].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-930) rumen should interpret job history log input paths with respect to default FS, not local FS
rumen should interpret job history log input paths with respect to default FS, not local FS --- Key: MAPREDUCE-930 URL: https://issues.apache.org/jira/browse/MAPREDUCE-930 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Dick King Assignee: Dick King Priority: Minor This allows job history log file/directory names that don't specify a file system to use the configured default FS instead of the local FS [when the configured default is not the local]. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-751) Rumen: a tool to extract job characterization data from job tracker logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748890#action_12748890 ] Hudson commented on MAPREDUCE-751: -- Integrated in Hadoop-Mapreduce-trunk-Commit #3 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/3/]) . Add Rumen, a tool for extracting statistics from job tracker logs and generating job traces for simulation and analysis. Contributed by Dick King and Guanying Wang > Rumen: a tool to extract job characterization data from job tracker logs > > > Key: MAPREDUCE-751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-751 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Dick King >Assignee: Dick King > Fix For: 0.21.0 > > Attachments: 2009-08-19--1030.patch, 2009-08-26--1513-patch.patch, > mapreduce-751--2009-07-23.patch, mapreduce-751-20090826.patch, > mapreduce-751-20090826.patch > > > We propose a new map/reduce component, rumen, which can be used to process > job history logs to produce any or all of the following: > * Retrospective info describing the statistical behavior of the > amount of time it would have taken to launch a job into a certain > percentage of the number of mapper slots in the log's cluster, given the > load over the period covered by the log > * Statistical info as to the runtimes and shuffle times, etc. of > the tasks and jobs covered by the log > * files describing detailed job trace information, and the > network topology as inferred from the host locations and rack IDs that > arise in the job tracker log. In addition to this facility, rumen > includes readers for this information to return job and detailed task > information to other tools. > These other tools include a more advanced version of gridmix, and > also includes mumak: see blocked issues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-421) mapred pipes might return exit code 0 even when failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748889#action_12748889 ] Hudson commented on MAPREDUCE-421: -- Integrated in Hadoop-Mapreduce-trunk-Commit #3 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/3/]) Pipes need to set the exit return code when something goes wrong. (Christian Kunz via omalley) > mapred pipes might return exit code 0 even when failing > --- > > Key: MAPREDUCE-421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-421 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.1 > > Attachments: MAPREDUCE-421.patch > > > up to hadoop 0.18.3 org.apache.hadoop.mapred.JobShell ensured that 'hadoop > jar' returns non-zero exit code when the job fails. > This is no longer true after moving this to org.apache.hadoop.util.RunJar. > Pipes jobs submitted through cli never returned proper exit code. > The main methods in org.apache.hadoop.util.RunJar. and > org.apache.hadoop.mapred.pipes.Submitter should be modified to return an exit > code similar to how org.apache.hadoop.mapred.JobShell did it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748891#action_12748891 ] Hudson commented on MAPREDUCE-370: -- Integrated in Hadoop-Mapreduce-trunk-Commit #3 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/3/]) Revert > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370-2.txt, patch-370-3.txt, > patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-925) Not able to run parallel instances of ant tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748818#action_12748818 ] Steve Loughran commented on MAPREDUCE-925: -- There is a way to run tests in parallel using vmware, virtualbox or equivalent: have a VM run the tests off a mounted copy of the files. To get things to work best, set build.dir to something local, so that generated results on one VM don't interfere with the others. > Not able to run parallel instances of ant tests > --- > > Key: MAPREDUCE-925 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-925 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.20.1 >Reporter: Suman Sehgal >Priority: Minor > > Not able to execute two or more parallel instances of ant test. > Scenario: > === > Executed "TestRecoveryManager" in a loop for 100 times and ran some other > tests in parallel. Encountered following error message in TestRecoveryManager > logs: > [junit] 2009-08-27 10:53:48,724 INFO mapred.TaskTracker > (TaskTracker.java:tryToGetOutputSize(1594)) - > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > taskTracker/jobcache///output/file.out in any of the > configured local directories > It seems that new ant test removes the existing mapred local directory itself > and recreates it for current test. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-839) unit test TestMiniMRChildTask fails on mac os-x
[ https://issues.apache.org/jira/browse/MAPREDUCE-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748817#action_12748817 ] Steve Loughran commented on MAPREDUCE-839: -- I see this too. Disable the test on OS-X? Or is there a way to come up with something that works everywhere. > unit test TestMiniMRChildTask fails on mac os-x > --- > > Key: MAPREDUCE-839 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-839 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Hong Tang >Priority: Minor > > The unit test TestMiniMRChildTask fails on Mac OS-X (10.5.8) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-887) After 4491, task cleaup directory some gets created under the ownershiptasktracker user instread job submitting.
[ https://issues.apache.org/jira/browse/MAPREDUCE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748815#action_12748815 ] Karam Singh commented on MAPREDUCE-887: --- It seems the issue observed is timing. As cleanup directories and very quickly. So seems because ls -lR command was launched ownership might changed. Checked with commenting out code of cleanup files found permissions are set correctly making issue as resloved invalid > After 4491, task cleaup directory some gets created under the > ownershiptasktracker user instread job submitting. > > > Key: MAPREDUCE-887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-887 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Karam Singh > > Some time, when task is killed, task cleanup directory is created under the > ownership tasktracker launching user instead job submitting user. > dr-xrws--- karams hadoop ] job_200908170914_0020 > |-- [drwxr-sr-x mapred hadoop ] > attempt_200908170914_0020_m_02_0.cleanup > `-- [drwxrws--- karams hadoop ] > attempt_200908170914_0020_m_12_0 > Here karams is user who submitted job and mapred is the use who launched TT. > taskattrempt.cleanup created with mapred user not with karams user. > This issue is intermittent, not always reproducible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-887) After 4491, task cleaup directory some gets created under the ownershiptasktracker user instread job submitting.
[ https://issues.apache.org/jira/browse/MAPREDUCE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karam Singh resolved MAPREDUCE-887. --- Resolution: Invalid > After 4491, task cleaup directory some gets created under the > ownershiptasktracker user instread job submitting. > > > Key: MAPREDUCE-887 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-887 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Karam Singh > > Some time, when task is killed, task cleanup directory is created under the > ownership tasktracker launching user instead job submitting user. > dr-xrws--- karams hadoop ] job_200908170914_0020 > |-- [drwxr-sr-x mapred hadoop ] > attempt_200908170914_0020_m_02_0.cleanup > `-- [drwxrws--- karams hadoop ] > attempt_200908170914_0020_m_12_0 > Here karams is user who submitted job and mapred is the use who launched TT. > taskattrempt.cleanup created with mapred user not with karams user. > This issue is intermittent, not always reproducible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-898) Change DistributedCache to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748800#action_12748800 ] Hadoop QA commented on MAPREDUCE-898: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417958/patch-898.txt against trunk revision 808730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2230 javac compiler warnings (more than the trunk's current 2226 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/534/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/534/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/534/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/534/console This message is automatically generated. > Change DistributedCache to use new api. > --- > > Key: MAPREDUCE-898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-898 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-898.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-861) Modify queue configuration format and parsing to support a hierarchy of queues.
[ https://issues.apache.org/jira/browse/MAPREDUCE-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748793#action_12748793 ] Hemanth Yamijala commented on MAPREDUCE-861: Opps. I didn't realize that my comments were posted partially *frown*, please ignore the first set. I am sorry as they are incomplete. > Modify queue configuration format and parsing to support a hierarchy of > queues. > --- > > Key: MAPREDUCE-861 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-861 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Hemanth Yamijala >Assignee: rahul k singh > Attachments: MAPREDUCE-861-1.patch > > > MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce > framework. This JIRA is for defining changes to the configuration related to > queues. > The current format for defining a queue and its properties is as follows: > mapred.queue... For e.g. > mapred.queue..acl-submit-job. The reason for using this verbose > format was to be able to reuse the Configuration parser in Hadoop. However, > administrators currently using the queue configuration have already indicated > a very strong desire for a more manageable format. Since, this becomes more > unwieldy with hierarchical queues, the time may be good to introduce a new > format for representing queue configuration. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-861) Modify queue configuration format and parsing to support a hierarchy of queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748792#action_12748792
]
Hemanth Yamijala commented on MAPREDUCE-861:
I've started taking a look at this code, and have come till reviewing the
framework changes. Here are some comments:
- I would suggest that we move out the parsing code for the new queues format
into a separate class like QueueHierarchyBuilder, in order to keep the
QueueManager class simple.
- I would also suggest that we make the deprecated and new class stick to a
unified interface that provides APIs like:
{code}
Queue getRoot();
boolean areAclsEnabled();
{code}
These can be used by the QueueManager on the appropriate builder depending on
whether old configuration is available or not. This way, the initialization
code (like in the constructor) of QueueManager will be consistent, irrespective
of where the queue configuration is.
- Also, since we want to discard the mapred-site.xml loaded properties ASAP, we
probably should parse and build the hierarchy from the constructor of the
builder itself.
- createHierarchy should be refactored to parse leaf level queues and non leaf
level queues separately. I would suggest something along the lines of:
{code}
for each queue node:
newQueue.name = parseName()
newQueue.properties = parseProperties()
if (node is a leaf level queue) {
parseLeafQueueProperties() // here we parse ACLs, state and other
properties.
} else {
for every queue node:
recursively parse queue nodes.
}
{code}
- Calling getInnerQueues without calling addChild can cause a null pointer
because children is null.
- hashCode is overridden, but not equals
- I assume the order of child elements of an XML tag will be in the order in
which they are defined in the file. I think we must explicitly ensure that
elements come in a fixed order. For e.g. currently, we are not enforcing an
order on how tags under the 'queue' node are defined. This could lead to cases
where if a 'queue' element comes before the 'name' element, we would call parse
hierarchy for the child queue without the parent's name.
- Shouldn't we check in populateProperties that the node is of type element.
- Since getNamedItem can be null, we should check for it in populateProperties.
This will allow us to throw a better exception.
- In QueueManager.initialize, allQueues.putAll(leafQueues) seems unnecessary,
since getInnerQueues already does this.
- setSchedulerInfo and getSchedulerInfo should work on all queues, not just
leaf queues.
- The current refresh code is broken with reference to what we want to do long
term. We should only change properties of leaf queues. Since we are planning to
change the refresh model in MAPREDUCE-893, I think for completeness sake the
refresh of the code should build the entire hierarchy, but pickup only leaf
level queues and copy properties like acls and state. Currently, since the
refresh code is adding to the root queues, hierarchy could be affected which is
not in the scope even of MAPREDUCE-893.
- getJobQueueInfos is calling getJobQueueInfo twice. We should avoid this. Not
a bug in the patch, as it was already there, but it will help to fix it.
- dumpConfiguration should be changed. It should be changed to the new JSON
format.
- DeprecatedQueueHierarchyBuilder should not hold on to an instance of the
configuration
Some minor comments on style:
- Import only required classes and not all classes in a package.
- getInnerQueues can be renamed as getDescendentQueues
- DeprecatedQueueHierarchyBuilder.loadingDeprecatedResource can be loadQueues()
or createQueues().
- I think we are using "acls" as an XML attribute of the queues element to
determine if ACLs are enabled or not. I would suggest we name it as aclsEnabled.
- QueueManager.validate should be better documented to specify what it's
validating.
- There's an empty statement in QueueManager.populateProperties ( just a ';' )
I am unable to decide on these points:
- Do we need leafQueues and allQueues stored separately ?
- Should we validate using an XSD ? I know you were planning on this. After
looking at the code, I am thinking this makes good sense.
- I am unable to decide if refreshConfiguration should be in the base class for
QueueHierarchyBuilder, because it would need a Configuration in the deprecated
case, and none in the new case.
> Modify queue configuration format and parsing to support a hierarchy of
> queues.
> ---
>
> Key: MAPREDUCE-861
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-861
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Hemanth Yamijala
>Assignee: rahul k singh
> Attachments: MAPREDUCE-861-1.patch
>
>
> MAPREDUCE-853 proposes to
[jira] Commented: (MAPREDUCE-861) Modify queue configuration format and parsing to support a hierarchy of queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748791#action_12748791
]
Hemanth Yamijala commented on MAPREDUCE-861:
I've started taking a look at this code, and have come till reviewing the
framework changes. Here are some comments:
- I would suggest that we move out the parsing code for the new queues format
into a separate class like QueueHierarchyBuilder, in order to keep the
QueueManager class simple.
- I would also suggest that we make the deprecated and new class stick to a
unified interface that provides APIs like:
{code}
Queue getRoot();
boolean areAclsEnabled();
{code}
These can be used by the QueueManager on the appropriate builder depending on
whether old configuration is available or not. This way, the initialization
code (like in the constructor) of QueueManager will be consistent, irrespective
of where the queue configuration is.
- Also, since we want to discard the mapred-site.xml loaded properties ASAP, we
probably should parse and build the hierarchy from the constructor of the
builder itself.
- createHierarchy should be refactored to parse leaf level queues and non leaf
level queues separately. I would suggest something along the lines of:
{code}
for each queue node:
newQueue.name = parseName()
newQueue.properties = parseProperties()
if (node is a leaf level queue) {
parseLeafQueueProperties() // here we parse ACLs, state and other
properties.
} else {
for every queue node:
recursively parse queue nodes.
}
{code}
- Calling getInnerQueues without calling addChild can cause a null pointer
because children is null.
- Also, the method getInnerQueues doesn't seem to be used. So, is it required ?
- hashCode is overridden, but not equals
- I assume the order of child elements of an XML tag will be in the order in
which they are defined in the file. I think we must explicitly ensure that
elements come in a fixed order. For e.g. currently, we are not enforcing an
order on how tags under the 'queue' node are defined. This could lead to cases
where if a 'queue' element comes before the 'name' element, we would call parse
hierarchy for the child queue without the parent's name.
- Shouldn't we check in populateProperties that the node is of type element.
- Since getNamedItem can be null, we should check for it in populateProperties.
This will allow us to throw a better exception.
- In QueueManager.initialize, allQueues.putAll(leafQueues) seems unnecessary,
since getInnerQueues already does this.
- setSchedulerInfo and getSchedulerInfo should work on all queues, not just
leaf queues.
- The current refresh code is broken with reference to what we want to do long
term. We should only change properties of leaf queues. Since we are planning to
change the refresh model in MAPREDUCE-893, I think for completeness sake the
refresh of the code should build the entire hierarchy, but pickup only leaf
level queues and copy properties like acls and state. Currently, since the
refresh code is adding to the root queues, hierarchy could be affected which is
not in the scope even of MAPREDUCE-893.
- getJobQueueInfos is calling getJobQueueInfo twice. We should avoid this. Not
a bug in the patch, as it was already there, but it will help to fix it.
- dumpConfiguration should be changed. It should be changed to the new JSON
format.
- DeprecatedQueueHierarchyBuilder should not hold on to an instance of the
configuration
Some minor comments on style:
> Modify queue configuration format and parsing to support a hierarchy of
> queues.
> ---
>
> Key: MAPREDUCE-861
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-861
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Hemanth Yamijala
>Assignee: rahul k singh
> Attachments: MAPREDUCE-861-1.patch
>
>
> MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce
> framework. This JIRA is for defining changes to the configuration related to
> queues.
> The current format for defining a queue and its properties is as follows:
> mapred.queue... For e.g.
> mapred.queue..acl-submit-job. The reason for using this verbose
> format was to be able to reuse the Configuration parser in Hadoop. However,
> administrators currently using the queue configuration have already indicated
> a very strong desire for a more manageable format. Since, this becomes more
> unwieldy with hierarchical queues, the time may be good to introduce a new
> format for representing queue configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-928) JobTracker startup can get confused if the mapred system dir isn't there
[
https://issues.apache.org/jira/browse/MAPREDUCE-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-928.
--
Resolution: Cannot Reproduce
rebuilding everything makes it go away, I think someone has found and fixed
this already. For the indexing tools out there, here is the stack trace.
{code}
[sf-startdaemon-debug] 09/08/28 12:21:06 [JobTracker] INFO mapred.JobTracker :
problem cleaning system directory:
hdfs://localhost:8020/tmp/hadoop/mapred/system: java.io.FileNotFoundException:
File hdfs://localhost:8020/tmp/hadoop/mapred/system does not exist.
[sf-startdaemon-debug] java.io.FileNotFoundException: File
hdfs://localhost:8020/tmp/hadoop/mapred/system does not exist.
[sf-startdaemon-debug] at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:262)
[sf-startdaemon-debug] at
org.apache.hadoop.mapred.JobTracker.innerStart(JobTracker.java:2012)
[sf-startdaemon-debug] at
org.apache.hadoop.util.Service.start(Service.java:185)
[sf-startdaemon-debug] at
org.smartfrog.services.hadoop.components.cluster.HadoopServiceImpl.innerDeploy(HadoopServiceImpl.java:522)
[sf-startdaemon-debug] at
org.smartfrog.services.hadoop.components.cluster.HadoopServiceImpl.access$000(HadoopServiceImpl.java:53)
[sf-startdaemon-debug] at
org.smartfrog.services.hadoop.components.cluster.HadoopServiceImpl$ServiceDeployerThread.execute(HadoopServiceImpl.java:786)
{code}
> JobTracker startup can get confused if the mapred system dir isn't there
>
>
> Key: MAPREDUCE-928
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-928
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.21.0
>Reporter: Steve Loughran
>
> I'm seeing this in my branch, the JobTracker is catching and ignoring a
> FileNotFoundException when it tries to delete a nonexistent system dir
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-928) JobTracker startup can get confused if the mapred system dir isn't there
JobTracker startup can get confused if the mapred system dir isn't there Key: MAPREDUCE-928 URL: https://issues.apache.org/jira/browse/MAPREDUCE-928 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.21.0 Reporter: Steve Loughran I'm seeing this in my branch, the JobTracker is catching and ignoring a FileNotFoundException when it tries to delete a nonexistent system dir -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-777) A method for finding and tracking jobs from the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748780#action_12748780
]
Amareshwari Sriramadasu commented on MAPREDUCE-777:
---
Some questions/comments on proposed api:
1.
{code}
+
+ /* mapred.Queue.QueueState needs to extend this class */
+ public static enum QueueState {
+STOPPED("stopped"), RUNNING("running");
{code}
enum cannot be extended in java. Owen, do you mean I should wrap this in a
class?
2. bq. public Job getJob(JobID job) throws IOException { return null; }
For returning Job handle from JobID, we should find a way to get configuration
of the job, through JobSubmissionProtocol.
3.
{code}
+ public QueueInfo getQueue(String name) throws IOException { return null; }
+ public Metrics getClusterStatus() throws IOException { return null; }
{code}
For these apis, JobSubmissionProtocol should be modified to return QueueInfo
and Metrics, instead of bigger objects JobQueueInfo and ClusterStatus, right?
> A method for finding and tracking jobs from the new API
> ---
>
> Key: MAPREDUCE-777
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-777
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: client
>Reporter: Owen O'Malley
>Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: m-777.patch, patch-777-1.txt, patch-777-2.txt,
> patch-777.txt
>
>
> We need to create a replacement interface for the JobClient API in the new
> interface. In particular, the user needs to be able to query and track jobs
> that were launched by other processes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-927) Cleanup of task-logs should happen in TaskTracker instead of the Child
Cleanup of task-logs should happen in TaskTracker instead of the Child -- Key: MAPREDUCE-927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-927 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Reporter: Vinod K V Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-926) History viewer on web UI should filter by job-id also
History viewer on web UI should filter by job-id also - Key: MAPREDUCE-926 URL: https://issues.apache.org/jira/browse/MAPREDUCE-926 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Vinod K V Job-id is the most famous handle to a job and there should be easier ways of hunting down a job from the history web viewer. Currently, filtering is supported to be based on job name and job owner's name. Job-id is a necessary addition to the list of filters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-849) Renaming of configuration property names in mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748763#action_12748763 ] Amareshwari Sriramadasu commented on MAPREDUCE-849: --- bq. All job-level keys will be added to org.apache.hadoop.mapreduce.JobContext as public static String. bq. These keys will be protected static String, to be in consistent with existing keys. My bad. Job level keys should be declared as public as for now, to access the keys everywhere. They cannot be protected Strings > Renaming of configuration property names in mapreduce > - > > Key: MAPREDUCE-849 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-849 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: Config changes.xls, Config changes.xls > > > In-line with HDFS-531, property names in configuration files should be > standardized in MAPREDUCE. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-925) Not able to run parallel instances of ant tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748744#action_12748744 ] Vinod K V commented on MAPREDUCE-925: - AFAIK this has always been like this. The test suite was never intended to be running parallel tests as most of the test share the same temporary space. Unless anyone objects, I'll close this as WON"T FIX. > Not able to run parallel instances of ant tests > --- > > Key: MAPREDUCE-925 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-925 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.20.1 >Reporter: Suman Sehgal >Priority: Minor > > Not able to execute two or more parallel instances of ant test. > Scenario: > === > Executed "TestRecoveryManager" in a loop for 100 times and ran some other > tests in parallel. Encountered following error message in TestRecoveryManager > logs: > [junit] 2009-08-27 10:53:48,724 INFO mapred.TaskTracker > (TaskTracker.java:tryToGetOutputSize(1594)) - > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > taskTracker/jobcache///output/file.out in any of the > configured local directories > It seems that new ant test removes the existing mapred local directory itself > and recreates it for current test. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-925) Not able to run parallel instances of ant tests
Not able to run parallel instances of ant tests --- Key: MAPREDUCE-925 URL: https://issues.apache.org/jira/browse/MAPREDUCE-925 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.20.1 Reporter: Suman Sehgal Priority: Minor Not able to execute two or more parallel instances of ant test. Scenario: === Executed "TestRecoveryManager" in a loop for 100 times and ran some other tests in parallel. Encountered following error message in TestRecoveryManager logs: [junit] 2009-08-27 10:53:48,724 INFO mapred.TaskTracker (TaskTracker.java:tryToGetOutputSize(1594)) - org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache///output/file.out in any of the configured local directories It seems that new ant test removes the existing mapred local directory itself and recreates it for current test. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748721#action_12748721 ] Amareshwari Sriramadasu commented on MAPREDUCE-370: --- -1 core tests. Test failure, TestRecoveryManager is due to MAPREDUCE-880. TestPipes is due to MAPREDUCE-924. TestSeveral crashed on hudson, but it passed on my local machine. > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370-2.txt, patch-370-3.txt, > patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-899: Attachment: MAPREDUCE-899-20090828.txt Attaching a patch that does the following: - Put in checks in taskcontroller to make sure that permissions are properly set.. For this, we check whether task-controller binary is -- set setuid and setgid bits -- user-owned by root -- group-owned by a special group to which only TT is a member. To check this, we --- scan the entry in group database for the special group and make sure it has only one member which is the tt_user --- scan all the entries in passwd database and make sure that only tt_user has the special group as its primary group. - The checks are made whenever task-controller binary is used in various operations like initialize_job, intialize_task etc. - The check is also made during TT start up so as to fail early in case. This is done by a plain run of the task-controller binary. No new tests are included, existing tests test this issue when combined with various combinations of ownership on the binary. > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-899-20090828.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-924) TestPipes crashes on trunk
TestPipes crashes on trunk -- Key: MAPREDUCE-924 URL: https://issues.apache.org/jira/browse/MAPREDUCE-924 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Reporter: Amareshwari Sriramadasu Fix For: 0.21.0 TestPipes crashes on trunk due to MAPREDUCE-421. Testcase should be modified to use Tool insteadof running main directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-832) Too many WARN messages about deprecated memorty config variables in JobTacker log
[ https://issues.apache.org/jira/browse/MAPREDUCE-832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-832: --- Release Note: Reduced the frequency of log messages printed when a deprecated memory management variable is found in configuration of a job. > Too many WARN messages about deprecated memorty config variables in JobTacker > log > - > > Key: MAPREDUCE-832 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-832 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Karam Singh >Assignee: rahul k singh > Fix For: 0.20.1 > > Attachments: mapreduce-832-20.patch, mapreduce-832-20.patch, > mapreduce-832.patch > > > When user submit a mapred job using old memory config vairiable > (mapred.task.maxmem) followinig message too many times in JobTracker logs -: > [ > WARN org.apache.hadoop.mapred.JobConf: The variable mapred.task.maxvmem is no > longer used instead use mapred.job.map.memory.mb and > mapred.job.reduce.memory.mb > ] -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881
[ https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-40: -- Release Note: Fixed backwards compatibility by re-introducing and deprecating removed memory monitoring related configuration options. (was: Backward compatibility is provided for memory management .) > Memory management variables need a backwards compatibility option after > HADOOP-5881 > --- > > Key: MAPREDUCE-40 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-40 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Hemanth Yamijala >Assignee: rahul k singh >Priority: Blocker > Fix For: 0.20.1 > > Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, > hadoop-5919-11.patch, hadoop-5919-12-20.patch, hadoop-5919-12.patch, > hadoop-5919-13-20.patch, hadoop-5919-13.patch, hadoop-5919-14-20.patch, > hadoop-5919-14.patch, hadoop-5919-15-20.patch, hadoop-5919-15.patch, > hadoop-5919-2.patch, hadoop-5919-3.patch, hadoop-5919-4.patch, > hadoop-5919-5.patch, hadoop-5919-6.patch, hadoop-5919-7.patch, > hadoop-5919-8.patch, hadoop-5919-9.patch > > > HADOOP-5881 modified variables related to memory management without looking > at the backwards compatibility angle. This JIRA is to adress the gap. Marking > it a blocker for 0.20.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-818) org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name doesnt exist.
[ https://issues.apache.org/jira/browse/MAPREDUCE-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-818: --- Release Note: Fixed a bug in the new org.apache.hadoop.mapreduce.Counters.getGroup() method to return an empty group if group name doesn't exist, instead of null, thus making sure that it is in sync with the Javadoc. > org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name > doesnt exist. > -- > > Key: MAPREDUCE-818 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-818 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 0.20.1 > > Attachments: patch-818.txt > > > org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name > doesnt exist. But the documentation says it returns an empty group if there > is none with the specified name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-383) pipes combiner does not reset properly after a spill
[ https://issues.apache.org/jira/browse/MAPREDUCE-383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Yamijala updated MAPREDUCE-383: --- Release Note: Fixed a bug in Pipes combiner to reset the spilled bytes count after the spill. > pipes combiner does not reset properly after a spill > > > Key: MAPREDUCE-383 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-383 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.1 > > Attachments: patch.HADOOP-6070 > > > When using a pipes combiner, the variable numBytes is not reset to 0 in > spillAll, effectively reducing the effect of running a combiner to the first > spill. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-370) Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api.
[ https://issues.apache.org/jira/browse/MAPREDUCE-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748701#action_12748701 ] Hadoop QA commented on MAPREDUCE-370: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417954/patch-370-3.txt against trunk revision 808730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/533/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/533/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/533/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/533/console This message is automatically generated. > Change org.apache.hadoop.mapred.lib.MultipleOutputs to use new api. > --- > > Key: MAPREDUCE-370 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-370 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.21.0 > > Attachments: patch-370-1.txt, patch-370-2.txt, patch-370-3.txt, > patch-370.txt > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-40) Memory management variables need a backwards compatibility option after HADOOP-5881
[ https://issues.apache.org/jira/browse/MAPREDUCE-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rahul k singh updated MAPREDUCE-40: --- Release Note: Backward compatibility is provided for memory management . > Memory management variables need a backwards compatibility option after > HADOOP-5881 > --- > > Key: MAPREDUCE-40 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-40 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Hemanth Yamijala >Assignee: rahul k singh >Priority: Blocker > Fix For: 0.20.1 > > Attachments: hadoop-5919-1.patch, hadoop-5919-10.patch, > hadoop-5919-11.patch, hadoop-5919-12-20.patch, hadoop-5919-12.patch, > hadoop-5919-13-20.patch, hadoop-5919-13.patch, hadoop-5919-14-20.patch, > hadoop-5919-14.patch, hadoop-5919-15-20.patch, hadoop-5919-15.patch, > hadoop-5919-2.patch, hadoop-5919-3.patch, hadoop-5919-4.patch, > hadoop-5919-5.patch, hadoop-5919-6.patch, hadoop-5919-7.patch, > hadoop-5919-8.patch, hadoop-5919-9.patch > > > HADOOP-5881 modified variables related to memory management without looking > at the backwards compatibility angle. This JIRA is to adress the gap. Marking > it a blocker for 0.20.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
