[jira] Updated: (MAPREDUCE-2062) speculative execution is too aggressive under certain conditions
[ https://issues.apache.org/jira/browse/MAPREDUCE-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joydeep Sen Sarma updated MAPREDUCE-2062: - Environment: hadoop-20 with HADOOP-2141 > speculative execution is too aggressive under certain conditions > > > Key: MAPREDUCE-2062 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2062 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker > Environment: hadoop-20 with HADOOP-2141 >Reporter: Joydeep Sen Sarma > > The function canBeSpeculated has subtle bugs that cause too much speculation > in certain cases. > - it compares the current progress of the task with the last observed mean of > all the tasks. if only one task is in question - then the progress rate > decays as time progresses (in the absence of updates) and std-dev is zero. So > a job with a single reducer or mapper is almost always speculated. > - is only a single task has reported progress - then the stddev is zero. so > other tasks may be speculated aggressively. > - several tasks take a while to report progress initially. they seem to get > speculated as soon as speculative-lag is over. the lag should be configurable > at the minimum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2062) speculative execution is too aggressive under certain conditions
speculative execution is too aggressive under certain conditions Key: MAPREDUCE-2062 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2062 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Reporter: Joydeep Sen Sarma The function canBeSpeculated has subtle bugs that cause too much speculation in certain cases. - it compares the current progress of the task with the last observed mean of all the tasks. if only one task is in question - then the progress rate decays as time progresses (in the absence of updates) and std-dev is zero. So a job with a single reducer or mapper is almost always speculated. - is only a single task has reported progress - then the stddev is zero. so other tasks may be speculated aggressively. - several tasks take a while to report progress initially. they seem to get speculated as soon as speculative-lag is over. the lag should be configurable at the minimum. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2061) child environment substitution is broken
child environment substitution is broken
Key: MAPREDUCE-2061
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2061
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Hong Tang
TaskRunner.getVMEnvironment is still broken in a couple of ways even after
HADOOP-5981:
* It does not recognize ${VAR} notation. This is necessary if we have both
"VAR" and "VAR_1" existent, and it becomes ambiguous whether $VAR_1 means
appending _1 following $VAR, or simply taking the value of $VAR_1
* It tries to do lazy-binding except for self-referencing like X=$X:Y. This
would cause some unexpected behavior. For instance, if I specify
"A=$LD_LIBRARY_PATH:xxx", A would be shown as ":xxx", while if I specify
"LD_LIBRARY_PATH=$Z:xxx,Z=$LD_LIBRARY_PATH:yyy", Z would be come the actual
value of LD_LIBRARY_PATH + ":yyy" (and LD_LIBRARY_PATH is not affected at all).
I think we should support eager binding (and we can support lazy binding later
through quoting or escaping).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2055) Retired job info cache should contain more details than just JobStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-2055: -- Attachment: fix4_MAPREDUCE-2055.patch By guessing the scope of the work I feel like the patch is going to break Herriot compilation. So, here's an approximate patch for the problem if it will every come out. > Retired job info cache should contain more details than just JobStatus > -- > > Key: MAPREDUCE-2055 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2055 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Krishna Ramachandran > Fix For: 0.22.0 > > Attachments: fix4_MAPREDUCE-2055.patch > > > RetireJobInfo cache currently holds only job status. Other data like profile, > counters are obtained from CompletedJobStatusStore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1434) Dynamic add input for one job
[ https://issues.apache.org/jira/browse/MAPREDUCE-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908171#action_12908171 ] Greg Roelofs commented on MAPREDUCE-1434: - I'm pretty sure "Submit Patch" is the preferred way to signal that a patch is ready for review; that will also trigger Hudson to perform automatic checks (or should, anyway--Hudson's been pretty flaky lately). Before you do that, however, you should review the official Hadoop code conventions, which are the same as the Sun/Oracle ones (http://www.oracle.com/technetwork/java/codeconv-138413.html) except with half the indentation (4 -> 2, 8 -> 4, etc.). I just skimmed over your patch and noticed a strange mix of 2-space, 3-space, 4-space, and 8-space indentation...that makes the code very hard to read. > Dynamic add input for one job > - > > Key: MAPREDUCE-1434 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1434 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 0.20.3 >Reporter: Xing Shi > Fix For: 0.20.3 > > Attachments: dynamic_input-v1.patch > > > Always we should firstly upload the data to hdfs, then we can analize the > data using hadoop mapreduce. > Sometimes, the upload process takes long time. So if we can add input during > one job, the time can be saved. > WHAT? > Client: > a) hadoop job -add-input jobId inputFormat ... > Add the input to jobid > b) hadoop job -add-input done > Tell the JobTracker, the input has been prepared over. > c) hadoop job -add-input status jobid > Show how many input the jobid has. > HOWTO? > Mainly, I think we should do three things: > 1. JobClinet: here JobClient should support add input to a job, indeed, > JobClient generate the split, and submit to JobTracker. > 2. JobTracker: JobTracker support addInput, and add the new tasks to the > original mapTasks. Because the uploaded data will be > processed quickly, so it also should update the scheduler to support pending > a map task till Client tells the Job input done. > 3. Reducer: the reducer should also update the mapNums, so it will shuffle > right. > This is the rough idea, and I will update it . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-1943: - Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch sorry attached a wrong file. > Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes > > > Key: MAPREDUCE-1943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1943-0.20-yahoo.patch, > MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S.patch > > > We have come across issues in production clusters wherein users abuse > counters, statusreport messages and split sizes. One such case was when one > of the users had 100 million counters. This leads to jobtracker going out of > memory and being unresponsive. In this jira I am proposing to put sane limits > on the status report length, the number of counters and the size of block > locations returned by the input split. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-1943: - Attachment: (was: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch) > Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes > > > Key: MAPREDUCE-1943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1943-0.20-yahoo.patch, > MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S.patch > > > We have come across issues in production clusters wherein users abuse > counters, statusreport messages and split sizes. One such case was when one > of the users had 100 million counters. This leads to jobtracker going out of > memory and being unresponsive. In this jira I am proposing to put sane limits > on the status report length, the number of counters and the size of block > locations returned by the input split. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1943) Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-1943: - Attachment: MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch fixes minor bug in my earlier patch with respect to configuration settings and calling getjobcounters without job being initialized. > Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes > > > Key: MAPREDUCE-1943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1943-0.20-yahoo.patch, > MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S.patch > > > We have come across issues in production clusters wherein users abuse > counters, statusreport messages and split sizes. One such case was when one > of the users had 100 million counters. This leads to jobtracker going out of > memory and being unresponsive. In this jira I am proposing to put sane limits > on the status report length, the number of counters and the size of block > locations returned by the input split. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2060) IOException: Filesystem closed on submitJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908087#action_12908087
]
Dan Adkins commented on MAPREDUCE-2060:
---
> Would It be possible that the JobTracker and the user (who submits jobs)
> using the same account?
It is certainly possible, though it is not the case for this particular job.
For all the talk about whether the JobTracker fs is opened under the JobTracker
user, it appears that it is not opened under any user:
src/java/org/apache/hadoop/mapred/JobTracker.java
@@ 4092,3 @@ JobTracker(final JobConf conf, Clock clock, boolean
ignoredForSimulation)
if (fs == null) {
fs = FileSystem.get(conf);
}
> IOException: Filesystem closed on submitJob
> ---
>
> Key: MAPREDUCE-2060
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2060
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.22.0
> Environment:
> https://svn.apache.org/repos/asf/hadoop/mapreduce/tr...@994941
> https://svn.apache.org/repos/asf/hadoop/hdfs/tr...@993542
>Reporter: Dan Adkins
>
> I get the following strange error on the jobtracker when attempting to submit
> a job:
> 10/09/09 20:31:35 INFO ipc.Server: IPC Server handler 7 on 31000, call
> submitJob(job_201009092028_0001,
> hdfs://hns4.sea1.qc:21000/tmp/hadoop-mr20/mapred/staging/dadkins/.staging/job_201009092028_0001,
> org.apache.hadoop.security.credenti...@20c87621) from 10.128.130.145:49253:
> error: java.io.IOException: Filesystem closed
> java.io.IOException: Filesystem closed
> at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:307)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:1212)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:494)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1491)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:395)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3078)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3014)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2996)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:349)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1380)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1376)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1105)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1374)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1548) Hadoop archives should be able to preserve times and other properties from original files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907937#action_12907937 ] Rodrigo Schmidt commented on MAPREDUCE-1548: @Mahadev: Yes, the propsplits part is a little cryptic. Adding more comments makes a lot of sense. I just uploaded a new version. Let me know if it is fine that way. Cheers! > Hadoop archives should be able to preserve times and other properties from > original files > - > > Key: MAPREDUCE-1548 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1548 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Affects Versions: 0.22.0 >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1548.0.patch, MAPREDUCE-1548.1.patch, > MAPREDUCE-1548.2.patch, MAPREDUCE-1548.3.patch, MAPREDUCE-1548.4.patch > > > Files inside hadoop archives don't keep their original: > - modification time > - access time > - permission > - owner > - group > all such properties are currently taken from the file storing the archive > index, and not the stored files. This doesn't look very correct. > There should be possible to preserve the original properties of the stored > files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1548) Hadoop archives should be able to preserve times and other properties from original files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt updated MAPREDUCE-1548: --- Status: Patch Available (was: Open) > Hadoop archives should be able to preserve times and other properties from > original files > - > > Key: MAPREDUCE-1548 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1548 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Affects Versions: 0.22.0 >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1548.0.patch, MAPREDUCE-1548.1.patch, > MAPREDUCE-1548.2.patch, MAPREDUCE-1548.3.patch, MAPREDUCE-1548.4.patch > > > Files inside hadoop archives don't keep their original: > - modification time > - access time > - permission > - owner > - group > all such properties are currently taken from the file storing the archive > index, and not the stored files. This doesn't look very correct. > There should be possible to preserve the original properties of the stored > files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1548) Hadoop archives should be able to preserve times and other properties from original files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt updated MAPREDUCE-1548: --- Status: Open (was: Patch Available) > Hadoop archives should be able to preserve times and other properties from > original files > - > > Key: MAPREDUCE-1548 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1548 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Affects Versions: 0.22.0 >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1548.0.patch, MAPREDUCE-1548.1.patch, > MAPREDUCE-1548.2.patch, MAPREDUCE-1548.3.patch, MAPREDUCE-1548.4.patch > > > Files inside hadoop archives don't keep their original: > - modification time > - access time > - permission > - owner > - group > all such properties are currently taken from the file storing the archive > index, and not the stored files. This doesn't look very correct. > There should be possible to preserve the original properties of the stored > files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1548) Hadoop archives should be able to preserve times and other properties from original files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rodrigo Schmidt updated MAPREDUCE-1548: --- Attachment: MAPREDUCE-1548.4.patch > Hadoop archives should be able to preserve times and other properties from > original files > - > > Key: MAPREDUCE-1548 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1548 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: harchive >Affects Versions: 0.22.0 >Reporter: Rodrigo Schmidt >Assignee: Rodrigo Schmidt > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1548.0.patch, MAPREDUCE-1548.1.patch, > MAPREDUCE-1548.2.patch, MAPREDUCE-1548.3.patch, MAPREDUCE-1548.4.patch > > > Files inside hadoop archives don't keep their original: > - modification time > - access time > - permission > - owner > - group > all such properties are currently taken from the file storing the archive > index, and not the stored files. This doesn't look very correct. > There should be possible to preserve the original properties of the stored > files. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
