[jira] [Created] (MAPREDUCE-5276) johistory.jsp hangs when a directory under ./done is empty
Alexander Alten-Lorenz created MAPREDUCE-5276: - Summary: johistory.jsp hangs when a directory under ./done is empty Key: MAPREDUCE-5276 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5276 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.23.4 Reporter: Alexander Alten-Lorenz Priority: Minor WARN from jobtracker.log 013-05-22 17:07:36,240 WARN org.apache.hadoop.mapred.JobHistory: JobHistory: existingDoneSubdirs doesn't contain file:/log/history/done/mr-hadoop1_1363076693931_/2013/03/21/000623, but should. the jobhistory page stops, which causes a empty page. The jobhistory.jsp should go over this WARN and proceed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4548) M/R jobs can not access S3 if Kerberos is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13667909#comment-13667909
]
Anoop Sam John commented on MAPREDUCE-4548:
---
I also met with this issue today. Need to check with the patch any way.
[~mdeferran] you want create and attach patch?
M/R jobs can not access S3 if Kerberos is enabled
-
Key: MAPREDUCE-4548
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4548
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: hadoop-1.0.0;MIT kerberos;java 1.6.0_26
Reporter: Manuel DE FERRAN
With Kerberos enabled, any job that is taking as input or output s3 files
fails.
It can be easily reproduced with wordcount shipped in hadoop-examples.jar and
a public S3 file:
{code}
/opt/hadoop/bin/hadoop --config /opt/hadoop/conf/ jar
/opt/hadoop/hadoop-examples-1.0.0.jar wordcount s3n://ubikodpublic/test out01
{code}
returns:
{code}
12/08/10 12:40:19 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
192 for hadoop on 10.85.151.233:9000
12/08/10 12:40:19 INFO security.TokenCache: Got dt for
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004;uri=10.85.151.233:9000;t.service=10.85.151.233:9000
12/08/10 12:40:19 INFO mapred.JobClient: Cleaning up the staging area
hdfs://aws04.machine.com:9000/mapred/staging/hadoop/.staging/job_201208101229_0004
java.lang.IllegalArgumentException: java.net.UnknownHostException:
ubikodpublic
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:293)
at
org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:317)
at
org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:189)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:92)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:79)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:197)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252)
SNIP
{code}
This patch seems to fix it.
{code}
Index: core/org/apache/hadoop/security/SecurityUtil.java
===
--- core/org/apache/hadoop/security/SecurityUtil.java (révision 1305278)
+++ core/org/apache/hadoop/security/SecurityUtil.java (copie de travail)
@@ -313,6 +313,9 @@
if (authority == null || authority.isEmpty()) {
return null;
}
+if (uri.getScheme().equals(s3n) || uri.getScheme().equals(s3)) {
+ return null;
+}
InetSocketAddress addr = NetUtils.createSocketAddr(authority, defPort);
return buildTokenService(addr).toString();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5277) Job history completed location cannot be on a file system other than default
Ivan Mitic created MAPREDUCE-5277: - Summary: Job history completed location cannot be on a file system other than default Key: MAPREDUCE-5277 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5277 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic mapred.job.tracker.history.completed.location should be configurable to a location on any available file system. This can come handy for cases where HDFS is not the only file system in use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5277) Job history completed location cannot be on a file system other than default
[ https://issues.apache.org/jira/browse/MAPREDUCE-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated MAPREDUCE-5277: -- Attachment: MAPREDUCE-5277.branch-1-win.patch Attaching the patch. Job history completed location cannot be on a file system other than default Key: MAPREDUCE-5277 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5277 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: MAPREDUCE-5277.branch-1-win.patch mapred.job.tracker.history.completed.location should be configurable to a location on any available file system. This can come handy for cases where HDFS is not the only file system in use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13667999#comment-13667999
]
Ivan Mitic commented on MAPREDUCE-5224:
---
Thanks Xi for taking time to address all comments! Latest patch looks good to
me, +1
bq. There is no need to use the default file system for the jobhistory. There
is another (orthogonal) bug here. Job history completed location also assumes
the default FS what is not correct. This should be a separate Jira.
I filed a Jira on this: MAPREDUCE-5277
JobTracker should allow the system directory to be in non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
MAPREDUCE-5224.4.patch, MAPREDUCE-5224.5.patch, MAPREDUCE-5224.patch
JobTracker today expects the system directory to be in the default file
system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We
should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5224) JobTracker should allow the system directory to be in non-default FS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13668000#comment-13668000
]
Ivan Mitic commented on MAPREDUCE-5224:
---
PS. I verified that the new test passes on Linux and on Windows.
JobTracker should allow the system directory to be in non-default FS
Key: MAPREDUCE-5224
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5224
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Minor
Fix For: 1-win
Attachments: MAPREDUCE-5224.2.patch, MAPREDUCE-5224.3.patch,
MAPREDUCE-5224.4.patch, MAPREDUCE-5224.5.patch, MAPREDUCE-5224.patch
JobTracker today expects the system directory to be in the default file
system
if (fs == null) {
fs = mrOwner.doAs(new PrivilegedExceptionActionFileSystem() {
public FileSystem run() throws IOException {
return FileSystem.get(conf);
}});
}
...
public String getSystemDir() {
Path sysDir = new Path(conf.get(mapred.system.dir,
/tmp/hadoop/mapred/system));
return fs.makeQualified(sysDir).toString();
}
In Cloud like Azure the default file system is set as ASV (Windows Azure Blob
Storage), but we would still like the system directory to be in DFS. We
should change JobTracker to allow that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5269) Preemption of Reducer (and Shuffle) via checkpointing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-5269: -- Assignee: Carlo Curino Preemption of Reducer (and Shuffle) via checkpointing - Key: MAPREDUCE-5269 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5269 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Carlo Curino Assignee: Carlo Curino Attachments: MAPREDUCE-5269.patch This patch tracks the changes in the task runtime (shuffle, reducer context, etc.) that are required to implement checkpoint-based preemption of reducer tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated MAPREDUCE-5194:
--
Assignee: Chris Douglas
Heed interrupts during Fetcher shutdown
---
Key: MAPREDUCE-5194
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
Project: Hadoop Map/Reduce
Issue Type: Task
Components: task
Reporter: Chris Douglas
Assignee: Chris Douglas
Priority: Minor
Attachments: M5194-1.patch, M5194.patch
In the current implementation, {{Fetcher}} instances usually exit gracefully
when the shuffle succeeds. When it fails, threads are interrupted, but may
continue running harmlessly until the JVM shuts down.
However, to generate consistent checkpoints, these threads should exit
cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
