[jira] [Commented] (HADOOP-7682) taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554028#comment-13554028 ] FKorning commented on HADOOP-7682: -- yes. it is extremely slow, which sort of makes the whole windows thing a bit moot. then again, if you have a farm of windows boxes sitting idle, you may as well use their cycles... Sent from my iPhone > taskTracker could not start because "Failed to set permissions" to "ttprivate > to 0700" > -- > > Key: HADOOP-7682 > URL: https://issues.apache.org/jira/browse/HADOOP-7682 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 1.0.1 > Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, > jdk1.6.0_05 >Reporter: Magic Xie > > ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because > java.io.IOException:Failed to set permissions of > path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 > at > org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) > at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) > at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) > Since hadoop0.20.203 when the TaskTracker initialize, it checks the > permission(TaskTracker Line 624) of > (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, > > org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) > call setPermission(Line 481) to deal with it, setPermission works fine on > *nx, however,it dose not alway works on windows. > setPermission call setReadable of Java.io.File in the line 498, but according > to the Table1 below provided by oracle,setReadable(false) will always return > false on windows, the same as setExecutable(false). > http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ > is it cause the task tracker "Failed to set permissions" to "ttprivate to > 0700"? > Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8274) In pseudo or cluster model under Cygwin, tasktracker can not create a new job because of symlink problem.
[ https://issues.apache.org/jira/browse/HADOOP-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553191#comment-13553191 ] FKorning commented on HADOOP-8274: -- Yes, You'll need to make LinkedFile recursively traverse through symlinks. I just did a quick hack to get it to resolve the last basename as a link. > In pseudo or cluster model under Cygwin, tasktracker can not create a new job > because of symlink problem. > - > > Key: HADOOP-8274 > URL: https://issues.apache.org/jira/browse/HADOOP-8274 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.20.205.0, 1.0.0, 1.0.1, 0.22.0 > Environment: windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0 >Reporter: tim.wu > > The standalone model is ok. But, in pseudo or cluster model, it always throw > errors, even I just run wordcount example. > The HDFS works fine, but tasktracker can not create threads(jvm) for new job. > It is empty under /logs/userlogs/job-/attempt-/. > The reason looks like that in windows, Java can not recognize a symlink of > folder as a folder. > The detail description is as following, > == > First, the error log of tasktracker is like: > == > 12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID: > jvm_201203280212_0005_m_-1386636958 > 12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner > jvm_201203280212_0005_m_-1386636958 spawned. > 12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed > jvm_201203280212_0005_m_-1386636958 but just removed > 12/03/28 14:35:17 INFO mapred.JvmManager: JVM : > jvm_201203280212_0005_m_-1386636958 exited with exit code -1. Number of tasks > it ran: 0 > 12/03/28 14:35:17 WARN mapred.TaskRunner: > attempt_201203280212_0005_m_02_0 : Child Error > java.io.IOException: Task process exit with nonzero status of -1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > 12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots : > 2 > 12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask): > attempt_201203280212_0005_m_02_1 task's state:UNASSIGNED > 12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch : > attempt_201203280212_0005_m_02_1 which needs 1 slots > 12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free > slots : 2 and trying to launch attempt_201203280212_0005_m_02_1 which > needs 1 slots > 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for > task: attempt_201203280212_0005_m_02_0 > java.io.FileNotFoundException: > D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_02_0\log.index > (The system cannot find the path specified) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:120) > at > org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102) > at > org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188) > at org.apache.hadoop.mapred.TaskLog$Reader.(TaskLog.java:423) > at > org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81) > at > org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >
[jira] [Commented] (HADOOP-7682) taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397563#comment-13397563 ] FKorning commented on HADOOP-7682: -- Ismail, You misunderstand, I haven't patched the offical 1.0.1 codebase: I'm not an official hadoop contributor, I'm not really sure if the developers will approve all my fixes such as adding a java.io.File wrapper etc. (to wit, no persons from the hadoop team, not even those who were assigned to the various cygwin bugs, have contacted me on this). what I provide are instructions for you to patch 1.0.1 yourself, and hopefully provide guidance to the official hadoop developpers on porting the software properly to cygwin. and no, if you read the doc, you'll see why there is no quick workaround, short of falling back to 0.20. But I rather think most people would want a 1.x, especially compiled with 64-bit. it's not a simple code fix. you need to understand how cygwin works; you need to customize and configure cygwin and the pre-requisite sofware such as java, ant, ivy, maven; you need to patch scripts; you need to patch code; you need to add brand new classes to wrap java.io.File; you need to rebuild, reconfigure and redeploy it; for a complete working fix, I want to port the latest version and release the code on sourceforge, but I'm a bit busy now. > taskTracker could not start because "Failed to set permissions" to "ttprivate > to 0700" > -- > > Key: HADOOP-7682 > URL: https://issues.apache.org/jira/browse/HADOOP-7682 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 1.0.1 > Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, > jdk1.6.0_05 >Reporter: Magic Xie > > ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because > java.io.IOException:Failed to set permissions of > path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 > at > org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) > at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) > at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) > Since hadoop0.20.203 when the TaskTracker initialize, it checks the > permission(TaskTracker Line 624) of > (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, > > org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) > call setPermission(Line 481) to deal with it, setPermission works fine on > *nx, however,it dose not alway works on windows. > setPermission call setReadable of Java.io.File in the line 498, but according > to the Table1 below provided by oracle,setReadable(false) will always return > false on windows, the same as setExecutable(false). > http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ > is it cause the task tracker "Failed to set permissions" to "ttprivate to > 0700"? > Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8274) In pseudo or cluster model under Cygwin, tasktracker can not create a new job because of symlink problem.
[ https://issues.apache.org/jira/browse/HADOOP-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268985#comment-13268985 ] FKorning commented on HADOOP-8274: -- make that: http://en.wikisource.org/wiki/User:Fkorning/Code/Hadoop-on-Cygwin > In pseudo or cluster model under Cygwin, tasktracker can not create a new job > because of symlink problem. > - > > Key: HADOOP-8274 > URL: https://issues.apache.org/jira/browse/HADOOP-8274 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.20.205.0, 1.0.0, 1.0.1, 0.22.0 > Environment: windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0 >Reporter: tim.wu > > The standalone model is ok. But, in pseudo or cluster model, it always throw > errors, even I just run wordcount example. > The HDFS works fine, but tasktracker can not create threads(jvm) for new job. > It is empty under /logs/userlogs/job-/attempt-/. > The reason looks like that in windows, Java can not recognize a symlink of > folder as a folder. > The detail description is as following, > == > First, the error log of tasktracker is like: > == > 12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID: > jvm_201203280212_0005_m_-1386636958 > 12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner > jvm_201203280212_0005_m_-1386636958 spawned. > 12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed > jvm_201203280212_0005_m_-1386636958 but just removed > 12/03/28 14:35:17 INFO mapred.JvmManager: JVM : > jvm_201203280212_0005_m_-1386636958 exited with exit code -1. Number of tasks > it ran: 0 > 12/03/28 14:35:17 WARN mapred.TaskRunner: > attempt_201203280212_0005_m_02_0 : Child Error > java.io.IOException: Task process exit with nonzero status of -1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > 12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots : > 2 > 12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask): > attempt_201203280212_0005_m_02_1 task's state:UNASSIGNED > 12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch : > attempt_201203280212_0005_m_02_1 which needs 1 slots > 12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free > slots : 2 and trying to launch attempt_201203280212_0005_m_02_1 which > needs 1 slots > 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for > task: attempt_201203280212_0005_m_02_0 > java.io.FileNotFoundException: > D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_02_0\log.index > (The system cannot find the path specified) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:120) > at > org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102) > at > org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188) > at org.apache.hadoop.mapred.TaskLog$Reader.(TaskLog.java:423) > at > org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81) > at > org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerCompl
[jira] [Commented] (HADOOP-7682) taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268984#comment-13268984 ] FKorning commented on HADOOP-7682: -- make that: http://en.wikisource.org/wiki/User:Fkorning/Code/Hadoop-on-Cygwin > taskTracker could not start because "Failed to set permissions" to "ttprivate > to 0700" > -- > > Key: HADOOP-7682 > URL: https://issues.apache.org/jira/browse/HADOOP-7682 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0 > Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, > jdk1.6.0_05 >Reporter: Magic Xie > > ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because > java.io.IOException:Failed to set permissions of > path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 > at > org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) > at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) > at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) > Since hadoop0.20.203 when the TaskTracker initialize, it checks the > permission(TaskTracker Line 624) of > (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, > > org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) > call setPermission(Line 481) to deal with it, setPermission works fine on > *nx, however,it dose not alway works on windows. > setPermission call setReadable of Java.io.File in the line 498, but according > to the Table1 below provided by oracle,setReadable(false) will always return > false on windows, the same as setExecutable(false). > http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ > is it cause the task tracker "Failed to set permissions" to "ttprivate to > 0700"? > Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8274) In pseudo or cluster model under Cygwin, tasktracker can not create a new job because of symlink problem.
[ https://issues.apache.org/jira/browse/HADOOP-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268978#comment-13268978 ] FKorning commented on HADOOP-8274: -- I've managed to get this working to the point where jobs are dispatched, tasks executed, and results compiled. * [http://en.wikisource.org/wiki/User:Fkorning/Code/Haddoop-on-Cygwin] However we still need to get the servlets to understand cygwin symlinks. I have no idea how to do this in Jetty. These two links show how to allow Tomcat and jetty to follow symlinks, but I don't know if this works in cygwin. * http://www.lamoree.com/machblog/index.cfm?event=showEntry&entryId=A2F0ED76-A500-41A6-A1DFDE0D1996F925 * http://stackoverflow.com/questions/315093/configure-symlinks-for-single-directory-in-tomcat Otherwise we'll have to open up the jetty code and replace java.io.File with org.apache.hadoop.fs.LinkedFile. > In pseudo or cluster model under Cygwin, tasktracker can not create a new job > because of symlink problem. > - > > Key: HADOOP-8274 > URL: https://issues.apache.org/jira/browse/HADOOP-8274 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.20.205.0, 1.0.0, 1.0.1, 0.22.0 > Environment: windows7+cygwin 1.7.11-1+jdk1.6.0_31+hadoop 1.0.0 >Reporter: tim.wu > > The standalone model is ok. But, in pseudo or cluster model, it always throw > errors, even I just run wordcount example. > The HDFS works fine, but tasktracker can not create threads(jvm) for new job. > It is empty under /logs/userlogs/job-/attempt-/. > The reason looks like that in windows, Java can not recognize a symlink of > folder as a folder. > The detail description is as following, > == > First, the error log of tasktracker is like: > == > 12/03/28 14:35:13 INFO mapred.JvmManager: In JvmRunner constructed JVM ID: > jvm_201203280212_0005_m_-1386636958 > 12/03/28 14:35:13 INFO mapred.JvmManager: JVM Runner > jvm_201203280212_0005_m_-1386636958 spawned. > 12/03/28 14:35:17 INFO mapred.JvmManager: JVM Not killed > jvm_201203280212_0005_m_-1386636958 but just removed > 12/03/28 14:35:17 INFO mapred.JvmManager: JVM : > jvm_201203280212_0005_m_-1386636958 exited with exit code -1. Number of tasks > it ran: 0 > 12/03/28 14:35:17 WARN mapred.TaskRunner: > attempt_201203280212_0005_m_02_0 : Child Error > java.io.IOException: Task process exit with nonzero status of -1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > 12/03/28 14:35:21 INFO mapred.TaskTracker: addFreeSlot : current free slots : > 2 > 12/03/28 14:35:24 INFO mapred.TaskTracker: LaunchTaskAction (registerTask): > attempt_201203280212_0005_m_02_1 task's state:UNASSIGNED > 12/03/28 14:35:24 INFO mapred.TaskTracker: Trying to launch : > attempt_201203280212_0005_m_02_1 which needs 1 slots > 12/03/28 14:35:24 INFO mapred.TaskTracker: In TaskLauncher, current free > slots : 2 and trying to launch attempt_201203280212_0005_m_02_1 which > needs 1 slots > 12/03/28 14:35:24 WARN mapred.TaskLog: Failed to retrieve stdout log for > task: attempt_201203280212_0005_m_02_0 > java.io.FileNotFoundException: > D:\cygwin\home\timwu\hadoop-1.0.0\logs\userlogs\job_201203280212_0005\attempt_201203280212_0005_m_02_0\log.index > (The system cannot find the path specified) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:120) > at > org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102) > at > org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:188) > at org.apache.hadoop.mapred.TaskLog$Reader.(TaskLog.java:423) > at > org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81) > at > org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) > at > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jet
[jira] [Commented] (HADOOP-7682) taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
[ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268979#comment-13268979 ] FKorning commented on HADOOP-7682: -- I've managed to get this working to the point where jobs are dispatched, tasks executed, and results compiled. * [http://en.wikisource.org/wiki/User:Fkorning/Code/Haddoop-on-Cygwin] However we still need to get the servlets to understand cygwin symlinks. I have no idea how to do this in Jetty. These two links show how to allow Tomcat and jetty to follow symlinks, but I don't know if this works in cygwin. * http://www.lamoree.com/machblog/index.cfm?event=showEntry&entryId=A2F0ED76-A500-41A6-A1DFDE0D1996F925 * http://stackoverflow.com/questions/315093/configure-symlinks-for-single-directory-in-tomcat Otherwise we'll have to open up the jetty code and replace java.io.File with org.apache.hadoop.fs.LinkedFile. > taskTracker could not start because "Failed to set permissions" to "ttprivate > to 0700" > -- > > Key: HADOOP-7682 > URL: https://issues.apache.org/jira/browse/HADOOP-7682 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0 > Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, > jdk1.6.0_05 >Reporter: Magic Xie > > ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because > java.io.IOException:Failed to set permissions of > path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700 > at > org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499) > at > org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318) > at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183) > at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635) > at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328) > at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430) > Since hadoop0.20.203 when the TaskTracker initialize, it checks the > permission(TaskTracker Line 624) of > (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR, > > org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR).RawLocalFileSystem(http://svn.apache.org/viewvc/hadoop/common/tags/release-0.20.203.0/src/core/org/apache/hadoop/fs/RawLocalFileSystem.java?view=markup) > call setPermission(Line 481) to deal with it, setPermission works fine on > *nx, however,it dose not alway works on windows. > setPermission call setReadable of Java.io.File in the line 498, but according > to the Table1 below provided by oracle,setReadable(false) will always return > false on windows, the same as setExecutable(false). > http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/ > is it cause the task tracker "Failed to set permissions" to "ttprivate to > 0700"? > Hadoop 0.20.202 works fine in the same environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8274) In pseudo or cluster model under Cygwin, tasktracker can not create a new job because of symlink problem.
[
https://issues.apache.org/jira/browse/HADOOP-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264577#comment-13264577
]
FKorning commented on HADOOP-8274:
--
If the symlinks really are necessary, here's a java.io.File wrapper
that understands cygwin symlinks (and windows shortcuts).
{code}
package net.muxbus.io;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;
/**
*
* @author Francis Korning fkorn...@yahoo.ca
* @date2012-04-30
*
* @project muxbusmuxbus.net
* @source sourceforge https://sourceforge.net/p/muxbus/
*
* @copyright 2012 F.Korning
* @license LGPL http://www.gnu.org/copyleft/lesser.html
*
* @credit Jesse Hager jesseha...@iname.com(.lnk shortcut file
format).
*
*
* LinkedFile is a java.io.File wrapper that understands both Cygwin Symbolic
Links
* and Windows Explorer Shorcut Links (.lnk). If a given File is a Link, it
tries
* to resolve the File by following link paths recursively until a final
canonical
* target is found, and if the file exists will act as a wrapper for the link
* target. A LinkedFile always tracks its source and provides link reflection.
*
* A common frustration for POSIX platform developers is that Java on windows
does
* not understand symbolic Links which aren't handled by the windows io native
libs.
* This means that Cygwin or Interix (aka Winterix or Windows Service for Unix)
* integrated java applications that depend on POSIX tools may break when it
comes
* to resolution of paths with linked files and directories.
*
* The first workaround strategy, which works where links are predictably
created,
* is to ensure that cygwin paths and windows paths are equivalent and
identical.
* For example, you create a circular "/cygwin" link that maps back to the root
"/":
*
*
*
* /cygwin -> / -> \cygwin\ -> C:\cygwin\
* /cygwin/tmp -> /tmp/ -> \cygwin\tmp\ -> C:\cygwin\tmp\
*
*
*
* But you often have no control over much of the integrated codebase and tools.
* If some of the components create symlinks dynamically where java expects real
* files and directories this workaround will fail.
*
* Now the only OS-native IO alternative would be to use Windows NTFS Junctions.
* These are completely transparent and work in shells, in Explorer, and for
java.
* The best implementation would be the junction provided in the SysInternals
suite.
* You could wrap the cygwin 'ln' command and force it to use a junction
instead,
* but this breaks some POSIX compatibility and requires Administrator superuser
* rights which you may not want to assign to all your running service daemons.
*
* Next Interix symbolic links and the mklink command are built-in since
Windows Vista.
* These fall in between as they act like native symlinks in the shell and in
java,
* look like cygwin links inside the shell, but again they can only be created
by the
* Administrator, and also break POSIX. Unlike cygwin they do appear as
shortcuts in
* a Windows Explorer, with the caveat that these are broken because of
permissions.
*
* Note that both strategies only work if the tools create links via the 'ln'
command.
* But what if the offending integrated tools are in perl or python, or even
better yet in
* compiled c/c++ code making use of libc calls for which we have no source nor
license?
* This means that we will need to adopt a strategy to adapt java.io.File to
Cygwin libc.
* Also, most multi-platform java developers favour Cygwin instead of
Microsoft's Interix.
* All of the POSIX daemons we're looking at expect cygwin shells, and this
compatibility
* is our main focus here; we ideally an a model that works without having to
wrap 'ln'.
*
* Now Both Cygwin Symlinks and Explorer Shortcuts are just plain binary files
underneath
* with some information pointing to their link targets. They have the
advantage that
* we can parse these and thus this allows us to add some link reflection to
java.io.
* (This also creates a strange case where windows would have more
functionality than
* unix, unless one would write a unix java native libc wrapper for link
functions).
* They also require no special Administrator rights for creation and
maintenance.
*
* A side-effect is that shortcuts created under Windows Explorer have the
feature that
* a (.lnk) extension is appended, which is obscured from the explorer. As the
names
* differ, both a Cygwin Symlink and a Windows Shortcut can co-exist in the
same place
* and make for a consistent interface both in the cygwin shell and Windows
Explorer.
*
* This means that one could well mirror every cygwin symlink with a windows
shortcut,
* and thus have symbolic link
