[jira] [Commented] (HDFS-4940) namenode OOMs under Bigtop's TestCLI
[ https://issues.apache.org/jira/browse/HDFS-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701800#comment-13701800 ] Roman Shaposhnik commented on HDFS-4940: [~sureshms] Suresh, sorry for the belated reply -- I'm actually on vacation till last week of July. While I'm away you can just run a Bigtop's TestCLI against the latest 2.1-based bits. Drop an email on Bigtop's ML and I'm sure folks will be more than happy to help with the details. I'll totally pick this JIRA back up once I get back. namenode OOMs under Bigtop's TestCLI Key: HDFS-4940 URL: https://issues.apache.org/jira/browse/HDFS-4940 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Roman Shaposhnik Priority: Blocker Fix For: 2.1.0-beta Bigtop's TestCLI when executed against Hadoop 2.1.0 seems to make it OOM quite reliably regardless of the heap size settings. I'm attaching a heap dump URL. Alliteratively anybody can just take Bigtop's tests, compiled them against Hadoop 2.1.0 bits and try to reproduce it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4278) The DFS_BLOCK_ACCESS_TOKEN_ENABLE config should be automatically turned on when security is enabled.
[ https://issues.apache.org/jira/browse/HDFS-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701835#comment-13701835 ] Harsh J commented on HDFS-4278: --- Suresh - Given the security won't work without it, why log an ERROR thats a step more longer to catch? My preference is to keep it automatic as stated on the description as well. Toggling it off with security enabled is a special need and there could be a config for an override instead. Thoughts? The DFS_BLOCK_ACCESS_TOKEN_ENABLE config should be automatically turned on when security is enabled. Key: HDFS-4278 URL: https://issues.apache.org/jira/browse/HDFS-4278 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 2.0.0-alpha Reporter: Harsh J Labels: newbie Attachments: HDFS-4278.patch When enabling security, one has to manually enable the config DFS_BLOCK_ACCESS_TOKEN_ENABLE (dfs.block.access.token.enable). Since these two are coupled, we could make it turn itself on automatically if we find security to be enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-353) DFSClient could throw a FileNotFound exception when a file could not be opened
[ https://issues.apache.org/jira/browse/HDFS-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701909#comment-13701909 ] Steve Loughran commented on HDFS-353: - Its the specific case of: metadata exists, but there are no block locations for the file. IOE maybe better than FileNotFound, though I'm not 100% sure. What we could at least do is throw up the filename. DFSClient could throw a FileNotFound exception when a file could not be opened -- Key: HDFS-353 URL: https://issues.apache.org/jira/browse/HDFS-353 Project: Hadoop HDFS Issue Type: Improvement Reporter: Steve Loughran Assignee: Chu Tong Priority: Minor DfsClient.openInit() throws an IOE when a file can't be found, that is, it has no blocks [sf-startdaemon-debug] 09/02/16 12:38:47 [IPC Server handler 0 on 8012] INFO mapred.TaskInProgress : Error from attempt_200902161238_0001_m_00_2: java.io.IOException: Cannot open filename /tests/mrtestsequence/in/in.txt [sf-startdaemon-debug]at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1352) [sf-startdaemon-debug]at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1343) [sf-startdaemon-debug]at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:312) [sf-startdaemon-debug]at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:177) [sf-startdaemon-debug]at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:347) I propose turning this into a FileNotFoundException, which is more specific about the underlying problem. Including the full dfs URL would be useful too. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4645) Move from randomly generated block ID to sequentially generated block ID
[ https://issues.apache.org/jira/browse/HDFS-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701927#comment-13701927 ] Hudson commented on HDFS-4645: -- Integrated in Hadoop-Yarn-trunk #264 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/264/]) HDFS-4645. Move from randomly generated block ID to sequentially generated block ID. Contributed by Arpit Agarwal (Revision 1500580) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500580 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/OutOfV1GenerationStampsException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/RandomBlockIdGenerator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SequentialBlockIdGenerator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSequentialBlockId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Move from randomly generated block ID to sequentially generated block ID Key: HDFS-4645 URL: https://issues.apache.org/jira/browse/HDFS-4645 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Arpit Agarwal Fix For: 3.0.0 Attachments: editsStored, HDFS-4645.001.patch, HDFS-4645.002.patch, HDFS-4645.003.patch, HDFS-4645.004.patch, HDFS-4645.005.patch, HDFS-4645.006.patch, SequentialblockIDallocation.pdf Currently block IDs are randomly generated. This means there is no
[jira] [Commented] (HDFS-4645) Move from randomly generated block ID to sequentially generated block ID
[ https://issues.apache.org/jira/browse/HDFS-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701986#comment-13701986 ] Hudson commented on HDFS-4645: -- Integrated in Hadoop-Hdfs-trunk #1454 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1454/]) HDFS-4645. Move from randomly generated block ID to sequentially generated block ID. Contributed by Arpit Agarwal (Revision 1500580) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500580 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/OutOfV1GenerationStampsException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/RandomBlockIdGenerator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SequentialBlockIdGenerator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSequentialBlockId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Move from randomly generated block ID to sequentially generated block ID Key: HDFS-4645 URL: https://issues.apache.org/jira/browse/HDFS-4645 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Arpit Agarwal Fix For: 3.0.0 Attachments: editsStored, HDFS-4645.001.patch, HDFS-4645.002.patch, HDFS-4645.003.patch, HDFS-4645.004.patch, HDFS-4645.005.patch, HDFS-4645.006.patch, SequentialblockIDallocation.pdf Currently block IDs are randomly generated. This means there is no
[jira] [Commented] (HDFS-4645) Move from randomly generated block ID to sequentially generated block ID
[ https://issues.apache.org/jira/browse/HDFS-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701991#comment-13701991 ] Hudson commented on HDFS-4645: -- Integrated in Hadoop-Mapreduce-trunk #1481 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1481/]) HDFS-4645. Move from randomly generated block ID to sequentially generated block ID. Contributed by Arpit Agarwal (Revision 1500580) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1500580 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/OutOfV1GenerationStampsException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/RandomBlockIdGenerator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SequentialBlockIdGenerator.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageVisitor.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSequentialBlockId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/TestOfflineEditsViewer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Move from randomly generated block ID to sequentially generated block ID Key: HDFS-4645 URL: https://issues.apache.org/jira/browse/HDFS-4645 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Arpit Agarwal Fix For: 3.0.0 Attachments: editsStored, HDFS-4645.001.patch, HDFS-4645.002.patch, HDFS-4645.003.patch, HDFS-4645.004.patch, HDFS-4645.005.patch, HDFS-4645.006.patch, SequentialblockIDallocation.pdf Currently block IDs are randomly generated. This means
[jira] [Created] (HDFS-4962) Use enum for nfs constants
Tsz Wo (Nicholas), SZE created HDFS-4962: Summary: Use enum for nfs constants Key: HDFS-4962 URL: https://issues.apache.org/jira/browse/HDFS-4962 Project: Hadoop HDFS Issue Type: Sub-task Components: nfs Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor The constants defined in MountInterface and some other classes are better to use enum types instead of lists of int. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4943) WebHdfsFileSystem does not work when original file path has encoded chars
[ https://issues.apache.org/jira/browse/HDFS-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HDFS-4943: Attachment: HDFS-4943-b023.patch Patch for branch-0.23 WebHdfsFileSystem does not work when original file path has encoded chars -- Key: HDFS-4943 URL: https://issues.apache.org/jira/browse/HDFS-4943 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.2.0, 1.1.2, 2.0.4-alpha Reporter: Jerry He Assignee: Jerry He Priority: Minor Fix For: 2.1.0-beta Attachments: HDFS-4943-b023.patch, HDFS-4943-trunk.patch, HDFS-4943-trunk-v2.patch In HBase, the WAL (hlog) file name on hdfs is URL encoded. For example, hdtest010%2C60020%2C1371000602151.1371058984668 When we use webhdfs client to access the hlog file via httpfs, it does not work in this case. $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs Found 1 items -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 $ hadoop fs -ls hdfs:///user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 Found 1 items -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs Found 1 items -rw-r--r-- 3 biadmin supergroup 15049470 2013-06-12 10:45 /user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 $ $ hadoop fs -ls webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668 13/06/27 18:36:08 DEBUG web.WebHdfsFileSystem: Original exception is org.apache.hadoop.ipc.RemoteException:java.io.FileNotFoundException:File does not exist: /user/biadmin/hbase_hlogs/hdtest010,60020,1371000602151.1371058984668 at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:299) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:104) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:641) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:538) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:468) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:662) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:673) at org.apache.hadoop.fs.FileSystem.getFileStatus(FileSystem.java:1365) at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1048) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:965) at org.apache.hadoop.fs.FsShell.ls(FsShell.java:573) at org.apache.hadoop.fs.FsShell.doall(FsShell.java:1571) at org.apache.hadoop.fs.FsShell.run(FsShell.java:1789) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.fs.FsShell.main(FsShell.java:1895) ls: Cannot access webhdfs://hdtest010:14000/user/biadmin/hbase_hlogs/hdtest010%2C60020%2C1371000602151.1371058984668: No such file or directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4278) The DFS_BLOCK_ACCESS_TOKEN_ENABLE config should be automatically turned on when security is enabled.
[ https://issues.apache.org/jira/browse/HDFS-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702089#comment-13702089 ] Suresh Srinivas commented on HDFS-4278: --- bq. Suresh - Given the security won't work without it, why log an ERROR thats a step more longer to catch? I am okay with it. Perhaps we could log a warning DFS_BLOCK_ACCESS_TOKEN_ENABLE is turned off and it is automatically being turned on. How do you make it work for testing purpose? I am against doing this by adding conditional code and variables with restricted visibility and unnecessary code complication. In that case, I would rather error out on the user about incompatible configuration than doing things automatically. The DFS_BLOCK_ACCESS_TOKEN_ENABLE config should be automatically turned on when security is enabled. Key: HDFS-4278 URL: https://issues.apache.org/jira/browse/HDFS-4278 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 2.0.0-alpha Reporter: Harsh J Labels: newbie Attachments: HDFS-4278.patch When enabling security, one has to manually enable the config DFS_BLOCK_ACCESS_TOKEN_ENABLE (dfs.block.access.token.enable). Since these two are coupled, we could make it turn itself on automatically if we find security to be enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3182) Add class to manage JournalList
[ https://issues.apache.org/jira/browse/HDFS-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3182. --- Resolution: Won't Fix This was obsoleted by other work in HDFS-3077 Add class to manage JournalList --- Key: HDFS-3182 URL: https://issues.apache.org/jira/browse/HDFS-3182 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, namenode Reporter: Suresh Srinivas See the comment for details of the JournalList ZooKeeper znode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702147#comment-13702147 ] Todd Lipcon commented on HDFS-3092: --- Hey folks. Is anyone still working on this given that HDFS-3077 has been committed for a year or so and in use at lots of sites? Maybe we should close it as wontfix, given a lot of the original ideas got incorporated into 3077? Enable journal protocol based editlog streaming for standby namenode Key: HDFS-3092 URL: https://issues.apache.org/jira/browse/HDFS-3092 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0, 0.23.3 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: ComparisonofApproachesforHAJournals.pdf, JNStates.png, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, Removingfilerdependency.pdf Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4879) Add blocked ArrayList collection to avoid CMS full GCs
[
https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-4879:
--
Attachment: hdfs-4879.txt
New patch changes ChunkedArrayList to extend AbstractList, which just throws
exceptions on unsupported operations. I also addressed the above feedback.
Given that the list now implements the List interface, the patch is much
smaller and I think much improved. Thanks for the review, folks.
I ran TestLargeDirectoryDelete and it passed.
Add blocked ArrayList collection to avoid CMS full GCs
Key: HDFS-4879
URL: https://issues.apache.org/jira/browse/HDFS-4879
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0, 2.0.4-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: hdfs-4879.txt, hdfs-4879.txt, hdfs-4879.txt
We recently saw an issue where a large deletion was issued which caused 25M
blocks to be collected during {{deleteInternal}}. Currently, the list of
collected blocks is an ArrayList, meaning that we had to allocate a
contiguous 25M-entry array (~400MB). After a NN has been running for a long
amount of time, the old generation may become fragmented such that it's hard
to find a 400MB contiguous chunk of heap.
In general, we should try to design the NN such that the only large objects
are long-lived and created at startup time. We can improve this particular
case (and perhaps some others) by introducing a new List implementation which
is made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4879) Add blocked ArrayList collection to avoid CMS full GCs
[
https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-4879:
--
Status: Patch Available (was: Open)
Add blocked ArrayList collection to avoid CMS full GCs
Key: HDFS-4879
URL: https://issues.apache.org/jira/browse/HDFS-4879
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.4-alpha, 3.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: hdfs-4879.txt, hdfs-4879.txt, hdfs-4879.txt
We recently saw an issue where a large deletion was issued which caused 25M
blocks to be collected during {{deleteInternal}}. Currently, the list of
collected blocks is an ArrayList, meaning that we had to allocate a
contiguous 25M-entry array (~400MB). After a NN has been running for a long
amount of time, the old generation may become fragmented such that it's hard
to find a 400MB contiguous chunk of heap.
In general, we should try to design the NN such that the only large objects
are long-lived and created at startup time. We can improve this particular
case (and perhaps some others) by introducing a new List implementation which
is made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-3092. --- Resolution: Won't Fix Enable journal protocol based editlog streaming for standby namenode Key: HDFS-3092 URL: https://issues.apache.org/jira/browse/HDFS-3092 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0, 0.23.3 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: ComparisonofApproachesforHAJournals.pdf, JNStates.png, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, Removingfilerdependency.pdf Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702260#comment-13702260 ] Suresh Srinivas commented on HDFS-3092: --- Lets close this. If any work is needed, we can open another jira. Enable journal protocol based editlog streaming for standby namenode Key: HDFS-3092 URL: https://issues.apache.org/jira/browse/HDFS-3092 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0, 0.23.3 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: ComparisonofApproachesforHAJournals.pdf, JNStates.png, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, Removingfilerdependency.pdf Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4905) Add appendToFile command to hdfs dfs
[ https://issues.apache.org/jira/browse/HDFS-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4905: Attachment: HDFS-4905.003.patch Fix findbugs warning. Add appendToFile command to hdfs dfs -- Key: HDFS-4905 URL: https://issues.apache.org/jira/browse/HDFS-4905 Project: Hadoop HDFS Issue Type: Bug Components: tools Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Priority: Minor Attachments: HDFS-4905.002.patch, HDFS-4905.003.patch, HDFS-4905.patch A hdfs dfs -appendToFile... option would be quite useful for quick testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4879) Add blocked ArrayList collection to avoid CMS full GCs
[
https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702372#comment-13702372
]
Hadoop QA commented on HDFS-4879:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591245/hdfs-4879.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4603//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4603//console
This message is automatically generated.
Add blocked ArrayList collection to avoid CMS full GCs
Key: HDFS-4879
URL: https://issues.apache.org/jira/browse/HDFS-4879
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0, 2.0.4-alpha
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: hdfs-4879.txt, hdfs-4879.txt, hdfs-4879.txt
We recently saw an issue where a large deletion was issued which caused 25M
blocks to be collected during {{deleteInternal}}. Currently, the list of
collected blocks is an ArrayList, meaning that we had to allocate a
contiguous 25M-entry array (~400MB). After a NN has been running for a long
amount of time, the old generation may become fragmented such that it's hard
to find a 400MB contiguous chunk of heap.
In general, we should try to design the NN such that the only large objects
are long-lived and created at startup time. We can improve this particular
case (and perhaps some others) by introducing a new List implementation which
is made of a linked list of arrays, each of which is size-limited (eg to 1MB).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702417#comment-13702417
]
Colin Patrick McCabe commented on HDFS-4817:
1,2,4: fixed
3: I think {{CanSetDropBehind}} is the best we're going to get. I wish we had
a base class for all Hadoop output streams like we do for input streams. But I
think that ship has sailed already for compatibility reasons.
5: I am going to have it return false when the method is not supported. I
think most code will not check this return value, except unit tests and perhaps
high performance-conscious code. I'll also add throws IOException in case
any future implementations need to do that.
6, 8: ok
9: yes, I think we should have separate configs for read versus write, same as
on the datanode side. added.
10: I think it's an established convention in Hadoop to make readahead one
word at this point. We have {{ReadaheadPool}}, {{ReadaheadRequest}},
{{ReadaheadRequestImpl}}, {{dfs.datanode.readahead.bytes}}, etc.
make HDFS advisory caching configurable on a per-file basis
---
Key: HDFS-4817
URL: https://issues.apache.org/jira/browse/HDFS-4817
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
HDFS-4817.004.patch
HADOOP-7753 and related JIRAs introduced some performance optimizations for
the DataNode. One of them was readahead. When readahead is enabled, the
DataNode starts reading the next bytes it thinks it will need in the block
file, before the client requests them. This helps hide the latency of
rotational media and send larger reads down to the device. Another
optimization was drop-behind. Using this optimization, we could remove
files from the Linux page cache after they were no longer needed.
Using {{dfs.datanode.drop.cache.behind.writes}} and
{{dfs.datanode.drop.cache.behind.reads}} can improve performance
substantially on many MapReduce jobs. In our internal benchmarks, we have
seen speedups of 40% on certain workloads. The reason is because if we know
the block data will not be read again any time soon, keeping it out of memory
allows more memory to be used by the other processes on the system. See
HADOOP-7714 for more benchmarks.
We would like to turn on these configurations on a per-file or per-client
basis, rather than on the DataNode as a whole. This will allow more users to
actually make use of them. It would also be good to add unit tests for the
drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4905) Add appendToFile command to hdfs dfs
[
https://issues.apache.org/jira/browse/HDFS-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702430#comment-13702430
]
Hadoop QA commented on HDFS-4905:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591256/HDFS-4905.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4604//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4604//console
This message is automatically generated.
Add appendToFile command to hdfs dfs
--
Key: HDFS-4905
URL: https://issues.apache.org/jira/browse/HDFS-4905
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Priority: Minor
Attachments: HDFS-4905.002.patch, HDFS-4905.003.patch, HDFS-4905.patch
A hdfs dfs -appendToFile... option would be quite useful for quick testing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HDFS-4841:
Attachment: HDFS-4841.patch
The problem was basically that in the shutdown hook, we'd try to get the
filesystem which would add the shutdown hook (again). The patch simply adds a
check to not add the shutdown hook if we're already shutting down.
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013
[jira] [Updated] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HDFS-4841:
Status: Patch Available (was: Open)
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: application/json
Set-Cookie:
[jira] [Commented] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702472#comment-13702472
]
Robert Kanter commented on HDFS-4841:
-
With Stephen's help, we manually verified that the patch fixes the problem.
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: application/json
Set-Cookie:
[jira] [Created] (HDFS-4963) Improve multihoming support in namenode
Arpit Agarwal created HDFS-4963:
---
Summary: Improve multihoming support in namenode
Key: HDFS-4963
URL: https://issues.apache.org/jira/browse/HDFS-4963
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 1.3.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
HDFS does not work very well on multi-homed machines. A few open Jiras refer to
this:
# HDFS-1379
# HADOOP-8198
There are multiple issues involved here and some of them can be worked around
by using alternate DNS names and configuring {{slave.host.name}} on Datanodes
and task trackers.
However namenode issues cannot be worked around because it does not respect the
{{fs.default.name}} configuration. e.g. {{Namenode#initialize}} performs a
gratuitous reverse DNS lookup to regenerate the hostname. Similar issues exist
elsewhere.
This Jira is being filed to fix some of the more egregious problems. To avoid
affecting existing users a new config setting may be introduced.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702521#comment-13702521
]
Hadoop QA commented on HDFS-4841:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591287/HDFS-4841.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4605//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4605//console
This message is automatically generated.
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
[jira] [Updated] (HDFS-4372) Track NameNode startup progress
[ https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4372: Attachment: HDFS-4372.4.rebase.patch Thanks Chris! The latest patch looks pretty good to me. It only needs some rebase, and I did it to save time. Could you review the rebased patch to make sure I did not bring in any error? Besides of that, +1 for the patch. Track NameNode startup progress --- Key: HDFS-4372 URL: https://issues.apache.org/jira/browse/HDFS-4372 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4372.1.patch, HDFS-4372.2.patch, HDFS-4372.3.patch, HDFS-4372.4.patch, HDFS-4372.4.rebase.patch Track detailed progress information about the steps of NameNode startup to enable display to users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4956) DistributedFileSystem does not handle CreateFlag.APPEND in create call
[ https://issues.apache.org/jira/browse/HDFS-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702551#comment-13702551 ] Suresh Srinivas commented on HDFS-4956: --- bq. yes, It seems only fix in DistributedFileSystem and DFSClient, because FSNamesystem has been considered it. I am not sure what you mean here. Assuming what you mean is, DFSClient and DistributedFileSystem does not handle it and FSNamesystem handles it, the right way to do this is DistributedFileSystem/DFSClient should call append or create based on the flags passed to them. Currently FSNamesystem has code that makes it look like it handles it. However there are some differences between append method and create. For example, create does not check if append feature is supported. It also cannot return LocatedBlock with last incomplete block. The FSNamesystem implementation is incorrect and incomplete. DistributedFileSystem does not handle CreateFlag.APPEND in create call -- Key: HDFS-4956 URL: https://issues.apache.org/jira/browse/HDFS-4956 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Suresh Srinivas Currently DistributedFileSystem does not handle CreateFlag.APPEND in the implementation of FileSystem#create() method. It only support OVERWRITE CREATE or just CREATE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4964) Enable multiple QOP for NameNode
Benoy Antony created HDFS-4964: -- Summary: Enable multiple QOP for NameNode Key: HDFS-4964 URL: https://issues.apache.org/jira/browse/HDFS-4964 Project: Hadoop HDFS Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Currently Namenode supports only single QOP. The feature makes name node listen on two ports. One port supports only authentication , one port supports privacy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4964) Enable multiple QOP for NameNode
[ https://issues.apache.org/jira/browse/HDFS-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-4964: --- Attachment: hdfs-4964.patch I'll add the testcases once the approach is reviewed. Enable multiple QOP for NameNode Key: HDFS-4964 URL: https://issues.apache.org/jira/browse/HDFS-4964 Project: Hadoop HDFS Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Attachments: hdfs-4964.patch Currently Namenode supports only single QOP. The feature makes name node listen on two ports. One port supports only authentication , one port supports privacy. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702564#comment-13702564
]
Aaron T. Myers commented on HDFS-4841:
--
+1, the patch looks good to me.
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: application/json
Set-Cookie:
[jira] [Updated] (HDFS-4965) Make datanodes support multiple QOP for RPC and streaming protocols
[ https://issues.apache.org/jira/browse/HDFS-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-4965: --- Attachment: HDFS-4965.patch I'll add the testcase once the approach is removed. Make datanodes support multiple QOP for RPC and streaming protocols --- Key: HDFS-4965 URL: https://issues.apache.org/jira/browse/HDFS-4965 Project: Hadoop HDFS Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HDFS-4965.patch Currently datanode supports only single QOP. The feature makes data node listen on two ports for RPC and streaming. One RPC port supports only authentication , other RPC port supports privacy. Similarly one streaming port supports only authentication , other streaming port supports encryption in addition to authentication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4965) Make datanodes support multiple QOP for RPC and streaming protocols
Benoy Antony created HDFS-4965: -- Summary: Make datanodes support multiple QOP for RPC and streaming protocols Key: HDFS-4965 URL: https://issues.apache.org/jira/browse/HDFS-4965 Project: Hadoop HDFS Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HDFS-4965.patch Currently datanode supports only single QOP. The feature makes data node listen on two ports for RPC and streaming. One RPC port supports only authentication , other RPC port supports privacy. Similarly one streaming port supports only authentication , other streaming port supports encryption in addition to authentication. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4965) Make datanodes support multiple QOP for RPC and streaming protocols
[ https://issues.apache.org/jira/browse/HDFS-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-4965: --- Description: Currently datanode supports only single QOP. The feature makes data node listen on two ports for RPC and streaming. One RPC port supports only authentication , other RPC port supports privacy. Similarly one streaming port supports only authentication , other streaming port supports encryption in addition to authentication. Please see HADOOP-9709 for general requirements. was: Currently datanode supports only single QOP. The feature makes data node listen on two ports for RPC and streaming. One RPC port supports only authentication , other RPC port supports privacy. Similarly one streaming port supports only authentication , other streaming port supports encryption in addition to authentication. Make datanodes support multiple QOP for RPC and streaming protocols --- Key: HDFS-4965 URL: https://issues.apache.org/jira/browse/HDFS-4965 Project: Hadoop HDFS Issue Type: Improvement Reporter: Benoy Antony Assignee: Benoy Antony Attachments: HDFS-4965.patch Currently datanode supports only single QOP. The feature makes data node listen on two ports for RPC and streaming. One RPC port supports only authentication , other RPC port supports privacy. Similarly one streaming port supports only authentication , other streaming port supports encryption in addition to authentication. Please see HADOOP-9709 for general requirements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702588#comment-13702588
]
Alejandro Abdelnur commented on HDFS-4841:
--
+1, committing to trunk/branch-2/branch-2.1 momentarily.
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: application/json
Set-Cookie:
[jira] [Updated] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-4841:
-
Resolution: Fixed
Fix Version/s: 2.1.0-beta
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Robert. Committed to trunk, branch-2 and branch-2.1
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Fix For: 2.1.0-beta
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma:
[jira] [Commented] (HDFS-4841) FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
[
https://issues.apache.org/jira/browse/HDFS-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702603#comment-13702603
]
Hudson commented on HDFS-4841:
--
Integrated in Hadoop-trunk-Commit #4052 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4052/])
HDFS-4841. FsShell commands using secure webhfds fail ClientFinalizer
shutdown hook. (rkanter via tucu) (Revision 1501005)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1501005
Files :
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
FsShell commands using secure webhfds fail ClientFinalizer shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Robert Kanter
Fix For: 2.1.0-beta
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out,
HDFS-4841.patch, hdfs-site.xml, jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY;
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22
[jira] [Updated] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4817:
---
Attachment: HDFS-4817.006.patch
besides the above, this patch includes:
* LONG_READ_THRESHOLD_BYTES: changed comment to refer you to
BlockSender#isLongRead
* BlockReceiver: add cachingStrategy to the list of things we print out when
LOG.debug is enabled
* TestCachingStrategy: add another test, this time testing client-side defaults
make HDFS advisory caching configurable on a per-file basis
---
Key: HDFS-4817
URL: https://issues.apache.org/jira/browse/HDFS-4817
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
HDFS-4817.004.patch, HDFS-4817.006.patch
HADOOP-7753 and related JIRAs introduced some performance optimizations for
the DataNode. One of them was readahead. When readahead is enabled, the
DataNode starts reading the next bytes it thinks it will need in the block
file, before the client requests them. This helps hide the latency of
rotational media and send larger reads down to the device. Another
optimization was drop-behind. Using this optimization, we could remove
files from the Linux page cache after they were no longer needed.
Using {{dfs.datanode.drop.cache.behind.writes}} and
{{dfs.datanode.drop.cache.behind.reads}} can improve performance
substantially on many MapReduce jobs. In our internal benchmarks, we have
seen speedups of 40% on certain workloads. The reason is because if we know
the block data will not be read again any time soon, keeping it out of memory
allows more memory to be used by the other processes on the system. See
HADOOP-7714 for more benchmarks.
We would like to turn on these configurations on a per-file or per-client
basis, rather than on the DataNode as a whole. This will allow more users to
actually make use of them. It would also be good to add unit tests for the
drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4951) FsShell commands using secure httpfs throw exceptions due to missing TokenRenewer
[
https://issues.apache.org/jira/browse/HDFS-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HDFS-4951:
Attachment: HDFS-4951.patch
Instead of essentially redoing the delegation token renewer (and related) code
from WebHDFS for HttpFS, and because the server is using WebHDFSFileSystem
anyway, I think the best and simplest solution is to make HttpFS use the same
delegation token kind as WebHDFS is using.
The patch simply changes the token kind that HttpFS's DelegationTokenIdentifier
is using from HTTPFS_DELEGATION_TOKEN to WEBHDFS delegation. I manually
verified that using the FsShell with HttpFS now works properly.
FsShell commands using secure httpfs throw exceptions due to missing
TokenRenewer
-
Key: HDFS-4951
URL: https://issues.apache.org/jira/browse/HDFS-4951
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: HDFS-4951.patch
It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
throws an exception:
{noformat}
$ hadoop fs -ls webhdfs://host:14000
// File listing omitted
13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
HTTPFS_DELEGATION_TOKEN
13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook
'ClientFinalizer' failed, java.lang.UnsupportedOperationException: Token
cancel is not supported for HTTPFS_DELEGATION_TOKEN tokens
java.lang.UnsupportedOperationException: Token cancel is not supported for
HTTPFS_DELEGATION_TOKEN tokens
at
org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:233)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:790)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2398)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2414)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{noformat}
WebHDFS doesn't have this problem because it has a {{TokenRenewer}} for its
delegation tokens ({{WEBHDFS delegation}} tokens).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4951) FsShell commands using secure httpfs throw exceptions due to missing TokenRenewer
[
https://issues.apache.org/jira/browse/HDFS-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HDFS-4951:
Status: Patch Available (was: Open)
FsShell commands using secure httpfs throw exceptions due to missing
TokenRenewer
-
Key: HDFS-4951
URL: https://issues.apache.org/jira/browse/HDFS-4951
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: HDFS-4951.patch
It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
throws an exception:
{noformat}
$ hadoop fs -ls webhdfs://host:14000
// File listing omitted
13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
HTTPFS_DELEGATION_TOKEN
13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook
'ClientFinalizer' failed, java.lang.UnsupportedOperationException: Token
cancel is not supported for HTTPFS_DELEGATION_TOKEN tokens
java.lang.UnsupportedOperationException: Token cancel is not supported for
HTTPFS_DELEGATION_TOKEN tokens
at
org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:233)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:790)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2398)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2414)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{noformat}
WebHDFS doesn't have this problem because it has a {{TokenRenewer}} for its
delegation tokens ({{WEBHDFS delegation}} tokens).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4373) Add HTTP API for querying NameNode startup progress
[ https://issues.apache.org/jira/browse/HDFS-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702655#comment-13702655 ] Jing Zhao commented on HDFS-4373: - The patch looks good to me. Some minors: 1. Based on the current changes in HDFS-4372, StartupProgressServlet#putIfNotNull may need to be updated since we no longer use Long and null for regular and uninitialized fields of Step. 2. In StartupProgressServlet, instead of putting all the information contained in a StartupProgressView into a newly-created map and converting the map into Json, can we directly dump the StartupProgressView instance into the response's writer using Json (just like Configuration#dumpConfiguration)? 3. We can clean the imports in both TestStartupProgressServlet.java and StartupProgressServlet.java Add HTTP API for querying NameNode startup progress --- Key: HDFS-4373 URL: https://issues.apache.org/jira/browse/HDFS-4373 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HDFS-4373.1.patch Provide an HTTP API for non-browser clients to query the NameNode's current progress through startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4372) Track NameNode startup progress
[
https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702712#comment-13702712
]
Hadoop QA commented on HDFS-4372:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591300/HDFS-4372.4.rebase.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4606//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4606//console
This message is automatically generated.
Track NameNode startup progress
---
Key: HDFS-4372
URL: https://issues.apache.org/jira/browse/HDFS-4372
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-4372.1.patch, HDFS-4372.2.patch, HDFS-4372.3.patch,
HDFS-4372.4.patch, HDFS-4372.4.rebase.patch
Track detailed progress information about the steps of NameNode startup to
enable display to users.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4951) FsShell commands using secure httpfs throw exceptions due to missing TokenRenewer
[
https://issues.apache.org/jira/browse/HDFS-4951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702711#comment-13702711
]
Hadoop QA commented on HDFS-4951:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591321/HDFS-4951.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs-httpfs:
org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4608//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4608//console
This message is automatically generated.
FsShell commands using secure httpfs throw exceptions due to missing
TokenRenewer
-
Key: HDFS-4951
URL: https://issues.apache.org/jira/browse/HDFS-4951
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 3.0.0
Reporter: Robert Kanter
Assignee: Robert Kanter
Attachments: HDFS-4951.patch
It looks like there isn't a {{TokenRenewer}} for HttpFS delegation tokens
({{HTTPFS_DELEGATION_TOKENS}} tokens, so when it goes to cancel the token, it
throws an exception:
{noformat}
$ hadoop fs -ls webhdfs://host:14000
// File listing omitted
13/06/21 13:09:04 WARN token.Token: No TokenRenewer defined for token kind
HTTPFS_DELEGATION_TOKEN
13/06/21 13:09:04 WARN util.ShutdownHookManager: ShutdownHook
'ClientFinalizer' failed, java.lang.UnsupportedOperationException: Token
cancel is not supported for HTTPFS_DELEGATION_TOKEN tokens
java.lang.UnsupportedOperationException: Token cancel is not supported for
HTTPFS_DELEGATION_TOKEN tokens
at
org.apache.hadoop.security.token.Token$TrivialRenewer.cancel(Token.java:417)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:146)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:233)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:790)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2398)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2414)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{noformat}
WebHDFS doesn't have this problem because it has a {{TokenRenewer}} for its
delegation tokens ({{WEBHDFS delegation}} tokens).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4966) implement advisory caching for RawLocalFilesystem
Colin Patrick McCabe created HDFS-4966: -- Summary: implement advisory caching for RawLocalFilesystem Key: HDFS-4966 URL: https://issues.apache.org/jira/browse/HDFS-4966 Project: Hadoop HDFS Issue Type: Improvement Reporter: Colin Patrick McCabe Priority: Minor It should be pretty straightforward to implement the advisory caching tunables (dropBehind, readahead) introduced in HDFS-4817 for RawLocalFilesystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4817:
---
Attachment: HDFS-4817.007.patch
* Remove unnecessary changes to TestStreamFile.java
* RawLocalFilesystem: remove TODO, create HDFS-4966 instead.
make HDFS advisory caching configurable on a per-file basis
---
Key: HDFS-4817
URL: https://issues.apache.org/jira/browse/HDFS-4817
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch
HADOOP-7753 and related JIRAs introduced some performance optimizations for
the DataNode. One of them was readahead. When readahead is enabled, the
DataNode starts reading the next bytes it thinks it will need in the block
file, before the client requests them. This helps hide the latency of
rotational media and send larger reads down to the device. Another
optimization was drop-behind. Using this optimization, we could remove
files from the Linux page cache after they were no longer needed.
Using {{dfs.datanode.drop.cache.behind.writes}} and
{{dfs.datanode.drop.cache.behind.reads}} can improve performance
substantially on many MapReduce jobs. In our internal benchmarks, we have
seen speedups of 40% on certain workloads. The reason is because if we know
the block data will not be read again any time soon, keeping it out of memory
allows more memory to be used by the other processes on the system. See
HADOOP-7714 for more benchmarks.
We would like to turn on these configurations on a per-file or per-client
basis, rather than on the DataNode as a whole. This will allow more users to
actually make use of them. It would also be good to add unit tests for the
drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702727#comment-13702727
]
Colin Patrick McCabe commented on HDFS-4817:
bq. What's the reason for having CachingStrategy#dropBehind and
CachingStrategy#readahead public? They both have a getter and a setter.
By the way, this point was also addressed in the latest patch (as well as
HDFS-4817.006.patch)
make HDFS advisory caching configurable on a per-file basis
---
Key: HDFS-4817
URL: https://issues.apache.org/jira/browse/HDFS-4817
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch
HADOOP-7753 and related JIRAs introduced some performance optimizations for
the DataNode. One of them was readahead. When readahead is enabled, the
DataNode starts reading the next bytes it thinks it will need in the block
file, before the client requests them. This helps hide the latency of
rotational media and send larger reads down to the device. Another
optimization was drop-behind. Using this optimization, we could remove
files from the Linux page cache after they were no longer needed.
Using {{dfs.datanode.drop.cache.behind.writes}} and
{{dfs.datanode.drop.cache.behind.reads}} can improve performance
substantially on many MapReduce jobs. In our internal benchmarks, we have
seen speedups of 40% on certain workloads. The reason is because if we know
the block data will not be read again any time soon, keeping it out of memory
allows more memory to be used by the other processes on the system. See
HADOOP-7714 for more benchmarks.
We would like to turn on these configurations on a per-file or per-client
basis, rather than on the DataNode as a whole. This will allow more users to
actually make use of them. It would also be good to add unit tests for the
drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702761#comment-13702761
]
Hadoop QA commented on HDFS-4817:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591314/HDFS-4817.006.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4607//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4607//console
This message is automatically generated.
make HDFS advisory caching configurable on a per-file basis
---
Key: HDFS-4817
URL: https://issues.apache.org/jira/browse/HDFS-4817
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch
HADOOP-7753 and related JIRAs introduced some performance optimizations for
the DataNode. One of them was readahead. When readahead is enabled, the
DataNode starts reading the next bytes it thinks it will need in the block
file, before the client requests them. This helps hide the latency of
rotational media and send larger reads down to the device. Another
optimization was drop-behind. Using this optimization, we could remove
files from the Linux page cache after they were no longer needed.
Using {{dfs.datanode.drop.cache.behind.writes}} and
{{dfs.datanode.drop.cache.behind.reads}} can improve performance
substantially on many MapReduce jobs. In our internal benchmarks, we have
seen speedups of 40% on certain workloads. The reason is because if we know
the block data will not be read again any time soon, keeping it out of memory
allows more memory to be used by the other processes on the system. See
HADOOP-7714 for more benchmarks.
We would like to turn on these configurations on a per-file or per-client
basis, rather than on the DataNode as a whole. This will allow more users to
actually make use of them. It would also be good to add unit tests for the
drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4960) Unnecessary .meta seeks even when skip checksum is true
[ https://issues.apache.org/jira/browse/HDFS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702787#comment-13702787 ] Colin Patrick McCabe commented on HDFS-4960: This change seems problematic. The .meta header exists so that we can check the version of the block file. If we ignore it, we've effectively locked ourselves into a single version forever. I don't see how we can accept this change unless an alternate way of versioning the block files is also proposed. One possible way would be through the naming of the block files. But this is something we should discuss before throwing out the existing mechanism. Also, if we're going to ignore the header for no-checksum, we should ignore it in the checksum case as well. Also, did you measure the performance impact? Unnecessary .meta seeks even when skip checksum is true --- Key: HDFS-4960 URL: https://issues.apache.org/jira/browse/HDFS-4960 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Varun Sharma Assignee: Varun Sharma Attachments: 4960-branch2.patch, 4960-trunk.patch While attempting to benchmark an HBase + Hadoop 2.0 setup on SSDs, we found unnecessary seeks into .meta files, each seek was a 7 byte read at the head of the file - this attempts to validate the version #. Since the client is requesting no-checksum, we should not be needing to touch the .meta file at all. Since the purpose of skip checksum is to also avoid the performance penalty of the extra seek, we should not be seeking into .meta if skip checksum is true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4960) Unnecessary .meta seeks even when skip checksum is true
[ https://issues.apache.org/jira/browse/HDFS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702832#comment-13702832 ] Varun Sharma commented on HDFS-4960: I don't know about the future plans of .meta - however, I think currently its only being used for storing checksums and if checksums are not being asked for, there is no need for seeking into the .meta file. I think the FB branch already has this change. I personally think that inlining metadata in blocks is the way to go, for the future, instead of storing separately in a .meta file - it cripples hdfs for real time work loads. I did not have a large enough data set so probably all the .meta files were in fs cache. lseek in .meta ~50 microseconds lseek in block ~50 microseconds read .meta ~50 microseconds read block ~ 300 microseconds So, just looking from system calls perspective, it is ~ 20 % however, it does not show up in the end to end test, because in general, there are a bunch of other contention issues inside HDFS + HBase which hog CPU resources. Unnecessary .meta seeks even when skip checksum is true --- Key: HDFS-4960 URL: https://issues.apache.org/jira/browse/HDFS-4960 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Varun Sharma Assignee: Varun Sharma Attachments: 4960-branch2.patch, 4960-trunk.patch While attempting to benchmark an HBase + Hadoop 2.0 setup on SSDs, we found unnecessary seeks into .meta files, each seek was a 7 byte read at the head of the file - this attempts to validate the version #. Since the client is requesting no-checksum, we should not be needing to touch the .meta file at all. Since the purpose of skip checksum is to also avoid the performance penalty of the extra seek, we should not be seeking into .meta if skip checksum is true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
Fengdong Yu created HDFS-4967: - Summary: Generate block ID sequentially cannot work with QJM HA Key: HDFS-4967 URL: https://issues.apache.org/jira/browse/HDFS-4967 Project: Hadoop HDFS Issue Type: Bug Components: ha, hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Fengdong Yu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-4967:
--
Description:
After HDFS-4645 committed in the trunk, then the following error showed during
name node start:
{code}
2013-07-09 11:28:45,394 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode:
Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-4967:
--
Description:
There are two name nodes, one is active, another acts as standby name node. QJM
Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed during
name node start:
{code}
2013-07-09 11:28:45,394 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode:
Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
was:
After HDFS-4645 committed in the trunk, then the following error showed during
name node start:
{code}
2013-07-09 11:28:45,394 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode:
Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
There are two name nodes, one is active, another acts as standby name node. QJM
Ha configured.
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
There are two name nodes, one is active, another acts as standby name node.
QJM Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the
[jira] [Updated] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fengdong Yu updated HDFS-4967:
--
Description:
After HDFS-4645 committed in the trunk, then the following error showed during
name node start:
{code}
2013-07-09 11:28:45,394 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode:
Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
There are two name nodes, one is active, another acts as standby name node. QJM
Ha configured.
was:
After HDFS-4645 committed in the trunk, then the following error showed during
name node start:
{code}
2013-07-09 11:28:45,394 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode:
Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
[jira] [Commented] (HDFS-4817) make HDFS advisory caching configurable on a per-file basis
[
https://issues.apache.org/jira/browse/HDFS-4817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702852#comment-13702852
]
Hadoop QA commented on HDFS-4817:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591346/HDFS-4817.007.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4609//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4609//console
This message is automatically generated.
make HDFS advisory caching configurable on a per-file basis
---
Key: HDFS-4817
URL: https://issues.apache.org/jira/browse/HDFS-4817
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-4817.001.patch, HDFS-4817.002.patch,
HDFS-4817.004.patch, HDFS-4817.006.patch, HDFS-4817.007.patch
HADOOP-7753 and related JIRAs introduced some performance optimizations for
the DataNode. One of them was readahead. When readahead is enabled, the
DataNode starts reading the next bytes it thinks it will need in the block
file, before the client requests them. This helps hide the latency of
rotational media and send larger reads down to the device. Another
optimization was drop-behind. Using this optimization, we could remove
files from the Linux page cache after they were no longer needed.
Using {{dfs.datanode.drop.cache.behind.writes}} and
{{dfs.datanode.drop.cache.behind.reads}} can improve performance
substantially on many MapReduce jobs. In our internal benchmarks, we have
seen speedups of 40% on certain workloads. The reason is because if we know
the block data will not be read again any time soon, keeping it out of memory
allows more memory to be used by the other processes on the system. See
HADOOP-7714 for more benchmarks.
We would like to turn on these configurations on a per-file or per-client
basis, rather than on the DataNode as a whole. This will allow more users to
actually make use of them. It would also be good to add unit tests for the
drop-cache code path, to ensure that it is functioning as we expect.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702864#comment-13702864
]
Suresh Srinivas commented on HDFS-4967:
---
Thanks for reporting this issue. [~arpitagarwal] while fixing this issue, we
should add a unit test for this condition to make sure we catch this type of
issue.
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
There are two name nodes, one is active, another acts as standby name node.
QJM Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HDFS-4967:
--
Assignee: Arpit Agarwal
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
Assignee: Arpit Agarwal
There are two name nodes, one is active, another acts as standby name node.
QJM Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4960) Unnecessary .meta seeks even when skip checksum is true
[ https://issues.apache.org/jira/browse/HDFS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702877#comment-13702877 ] stack commented on HDFS-4960: - bq. The .meta header exists so that we can check the version of the block file. Thanks for taking a looksee mighty [~cmccabe]. What if we made a patch with a configuration to skip the reading of metadata? Or, added a setSkipMetadata API as we have a setVerifyChecksum API? Agree, it is handy having blocks versioned so can be evolved at later date. No harm having version in side file either since we are probably going to do an extra seek to get metadata anyways (would be coolio if DN cached block metadata so could save a seek but maybe this would be more trouble than it is worth -- we'd need to prove it useful in say a heavy random read scenario). But the metadata doesn't look to have ever changed. It is version 1 (It looks like the version used to be a define out of FSDataset before it was defined inside this file but even then the version was 1). Meantime the lads here are paying a seek to learn something that is unlikely ever to change. (Filename would be good place for version but would be a massive change). Thanks Colin. Unnecessary .meta seeks even when skip checksum is true --- Key: HDFS-4960 URL: https://issues.apache.org/jira/browse/HDFS-4960 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.1.0-beta Reporter: Varun Sharma Assignee: Varun Sharma Attachments: 4960-branch2.patch, 4960-trunk.patch While attempting to benchmark an HBase + Hadoop 2.0 setup on SSDs, we found unnecessary seeks into .meta files, each seek was a 7 byte read at the head of the file - this attempts to validate the version #. Since the client is requesting no-checksum, we should not be needing to touch the .meta file at all. Since the purpose of skip checksum is to also avoid the performance penalty of the extra seek, we should not be seeking into .meta if skip checksum is true -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702921#comment-13702921
]
Arpit Agarwal commented on HDFS-4967:
-
Thanks for reporting Fengdong! I am taking a look.
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
Assignee: Arpit Agarwal
There are two name nodes, one is active, another acts as standby name node.
QJM Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4968) Provide configuration option for FileSystem symlink resolution
Andrew Wang created HDFS-4968:
-
Summary: Provide configuration option for FileSystem symlink
resolution
Key: HDFS-4968
URL: https://issues.apache.org/jira/browse/HDFS-4968
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.0.0, 2.2.0
Reporter: Andrew Wang
Assignee: Andrew Wang
With FileSystem symlink support incoming in HADOOP-8040, some clients will wish
to not transparently resolve symlinks. This is somewhat similar to O_NOFOLLOW
in open(2).
Rationale for is that in the new Hiveserver2 security model, all Hive queries
run as the Hive user. Users can also use Hive to query data in their homedirs,
provided it's readable by the Hive user. This leads to a security issue with
symlinks:
# User Mallory invokes Hive to process data files in {{/user/mallory/hive/}}
# Hiveserver2 checks permissions on the files in {{/user/mallory/hive/}} and
allows the query to proceed.
# RACE: Mallory replaces the files in {{/user/mallory/hive}} with symlinks that
point to user Ann's Hive files in {{/user/ann/hive}}. These files aren't
readable by Mallory, but she can create whatever symlinks she wants in her own
scratch directory.
# Hive's MR jobs happily resolve the symlinks and accesses Ann's private data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702937#comment-13702937
]
Arpit Agarwal commented on HDFS-4967:
-
[~azuryy] Was this an upgrade or a freshly formatted cluster? I am not sure how
we got an image with a zero {{LastAllocatedBlockId}}. I'll see if I can repro
it.
Thanks.
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
Assignee: Arpit Agarwal
There are two name nodes, one is active, another acts as standby name node.
QJM Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4967) Generate block ID sequentially cannot work with QJM HA
[
https://issues.apache.org/jira/browse/HDFS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702952#comment-13702952
]
Fengdong Yu commented on HDFS-4967:
---
Hi Arpit,
I will add some reprduce steps here shortly.
Generate block ID sequentially cannot work with QJM HA
--
Key: HDFS-4967
URL: https://issues.apache.org/jira/browse/HDFS-4967
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client, namenode
Affects Versions: 3.0.0
Reporter: Fengdong Yu
Assignee: Arpit Agarwal
There are two name nodes, one is active, another acts as standby name node.
QJM Ha configured.
After HDFS-4645 committed in the trunk, then the following error showed
during name node start:
{code}
2013-07-09 11:28:45,394 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.lang.IllegalStateException: Cannot skip to less than the current value
(=1073741824), where newValue=0
at
org.apache.hadoop.util.SequentialNumber.skipTo(SequentialNumber.java:58)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setLastAllocatedBlockId(FSNamesystem.java:5124)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Loader.load(FSImageFormat.java:278)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:809)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:798)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:623)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:719)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:552)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:401)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:435)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:607)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:592)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1172)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1238)
2013-07-09 11:28:45,397 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
