[jira] [Updated] (HDFS-7470) SecondaryNameNode need twice memory when calling reloadFromImageFile
[
https://issues.apache.org/jira/browse/HDFS-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhaoyunjiong updated HDFS-7470:
---
Attachment: HDFS-7470.1.patch
Update patch to fix test failure.
SecondaryNameNode need twice memory when calling reloadFromImageFile
Key: HDFS-7470
URL: https://issues.apache.org/jira/browse/HDFS-7470
Project: Hadoop HDFS
Issue Type: Bug
Reporter: zhaoyunjiong
Assignee: zhaoyunjiong
Attachments: HDFS-7470.1.patch, HDFS-7470.patch
histo information at 2014-12-02 01:19
{quote}
num #instances #bytes class name
--
1: 18644963019326123016 [Ljava.lang.Object;
2: 15736664915107198304
org.apache.hadoop.hdfs.server.namenode.INodeFile
3: 18340903011738177920
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo
4: 157358401 5244264024
[Lorg.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
5: 3 3489661000
[Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement;
6: 29253275 1872719664 [B
7: 3230821 284312248
org.apache.hadoop.hdfs.server.namenode.INodeDirectory
8: 2756284 110251360 java.util.ArrayList
9:469158 22519584 org.apache.hadoop.fs.permission.AclEntry
10: 847 17133032 [Ljava.util.HashMap$Entry;
11:188471 17059632 [C
12:314614 10067656
[Lorg.apache.hadoop.hdfs.server.namenode.INode$Feature;
13:2345799383160
com.google.common.collect.RegularImmutableList
14: 495846850280 constMethodKlass
15: 495846356704 methodKlass
16:1872705992640 java.lang.String
17:2345795629896
org.apache.hadoop.hdfs.server.namenode.AclFeature
{quote}
histo information at 2014-12-02 01:32
{quote}
num #instances #bytes class name
--
1: 35583805135566651032 [Ljava.lang.Object;
2: 30227275829018184768
org.apache.hadoop.hdfs.server.namenode.INodeFile
3: 35250072322560046272
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo
4: 30226451010075087952
[Lorg.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
5: 177120233 9374983920 [B
6: 3 3489661000
[Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement;
7: 6191688 544868544
org.apache.hadoop.hdfs.server.namenode.INodeDirectory
8: 2799256 111970240 java.util.ArrayList
9:890728 42754944 org.apache.hadoop.fs.permission.AclEntry
10:330986 29974408 [C
11:596871 19099880
[Lorg.apache.hadoop.hdfs.server.namenode.INode$Feature;
12:445364 17814560
com.google.common.collect.RegularImmutableList
13: 844 17132816 [Ljava.util.HashMap$Entry;
14:445364 10688736
org.apache.hadoop.hdfs.server.namenode.AclFeature
15:329789 10553248 java.lang.String
16: 917418807136
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
17: 495846850280 constMethodKlass
{quote}
And the stack trace shows it was doing reloadFromImageFile:
{quote}
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getInode(FSDirectory.java:2426)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:160)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:121)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:902)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:888)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.reloadFromImageFile(FSImage.java:562)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:1048)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:536)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:388)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:354)
at java.security.AccessController.doPrivileged(Native Method)
at
[jira] [Commented] (HDFS-7414) Namenode got shutdown and can't recover where edit update might be missed
[
https://issues.apache.org/jira/browse/HDFS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234088#comment-14234088
]
Brahma Reddy Battula commented on HDFS-7414:
Hi [~yzhangal]
Thanks for taking a look into this issue..
Yes, I have the HDFS-6527 fix..Please check the following for same...
{code}
private boolean isFileDeleted(INodeFile file)
{
return (this.dir.getInode(file.getId()) == null) || (file.getParent() ==
null) || ((file.isWithSnapshot())
(file.getFileWithSnapshotFeature().isCurrentFileDeleted()));
}
{code}
Namenode got shutdown and can't recover where edit update might be missed
-
Key: HDFS-7414
URL: https://issues.apache.org/jira/browse/HDFS-7414
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.1, 2.5.1
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Priority: Blocker
Scenario:
Was running mapreduce job.
CPU usage crossed 190% for Datanode and machine became slow..
and seen the following exception ..
*Did not get the exact root cause, but as cpu usage more edit log updation
might be missed...Need dig to more...anyone have any thoughts.*
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
CloseOp [length=0, inodeId=0,
path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025,
replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
clientMachine=, opCode=OP_CLOSE, txid=162982] |
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:741)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading
fsimage |
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
[jira] [Created] (HDFS-7471) TestDatanodeManager#testNumVersionsReportedCorrect occasionally fails
Ted Yu created HDFS-7471:
Summary: TestDatanodeManager#testNumVersionsReportedCorrect
occasionally fails
Key: HDFS-7471
URL: https://issues.apache.org/jira/browse/HDFS-7471
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1957/ :
{code}
FAILED:
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect
Error Message:
The map of version counts returned by DatanodeManager was not what it was
expected to be on iteration 237 expected:0 but was:1
Stack Trace:
java.lang.AssertionError: The map of version counts returned by DatanodeManager
was not what it was expected to be on iteration 237 expected:0 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect(TestDatanodeManager.java:150)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7472) Fix typo in message of ReplicaNotFoundException
Masatake Iwasaki created HDFS-7472: -- Summary: Fix typo in message of ReplicaNotFoundException Key: HDFS-7472 URL: https://issues.apache.org/jira/browse/HDFS-7472 Project: Hadoop HDFS Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial unexpeted - unexpected -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7472) Fix typo in message of ReplicaNotFoundException
[ https://issues.apache.org/jira/browse/HDFS-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-7472: --- Status: Patch Available (was: Open) Fix typo in message of ReplicaNotFoundException --- Key: HDFS-7472 URL: https://issues.apache.org/jira/browse/HDFS-7472 Project: Hadoop HDFS Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Attachments: HDFS-7472.1.patch unexpeted - unexpected -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7472) Fix typo in message of ReplicaNotFoundException
[ https://issues.apache.org/jira/browse/HDFS-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-7472: --- Attachment: HDFS-7472.1.patch Fix typo in message of ReplicaNotFoundException --- Key: HDFS-7472 URL: https://issues.apache.org/jira/browse/HDFS-7472 Project: Hadoop HDFS Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Attachments: HDFS-7472.1.patch unexpeted - unexpected -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6833) DirectoryScanner should not register a deleting block with memory of DataNode
[
https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234282#comment-14234282
]
Shinichi Yamashita commented on HDFS-6833:
--
Hi [~cnauroth], thank you for your care, and I'm sorry for the late reply.
I revise a patch based on the latest comment of Yongjun.
After the current of my work was settled, and I attach new patch next week.
DirectoryScanner should not register a deleting block with memory of DataNode
-
Key: HDFS-6833
URL: https://issues.apache.org/jira/browse/HDFS-6833
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.0.0, 2.5.0, 2.5.1
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Priority: Critical
Attachments: HDFS-6833-6-2.patch, HDFS-6833-6-3.patch,
HDFS-6833-6.patch, HDFS-6833-7-2.patch, HDFS-6833-7.patch, HDFS-6833.8.patch,
HDFS-6833.9.patch, HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch,
HDFS-6833.patch, HDFS-6833.patch
When a block is deleted in DataNode, the following messages are usually
output.
{code}
2014-08-07 17:53:11,606 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
for deletion
2014-08-07 17:53:11,617 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
{code}
However, DirectoryScanner may be executed when DataNode deletes the block in
the current implementation. And the following messsages are output.
{code}
2014-08-07 17:53:30,519 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
for deletion
2014-08-07 17:53:31,426 INFO
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata
files:0, missing block files:0, missing blocks in memory:1, mismatched
blocks:0
2014-08-07 17:53:31,426 WARN
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added
missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
getNumBytes() = 21230663
getBytesOnDisk() = 21230663
getVisibleLength()= 21230663
getVolume() = /hadoop/data1/dfs/data/current
getBlockFile()=
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
unlinked =false
2014-08-07 17:53:31,531 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
{code}
Deleting block information is registered in DataNode's memory.
And when DataNode sends a block report, NameNode receives wrong block
information.
For example, when we execute recommission or change the number of
replication, NameNode may delete the right block as ExcessReplicate by this
problem.
And Under-Replicated Blocks and Missing Blocks occur.
When DataNode run DirectoryScanner, DataNode should not register a deleting
block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7472) Fix typo in message of ReplicaNotFoundException
[ https://issues.apache.org/jira/browse/HDFS-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234330#comment-14234330 ] Akira AJISAKA commented on HDFS-7472: - LGTM, +1 (non-binding) pending Jenkins. Fix typo in message of ReplicaNotFoundException --- Key: HDFS-7472 URL: https://issues.apache.org/jira/browse/HDFS-7472 Project: Hadoop HDFS Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Attachments: HDFS-7472.1.patch unexpeted - unexpected -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
[
https://issues.apache.org/jira/browse/HDFS-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234364#comment-14234364
]
Yongjun Zhang commented on HDFS-7448:
-
Good find [~ajisakaa], thanks!
TestBookKeeperHACheckpoints fails in trunk build
Key: HDFS-7448
URL: https://issues.apache.org/jira/browse/HDFS-7448
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Akira AJISAKA
Priority: Minor
Fix For: 3.0.0, 2.7.0
Attachments: HDFS-7448-001.patch
The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
Time elapsed: 6.822 sec ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be
replicated to 0 nodes instead of minReplication (=1). There are 0
datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6833) DirectoryScanner should not register a deleting block with memory of DataNode
[
https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234366#comment-14234366
]
Chris Nauroth commented on HDFS-6833:
-
Thank you for the update, [~yamashitasni].
DirectoryScanner should not register a deleting block with memory of DataNode
-
Key: HDFS-6833
URL: https://issues.apache.org/jira/browse/HDFS-6833
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.0.0, 2.5.0, 2.5.1
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Priority: Critical
Attachments: HDFS-6833-6-2.patch, HDFS-6833-6-3.patch,
HDFS-6833-6.patch, HDFS-6833-7-2.patch, HDFS-6833-7.patch, HDFS-6833.8.patch,
HDFS-6833.9.patch, HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch,
HDFS-6833.patch, HDFS-6833.patch
When a block is deleted in DataNode, the following messages are usually
output.
{code}
2014-08-07 17:53:11,606 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
for deletion
2014-08-07 17:53:11,617 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
{code}
However, DirectoryScanner may be executed when DataNode deletes the block in
the current implementation. And the following messsages are output.
{code}
2014-08-07 17:53:30,519 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
for deletion
2014-08-07 17:53:31,426 INFO
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata
files:0, missing block files:0, missing blocks in memory:1, mismatched
blocks:0
2014-08-07 17:53:31,426 WARN
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added
missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
getNumBytes() = 21230663
getBytesOnDisk() = 21230663
getVisibleLength()= 21230663
getVolume() = /hadoop/data1/dfs/data/current
getBlockFile()=
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
unlinked =false
2014-08-07 17:53:31,531 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
{code}
Deleting block information is registered in DataNode's memory.
And when DataNode sends a block report, NameNode receives wrong block
information.
For example, when we execute recommission or change the number of
replication, NameNode may delete the right block as ExcessReplicate by this
problem.
And Under-Replicated Blocks and Missing Blocks occur.
When DataNode run DirectoryScanner, DataNode should not register a deleting
block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7426) Change nntop JMX format to be a JSON blob
[ https://issues.apache.org/jira/browse/HDFS-7426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234451#comment-14234451 ] Andrew Wang commented on HDFS-7426: --- Hey Maysam, I think I kept all the information there. Rather an ALL_USERS user, I added a totalCount field for each operation which I think does the same thing. I agree that having the total count is important to get a sense of the whole picture. If you want to forego the HTML view that's also fine with me, also agree that the blob can suffice. Change nntop JMX format to be a JSON blob - Key: HDFS-7426 URL: https://issues.apache.org/jira/browse/HDFS-7426 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.7.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-7426.001.patch, hdfs-7426.002.patch After discussion with [~maysamyabandeh], we think we can adjust the JMX output to instead be a richer JSON blob. This should be easier to parse and also be more informative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234457#comment-14234457 ] Suresh Srinivas commented on HDFS-6826: --- [~daryn], I agree with some of the comments you have made. One of the main goals from Ranger (previously Argus) perspective is to allow plugin for checking permission. This will help implement in Argus project a centralized permission model. This is lot smaller and self contained than the current scope of changes. While this discussion can continue, I want to go ahead and create a jira for the narrow scope of pluggability that Ranger is looking for. Let me know what you guys think. I will create a separate jira, if there is general consensus. Plugin interface to enable delegation of HDFS authorization assertions -- Key: HDFS-6826 URL: https://issues.apache.org/jira/browse/HDFS-6826 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch, HDFS-6826v8.patch, HDFS-6826v9.patch, HDFSPluggableAuthorizationProposal-v2.pdf, HDFSPluggableAuthorizationProposal.pdf When Hbase data, HiveMetaStore data or Search data is accessed via services (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce permissions on corresponding entities (databases, tables, views, columns, search collections, documents). It is desirable, when the data is accessed directly by users accessing the underlying data files (i.e. from a MapReduce job), that the permission of the data files map to the permissions of the corresponding data entity (i.e. table, column family or search collection). To enable this we need to have the necessary hooks in place in the NameNode to delegate authorization to an external system that can map HDFS files/directories to data entities and resolve their permissions based on the data entities permissions. I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234462#comment-14234462 ] Brandon Li commented on HDFS-7424: -- I've committed the patch. Thank you, Haohui, for the review! Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch, HDFS-7424.004.patch, HDFS-7424.005.patch, HDFS-7424.006.patch, HDFS-7424.007.patch, HDFS-7424.008.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-7424: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch, HDFS-7424.004.patch, HDFS-7424.005.patch, HDFS-7424.006.patch, HDFS-7424.007.patch, HDFS-7424.008.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234472#comment-14234472 ] Hudson commented on HDFS-7424: -- FAILURE: Integrated in Hadoop-trunk-Commit #6648 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6648/]) HDFS-7424. Add web UI for NFS gateway. Contributed by Brandon Li (brandonli: rev 1bbcc3d0320b9435317bfeaa078af22d4de8d00c) * hadoop-hdfs-project/hadoop-hdfs-nfs/pom.xml * hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/pom.xml * hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3HttpServer.java * hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestNfs3HttpServer.java Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch, HDFS-7424.004.patch, HDFS-7424.005.patch, HDFS-7424.006.patch, HDFS-7424.007.patch, HDFS-7424.008.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7424) Add web UI for NFS gateway
[ https://issues.apache.org/jira/browse/HDFS-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-7424: - Fix Version/s: 2.7.0 Add web UI for NFS gateway -- Key: HDFS-7424 URL: https://issues.apache.org/jira/browse/HDFS-7424 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 2.2.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 2.7.0 Attachments: HDFS-7424.001.patch, HDFS-7424.002.patch, HDFS-7424.003.patch, HDFS-7424.004.patch, HDFS-7424.005.patch, HDFS-7424.006.patch, HDFS-7424.007.patch, HDFS-7424.008.patch This JIRA is to track the effort to add web UI for NFS gateway to show some metrics and configuration related information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234476#comment-14234476 ] Aaron T. Myers commented on HDFS-6826: -- The problem I have with just an FsPermissionChecker plugin is that a user or admin who goes to look at the permissions of the relevant files in HDFS will see completely different permissions than what are in fact being enforced by the NN. I'm certainly not wedded to the current implementation that's been posted on this JIRA, but I feel pretty strongly that any solution that allows for outsourcing the HDFS authorization decisions should also show to the user the permissions that are being enforced. If your proposed alternate implementation supports that, then it'll be fine by me. Plugin interface to enable delegation of HDFS authorization assertions -- Key: HDFS-6826 URL: https://issues.apache.org/jira/browse/HDFS-6826 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch, HDFS-6826v8.patch, HDFS-6826v9.patch, HDFSPluggableAuthorizationProposal-v2.pdf, HDFSPluggableAuthorizationProposal.pdf When Hbase data, HiveMetaStore data or Search data is accessed via services (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce permissions on corresponding entities (databases, tables, views, columns, search collections, documents). It is desirable, when the data is accessed directly by users accessing the underlying data files (i.e. from a MapReduce job), that the permission of the data files map to the permissions of the corresponding data entity (i.e. table, column family or search collection). To enable this we need to have the necessary hooks in place in the NameNode to delegate authorization to an external system that can map HDFS files/directories to data entities and resolve their permissions based on the data entities permissions. I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234651#comment-14234651 ] Suresh Srinivas commented on HDFS-6826: --- [~atm], I see your point. I agree with it. The root cause of this seems to be the place where we are adding pluggability - FSPermissionChecker, which ties the access control implementation to permissions. How about this? Lets say we add pluggable AccessControlPolicy that gets called before permission checker. One could have custom policies that can check whether access is allowed based on wild card paths, where the access is coming from etc. and allow the operation.Policy implementation decides to allow access, disallow access, or continue (to FSPermissionChecker). Would that work? Thought? Plugin interface to enable delegation of HDFS authorization assertions -- Key: HDFS-6826 URL: https://issues.apache.org/jira/browse/HDFS-6826 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch, HDFS-6826v8.patch, HDFS-6826v9.patch, HDFSPluggableAuthorizationProposal-v2.pdf, HDFSPluggableAuthorizationProposal.pdf When Hbase data, HiveMetaStore data or Search data is accessed via services (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce permissions on corresponding entities (databases, tables, views, columns, search collections, documents). It is desirable, when the data is accessed directly by users accessing the underlying data files (i.e. from a MapReduce job), that the permission of the data files map to the permissions of the corresponding data entity (i.e. table, column family or search collection). To enable this we need to have the necessary hooks in place in the NameNode to delegate authorization to an external system that can map HDFS files/directories to data entities and resolve their permissions based on the data entities permissions. I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7414) Namenode got shutdown and can't recover where edit update might be missed
[
https://issues.apache.org/jira/browse/HDFS-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234684#comment-14234684
]
Yongjun Zhang commented on HDFS-7414:
-
HI [~brahmareddy],
There is another related jira to be aware of: HDFS-6825.
If you could describe the steps that reproduce the issue, that will help.
Thanks.
Namenode got shutdown and can't recover where edit update might be missed
-
Key: HDFS-7414
URL: https://issues.apache.org/jira/browse/HDFS-7414
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.1, 2.5.1
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Priority: Blocker
Scenario:
Was running mapreduce job.
CPU usage crossed 190% for Datanode and machine became slow..
and seen the following exception ..
*Did not get the exact root cause, but as cpu usage more edit log updation
might be missed...Need dig to more...anyone have any thoughts.*
{noformat}
2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation
CloseOp [length=0, inodeId=0,
path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025,
replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864,
blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337],
permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=,
clientMachine=, opCode=OP_CLOSE, txid=162982] |
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:741)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:724)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
2014-11-20 05:01:18,654 | WARN | main | Encountered exception loading
fsimage |
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
java.io.FileNotFoundException: File does not exist:
/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_25_1/part-m-00025
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
at
[jira] [Created] (HDFS-7473) Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error
Jason Keller created HDFS-7473: -- Summary: Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error Key: HDFS-7473 URL: https://issues.apache.org/jira/browse/HDFS-7473 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.2 Reporter: Jason Keller When setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml, the error java.lang.IllegalArgumentException: Cannot set dfs.namenode.fs-limits.max-directory-items to a value less than 0 or greater than 640 is produced. However, the documentation shows that 0 is a valid setting for dfs.namenode.fs-limits.max-directory-items, turning the check off. Looking into the code in hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java shows that the culprit is Preconditions.checkArgument(maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set + DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY+ to a value less than 0 or greater than + MAX_DIR_ITEMS); This checks if maxDirItems is greater than 0. Since 0 is not greater than 0, it produces an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7468) Moving verify* functions to corresponding classes
[ https://issues.apache.org/jira/browse/HDFS-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7468: - Summary: Moving verify* functions to corresponding classes (was: Code cleanup for verify* in FSDirectory) Moving verify* functions to corresponding classes - Key: HDFS-7468 URL: https://issues.apache.org/jira/browse/HDFS-7468 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Li Lu Assignee: Li Lu Attachments: HDFS-7468-120214-1.patch, HDFS-7468-120214.patch We need to clean up verify* methods in FSDirectory. It would be cleaner if those methods are close to their call sites. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7468) Moving verify* functions to corresponding classes
[ https://issues.apache.org/jira/browse/HDFS-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7468: - Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed the patch to trunk and branch-2. Thanks [~gtCarrera9] for the contribution. Moving verify* functions to corresponding classes - Key: HDFS-7468 URL: https://issues.apache.org/jira/browse/HDFS-7468 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Li Lu Assignee: Li Lu Fix For: 2.7.0 Attachments: HDFS-7468-120214-1.patch, HDFS-7468-120214.patch We need to clean up verify* methods in FSDirectory. It would be cleaner if those methods are close to their call sites. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6264) Provide FileSystem#create() variant which throws exception if parent directory doesn't exist
[ https://issues.apache.org/jira/browse/HDFS-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234711#comment-14234711 ] Ted Yu commented on HDFS-6264: -- I wonder if sub-task(s) should be created according to Steve's comment above. Provide FileSystem#create() variant which throws exception if parent directory doesn't exist Key: HDFS-6264 URL: https://issues.apache.org/jira/browse/HDFS-6264 Project: Hadoop HDFS Issue Type: Task Components: namenode Affects Versions: 2.4.0 Reporter: Ted Yu Labels: hbase Attachments: hdfs-6264-v1.txt FileSystem#createNonRecursive() is deprecated. However, there is no DistributedFileSystem#create() implementation which throws exception if parent directory doesn't exist. This limits clients' migration away from the deprecated method. For HBase, IO fencing relies on the behavior of FileSystem#createNonRecursive(). Variant of create() method should be added which throws exception if parent directory doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7468) Moving verify* functions to corresponding classes
[ https://issues.apache.org/jira/browse/HDFS-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234716#comment-14234716 ] Hudson commented on HDFS-7468: -- FAILURE: Integrated in Hadoop-trunk-Commit #6650 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6650/]) HDFS-7468. Moving verify* functions to corresponding classes. Contributed by Li Lu. (wheat9: rev 26d8dec756da1d9bd3df3b41a4dd5d8ff03bc5b2) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirSnapshotOp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java Moving verify* functions to corresponding classes - Key: HDFS-7468 URL: https://issues.apache.org/jira/browse/HDFS-7468 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Li Lu Assignee: Li Lu Fix For: 2.7.0 Attachments: HDFS-7468-120214-1.patch, HDFS-7468-120214.patch We need to clean up verify* methods in FSDirectory. It would be cleaner if those methods are close to their call sites. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7463) Simplify FSNamesystem#getBlockLocationsUpdateTimes
[
https://issues.apache.org/jira/browse/HDFS-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7463:
-
Attachment: HDFS-7463.002.patch
Simplify FSNamesystem#getBlockLocationsUpdateTimes
--
Key: HDFS-7463
URL: https://issues.apache.org/jira/browse/HDFS-7463
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7463.000.patch, HDFS-7463.001.patch,
HDFS-7463.002.patch
Currently {{FSNamesystem#getBlockLocationsUpdateTimes}} holds the read lock
to access the blocks. It releases the read lock and then acquires the write
lock when it needs to update the access time of the {{INode}}.
This jira proposes to move the responsibility of the latter steps to the
caller to simplify the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7474) Avoid resolve path in FSPermissionChecker
Jing Zhao created HDFS-7474: --- Summary: Avoid resolve path in FSPermissionChecker Key: HDFS-7474 URL: https://issues.apache.org/jira/browse/HDFS-7474 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Currently for an typical namenode operation a path can be resolved multiple times. Usually the FSPermissionChecker first takes the path string as input and resolve the path before the real checking. After permission checking the path is then resolved again (and again) for real operations in FSNamesystem and FSDirectory. Ideally we should resolve the path only once, and use INodesInPath internally afterwards. As a first step, we can start from avoiding path resolving in FSPermissionChecker. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7463) Simplify FSNamesystem#getBlockLocationsUpdateTimes
[
https://issues.apache.org/jira/browse/HDFS-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234737#comment-14234737
]
Haohui Mai commented on HDFS-7463:
--
Thanks for the review. The v2 patch moves the {{now()}} into the write-lock.
bq. Often it is good to have those for figuring out what exceptions to unwrap
on the client side, right?
I agree the the complete lists have some value in terms of providing some hints
to the client. My concern is that the lists are neither complete nor consistent
in most cases. Take this patch for example:
{code}
LocatedBlocks getBlockLocations(String src, long offset, long length,
boolean doAccessTime, boolean needBlockToken, boolean checkSafeMode)
throws FileNotFoundException, UnresolvedLinkException, IOException {
{code}
Which miss the {{StandbyException}.
For {{GetFileInfo}} in {{NameNodeAdaptor}}:
{code}
public static HdfsFileStatus getFileInfo(NameNode namenode, String src,
boolean resolveLink) throws AccessControlException,
UnresolvedLinkException,
StandbyException, IOException {
{code}
Which misses {{FileNotFoundException}}.
Currently there is no effective automated mechanisms to enforce the lists are
complete and consistent. Occasionally it creates more confusions than the hints
provided to the client if you look through all operations in {{FSNamesystem}}.
Simplify FSNamesystem#getBlockLocationsUpdateTimes
--
Key: HDFS-7463
URL: https://issues.apache.org/jira/browse/HDFS-7463
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7463.000.patch, HDFS-7463.001.patch,
HDFS-7463.002.patch
Currently {{FSNamesystem#getBlockLocationsUpdateTimes}} holds the read lock
to access the blocks. It releases the read lock and then acquires the write
lock when it needs to update the access time of the {{INode}}.
This jira proposes to move the responsibility of the latter steps to the
caller to simplify the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[
https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234748#comment-14234748
]
Haohui Mai commented on HDFS-6407:
--
bq. It will be a good idea to standardize on one sort/pagination plugin across
hdfs and yarn UIs.
Just to reiterate, I don't see a way to do it correctly by simply sorting rows
in the HTML tables. What are the expected results of the following case?
{code}
|| DN || DFS Usage ||
| dn1 | 92 GB |
| dn2 | 110 GB |
| dn3 | 4 MB |
{code}
If you sort them by DFS Usage alphabetically (which is what the plugin does),
you'll get:
{code}
|| DN || DFS Usage ||
| dn2 | 110 GB |
| dn3 | 4 MB |
| dn1 | 92 GB |
{code}
The results do not make sense to me. What it implies is that the sorting should
be done at the JSON data, and it should be done at the same time when
pagination is done.
new namenode UI, lost ability to sort columns in datanode tab
-
Key: HDFS-6407
URL: https://issues.apache.org/jira/browse/HDFS-6407
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.4.0
Reporter: Nathan Roberts
Assignee: Benoy Antony
Priority: Minor
Attachments: HDFS-6407.patch, browse_directory.png, datanodes.png,
snapshots.png
old ui supported clicking on column header to sort on that column. The new ui
seems to have dropped this very useful feature.
There are a few tables in the Namenode UI to display datanodes information,
directory listings and snapshots.
When there are many items in the tables, it is useful to have ability to sort
on the different columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7475) Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
Xiaoyu Yao created HDFS-7475:
Summary: Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved
deterministic
Key: HDFS-7475
URL: https://issues.apache.org/jira/browse/HDFS-7475
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled. The
task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist task
count reduce to 0 before verifying the block file exist in lazy persist dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7475) Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
[
https://issues.apache.org/jira/browse/HDFS-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-7475:
-
Description:
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled. The
task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist task
count reduce to 0 before verifying the block file exist in lazy persist dir.
was:
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled. The
task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist task
count reduce to 0 before verifying the block file exist in lazy persist dir.
Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
--
Key: HDFS-7475
URL: https://issues.apache.org/jira/browse/HDFS-7475
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled.
The task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist
task count reduce to 0 before verifying the block file exist in lazy persist
dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7475) Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
[
https://issues.apache.org/jira/browse/HDFS-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-7475:
-
Attachment: HDFS-7475.0.patch
Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
--
Key: HDFS-7475
URL: https://issues.apache.org/jira/browse/HDFS-7475
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7475.0.patch
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled.
The task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist
task count reduce to 0 before verifying the block file exist in lazy persist
dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7474) Avoid resolve path in FSPermissionChecker
[ https://issues.apache.org/jira/browse/HDFS-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7474: Status: Patch Available (was: Open) Avoid resolve path in FSPermissionChecker - Key: HDFS-7474 URL: https://issues.apache.org/jira/browse/HDFS-7474 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-7474.000.patch Currently for an typical namenode operation a path can be resolved multiple times. Usually the FSPermissionChecker first takes the path string as input and resolve the path before the real checking. After permission checking the path is then resolved again (and again) for real operations in FSNamesystem and FSDirectory. Ideally we should resolve the path only once, and use INodesInPath internally afterwards. As a first step, we can start from avoiding path resolving in FSPermissionChecker. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7474) Avoid resolve path in FSPermissionChecker
[ https://issues.apache.org/jira/browse/HDFS-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7474: Attachment: HDFS-7474.000.patch Initial patch. Also include some changes to avoid unnecessary path resolving for snapshot related ops. Avoid resolve path in FSPermissionChecker - Key: HDFS-7474 URL: https://issues.apache.org/jira/browse/HDFS-7474 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-7474.000.patch Currently for an typical namenode operation a path can be resolved multiple times. Usually the FSPermissionChecker first takes the path string as input and resolve the path before the real checking. After permission checking the path is then resolved again (and again) for real operations in FSNamesystem and FSDirectory. Ideally we should resolve the path only once, and use INodesInPath internally afterwards. As a first step, we can start from avoiding path resolving in FSPermissionChecker. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7475) Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
[
https://issues.apache.org/jira/browse/HDFS-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-7475:
-
Status: Patch Available (was: Open)
Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
--
Key: HDFS-7475
URL: https://issues.apache.org/jira/browse/HDFS-7475
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7475.0.patch
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled.
The task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist
task count reduce to 0 before verifying the block file exist in lazy persist
dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7426) Change nntop JMX format to be a JSON blob
[ https://issues.apache.org/jira/browse/HDFS-7426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234828#comment-14234828 ] Maysam Yabandeh commented on HDFS-7426: --- Ah, yeah you are right. I missed that. I am +1 with the patch. Change nntop JMX format to be a JSON blob - Key: HDFS-7426 URL: https://issues.apache.org/jira/browse/HDFS-7426 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.7.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-7426.001.patch, hdfs-7426.002.patch After discussion with [~maysamyabandeh], we think we can adjust the JMX output to instead be a richer JSON blob. This should be easier to parse and also be more informative. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7465) Html view for nntop: top users of name node
[ https://issues.apache.org/jira/browse/HDFS-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh resolved HDFS-7465. --- Resolution: Not a Problem Release Note: As discussed in HDFS-7426 there is not much demand for this feature. Canceling it for the moment and we can reopen it later if some interest showed up. Html view for nntop: top users of name node --- Key: HDFS-7465 URL: https://issues.apache.org/jira/browse/HDFS-7465 Project: Hadoop HDFS Issue Type: Improvement Reporter: Maysam Yabandeh Priority: Minor Fix For: 2.7.0 nntop (HDFS-6982) maintains the list of top users of the name node and export the list via jmx. An external monitoring tool can then draw graphs of the top users using jmx interface. For small-scale deployments that are not yet equipped with an external monitoring tool, however, it would be useful to show the list of recent top users using a html view. The html page lists the top users of each name node command in the past 1, 5, 25 minutes (configurable). The details can be found in the design doc of nntop: https://issues.apache.org/jira/secure/attachment/12665990/nntop-design-v1.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7469) Inline permission checking done by FSPermissionChecker#checkSubAccess
[
https://issues.apache.org/jira/browse/HDFS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7469:
-
Attachment: HDFS-7469.003.patch
Inline permission checking done by FSPermissionChecker#checkSubAccess
-
Key: HDFS-7469
URL: https://issues.apache.org/jira/browse/HDFS-7469
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7469.000.patch, HDFS-7469.001.patch,
HDFS-7469.002.patch, HDFS-7469.003.patch
{{FSPermissionChecker#checkSubAccess}} is used by {{getContentSummary()}} and
{{computeSnapshotDiff()}} to validate the permission of the entire subtree
through BFS.
As these two calls traverse subtrees to compute the result, the permission
checking should be done during the same traversal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7469) Inline permission checking done by FSPermissionChecker#checkSubAccess
[
https://issues.apache.org/jira/browse/HDFS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234841#comment-14234841
]
Haohui Mai commented on HDFS-7469:
--
The v3 patch inlines the checks for {{getContentSummary()}} and
{{snapshotDiff()}}. For {{delete()}} the check cannot be inlined, thus the
patch creates a specialized function for {{delete()}} to simplify the common
code path.
Inline permission checking done by FSPermissionChecker#checkSubAccess
-
Key: HDFS-7469
URL: https://issues.apache.org/jira/browse/HDFS-7469
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7469.000.patch, HDFS-7469.001.patch,
HDFS-7469.002.patch, HDFS-7469.003.patch
{{FSPermissionChecker#checkSubAccess}} is used by {{getContentSummary()}} and
{{computeSnapshotDiff()}} to validate the permission of the entire subtree
through BFS.
As these two calls traverse subtrees to compute the result, the permission
checking should be done during the same traversal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234854#comment-14234854 ] Zhe Zhang commented on HDFS-7285: - [~sureshms] Sure, we are working on an updated design document and will post it soon. Erasure Coding Support inside HDFS -- Key: HDFS-7285 URL: https://issues.apache.org/jira/browse/HDFS-7285 Project: Hadoop HDFS Issue Type: New Feature Reporter: Weihua Jiang Assignee: Zhe Zhang Attachments: HDFSErasureCodingDesign-20141028.pdf Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data. Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back. We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7457) DatanodeID generates excessive garbage
[
https://issues.apache.org/jira/browse/HDFS-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234895#comment-14234895
]
Lei (Eddy) Xu commented on HDFS-7457:
-
[~daryn] This is an interesting finding and thanks for working on it.
I just have a small question:
{code:java}
if (hashCode == -1) {
int newHashCode = xferAddr.hashCode() ^ datanodeUuid.hashCode();
hashCode = newHashCode Integer.MAX_VALUE;
}
return hashCode;
{code}
Do we need to change this? {{int}} in {{hashCode}}, as primitive, should not
be garbage collected.
Other than that, this patch looks very good overall.
DatanodeID generates excessive garbage
--
Key: HDFS-7457
URL: https://issues.apache.org/jira/browse/HDFS-7457
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-7457.patch
{{DatanodeID#getXferAddr}} is a dynamically generated string. This string is
repeatedly generated for the hash code, equality, comparisons, and
stringification. Every DN-NN RPC method calls
{{DatanodeManager#getDatanode}} to validate if the node is registered, which
involves a call to {{getXferAddr}}.
The dynamic computation generates unnecessary trash that puts unnecessary
pressure on the GC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7463) Simplify FSNamesystem#getBlockLocationsUpdateTimes
[
https://issues.apache.org/jira/browse/HDFS-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234900#comment-14234900
]
Charles Lamb commented on HDFS-7463:
Hi [~wheat9],
The changes to now() seem fine in .002.
bq. My concern is that the lists are neither complete nor consistent in most
cases.
I feel your pain. I might suggest that this is an issue for a different Jira
perhaps? It feels like something of a policy change that should be addressed in
a more all-or-nothing scale rather than a one-off targeted approach.
Charles
Simplify FSNamesystem#getBlockLocationsUpdateTimes
--
Key: HDFS-7463
URL: https://issues.apache.org/jira/browse/HDFS-7463
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7463.000.patch, HDFS-7463.001.patch,
HDFS-7463.002.patch
Currently {{FSNamesystem#getBlockLocationsUpdateTimes}} holds the read lock
to access the blocks. It releases the read lock and then acquires the write
lock when it needs to update the access time of the {{INode}}.
This jira proposes to move the responsibility of the latter steps to the
caller to simplify the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7422) TestEncryptionZonesWithKMS fails against Java 8
[
https://issues.apache.org/jira/browse/HDFS-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234918#comment-14234918
]
Mohammad Kamrul Islam commented on HDFS-7422:
-
[~tedyu] i ran against JDK 1.8.0.5, it worked fine on Linux.
Did it fail multiple times?
TestEncryptionZonesWithKMS fails against Java 8
---
Key: HDFS-7422
URL: https://issues.apache.org/jira/browse/HDFS-7422
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/12/ :
{code}
REGRESSION:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testReadWriteUsingWebHdfs
Error Message:
Stream closed.
Stack Trace:
java.io.IOException: Stream closed.
at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:385)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:656)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:622)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:104)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.init(ByteRangeInputStream.java:89)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.init(WebHdfsFileSystem.java:1261)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1175)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at
org.apache.hadoop.hdfs.DFSTestUtil.verifyFilesEqual(DFSTestUtil.java:1399)
at
org.apache.hadoop.hdfs.TestEncryptionZones.testReadWriteUsingWebHdfs(TestEncryptionZones.java:634)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.ipc.RemoteException: Stream closed.
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:353)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
[jira] [Commented] (HDFS-7422) TestEncryptionZonesWithKMS fails against Java 8
[
https://issues.apache.org/jira/browse/HDFS-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235005#comment-14235005
]
Ted Yu commented on HDFS-7422:
--
Unfortunately Jenkins doesn't keep build #12.
I have increased the number of builds Jenkins retains.
Also trying to reproduce locally.
TestEncryptionZonesWithKMS fails against Java 8
---
Key: HDFS-7422
URL: https://issues.apache.org/jira/browse/HDFS-7422
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/12/ :
{code}
REGRESSION:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testReadWriteUsingWebHdfs
Error Message:
Stream closed.
Stack Trace:
java.io.IOException: Stream closed.
at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:385)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:656)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:622)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:104)
at
org.apache.hadoop.hdfs.web.ByteRangeInputStream.init(ByteRangeInputStream.java:89)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.init(WebHdfsFileSystem.java:1261)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1175)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at
org.apache.hadoop.hdfs.DFSTestUtil.verifyFilesEqual(DFSTestUtil.java:1399)
at
org.apache.hadoop.hdfs.TestEncryptionZones.testReadWriteUsingWebHdfs(TestEncryptionZones.java:634)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.ipc.RemoteException: Stream closed.
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:353)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235016#comment-14235016 ] Benoy Antony commented on HDFS-6407: It is possible to add custom sorting datatables plugin (used in yarn) to address these cases. A similar example is shown here: http://www.datatables.net/plug-ins/sorting/file-size. This can be modified to include TB and PB. new namenode UI, lost ability to sort columns in datanode tab - Key: HDFS-6407 URL: https://issues.apache.org/jira/browse/HDFS-6407 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: Nathan Roberts Assignee: Benoy Antony Priority: Minor Attachments: HDFS-6407.patch, browse_directory.png, datanodes.png, snapshots.png old ui supported clicking on column header to sort on that column. The new ui seems to have dropped this very useful feature. There are a few tables in the Namenode UI to display datanodes information, directory listings and snapshots. When there are many items in the tables, it is useful to have ability to sort on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235016#comment-14235016 ] Benoy Antony edited comment on HDFS-6407 at 12/5/14 4:03 AM: - It is possible to add custom sorting to datatables plugin (used in yarn) to address these cases. A similar example is shown here: http://www.datatables.net/plug-ins/sorting/file-size. This can be modified to include TB and PB. was (Author: benoyantony): It is possible to add custom sorting datatables plugin (used in yarn) to address these cases. A similar example is shown here: http://www.datatables.net/plug-ins/sorting/file-size. This can be modified to include TB and PB. new namenode UI, lost ability to sort columns in datanode tab - Key: HDFS-6407 URL: https://issues.apache.org/jira/browse/HDFS-6407 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: Nathan Roberts Assignee: Benoy Antony Priority: Minor Attachments: HDFS-6407.patch, browse_directory.png, datanodes.png, snapshots.png old ui supported clicking on column header to sort on that column. The new ui seems to have dropped this very useful feature. There are a few tables in the Namenode UI to display datanodes information, directory listings and snapshots. When there are many items in the tables, it is useful to have ability to sort on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[
https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235041#comment-14235041
]
Hadoop QA commented on HDFS-6585:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651837/patch_f15b7d505f12213f1ee9fb5ddb4bdaa64f9f623d.txt
against trunk revision 7896815.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8917//console
This message is automatically generated.
INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
---
Key: HDFS-6585
URL: https://issues.apache.org/jira/browse/HDFS-6585
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Zhilei Xu
Assignee: Zhilei Xu
Labels: patch
Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt,
patch_f15b7d505f12213f1ee9fb5ddb4bdaa64f9f623d.txt
Most of the APIs (both internal and external) in FSNamesystem calls
INodesInPath.resolve() to get the list of INodes corresponding to a file
path. Usually one API will call resolve() multiple times and that's a waste
of time.
This issue particularly refers to FSNamesystem.setPermission, which calls
resolve() twice indirectly: one from checkOwner(), another from
dir.setPermission().
Should save the result of resolve(), and use it whenever possible throughout
the lifetime of an API call, instead of making new resolve() calls.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235043#comment-14235043
]
Hadoop QA commented on HDFS-6826:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668911/HDFS-6826v9.patch
against trunk revision 7896815.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8919//console
This message is automatically generated.
Plugin interface to enable delegation of HDFS authorization assertions
--
Key: HDFS-6826
URL: https://issues.apache.org/jira/browse/HDFS-6826
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch,
HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch,
HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch,
HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch,
HDFS-6826v8.patch, HDFS-6826v9.patch,
HDFSPluggableAuthorizationProposal-v2.pdf,
HDFSPluggableAuthorizationProposal.pdf
When Hbase data, HiveMetaStore data or Search data is accessed via services
(Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
permissions on corresponding entities (databases, tables, views, columns,
search collections, documents). It is desirable, when the data is accessed
directly by users accessing the underlying data files (i.e. from a MapReduce
job), that the permission of the data files map to the permissions of the
corresponding data entity (i.e. table, column family or search collection).
To enable this we need to have the necessary hooks in place in the NameNode
to delegate authorization to an external system that can map HDFS
files/directories to data entities and resolve their permissions based on the
data entities permissions.
I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7463) Simplify FSNamesystem#getBlockLocationsUpdateTimes
[
https://issues.apache.org/jira/browse/HDFS-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7463:
-
Attachment: HDFS-7463.003.patch
Simplify FSNamesystem#getBlockLocationsUpdateTimes
--
Key: HDFS-7463
URL: https://issues.apache.org/jira/browse/HDFS-7463
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7463.000.patch, HDFS-7463.001.patch,
HDFS-7463.002.patch, HDFS-7463.003.patch
Currently {{FSNamesystem#getBlockLocationsUpdateTimes}} holds the read lock
to access the blocks. It releases the read lock and then acquires the write
lock when it needs to update the access time of the {{INode}}.
This jira proposes to move the responsibility of the latter steps to the
caller to simplify the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7454) Reduce memory footprint for AclEntries in NameNode
[
https://issues.apache.org/jira/browse/HDFS-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235065#comment-14235065
]
Haohui Mai commented on HDFS-7454:
--
The v4 patch looks good to me. +1
Reduce memory footprint for AclEntries in NameNode
--
Key: HDFS-7454
URL: https://issues.apache.org/jira/browse/HDFS-7454
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Attachments: HDFS-7454-001.patch, HDFS-7454-002.patch,
HDFS-7454-003.patch, HDFS-7454-004.patch
HDFS-5620 indicated a GlobalAclSet containing unique {{AclFeature}} can be
de-duplicated to save the memory in NameNode. However it was not implemented
at that time.
This Jira re-proposes same implementation, along with de-duplication of
unique {{AclEntry}} across all ACLs.
One simple usecase is:
A mapreduce user's home directory with the set of default ACLs, under which
lot of other files/directories could be created when jobs is run. Here all
the default ACLs of parent directory will be duplicated till the explicit
delete of those ACLs. With de-duplication,only one object will be in memory
for the same Entry across all ACLs of all files/directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7454) Reduce memory footprint for AclEntries in NameNode
[
https://issues.apache.org/jira/browse/HDFS-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7454:
-
Issue Type: Improvement (was: Bug)
Reduce memory footprint for AclEntries in NameNode
--
Key: HDFS-7454
URL: https://issues.apache.org/jira/browse/HDFS-7454
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Attachments: HDFS-7454-001.patch, HDFS-7454-002.patch,
HDFS-7454-003.patch, HDFS-7454-004.patch
HDFS-5620 indicated a GlobalAclSet containing unique {{AclFeature}} can be
de-duplicated to save the memory in NameNode. However it was not implemented
at that time.
This Jira re-proposes same implementation, along with de-duplication of
unique {{AclEntry}} across all ACLs.
One simple usecase is:
A mapreduce user's home directory with the set of default ACLs, under which
lot of other files/directories could be created when jobs is run. Here all
the default ACLs of parent directory will be duplicated till the explicit
delete of those ACLs. With de-duplication,only one object will be in memory
for the same Entry across all ACLs of all files/directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7454) Reduce memory footprint for AclEntries in NameNode
[
https://issues.apache.org/jira/browse/HDFS-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7454:
-
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~vinayrpet] for the
contribution.
Reduce memory footprint for AclEntries in NameNode
--
Key: HDFS-7454
URL: https://issues.apache.org/jira/browse/HDFS-7454
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Fix For: 2.7.0
Attachments: HDFS-7454-001.patch, HDFS-7454-002.patch,
HDFS-7454-003.patch, HDFS-7454-004.patch
HDFS-5620 indicated a GlobalAclSet containing unique {{AclFeature}} can be
de-duplicated to save the memory in NameNode. However it was not implemented
at that time.
This Jira re-proposes same implementation, along with de-duplication of
unique {{AclEntry}} across all ACLs.
One simple usecase is:
A mapreduce user's home directory with the set of default ACLs, under which
lot of other files/directories could be created when jobs is run. Here all
the default ACLs of parent directory will be duplicated till the explicit
delete of those ACLs. With de-duplication,only one object will be in memory
for the same Entry across all ACLs of all files/directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7454) Reduce memory footprint for AclEntries in NameNode
[
https://issues.apache.org/jira/browse/HDFS-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235107#comment-14235107
]
Hudson commented on HDFS-7454:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #6652 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6652/])
HDFS-7454. Reduce memory footprint for AclEntries in NameNode. Contributed by
Vinayakumar B. (wheat9: rev 0653918dad855b394e8e3b8b3f512f474d872ee9)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclEntryStatusFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSAclBaseTest.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/AclFeature.java
Reduce memory footprint for AclEntries in NameNode
--
Key: HDFS-7454
URL: https://issues.apache.org/jira/browse/HDFS-7454
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Fix For: 2.7.0
Attachments: HDFS-7454-001.patch, HDFS-7454-002.patch,
HDFS-7454-003.patch, HDFS-7454-004.patch
HDFS-5620 indicated a GlobalAclSet containing unique {{AclFeature}} can be
de-duplicated to save the memory in NameNode. However it was not implemented
at that time.
This Jira re-proposes same implementation, along with de-duplication of
unique {{AclEntry}} across all ACLs.
One simple usecase is:
A mapreduce user's home directory with the set of default ACLs, under which
lot of other files/directories could be created when jobs is run. Here all
the default ACLs of parent directory will be duplicated till the explicit
delete of those ACLs. With de-duplication,only one object will be in memory
for the same Entry across all ACLs of all files/directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7454) Reduce memory footprint for AclEntries in NameNode
[
https://issues.apache.org/jira/browse/HDFS-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235115#comment-14235115
]
Chris Nauroth commented on HDFS-7454:
-
[~vinayrpet], thank you for the patch. [~wheat9], thank you for taking care of
code review and commit. I see there was one specific question directed to me,
and I apologize for not being able to reply sooner.
{quote}
I have one doubt here..
whether we really need to append all ACL entries along with permission bits in
the exception message?
By seeing these AclEntries, caller could easily access by impersonating one of
the user in the entries? Right?
{quote}
File system permissions and ACLs assume strong authentication is in place
first. In a cluster using Kerberos, I don't expect seeing ACL entries alone
would compromise our security. The user wouldn't be able to impersonate
another user anyway, unless there was some other misconfiguration, such as
allowing the user access to private keytab files.
I'd suggest we either restore the old exception message or just append the '+'
indicator if an ACL is present, like the ls command. This will let users know
that they should consider ACLs if they are dealing with an unexpected access
denied. We can do it in a follow-up jira.
Thanks again, Vinay! I'm aiming to review HDFS-7456 tomorrow, and of course
finishing out HDFS-7384 too.
Reduce memory footprint for AclEntries in NameNode
--
Key: HDFS-7454
URL: https://issues.apache.org/jira/browse/HDFS-7454
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Fix For: 2.7.0
Attachments: HDFS-7454-001.patch, HDFS-7454-002.patch,
HDFS-7454-003.patch, HDFS-7454-004.patch
HDFS-5620 indicated a GlobalAclSet containing unique {{AclFeature}} can be
de-duplicated to save the memory in NameNode. However it was not implemented
at that time.
This Jira re-proposes same implementation, along with de-duplication of
unique {{AclEntry}} across all ACLs.
One simple usecase is:
A mapreduce user's home directory with the set of default ACLs, under which
lot of other files/directories could be created when jobs is run. Here all
the default ACLs of parent directory will be duplicated till the explicit
delete of those ACLs. With de-duplication,only one object will be in memory
for the same Entry across all ACLs of all files/directories.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7476) Consolidate ACL-related operations to a single class
Haohui Mai created HDFS-7476:
Summary: Consolidate ACL-related operations to a single class
Key: HDFS-7476
URL: https://issues.apache.org/jira/browse/HDFS-7476
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
This jira proposes to consolidate ACL-related operations scattered in
{{FSNamesystem}} and {{FSDirectory}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7476) Consolidate ACL-related operations to a single class
[
https://issues.apache.org/jira/browse/HDFS-7476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7476:
-
Attachment: HDFS-7476.000.patch
Consolidate ACL-related operations to a single class
Key: HDFS-7476
URL: https://issues.apache.org/jira/browse/HDFS-7476
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7476.000.patch
This jira proposes to consolidate ACL-related operations scattered in
{{FSNamesystem}} and {{FSDirectory}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7476) Consolidate ACL-related operations to a single class
[
https://issues.apache.org/jira/browse/HDFS-7476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7476:
-
Status: Patch Available (was: Open)
Consolidate ACL-related operations to a single class
Key: HDFS-7476
URL: https://issues.apache.org/jira/browse/HDFS-7476
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7476.000.patch
This jira proposes to consolidate ACL-related operations scattered in
{{FSNamesystem}} and {{FSDirectory}} into a single class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7337) Configurable and pluggable Erasure Codec and schema
[ https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-7337: Attachment: HDFS-7337-prototype-v3.zip Updated the prototype codes: 1. Refined the APIs, ErasureCoder handles BlockGroup, ECWorker not bothering to understand it; 2. Included unit test codes for example to see how ErasureCodec/ErasureCoder can be used. Configurable and pluggable Erasure Codec and schema --- Key: HDFS-7337 URL: https://issues.apache.org/jira/browse/HDFS-7337 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Kai Zheng Attachments: HDFS-7337-prototype-v1.patch, HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip, PluggableErasureCodec.pdf According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via pluggable approach. It allows to define and configure multiple codec schemas with different coding algorithms and parameters. The resultant codec schemas can be utilized and specified via command tool for different file folders. While design and implement such pluggable framework, it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework is useful and workable. Separate JIRA could be opened for the RS codec implementation. Note HDFS-7353 will focus on the very low level codec API and implementation to make concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level stuffs that interact with configuration, schema and etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7477) Replace ACLException with AccessControlException
Haohui Mai created HDFS-7477:
Summary: Replace ACLException with AccessControlException
Key: HDFS-7477
URL: https://issues.apache.org/jira/browse/HDFS-7477
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Currently many functions logs audit log during failures only when
{{AccessControlException}} is thrown, thus no audit logs are logged if
{{AclException}} is thrown when the ACLs deny the access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7478) Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
Li Lu created HDFS-7478: --- Summary: Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem Key: HDFS-7478 URL: https://issues.apache.org/jira/browse/HDFS-7478 Project: Hadoop HDFS Issue Type: Bug Reporter: Li Lu Assignee: Li Lu The NNConf class appears to be straightforward and is only used by FSNamesystem. We may want to move it into FSNamesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-7477) Replace ACLException with AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Lu reassigned HDFS-7477:
---
Assignee: Li Lu
Replace ACLException with AccessControlException
Key: HDFS-7477
URL: https://issues.apache.org/jira/browse/HDFS-7477
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
Currently many functions logs audit log during failures only when
{{AccessControlException}} is thrown, thus no audit logs are logged if
{{AclException}} is thrown when the ACLs deny the access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7479) Removed deprecated o.a.h.fs.permission.AccessControlException
Haohui Mai created HDFS-7479:
Summary: Removed deprecated
o.a.h.fs.permission.AccessControlException
Key: HDFS-7479
URL: https://issues.apache.org/jira/browse/HDFS-7479
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
The {{o.a.h.fs.permission.AccessControlException}} has been deprecated for last
major releases and it should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-7479) Removed deprecated o.a.h.fs.permission.AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Lu reassigned HDFS-7479:
---
Assignee: Li Lu
Removed deprecated o.a.h.fs.permission.AccessControlException
-
Key: HDFS-7479
URL: https://issues.apache.org/jira/browse/HDFS-7479
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
The {{o.a.h.fs.permission.AccessControlException}} has been deprecated for
last major releases and it should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7479) Removed deprecated o.a.h.fs.permission.AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Lu updated HDFS-7479:
Attachment: HDFS-7479-120414.patch
Hi [~wheat9], in this patch I removed the deprecated
o.a.h.fs.permission.AccessControlException, and updated
o.a.h.security.AccessControlException class according to the removal. If you
have time please feel free to take a look at it. Thanks!
Removed deprecated o.a.h.fs.permission.AccessControlException
-
Key: HDFS-7479
URL: https://issues.apache.org/jira/browse/HDFS-7479
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
Attachments: HDFS-7479-120414.patch
The {{o.a.h.fs.permission.AccessControlException}} has been deprecated for
last major releases and it should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-7473) Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error
[ https://issues.apache.org/jira/browse/HDFS-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HDFS-7473: --- Assignee: Akira AJISAKA Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error --- Key: HDFS-7473 URL: https://issues.apache.org/jira/browse/HDFS-7473 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.2 Reporter: Jason Keller Assignee: Akira AJISAKA When setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml, the error java.lang.IllegalArgumentException: Cannot set dfs.namenode.fs-limits.max-directory-items to a value less than 0 or greater than 640 is produced. However, the documentation shows that 0 is a valid setting for dfs.namenode.fs-limits.max-directory-items, turning the check off. Looking into the code in hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java shows that the culprit is Preconditions.checkArgument(maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set + DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY+ to a value less than 0 or greater than + MAX_DIR_ITEMS); This checks if maxDirItems is greater than 0. Since 0 is not greater than 0, it produces an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7473) Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error
[
https://issues.apache.org/jira/browse/HDFS-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235153#comment-14235153
]
Akira AJISAKA commented on HDFS-7473:
-
Thanks for the report, [~keller-jason].
After HDFS-6102, setting the property to 0 is invalid. We should update the
error message
{code}
Preconditions.checkArgument(
maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set
+ DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY
+ to a value less than 0 or greater than + MAX_DIR_ITEMS);
{code}
to
{code}
Preconditions.checkArgument(
maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set
+ DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY
+ to a value less than 1 or greater than + MAX_DIR_ITEMS);
{code}
We should update hdfs-default.xml also.
Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml
leads to error
---
Key: HDFS-7473
URL: https://issues.apache.org/jira/browse/HDFS-7473
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.2
Reporter: Jason Keller
When setting dfs.namenode.fs-limits.max-directory-items to 0 in
hdfs-site.xml, the error java.lang.IllegalArgumentException: Cannot set
dfs.namenode.fs-limits.max-directory-items to a value less than 0 or greater
than 640 is produced. However, the documentation shows that 0 is a
valid setting for dfs.namenode.fs-limits.max-directory-items, turning the
check off.
Looking into the code in
hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
shows that the culprit is
Preconditions.checkArgument(maxDirItems 0 maxDirItems = MAX_DIR_ITEMS,
Cannot set + DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY+ to a
value less than 0 or greater than + MAX_DIR_ITEMS);
This checks if maxDirItems is greater than 0. Since 0 is not greater than 0,
it produces an error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7478) Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated HDFS-7478: Status: Patch Available (was: Open) Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem -- Key: HDFS-7478 URL: https://issues.apache.org/jira/browse/HDFS-7478 Project: Hadoop HDFS Issue Type: Bug Reporter: Li Lu Assignee: Li Lu Attachments: HDFS-7478-120414.patch The NNConf class appears to be straightforward and is only used by FSNamesystem. We may want to move it into FSNamesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7478) Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated HDFS-7478: Attachment: HDFS-7478-120414.patch In this patch I moved o.a.h.fs.server.namenode.NNConf to FSNamesystem, then removed the original file. Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem -- Key: HDFS-7478 URL: https://issues.apache.org/jira/browse/HDFS-7478 Project: Hadoop HDFS Issue Type: Bug Reporter: Li Lu Assignee: Li Lu Attachments: HDFS-7478-120414.patch The NNConf class appears to be straightforward and is only used by FSNamesystem. We may want to move it into FSNamesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7479) Removed deprecated o.a.h.fs.permission.AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Lu updated HDFS-7479:
Status: Patch Available (was: Open)
Removed deprecated o.a.h.fs.permission.AccessControlException
-
Key: HDFS-7479
URL: https://issues.apache.org/jira/browse/HDFS-7479
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
Attachments: HDFS-7479-120414.patch
The {{o.a.h.fs.permission.AccessControlException}} has been deprecated for
last major releases and it should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7478) Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235164#comment-14235164
]
Haohui Mai commented on HDFS-7478:
--
{code}
+ private static class NNConf {
+/**
+ * Support for ACLs is controlled by a configuration flag. If the
+ * configuration flag is false, then the NameNode will reject all
+ * ACL-related operations.
+ */
+private final boolean aclsEnabled;
+
{code}
Let's just inline the configuration and function into {{FSNamesystem}} directly
instead of having a inner static class.
Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
--
Key: HDFS-7478
URL: https://issues.apache.org/jira/browse/HDFS-7478
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Li Lu
Assignee: Li Lu
Attachments: HDFS-7478-120414.patch
The NNConf class appears to be straightforward and is only used by
FSNamesystem. We may want to move it into FSNamesystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7478) Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-7478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Lu updated HDFS-7478: Attachment: HDFS-7478-120414-1.patch Thanks for the suggestion. In this version I completely eliminated the NNConf class and merged all its fields and methods to FSNamesystem. Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem -- Key: HDFS-7478 URL: https://issues.apache.org/jira/browse/HDFS-7478 Project: Hadoop HDFS Issue Type: Bug Reporter: Li Lu Assignee: Li Lu Attachments: HDFS-7478-120414-1.patch, HDFS-7478-120414.patch The NNConf class appears to be straightforward and is only used by FSNamesystem. We may want to move it into FSNamesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7473) Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error
[ https://issues.apache.org/jira/browse/HDFS-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-7473: Attachment: HDFS-7473-001.patch Attaching a patch to fix the documentation. Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error --- Key: HDFS-7473 URL: https://issues.apache.org/jira/browse/HDFS-7473 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.2 Reporter: Jason Keller Assignee: Akira AJISAKA Attachments: HDFS-7473-001.patch When setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml, the error java.lang.IllegalArgumentException: Cannot set dfs.namenode.fs-limits.max-directory-items to a value less than 0 or greater than 640 is produced. However, the documentation shows that 0 is a valid setting for dfs.namenode.fs-limits.max-directory-items, turning the check off. Looking into the code in hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java shows that the culprit is Preconditions.checkArgument(maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set + DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY+ to a value less than 0 or greater than + MAX_DIR_ITEMS); This checks if maxDirItems is greater than 0. Since 0 is not greater than 0, it produces an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7473) Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error
[ https://issues.apache.org/jira/browse/HDFS-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-7473: Labels: newbie (was: ) Target Version/s: 2.7.0 Affects Version/s: 2.4.0 Status: Patch Available (was: Open) Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error --- Key: HDFS-7473 URL: https://issues.apache.org/jira/browse/HDFS-7473 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.2, 2.4.0 Reporter: Jason Keller Assignee: Akira AJISAKA Labels: newbie Attachments: HDFS-7473-001.patch When setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml, the error java.lang.IllegalArgumentException: Cannot set dfs.namenode.fs-limits.max-directory-items to a value less than 0 or greater than 640 is produced. However, the documentation shows that 0 is a valid setting for dfs.namenode.fs-limits.max-directory-items, turning the check off. Looking into the code in hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java shows that the culprit is Preconditions.checkArgument(maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set + DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY+ to a value less than 0 or greater than + MAX_DIR_ITEMS); This checks if maxDirItems is greater than 0. Since 0 is not greater than 0, it produces an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7473) Document setting dfs.namenode.fs-limits.max-directory-items to 0 is invalid
[ https://issues.apache.org/jira/browse/HDFS-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-7473: Component/s: documentation Summary: Document setting dfs.namenode.fs-limits.max-directory-items to 0 is invalid (was: Setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml leads to error) Document setting dfs.namenode.fs-limits.max-directory-items to 0 is invalid --- Key: HDFS-7473 URL: https://issues.apache.org/jira/browse/HDFS-7473 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.2 Reporter: Jason Keller Assignee: Akira AJISAKA Labels: newbie Attachments: HDFS-7473-001.patch When setting dfs.namenode.fs-limits.max-directory-items to 0 in hdfs-site.xml, the error java.lang.IllegalArgumentException: Cannot set dfs.namenode.fs-limits.max-directory-items to a value less than 0 or greater than 640 is produced. However, the documentation shows that 0 is a valid setting for dfs.namenode.fs-limits.max-directory-items, turning the check off. Looking into the code in hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java shows that the culprit is Preconditions.checkArgument(maxDirItems 0 maxDirItems = MAX_DIR_ITEMS, Cannot set + DFSConfigKeys.DFS_NAMENODE_MAX_DIRECTORY_ITEMS_KEY+ to a value less than 0 or greater than + MAX_DIR_ITEMS); This checks if maxDirItems is greater than 0. Since 0 is not greater than 0, it produces an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7463) Simplify FSNamesystem#getBlockLocationsUpdateTimes
[
https://issues.apache.org/jira/browse/HDFS-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235195#comment-14235195
]
Hadoop QA commented on HDFS-7463:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685182/HDFS-7463.002.patch
against trunk revision 7896815.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.TestEncryptionZones
org.apache.hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
org.apache.hadoop.hdfs.TestGetBlocks
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8914//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8914//console
This message is automatically generated.
Simplify FSNamesystem#getBlockLocationsUpdateTimes
--
Key: HDFS-7463
URL: https://issues.apache.org/jira/browse/HDFS-7463
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7463.000.patch, HDFS-7463.001.patch,
HDFS-7463.002.patch, HDFS-7463.003.patch
Currently {{FSNamesystem#getBlockLocationsUpdateTimes}} holds the read lock
to access the blocks. It releases the read lock and then acquires the write
lock when it needs to update the access time of the {{INode}}.
This jira proposes to move the responsibility of the latter steps to the
caller to simplify the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7474) Avoid resolve path in FSPermissionChecker
[
https://issues.apache.org/jira/browse/HDFS-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235197#comment-14235197
]
Hadoop QA commented on HDFS-7474:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685198/HDFS-7474.000.patch
against trunk revision 7896815.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8918//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8918//console
This message is automatically generated.
Avoid resolve path in FSPermissionChecker
-
Key: HDFS-7474
URL: https://issues.apache.org/jira/browse/HDFS-7474
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7474.000.patch
Currently for an typical namenode operation a path can be resolved multiple
times. Usually the FSPermissionChecker first takes the path string as input
and resolve the path before the real checking. After permission checking the
path is then resolved again (and again) for real operations in FSNamesystem
and FSDirectory. Ideally we should resolve the path only once, and use
INodesInPath internally afterwards. As a first step, we can start from
avoiding path resolving in FSPermissionChecker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7472) Fix typo in message of ReplicaNotFoundException
[ https://issues.apache.org/jira/browse/HDFS-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-7472: Target Version/s: 2.7.0 Labels: newbie (was: ) Fix typo in message of ReplicaNotFoundException --- Key: HDFS-7472 URL: https://issues.apache.org/jira/browse/HDFS-7472 Project: Hadoop HDFS Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Labels: newbie Attachments: HDFS-7472.1.patch unexpeted - unexpected -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7479) Removed deprecated o.a.h.fs.permission.AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-7479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235217#comment-14235217
]
Hadoop QA commented on HDFS-7479:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685254/HDFS-7479-120414.patch
against trunk revision 0653918.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8922//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8922//console
This message is automatically generated.
Removed deprecated o.a.h.fs.permission.AccessControlException
-
Key: HDFS-7479
URL: https://issues.apache.org/jira/browse/HDFS-7479
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Assignee: Li Lu
Attachments: HDFS-7479-120414.patch
The {{o.a.h.fs.permission.AccessControlException}} has been deprecated for
last major releases and it should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7475) Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
[
https://issues.apache.org/jira/browse/HDFS-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235222#comment-14235222
]
Hadoop QA commented on HDFS-7475:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685197/HDFS-7475.0.patch
against trunk revision 7896815.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8915//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8915//console
This message is automatically generated.
Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
--
Key: HDFS-7475
URL: https://issues.apache.org/jira/browse/HDFS-7475
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7475.0.patch
TestLazyPersistFiles#testLazyPersistBlocksAreSaved verifies that the memory
block is saved in lazy persist directory after some sleep with the following
code.
{code}
// Sleep for a short time to allow the lazy writer thread to do its job
Thread.sleep(6 * LAZY_WRITER_INTERVAL_SEC * 1000);
{code}
However, the sleep only guarantees an async lazy persist task is scheduled.
The task running on a per volume thread pool thread can take longer to finish
persisting. It will be more deterministic to wait the pending lazy persist
task count reduce to 0 before verifying the block file exist in lazy persist
dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7469) Inline permission checking done by FSPermissionChecker#checkSubAccess
[
https://issues.apache.org/jira/browse/HDFS-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235220#comment-14235220
]
Hadoop QA commented on HDFS-7469:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685210/HDFS-7469.003.patch
against trunk revision 7896815.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSPermission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8916//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8916//console
This message is automatically generated.
Inline permission checking done by FSPermissionChecker#checkSubAccess
-
Key: HDFS-7469
URL: https://issues.apache.org/jira/browse/HDFS-7469
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7469.000.patch, HDFS-7469.001.patch,
HDFS-7469.002.patch, HDFS-7469.003.patch
{{FSPermissionChecker#checkSubAccess}} is used by {{getContentSummary()}} and
{{computeSnapshotDiff()}} to validate the permission of the entire subtree
through BFS.
As these two calls traverse subtrees to compute the result, the permission
checking should be done during the same traversal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7474) Avoid resolve path in FSPermissionChecker
[
https://issues.apache.org/jira/browse/HDFS-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235225#comment-14235225
]
Hadoop QA commented on HDFS-7474:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12685198/HDFS-7474.000.patch
against trunk revision 0653918.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8927//console
This message is automatically generated.
Avoid resolve path in FSPermissionChecker
-
Key: HDFS-7474
URL: https://issues.apache.org/jira/browse/HDFS-7474
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7474.000.patch
Currently for an typical namenode operation a path can be resolved multiple
times. Usually the FSPermissionChecker first takes the path string as input
and resolve the path before the real checking. After permission checking the
path is then resolved again (and again) for real operations in FSNamesystem
and FSDirectory. Ideally we should resolve the path only once, and use
INodesInPath internally afterwards. As a first step, we can start from
avoiding path resolving in FSPermissionChecker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
