[
https://issues.apache.org/jira/browse/HDFS-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384723#comment-15384723
]
Chen Liang commented on HDFS-10613:
-----------------------------------
Hi Xiaohe,
Some initial tests show that this is related to a couple factors. For example,
if I have replication factor 3 and block size 64MB, then when writing a file of
10MB, the system will try to creates 3 blocks, with total size being 64 * 3 =
192 MB. So your quota is less then 192 MB, this exception will occur, despite
my file is only 10 MB.
But this being said, 384 MB consumption for 14 MB file still seems weird. Could
you please list you settings here? e.g. replication factor, block size, etc.
Also, I believe open question would be, since currently the numbers are
confusing, would it be better to include more information in the exception
messages.
> Wrong real size when a DSQuotaExceededException occur
> -----------------------------------------------------
>
> Key: HDFS-10613
> URL: https://issues.apache.org/jira/browse/HDFS-10613
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.7.2
> Environment: Linux x86_64
> Reporter: Xiaohe Lan
> Assignee: Chen Liang
> Priority: Minor
>
> When put a file from local larger than the quota of a HDFS directory, there
> will be a DSQuotaExceededException, the diskspace consumed in the error
> message seems to be unreasonable.
> Why the diskspace consumed is 384M while test.zip is a 14M file ?
> {code}
> bash-4.1$ ls -lh test.zip
> -rw-r--r-- 1 xilan dba 14M Jul 12 00:54 test.zip
> bash-4.1$ hdfs dfs -mkdir /user/foobar
> bash-4.1$ hdfs dfsadmin -setSpaceQuota 10m /user/foobar
> bash-4.1$ hdfs dfs -put test.zip /user/foobar/
> 16/07/12 00:57:11 WARN hdfs.DFSClient: DataStreamer Exception
> org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota
> of /user/foobar is exceeded: quota = 10485760 B = 10 MB but diskspace
> consumed = 402653184 B = 384 MB
> at
> org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211)
> at
> org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:874)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:707)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:666)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:491)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3571)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.storeAllocatedBlock(FSNamesystem.java:3157)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3038)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1462)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1255)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
