[jira] [Commented] (HDFS-1321) If service port and main port are the same, there is no clear log message explaining the issue.
[ https://issues.apache.org/jira/browse/HDFS-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055066#comment-13055066 ] Hudson commented on HDFS-1321: -- Integrated in Hadoop-Hdfs-trunk #707 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/707/]) If service port and main port are the same, there is no clear log message explaining the issue. --- Key: HDFS-1321 URL: https://issues.apache.org/jira/browse/HDFS-1321 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: gary murry Assignee: Jim Plush Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1321-take2.txt, HDFS-1321-take3.txt, HDFS-1321-take4.txt, HDFS-1321.patch With the introduction of a service port to the namenode, there is now a chance for user error to set the two port equal. This will cause the namenode to fail to start up. It would be nice if there was a log message explaining the port clash. Or just treat things as if the service port was not specified. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1381) HDFS javadocs hard-code references to dfs.namenode.name.dir and dfs.datanode.data.dir parameters
[ https://issues.apache.org/jira/browse/HDFS-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055067#comment-13055067 ] Hudson commented on HDFS-1381: -- Integrated in Hadoop-Hdfs-trunk #707 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/707/]) HDFS-1381. HDFS javadocs hard-code references to dfs.namenode.name.dir and dfs.datanode.data.dir parameters (Jim Plush via atm) atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1139715 Files : * /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSStorageStateRecovery.java * /hadoop/common/trunk/hdfs/CHANGES.txt * /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/UpgradeUtilities.java HDFS javadocs hard-code references to dfs.namenode.name.dir and dfs.datanode.data.dir parameters Key: HDFS-1381 URL: https://issues.apache.org/jira/browse/HDFS-1381 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.20.1 Reporter: Jakob Homan Assignee: Jim Plush Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1381-take1.txt, HDFS-1381-take2.txt The javadoc for MiniDFSCluster makes repeated references to setting dfs.name.dir and dfs.data.dir. These should be replaced with references to DFSConfigKeys' DFS_NAMENODE_NAME_DIR_KEY and DFS_DATANODE_DATA_DIR_KEY, respectively. The old values are deprecated in DFSConfigKeys, but we should switch to the new values where ever we can. Also, a quick search the code shows that TestDFSStorageStateRecovery.java and UpgradeUtilities.java should be updated as well. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1723) quota errors messages should use the same scale
[ https://issues.apache.org/jira/browse/HDFS-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055129#comment-13055129 ] Jim Plush commented on HDFS-1723: - good catch, I'll update the patch and re-submit quota errors messages should use the same scale --- Key: HDFS-1723 URL: https://issues.apache.org/jira/browse/HDFS-1723 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Allen Wittenauer Assignee: Jim Plush Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1723-take1.txt, HDFS-1723-take2.txt A typical error message looks like this: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /dir is exceeded: quota=3298534883328 diskspace consumed=5246.0g Since the two values are in difference scales and one is replicated vs. not replicated (I think), this isn't very easy for the user to understand. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-929) DFSClient#getBlockSize is unused
[ https://issues.apache.org/jira/browse/HDFS-929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055130#comment-13055130 ] Jim Plush commented on HDFS-929: apologies, I read your comment Maybe you can mark it as deprecated? as you wanted it marked deprecated. DFSClient#getBlockSize is unused Key: HDFS-929 URL: https://issues.apache.org/jira/browse/HDFS-929 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Eli Collins Assignee: Jim Plush Priority: Minor Fix For: 0.23.0 Attachments: HDFS-929-take1.txt DFSClient#getBlockSize is unused. Since it's a public class internal to HDFS we just remove it? If not then we should add a unit test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1723) quota errors messages should use the same scale
[ https://issues.apache.org/jira/browse/HDFS-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Plush updated HDFS-1723: Attachment: HDFS-1723-take3.txt refactoring based on Aaron's comments regarding removing the NSQuotaExceededException from the patch as it's not required for the fix. quota errors messages should use the same scale --- Key: HDFS-1723 URL: https://issues.apache.org/jira/browse/HDFS-1723 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Allen Wittenauer Assignee: Jim Plush Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1723-take1.txt, HDFS-1723-take2.txt, HDFS-1723-take3.txt A typical error message looks like this: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /dir is exceeded: quota=3298534883328 diskspace consumed=5246.0g Since the two values are in difference scales and one is replicated vs. not replicated (I think), this isn't very easy for the user to understand. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[
https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055154#comment-13055154
]
Jim Plush commented on HDFS-979:
There seem to be several other places where a generic message like this is
displayed...
➺ Ack --java All specified directories are not accessible
hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
134: All specified directories are not accessible or do not exist.);
hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
188: All specified directories are not accessible or do not exist.);
hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
201: All specified directories are not accessible or do not exist.);
FSImage should specify which dirs are missing when refusing to come up
--
Key: HDFS-979
URL: https://issues.apache.org/jira/browse/HDFS-979
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Priority: Minor
When {{FSImage}} can't come up as either it has no data or edit dirs, it
tells me this
{code}
java.io.IOException: All specified directories are not accessible or do not
exist.
{code}
What it doesn't do is say which of the two attributes are missing. This would
be beneficial to anyone trying to track down the problem. Also, I don't think
the message is correct. It's bailing out because dataDirs.size() == 0 ||
editsDirs.size() == 0 , because a list is empty -not because the dirs aren't
there, as there hasn't been any validation yet.
More useful would be
# Explicit mention of which attributes are null
# Declare that this is because they are not in the config
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1723) quota errors messages should use the same scale
[ https://issues.apache.org/jira/browse/HDFS-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055157#comment-13055157 ] Hadoop QA commented on HDFS-1723: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483868/HDFS-1723-take3.txt against trunk revision 1139715. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/842//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/842//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/842//console This message is automatically generated. quota errors messages should use the same scale --- Key: HDFS-1723 URL: https://issues.apache.org/jira/browse/HDFS-1723 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Allen Wittenauer Assignee: Jim Plush Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1723-take1.txt, HDFS-1723-take2.txt, HDFS-1723-take3.txt A typical error message looks like this: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /dir is exceeded: quota=3298534883328 diskspace consumed=5246.0g Since the two values are in difference scales and one is replicated vs. not replicated (I think), this isn't very easy for the user to understand. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2107) Move block management code to a package
[
https://issues.apache.org/jira/browse/HDFS-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2107:
-
Attachment: h2107_20110626.patch
All the findbugs and javadoc warnings are existing problems.
h2107_20110626.patch: fixed the warnings.
Move block management code to a package
---
Key: HDFS-2107
URL: https://issues.apache.org/jira/browse/HDFS-2107
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Affects Versions: 0.23.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h2107_20110624.patch, h2107_20110626.patch, svn_mv.sh
In namenode, everything (except metrics) is in
{{org.apache.hadoop.hdfs.server.namenode}}. Let's create a package for block
management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2107) Move block management code to a package
[
https://issues.apache.org/jira/browse/HDFS-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055220#comment-13055220
]
Hadoop QA commented on HDFS-2107:
-
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483875/h2107_20110626.patch
against trunk revision 1139715.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 86 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/843//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/843//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/843//console
This message is automatically generated.
Move block management code to a package
---
Key: HDFS-2107
URL: https://issues.apache.org/jira/browse/HDFS-2107
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Affects Versions: 0.23.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h2107_20110624.patch, h2107_20110626.patch, svn_mv.sh
In namenode, everything (except metrics) is in
{{org.apache.hadoop.hdfs.server.namenode}}. Let's create a package for block
management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[
https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Plush updated HDFS-979:
---
Attachment: HDFS-979-take1.txt
Adding a verbose (the more verbose the more helpful?) Exception message should
either the dataDirs or the editDirs show a size of 0 in the
recoverTransitionRead method. It will also loop through the directories and if
they are a file scheme, check to make sure those directories actually do exist
on disk.
For example if the Edit dirs was showing an empty list and the directory did
not exist locally the message would be:
All specified directories are not accessible or do not exist. Specifically:
FSImage reports 0 NameNode edit dirs. The config value of
'dfs.namenode.edits.dir' is: file:///opt/hadoop-doesnotexist/dfs/name.
Directory: /opt/hadoop-doesnotexist/dfs/name DOES NOT EXIST.
FSImage should specify which dirs are missing when refusing to come up
--
Key: HDFS-979
URL: https://issues.apache.org/jira/browse/HDFS-979
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Priority: Minor
Attachments: HDFS-979-take1.txt
When {{FSImage}} can't come up as either it has no data or edit dirs, it
tells me this
{code}
java.io.IOException: All specified directories are not accessible or do not
exist.
{code}
What it doesn't do is say which of the two attributes are missing. This would
be beneficial to anyone trying to track down the problem. Also, I don't think
the message is correct. It's bailing out because dataDirs.size() == 0 ||
editsDirs.size() == 0 , because a list is empty -not because the dirs aren't
there, as there hasn't been any validation yet.
More useful would be
# Explicit mention of which attributes are null
# Declare that this is because they are not in the config
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[
https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Plush updated HDFS-979:
---
Fix Version/s: 0.23.0
Assignee: Jim Plush
Release Note: Added more verbose error logging in FSImage should there be
an issue getting the size of the NameNode data or edits directories.
Status: Patch Available (was: Open)
FSImage should specify which dirs are missing when refusing to come up
--
Key: HDFS-979
URL: https://issues.apache.org/jira/browse/HDFS-979
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Jim Plush
Priority: Minor
Fix For: 0.23.0
Attachments: HDFS-979-take1.txt
When {{FSImage}} can't come up as either it has no data or edit dirs, it
tells me this
{code}
java.io.IOException: All specified directories are not accessible or do not
exist.
{code}
What it doesn't do is say which of the two attributes are missing. This would
be beneficial to anyone trying to track down the problem. Also, I don't think
the message is correct. It's bailing out because dataDirs.size() == 0 ||
editsDirs.size() == 0 , because a list is empty -not because the dirs aren't
there, as there hasn't been any validation yet.
More useful would be
# Explicit mention of which attributes are null
# Declare that this is because they are not in the config
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[
https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055240#comment-13055240
]
Hadoop QA commented on HDFS-979:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483878/HDFS-979-take1.txt
against trunk revision 1139715.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/844//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/844//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/844//console
This message is automatically generated.
FSImage should specify which dirs are missing when refusing to come up
--
Key: HDFS-979
URL: https://issues.apache.org/jira/browse/HDFS-979
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Jim Plush
Priority: Minor
Fix For: 0.23.0
Attachments: HDFS-979-take1.txt
When {{FSImage}} can't come up as either it has no data or edit dirs, it
tells me this
{code}
java.io.IOException: All specified directories are not accessible or do not
exist.
{code}
What it doesn't do is say which of the two attributes are missing. This would
be beneficial to anyone trying to track down the problem. Also, I don't think
the message is correct. It's bailing out because dataDirs.size() == 0 ||
editsDirs.size() == 0 , because a list is empty -not because the dirs aren't
there, as there hasn't been any validation yet.
More useful would be
# Explicit mention of which attributes are null
# Declare that this is because they are not in the config
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[
https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Plush updated HDFS-979:
---
Attachment: HDFS-979-take2.txt
Updated to use a StringBuilder in the loop after complaints from Findbugs
report.
FSImage should specify which dirs are missing when refusing to come up
--
Key: HDFS-979
URL: https://issues.apache.org/jira/browse/HDFS-979
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Jim Plush
Priority: Minor
Fix For: 0.23.0
Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt
When {{FSImage}} can't come up as either it has no data or edit dirs, it
tells me this
{code}
java.io.IOException: All specified directories are not accessible or do not
exist.
{code}
What it doesn't do is say which of the two attributes are missing. This would
be beneficial to anyone trying to track down the problem. Also, I don't think
the message is correct. It's bailing out because dataDirs.size() == 0 ||
editsDirs.size() == 0 , because a list is empty -not because the dirs aren't
there, as there hasn't been any validation yet.
More useful would be
# Explicit mention of which attributes are null
# Declare that this is because they are not in the config
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-979) FSImage should specify which dirs are missing when refusing to come up
[
https://issues.apache.org/jira/browse/HDFS-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055276#comment-13055276
]
Hadoop QA commented on HDFS-979:
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12483882/HDFS-979-take2.txt
against trunk revision 1139715.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/845//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/845//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/845//console
This message is automatically generated.
FSImage should specify which dirs are missing when refusing to come up
--
Key: HDFS-979
URL: https://issues.apache.org/jira/browse/HDFS-979
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Jim Plush
Priority: Minor
Fix For: 0.23.0
Attachments: HDFS-979-take1.txt, HDFS-979-take2.txt
When {{FSImage}} can't come up as either it has no data or edit dirs, it
tells me this
{code}
java.io.IOException: All specified directories are not accessible or do not
exist.
{code}
What it doesn't do is say which of the two attributes are missing. This would
be beneficial to anyone trying to track down the problem. Also, I don't think
the message is correct. It's bailing out because dataDirs.size() == 0 ||
editsDirs.size() == 0 , because a list is empty -not because the dirs aren't
there, as there hasn't been any validation yet.
More useful would be
# Explicit mention of which attributes are null
# Declare that this is because they are not in the config
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2109) Store uMask as member variable to DFSClient.Conf
Store uMask as member variable to DFSClient.Conf Key: HDFS-2109 URL: https://issues.apache.org/jira/browse/HDFS-2109 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.23.0 As a part of removing reference to conf in DFSClient, I am proposing replacing FsPermission.getUMask(conf) everywhere in DFSClient class with dfsClientConf.uMask. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2109) Store uMask as member variable to DFSClient.Conf
[ https://issues.apache.org/jira/browse/HDFS-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Mundlapudi updated HDFS-2109: - Description: As a part of removing reference to conf in DFSClient, I am proposing replacing FsPermission.getUMask(conf) everywhere in DFSClient class with dfsClientConf.uMask by storing uMask as a member variable to DFSClient.Conf. was: As a part of removing reference to conf in DFSClient, I am proposing replacing FsPermission.getUMask(conf) everywhere in DFSClient class with dfsClientConf.uMask. Store uMask as member variable to DFSClient.Conf Key: HDFS-2109 URL: https://issues.apache.org/jira/browse/HDFS-2109 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.23.0 As a part of removing reference to conf in DFSClient, I am proposing replacing FsPermission.getUMask(conf) everywhere in DFSClient class with dfsClientConf.uMask by storing uMask as a member variable to DFSClient.Conf. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2109) Store uMask as member variable to DFSClient.Conf
[ https://issues.apache.org/jira/browse/HDFS-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Mundlapudi updated HDFS-2109: - Attachment: HDFS-2109-1.patch Attaching the patch. Store uMask as member variable to DFSClient.Conf Key: HDFS-2109 URL: https://issues.apache.org/jira/browse/HDFS-2109 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.23.0 Attachments: HDFS-2109-1.patch As a part of removing reference to conf in DFSClient, I am proposing replacing FsPermission.getUMask(conf) everywhere in DFSClient class with dfsClientConf.uMask by storing uMask as a member variable to DFSClient.Conf. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1723) quota errors messages should use the same scale
[ https://issues.apache.org/jira/browse/HDFS-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055292#comment-13055292 ] Aaron T. Myers commented on HDFS-1723: -- Thanks for addressing my comments, Jim. I should have noticed this earlier, but I have one final comment and then I'll give it a +1 / commit it: Earlier in the test we're explicitly setting the quota value to 1k, so we might as well explicitly test that the output is 1.0k, rather than have a regex which will match it. We wouldn't want this test to pass if it output, for example, quota=9.5k quota errors messages should use the same scale --- Key: HDFS-1723 URL: https://issues.apache.org/jira/browse/HDFS-1723 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Allen Wittenauer Assignee: Jim Plush Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1723-take1.txt, HDFS-1723-take2.txt, HDFS-1723-take3.txt A typical error message looks like this: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /dir is exceeded: quota=3298534883328 diskspace consumed=5246.0g Since the two values are in difference scales and one is replicated vs. not replicated (I think), this isn't very easy for the user to understand. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1026) Quota checks fail for small files and quotas
[
https://issues.apache.org/jira/browse/HDFS-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055293#comment-13055293
]
Eli Collins commented on HDFS-1026:
---
Yes, add a comment in FSDirectory#addBlock and in the code that generates a
quota violation exception LOG.warn that the quota check failed because the
quota is smaller than block size * # replicas.
Quota checks fail for small files and quotas
Key: HDFS-1026
URL: https://issues.apache.org/jira/browse/HDFS-1026
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation, name-node
Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
Reporter: Eli Collins
Priority: Blocker
If a directory has a quota less than blockSize * numReplicas then you can't
add a file to it, even if the file size is less than the quota. This is
because FSDirectory#addBlock updates the count assuming at least one block is
written in full. We don't know how much of the block will be written when
addBlock is called and supporting such small quotas is not important so
perhaps we should document this and log an error message instead of making
small (blockSize * numReplicas) quotas work.
{code}
// check quota limits and updated space consumed
updateCount(inodes, inodes.length-1, 0,
fileINode.getPreferredBlockSize()*fileINode.getReplication(), true);
{code}
You can reproduce with the following commands:
{code}
$ dd if=/dev/zero of=temp bs=1000 count=64
$ hadoop fs -mkdir /user/eli/dir
$ hdfs dfsadmin -setSpaceQuota 191M /user/eli/dir
$ hadoop fs -put temp /user/eli/dir # Causes DSQuotaExceededException
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1723) quota errors messages should use the same scale
[ https://issues.apache.org/jira/browse/HDFS-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055323#comment-13055323 ] Hadoop QA commented on HDFS-1723: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483898/HDFS-1723-take4.txt against trunk revision 1139715. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/846//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/846//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/846//console This message is automatically generated. quota errors messages should use the same scale --- Key: HDFS-1723 URL: https://issues.apache.org/jira/browse/HDFS-1723 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.0 Reporter: Allen Wittenauer Assignee: Jim Plush Priority: Minor Labels: newbie Fix For: 0.23.0 Attachments: HDFS-1723-take1.txt, HDFS-1723-take2.txt, HDFS-1723-take3.txt, HDFS-1723-take4.txt A typical error message looks like this: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /dir is exceeded: quota=3298534883328 diskspace consumed=5246.0g Since the two values are in difference scales and one is replicated vs. not replicated (I think), this isn't very easy for the user to understand. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
