[jira] [Resolved] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-3201. -- Resolution: Unresolved Fix Version/s: Shared journals (HDFS-3092) Hadoop Flags: Reviewed +1 patch looks good. I have committed this. > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: Shared journals (HDFS-3092) > > Attachments: HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch, > HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3302) Review and improve HDFS trash documentation
Review and improve HDFS trash documentation --- Key: HDFS-3302 URL: https://issues.apache.org/jira/browse/HDFS-3302 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Improve Trash documentation for users. (0.23 published release docs are missing original HDFS docs btw...) A set of FAQ-like questions can be found on HDFS-2740 I'll update the ticket shortly with the areas to cover in the docs, as enabling trash by default (HDFS-2740) would be considered as a wide behavior change per its follow ups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3301) Add public waitOnSafeMode API with HdfsUtils
[
https://issues.apache.org/jira/browse/HDFS-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-3301:
--
Summary: Add public waitOnSafeMode API with HdfsUtils (was: Add public
waitOnSafeMode API as HdfsUtils )
> Add public waitOnSafeMode API with HdfsUtils
> -
>
> Key: HDFS-3301
> URL: https://issues.apache.org/jira/browse/HDFS-3301
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> Add public waitOnSafeMode API as HdfsUtils.
> I have seen this util api in Hbase and using FSCOnstants in it. Currently
> that is deprecated and moved the SafeModeActions to HdfsConstants and also
> marked as private audience. So, it wil help adding such api in HdfsUtils
> itself to avoid the need of accessing HdfsConstants.
> from Hbase FSUtils class.
> {code}
> /**
>* If DFS, check safe mode and if so, wait until we clear it.
>* @param conf configuration
>* @param wait Sleep between retries
>* @throws IOException e
>*/
> public static void waitOnSafeMode(final Configuration conf,
> final long wait)
> throws IOException {
> FileSystem fs = FileSystem.get(conf);
> if (!(fs instanceof DistributedFileSystem)) return;
> DistributedFileSystem dfs = (DistributedFileSystem)fs;
> // Make sure dfs is not in safe mode
> while
> (dfs.setSafeMode(org.apache.hadoop.hdfs.protocol.FSConstants.SafeModeAction.SAFEMODE_GET))
> {
> LOG.info("Waiting for dfs to exit safe mode...");
> try {
> Thread.sleep(wait);
> } catch (InterruptedException e) {
> //continue
> }
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3301) Add public waitOnSafeMode API as HdfsUtils
Add public waitOnSafeMode API as HdfsUtils
---
Key: HDFS-3301
URL: https://issues.apache.org/jira/browse/HDFS-3301
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs client
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Add public waitOnSafeMode API as HdfsUtils.
I have seen this util api in Hbase and using FSCOnstants in it. Currently that
is deprecated and moved the SafeModeActions to HdfsConstants and also marked as
private audience. So, it wil help adding such api in HdfsUtils itself to avoid
the need of accessing HdfsConstants.
from Hbase FSUtils class.
{code}
/**
* If DFS, check safe mode and if so, wait until we clear it.
* @param conf configuration
* @param wait Sleep between retries
* @throws IOException e
*/
public static void waitOnSafeMode(final Configuration conf,
final long wait)
throws IOException {
FileSystem fs = FileSystem.get(conf);
if (!(fs instanceof DistributedFileSystem)) return;
DistributedFileSystem dfs = (DistributedFileSystem)fs;
// Make sure dfs is not in safe mode
while
(dfs.setSafeMode(org.apache.hadoop.hdfs.protocol.FSConstants.SafeModeAction.SAFEMODE_GET))
{
LOG.info("Waiting for dfs to exit safe mode...");
try {
Thread.sleep(wait);
} catch (InterruptedException e) {
//continue
}
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257204#comment-13257204 ] Hudson commented on HDFS-3282: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2120 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2120/]) iHDFS-3282. Expose getFileLength API. Contributed by Uma Maheswara Rao G. (Revision 1327790) HDFS-3282. Expose getFileLength API. Contributed by Uma Maheswara Rao G. (Revision 1327788) Result = ABORTED umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327790 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsDataInputStream.java umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327788 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0 > > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3169) TestFsck should test multiple -move operations in a row
[ https://issues.apache.org/jira/browse/HDFS-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257188#comment-13257188 ] Hudson commented on HDFS-3169: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2119 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2119/]) HDFS-3169. TestFsck should test multiple -move operations in a row. Contributed by Colin Patrick McCabe (Revision 1327776) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327776 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > TestFsck should test multiple -move operations in a row > --- > > Key: HDFS-3169 > URL: https://issues.apache.org/jira/browse/HDFS-3169 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3169.001.patch > > > TestFsck should test multiple -move operations in a row. Overall, it would > be nice to have more coverage on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257185#comment-13257185 ] Uma Maheswara Rao G commented on HDFS-3282: --- I just committed this to trunk! > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0 > > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-3282: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0 > > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257184#comment-13257184 ] Uma Maheswara Rao G commented on HDFS-3282: --- Thanks a lot Eli and Nicholas for taking a look on Javac warnings. > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257178#comment-13257178 ] Hudson commented on HDFS-3282: -- Integrated in Hadoop-Common-trunk-Commit #2103 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2103/]) iHDFS-3282. Expose getFileLength API. Contributed by Uma Maheswara Rao G. (Revision 1327790) HDFS-3282. Expose getFileLength API. Contributed by Uma Maheswara Rao G. (Revision 1327788) Result = SUCCESS umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327790 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsDataInputStream.java umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327788 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257176#comment-13257176 ] Hudson commented on HDFS-3282: -- Integrated in Hadoop-Hdfs-trunk-Commit #2176 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2176/]) HDFS-3282. Expose getFileLength API. Contributed by Uma Maheswara Rao G. (Revision 1327788) Result = FAILURE umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327788 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3275) Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
[ https://issues.apache.org/jira/browse/HDFS-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257171#comment-13257171 ] amith commented on HDFS-3275: - Thanks for your comments uma Surely I will work on it and provide a patch > Format command overwrites contents of non-empty shared edits dir if name dirs > are empty without any prompting > - > > Key: HDFS-3275 > URL: https://issues.apache.org/jira/browse/HDFS-3275 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Vinithra Varadharajan >Assignee: amith > Attachments: HDFS-3275.patch > > > To reproduce: > # Configure a NameNode with namedirs and a shared edits dir, all of which are > empty. > # Run hdfs namenode -format. Namedirs and shared edits dir gets populated. > # Delete the contents of the namedirs. Leave the shared edits dir as is. > Check the timestamps of the shared edits dir contents. > # Run format again. The namedirs as well as the shared edits dir get > formatted. The shared edits dir's contents have been replaced without any > prompting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) Miscellaneous xml cleanups for OEV
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257170#comment-13257170 ] Hudson commented on HDFS-3206: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2118 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2118/]) HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Miscellaneous xml cleanups for OEV > -- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3201: - Attachment: HDFS-3201.HDFS-3092.patch The new patch addressed Nicholas' comments. Thanks! > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch, > HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3169) TestFsck should test multiple -move operations in a row
[ https://issues.apache.org/jira/browse/HDFS-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3169: -- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: (was: 2.0.0) Status: Resolved (was: Patch Available) I've committed this and merged to branch-2. Thanks Colin. > TestFsck should test multiple -move operations in a row > --- > > Key: HDFS-3169 > URL: https://issues.apache.org/jira/browse/HDFS-3169 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3169.001.patch > > > TestFsck should test multiple -move operations in a row. Overall, it would > be nice to have more coverage on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3169) TestFsck should test multiple -move operations in a row
[ https://issues.apache.org/jira/browse/HDFS-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257158#comment-13257158 ] Hudson commented on HDFS-3169: -- Integrated in Hadoop-Common-trunk-Commit #2102 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2102/]) HDFS-3169. TestFsck should test multiple -move operations in a row. Contributed by Colin Patrick McCabe (Revision 1327776) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327776 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > TestFsck should test multiple -move operations in a row > --- > > Key: HDFS-3169 > URL: https://issues.apache.org/jira/browse/HDFS-3169 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3169.001.patch > > > TestFsck should test multiple -move operations in a row. Overall, it would > be nice to have more coverage on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3169) TestFsck should test multiple -move operations in a row
[ https://issues.apache.org/jira/browse/HDFS-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257157#comment-13257157 ] Hudson commented on HDFS-3169: -- Integrated in Hadoop-Hdfs-trunk-Commit #2175 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2175/]) HDFS-3169. TestFsck should test multiple -move operations in a row. Contributed by Colin Patrick McCabe (Revision 1327776) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327776 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java > TestFsck should test multiple -move operations in a row > --- > > Key: HDFS-3169 > URL: https://issues.apache.org/jira/browse/HDFS-3169 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3169.001.patch > > > TestFsck should test multiple -move operations in a row. Overall, it would > be nice to have more coverage on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) Miscellaneous xml cleanups for OEV
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257153#comment-13257153 ] Hudson commented on HDFS-3206: -- Integrated in Hadoop-Common-trunk-Commit #2101 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2101/]) HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Miscellaneous xml cleanups for OEV > -- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) Miscellaneous xml cleanups for OEV
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257152#comment-13257152 ] Hudson commented on HDFS-3206: -- Integrated in Hadoop-Hdfs-trunk-Commit #2174 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2174/]) HDFS-3206. Miscellaneous xml cleanups for OEV. Contributed by Colin Patrick McCabe (Revision 1327768) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327768 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Miscellaneous xml cleanups for OEV > -- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) Miscellaneous xml cleanups for OEV
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257149#comment-13257149 ] Eli Collins commented on HDFS-3206: --- I've committed this and merged to branch-2. Thanks Colin. > Miscellaneous xml cleanups for OEV > -- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3169) TestFsck should test multiple -move operations in a row
[ https://issues.apache.org/jira/browse/HDFS-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3169: -- Component/s: test Target Version/s: 2.0.0 Affects Version/s: 2.0.0 Hadoop Flags: Reviewed +1 looks good > TestFsck should test multiple -move operations in a row > --- > > Key: HDFS-3169 > URL: https://issues.apache.org/jira/browse/HDFS-3169 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3169.001.patch > > > TestFsck should test multiple -move operations in a row. Overall, it would > be nice to have more coverage on this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3206) Miscellaneous xml cleanups for OEV
[
https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3206:
--
Component/s: tools
Target Version/s: 2.0.0
Affects Version/s: 2.0.0
Issue Type: Improvement (was: Bug)
Hadoop Flags: Reviewed
Summary: Miscellaneous xml cleanups for OEV (was: oev:
miscellaneous xml cleanups)
+1 looks good. The binary patch didn't apply for me so I generated the new
editsStored via:
{code}
./bin/hdfs oev -i
~/src/hadoop-trunk1/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
-o editsStored -p binary
{code}
and ran TestOfflineEditsViewer for sanity.
> Miscellaneous xml cleanups for OEV
> --
>
> Key: HDFS-3206
> URL: https://issues.apache.org/jira/browse/HDFS-3206
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.0.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch,
> HDFS-3206.003.patch, HDFS-3206.004.patch
>
>
> * SetOwner operations can change both the user and group which a file or
> directory belongs to, or just one of those. Currently, in the XML
> serialization/deserialization code, we don't handle the case where just the
> group is set, not the user. We should handle this case.
> * consistently serialize generation stamp as GENSTAMP.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257138#comment-13257138
]
Hudson commented on HDFS-891:
-
Integrated in Hadoop-Mapreduce-trunk-Commit #2117 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2117/])
HDFS-891. DataNode no longer needs to check for dfs.network.script.
Contributed by Harsh J (Revision 1327762)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327762
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-891.patch, hdfs-891.txt
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[
https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257136#comment-13257136
]
Tsz Wo (Nicholas), SZE commented on HDFS-3201:
--
- You are actually using DFS_JOURNAL_USER_NAME_KEY in the code, please update
the TODO.
{code}
+ // TODO: journal service should use DFS_JOURNAL_USER_NAME_KEY when security
is
+ // enabled
{code}
- Please remove Journal.getStorageInfo()
- Please use JOURNAL_ATTRIBUTE_KEY
{code}
+ httpServer.setAttribute("localjournal", localJournal);
{code}
- Move editsToDownload(..) to the test since it is only used there.
> Implement Journal reader for reading transactions from local edits log or
> remote Journal node
> -
>
> Key: HDFS-3201
> URL: https://issues.apache.org/jira/browse/HDFS-3201
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch,
> HDFS-3201.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3201: - Status: Open (was: Patch Available) Hi Brandon, you don't have to submit the patch since it is not for trunk. > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch, > HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257131#comment-13257131
]
Hudson commented on HDFS-891:
-
Integrated in Hadoop-Common-trunk-Commit #2100 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2100/])
HDFS-891. DataNode no longer needs to check for dfs.network.script.
Contributed by Harsh J (Revision 1327762)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327762
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-891.patch, hdfs-891.txt
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257130#comment-13257130
]
Hudson commented on HDFS-891:
-
Integrated in Hadoop-Hdfs-trunk-Commit #2173 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2173/])
HDFS-891. DataNode no longer needs to check for dfs.network.script.
Contributed by Harsh J (Revision 1327762)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327762
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-891.patch, hdfs-891.txt
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-891:
-
Resolution: Fixed
Fix Version/s: 2.0.0
Target Version/s: (was: 2.0.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
New jenkins run is clean. No new test is necessary since this just removes the
old check. I've committed this and merged to branch-2. Thanks Steve and Harsh!
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-891.patch, hdfs-891.txt
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3300) stream the edit segments to NameNode when NameNode starts up
stream the edit segments to NameNode when NameNode starts up Key: HDFS-3300 URL: https://issues.apache.org/jira/browse/HDFS-3300 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Brandon Li Edit logs are saved on Journal daemon. When NameNode starts, it loads the latest image file and then streams the edit logs from an active Journal daemon. Currently we are using http to transfer edit files between two journal daemons/nodes or between a journal daemon and a NameNode. To get edit file from Journal daemon, the NameNode has to download it first and then read it from the disk. To avoid the slow start-up time, NameNode should be enhance to read the http data stream and update its in memory name space instead of saving the streamed data on disk first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257102#comment-13257102 ] Hadoop QA commented on HDFS-3201: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523278/HDFS-3201.HDFS-3092.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2303//console This message is automatically generated. > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch, > HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3201: - Attachment: HDFS-3201.HDFS-3092.patch > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch, HDFS-3201.HDFS-3092.patch, > HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3299) handle creation time also in journal service
handle creation time also in journal service Key: HDFS-3299 URL: https://issues.apache.org/jira/browse/HDFS-3299 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) oev: miscellaneous xml cleanups
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257069#comment-13257069 ] Colin Patrick McCabe commented on HDFS-3206: I just want to comment here that the test failure is the result of Jenkins not applying my binary diff. I filed https://issues.apache.org/jira/browse/HADOOP-8291 in hopes the Jenkins will someday support binary diffs. > oev: miscellaneous xml cleanups > --- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) oev: miscellaneous xml cleanups
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257068#comment-13257068 ] Colin Patrick McCabe commented on HDFS-3206: I just want to comment here that the test failure is the result of Jenkins not applying my binary diff. I filed https://issues.apache.org/jira/browse/HADOOP-8291 in hopes the Jenkins will someday support binary diffs. > oev: miscellaneous xml cleanups > --- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3290) Use a better local directory layout for the datanode
[ https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257065#comment-13257065 ] Colin Patrick McCabe commented on HDFS-3290: Hi Matt, You are right again-- and I was wrong. Todd makes a very good point, though. > Use a better local directory layout for the datanode > > > Key: HDFS-3290 > URL: https://issues.apache.org/jira/browse/HDFS-3290 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > When the HDFS DataNode stores chunks in a local directory, it currently puts > all of the chunk files into either one big directory, or a collection of > directories. However, there is no way to know which directory a given block > will end up in, given its ID. As the number of files increases, this does > not scale well. > Similar to the git version control system, HDFS should create a few different > top level directories keyed off of a few bits in the chunk ID. Git uses 8 > bits. This substantially cuts down on the number of chunk files in the same > directory and gives increased performance, while not compromising O(1) lookup > of chunks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3206) oev: miscellaneous xml cleanups
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257040#comment-13257040 ] Hadoop QA commented on HDFS-3206: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523258/HDFS-3206.004.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2302//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2302//console This message is automatically generated. > oev: miscellaneous xml cleanups > --- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3298) Add HdfsDataOutputStream as a public API
[ https://issues.apache.org/jira/browse/HDFS-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3298: - Description: We need a public API to access HDFS specific features like getNumCurrentReplicas as mentioned in [Uma's comment|https://issues.apache.org/jira/browse/HDFS-1599?focusedCommentId=13243105&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13243105]. Summary: Add HdfsDataOutputStream as a public API (was: Add HdfsDataOutputStream) > Add HdfsDataOutputStream as a public API > > > Key: HDFS-3298 > URL: https://issues.apache.org/jira/browse/HDFS-3298 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > We need a public API to access HDFS specific features like > getNumCurrentReplicas as mentioned in [Uma's > comment|https://issues.apache.org/jira/browse/HDFS-1599?focusedCommentId=13243105&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13243105]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257038#comment-13257038 ] Tsz Wo (Nicholas), SZE commented on HDFS-3282: -- Thanks for looking at this, Eli. Unfortunately, we don't have something like OK_JAVAC_WARNINGS. The test-patch script simply compare the number of javac warnings with and without the patch. So, no additional change to the patch is required. > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257032#comment-13257032
]
Hadoop QA commented on HDFS-891:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523249/hdfs-891.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2301//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2301//console
This message is automatically generated.
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Attachments: HDFS-891.patch, hdfs-891.txt
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3290) Use a better local directory layout for the datanode
[ https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257001#comment-13257001 ] Matt Foley commented on HDFS-3290: -- bq. So you have to search every directory to find a given block. This clearly won't scale as the number of directories increases. Colin, your new assertion is also incorrect. I'll leave it as an exercise for the student to figure out what the datanode really does when it needs to look up a block. I'm glad you're enthusiastic about improving Hadoop. But please consider that this code is actually quite mature and works really well, in production in multiple companies with 100's of millions of blocks under management. Evidently it DOES scale. So please read the code and thoroughly understand how it really works before suggesting major changes. There are plenty of things that can be improved; but changing stuff that "ain't broke" induces errors and unnecessary risk. Thank you. > Use a better local directory layout for the datanode > > > Key: HDFS-3290 > URL: https://issues.apache.org/jira/browse/HDFS-3290 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > When the HDFS DataNode stores chunks in a local directory, it currently puts > all of the chunk files into either one big directory, or a collection of > directories. However, there is no way to know which directory a given block > will end up in, given its ID. As the number of files increases, this does > not scale well. > Similar to the git version control system, HDFS should create a few different > top level directories keyed off of a few bits in the chunk ID. Git uses 8 > bits. This substantially cuts down on the number of chunk files in the same > directory and gives increased performance, while not compromising O(1) lookup > of chunks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3290) Use a better local directory layout for the datanode
[ https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256999#comment-13256999 ] Todd Lipcon commented on HDFS-3290: --- It doesn't do a "search" for a block. The DN keeps the block map in memory. But, I do think this is a good idea, as it will make it easier in the future to avoid having to keep the block map in memory on the DNs. > Use a better local directory layout for the datanode > > > Key: HDFS-3290 > URL: https://issues.apache.org/jira/browse/HDFS-3290 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > When the HDFS DataNode stores chunks in a local directory, it currently puts > all of the chunk files into either one big directory, or a collection of > directories. However, there is no way to know which directory a given block > will end up in, given its ID. As the number of files increases, this does > not scale well. > Similar to the git version control system, HDFS should create a few different > top level directories keyed off of a few bits in the chunk ID. Git uses 8 > bits. This substantially cuts down on the number of chunk files in the same > directory and gives increased performance, while not compromising O(1) lookup > of chunks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3206) oev: miscellaneous xml cleanups
[ https://issues.apache.org/jira/browse/HDFS-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3206: --- Attachment: HDFS-3206.004.patch * here's a patch with the binary part included. Hopefully jenkins won't choke on it... > oev: miscellaneous xml cleanups > --- > > Key: HDFS-3206 > URL: https://issues.apache.org/jira/browse/HDFS-3206 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3206.001.patch, HDFS-3206.002.patch, > HDFS-3206.003.patch, HDFS-3206.004.patch > > > * SetOwner operations can change both the user and group which a file or > directory belongs to, or just one of those. Currently, in the XML > serialization/deserialization code, we don't handle the case where just the > group is set, not the user. We should handle this case. > * consistently serialize generation stamp as GENSTAMP. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256961#comment-13256961 ] Hudson commented on HDFS-3292: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2115 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2115/]) HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawCapacity() and getRawUsed() from DistributedFileSystem. Contributed by Arpit Gupta (Revision 1327664) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327664 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Fix For: 3.0.0 > > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256958#comment-13256958 ] Eli Collins commented on HDFS-3282: --- Oops, misread javac warnings for javadocs warnings, never mind. > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256952#comment-13256952 ] Eli Collins commented on HDFS-3282: --- If you're going to punt on the javadoc warnings this introduces the patch should bump OK_JAVADOC_WARNINGS so subsequent jenkins runs don't automatically -1. > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3298) Add HdfsDataOutputStream
Add HdfsDataOutputStream Key: HDFS-3298 URL: https://issues.apache.org/jira/browse/HDFS-3298 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-891:
-
Attachment: hdfs-891.txt
Re-uploading the patch to kick jenkins. I think the previous findbugs/audit
warning has to do with us removing a System.exit that is referenced in a
findbugs excludes file, want to see what the actual warnings are to be sure.
Will hold of committing since we may need to update the findbugs file as well.
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Attachments: HDFS-891.patch, hdfs-891.txt
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
[ https://issues.apache.org/jira/browse/HDFS-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256936#comment-13256936 ] Colin Patrick McCabe commented on HDFS-3271: Since we already have a configuration knob for using the re-entrant functions in the .xml files, that seems like the logical thing to use. Yes, it will be a bit of a pain to get it over to the FUSE side, but not impossible. For now, I'll wait for some before/after performance numbers before doing anything on this... > src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc > -- > > Key: HDFS-3271 > URL: https://issues.apache.org/jira/browse/HDFS-3271 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-891) DataNode no longer needs to check for dfs.network.script
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-891:
-
Target Version/s: 2.0.0 (was: 3.0.0, 2.0.0)
Affects Version/s: (was: 0.22.0)
0.23.0
Assignee: Harsh J
Summary: DataNode no longer needs to check for dfs.network.script
(was: DataNode.instantiateDataNode calls system.exit(-1) if
conf.get("dfs.network.script") != null)
Agree. +1 patch looks good
> DataNode no longer needs to check for dfs.network.script
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Steve Loughran
>Assignee: Harsh J
>Priority: Minor
> Attachments: HDFS-891.patch
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-891) DataNode.instantiateDataNode calls system.exit(-1) if conf.get("dfs.network.script") != null
[
https://issues.apache.org/jira/browse/HDFS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-891:
-
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1, 0.24.0)
> DataNode.instantiateDataNode calls system.exit(-1) if
> conf.get("dfs.network.script") != null
>
>
> Key: HDFS-891
> URL: https://issues.apache.org/jira/browse/HDFS-891
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.22.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HDFS-891.patch
>
>
> Looking at the code for {{DataNode.instantiateDataNode())} , I see that it
> calls {{system.exit(-1)}} if it is not happy with the configuration
> {code}
> if (conf.get("dfs.network.script") != null) {
> LOG.error("This configuration for rack identification is not supported"
> +
> " anymore. RackID resolution is handled by the NameNode.");
> System.exit(-1);
> }
> {code}
> This is excessive. It should throw an exception and let whoever called the
> method decide how to handle it. The {{DataNode.main()}} method will log the
> exception and exit with a -1 value, but other callers (such as anything using
> {{MiniDFSCluster}} will now see a meaningful message rather than some Junit
> "tests exited without completing" warning.
> Easy to write a test for the correct behaviour: start a {{MiniDFSCluster}}
> with this configuration set, see what happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2998) OfflineImageViewer and ImageVisitor should be annotated public
[ https://issues.apache.org/jira/browse/HDFS-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2998: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2) > OfflineImageViewer and ImageVisitor should be annotated public > -- > > Key: HDFS-2998 > URL: https://issues.apache.org/jira/browse/HDFS-2998 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.1 >Reporter: Aaron T. Myers > > The OfflineImageViewer is currently annotated as InterfaceAudience.Private. > It's intended for subclassing, so it should be annotated as the public API > that it is. > The ImageVisitor class should similarly be annotated public (evolving is > fine). Note that it should also be changed to be public; it's currently > package-private, which means that users have to cheat with their subclass > package name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2896) The 2NN incorrectly daemonizes
[
https://issues.apache.org/jira/browse/HDFS-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2896:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1)
> The 2NN incorrectly daemonizes
> --
>
> Key: HDFS-2896
> URL: https://issues.apache.org/jira/browse/HDFS-2896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Eli Collins
>Assignee: Eli Collins
>
> The SecondaryNameNode (and Checkpointer) confuse o.a.h.u.Daemon with a Unix
> daemon. Per below it intends to create a thread that never ends, but
> o.a.h.u.Daemon just marks a thread with Java's Thread#setDaemon which means
> Java will terminate the thread when there are no more non-daemon user threads
> running
> {code}
> // Create a never ending deamon
> Daemon checkpointThread = new Daemon(secondary);
> {code}
> Perhaps they thought they were using commons Daemon. We of course don't want
> the 2NN to exit unless it exits itself or is stopped explicitly. Currently it
> won't do this because the main thread is not marked as a daemon thread. In
> any case, let's make the 2NN consistent with the NN in this regard (exit when
> the RPC thread exits).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2075) Add "Number of Reporting Nodes" to namenode web UI
[ https://issues.apache.org/jira/browse/HDFS-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2075: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1, 0.24.0) > Add "Number of Reporting Nodes" to namenode web UI > -- > > Key: HDFS-2075 > URL: https://issues.apache.org/jira/browse/HDFS-2075 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node, tools >Affects Versions: 0.20.1, 0.20.2 >Reporter: Xing Jin >Priority: Minor > Labels: newbie > Fix For: 0.20.3 > > Attachments: HDFS-2075.patch.txt > > Original Estimate: 1h > Remaining Estimate: 1h > > The namenode web UI misses some information when safemode is on (e.g., the > number of reporting nodes). These information will help us understand the > system status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2930) bin/hdfs should print a message when an invalid command is specified
[
https://issues.apache.org/jira/browse/HDFS-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2930:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2)
> bin/hdfs should print a message when an invalid command is specified
>
>
> Key: HDFS-2930
> URL: https://issues.apache.org/jira/browse/HDFS-2930
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 0.23.1
>Reporter: Eli Collins
> Labels: newbie
> Attachments: HDFS-2930.patch
>
>
> hdfs will currently give a NoClassDefFoundError stacktrace if there's a typo
> specifying the command.
> {noformat}
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfadmin
> Exception in thread "main" java.lang.NoClassDefFoundError: dfadmin
> Caused by: java.lang.ClassNotFoundException: dfadmin
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2885) Remove "federation" from the nameservice config options
[
https://issues.apache.org/jira/browse/HDFS-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2885:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1)
> Remove "federation" from the nameservice config options
> ---
>
> Key: HDFS-2885
> URL: https://issues.apache.org/jira/browse/HDFS-2885
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.23.1
>Reporter: Eli Collins
>
> HDFS-1623, and potentially other HDFS features will use the nameservice
> abstraction, even if federation is not enabled (eg you need to configure
> {{dfs.federation.nameservices}} in HA even if you're not using federation
> just to declare your nameservice). This is confusing to users. We should
> consider deprecating and removing "federation" from the
> {{dfs.federation.nameservices}} and {{dfs.federation.nameservice.id}} config
> options, as {{dfs.nameservices}} and {{dfs.nameservice.id}} are more
> intuitive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2911) Gracefully handle OutOfMemoryErrors
[ https://issues.apache.org/jira/browse/HDFS-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2911: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Gracefully handle OutOfMemoryErrors > --- > > Key: HDFS-2911 > URL: https://issues.apache.org/jira/browse/HDFS-2911 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 0.23.0, 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > > We should gracefully handle j.l.OutOfMemoryError exceptions in the NN or DN. > We should catch them in a high-level handler, cleanly fail the RPC (vs > sending back the OOM stackrace) or background thread, and shutdown the NN or > DN. Currently the process is left in a not well-test tested state > (continuously fails RPCs and internal threads, may or may not recover and > doesn't shutdown gracefully). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3115) Update hdfs design doc to consider HA NNs
[ https://issues.apache.org/jira/browse/HDFS-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-3115: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > Update hdfs design doc to consider HA NNs > - > > Key: HDFS-3115 > URL: https://issues.apache.org/jira/browse/HDFS-3115 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 0.24.0, 0.23.3 >Reporter: Todd Lipcon >Priority: Minor > > The hdfs_design_doc.xml still references the NN as an SPOF, which is no > longer true. We should sweep docs for anything else that seems to be out of > date with HA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2936) Provide a better way to specify a HDFS-wide minimum replication requirement
[
https://issues.apache.org/jira/browse/HDFS-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2936:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2, 0.24.0)
> Provide a better way to specify a HDFS-wide minimum replication requirement
> ---
>
> Key: HDFS-2936
> URL: https://issues.apache.org/jira/browse/HDFS-2936
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>
> Currently, if an admin would like to enforce a replication factor for all
> files on his HDFS, he does not have a way. He may arguably set
> dfs.replication.min but that is a very hard guarantee and if the pipeline
> can't afford that number for some reason/failure, the close() does not
> succeed on the file being written and leads to several issues.
> After discussing with Todd, we feel it would make sense to introduce a second
> config (which is ${dfs.replication.min} by default) which would act as a
> minimum specified replication for files. This is different than
> dfs.replication.min which also ensures that many replicas are recorded before
> completeFile() returns... perhaps something like ${dfs.replication.min.user}.
> We can leave dfs.replication.min alone for hard-guarantees and add
> ${dfs.replication.min.for.block.completion} which could be left at 1 even if
> dfs.replication.min is >1, and let files complete normally but not be of a
> low replication factor (so can be monitored and accounted-for later).
> I'm prefering the second option myself. Will post a patch with tests soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2148) Address all the federation TODOs
[ https://issues.apache.org/jira/browse/HDFS-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2148: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Address all the federation TODOs > > > Key: HDFS-2148 > URL: https://issues.apache.org/jira/browse/HDFS-2148 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Eli Collins >Assignee: Suresh Srinivas > > The federation merge introduced a bunch of todos marked "TODO:FEDERATION" and > "TODO". We should have jiras for these and address the issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2884) TestDecommission.testDecommissionFederation fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2884:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1)
> TestDecommission.testDecommissionFederation fails intermittently
>
>
> Key: HDFS-2884
> URL: https://issues.apache.org/jira/browse/HDFS-2884
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.1
>Reporter: Eli Collins
>
> I saw the following assert fail on a jenkins job for branch HDFS-1623 but I
> don't think it's HA related.
>
> {noformat}
> java.lang.AssertionError: Number of Datanodes expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at
> org.apache.hadoop.hdfs.TestDecommission.validateCluster(TestDecommission.java:275)
> at
> org.apache.hadoop.hdfs.TestDecommission.startCluster(TestDecommission.java:288)
> at
> org.apache.hadoop.hdfs.TestDecommission.testDecommission(TestDecommission.java:384)
> at
> org.apache.hadoop.hdfs.TestDecommission.testDecommissionFederation(TestDecommission.java:344)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2798) Append may race with datanode block scanner, causing replica to be incorrectly marked corrupt
[ https://issues.apache.org/jira/browse/HDFS-2798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2798: -- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > Append may race with datanode block scanner, causing replica to be > incorrectly marked corrupt > - > > Key: HDFS-2798 > URL: https://issues.apache.org/jira/browse/HDFS-2798 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Todd Lipcon >Priority: Critical > > When a pipeline is setup for append, the block's metadata file is renamed > before the block is removed from the datanode block scanner queues. This can > cause a race condition where the block scanner incorrectly marks the block as > corrupt, since it tries to scan the file corresponding to the old genstamp. > This causes TestFileAppend2 to time out in extremely rare circumstances - the > corrupt replica prevents the writer thread from completing the file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2999) DN metrics should include per-disk utilization
[ https://issues.apache.org/jira/browse/HDFS-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2999: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2) > DN metrics should include per-disk utilization > -- > > Key: HDFS-2999 > URL: https://issues.apache.org/jira/browse/HDFS-2999 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.1 >Reporter: Aaron T. Myers > > We should have per-dfs.data.dir metrics in the DN's metrics report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2897) Enable a single 2nn to checkpoint multiple nameservices
[ https://issues.apache.org/jira/browse/HDFS-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2897: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Enable a single 2nn to checkpoint multiple nameservices > --- > > Key: HDFS-2897 > URL: https://issues.apache.org/jira/browse/HDFS-2897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 0.23.0 >Reporter: Eli Collins > Fix For: 0.23.0 > > > The dfs.namenode.secondary.http-address needs to be suffixed with a > particular nameservice. It would be useful to be able to be able to configure > a single 2NN to checkpoint all the nameservices for a NN rather than having > to run a separate 2NN per nameservice. It could potentially checkpoint all > namenode IDs for a nameservice as well but given that the standby is capable > of checkpointing and is required I think we can ignore this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2771) Move Federation and WebHDFS documentation into HDFS project
[ https://issues.apache.org/jira/browse/HDFS-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2771: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Move Federation and WebHDFS documentation into HDFS project > --- > > Key: HDFS-2771 > URL: https://issues.apache.org/jira/browse/HDFS-2771 > Project: Hadoop HDFS > Issue Type: Task > Components: documentation >Affects Versions: 0.23.0 >Reporter: Todd Lipcon > > For some strange reason, the WebHDFS and Federation documentation is > currently in the hadoop-yarn site. This is counter-intuitive. We should move > these documents to an hdfs site, or if we think that all documentation should > go on one site, it should go into the hadoop-common project somewhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2892) Some of property descriptions are not given(hdfs-default.xml)
[
https://issues.apache.org/jira/browse/HDFS-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2892:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1, 0.23.0)
> Some of property descriptions are not given(hdfs-default.xml)
> --
>
> Key: HDFS-2892
> URL: https://issues.apache.org/jira/browse/HDFS-2892
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Brahma Reddy Battula
>Priority: Trivial
>
> Hi..I taken 23.0 release form
> http://hadoop.apache.org/common/releases.html#11+Nov%2C+2011%3A+release+0.23.0+available
> I just gone through all properties provided in the hdfs-default.xml..Some of
> the property description not mentioned..It's better to give description of
> property and usage(how to configure ) and Only MapReduce related jars only
> provided..Please check following two configurations
> *No Description*
> {noformat}
>
> dfs.datanode.https.address
> 0.0.0.0:50475
>
>
> dfs.namenode.https-address
> 0.0.0.0:50470
>
> {noformat}
> Better to mention example usage (what to configure...format(syntax))in
> desc,here I did not get what default mean whether this name of n/w interface
> or something else
>
> dfs.datanode.dns.interface
> default
> The name of the Network Interface from which a data node
> should
> report its IP address.
>
>
> The following property is commented..If it is not supported better to remove.
>
>dfs.cluster.administrators
>ACL for the admins
>This configuration is used to control who can access the
> default servlets in the namenode, etc.
>
>
> Small clarification for following property..if some value configured this
> then NN will be safe mode upto this much time..
> May I know usage of the following property...
>
> dfs.blockreport.initialDelay 0
> Delay for first block report in seconds.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256920#comment-13256920 ] Hudson commented on HDFS-3292: -- Integrated in Hadoop-Common-trunk-Commit #2098 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2098/]) HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawCapacity() and getRawUsed() from DistributedFileSystem. Contributed by Arpit Gupta (Revision 1327664) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327664 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Fix For: 3.0.0 > > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256918#comment-13256918 ] Hudson commented on HDFS-3292: -- Integrated in Hadoop-Hdfs-trunk-Commit #2171 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2171/]) HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawCapacity() and getRawUsed() from DistributedFileSystem. Contributed by Arpit Gupta (Revision 1327664) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327664 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Fix For: 3.0.0 > > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256916#comment-13256916 ] Tsz Wo (Nicholas), SZE commented on HDFS-3282: -- All the new javac warnings are the deprecated warnings of the use of DFSDataInputStream. Let's fix them separately. > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3290) Use a better local directory layout for the datanode
[ https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3290: --- Description: When the HDFS DataNode stores chunks in a local directory, it currently puts all of the chunk files into either one big directory, or a collection of directories. However, there is no way to know which directory a given block will end up in, given its ID. As the number of files increases, this does not scale well. Similar to the git version control system, HDFS should create a few different top level directories keyed off of a few bits in the chunk ID. Git uses 8 bits. This substantially cuts down on the number of chunk files in the same directory and gives increased performance, while not compromising O(1) lookup of chunks. was: When the HDFS DataNode stores chunks in a local directory, it currently puts all of the chunk files into one big directory. As the number of files increases, this does not work well at all. Local filesystems are not optimized for the case where there are hundreds of thousands of files in the same directory. It also makes inspecting directories with standard UNIX tools difficult. Similar to the git version control system, HDFS should create a few different top level directories keyed off of a few bits in the chunk ID. Git uses 8 bits. This substantially cuts down on the number of chunk files in the same directory and gives increased performance. > Use a better local directory layout for the datanode > > > Key: HDFS-3290 > URL: https://issues.apache.org/jira/browse/HDFS-3290 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > When the HDFS DataNode stores chunks in a local directory, it currently puts > all of the chunk files into either one big directory, or a collection of > directories. However, there is no way to know which directory a given block > will end up in, given its ID. As the number of files increases, this does > not scale well. > Similar to the git version control system, HDFS should create a few different > top level directories keyed off of a few bits in the chunk ID. Git uses 8 > bits. This substantially cuts down on the number of chunk files in the same > directory and gives increased performance, while not compromising O(1) lookup > of chunks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3290) Use a better local directory layout for the datanode
[ https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256914#comment-13256914 ] Colin Patrick McCabe commented on HDFS-3290: Hi Matt, Thanks for your comments. I misunderstood how the DataNode functions, probably because I didn't create enough blocks to see the multiple directories. It seems like we do indeed create multiple directories. However, there is no way to tell where a given block will be given its ID. So you have to search every directory to find a given block. This clearly won't scale as the number of directories increases. So I guess the title of this JIRA can remain the same-- a "better" directory structure that doesn't require an O(N / maxBlocksPerDir) search to read a block would be a good thing... Colin > Use a better local directory layout for the datanode > > > Key: HDFS-3290 > URL: https://issues.apache.org/jira/browse/HDFS-3290 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > When the HDFS DataNode stores chunks in a local directory, it currently puts > all of the chunk files into one big directory. As the number of files > increases, this does not work well at all. Local filesystems are not > optimized for the case where there are hundreds of thousands of files in the > same directory. It also makes inspecting directories with standard UNIX > tools difficult. > Similar to the git version control system, HDFS should create a few different > top level directories keyed off of a few bits in the chunk ID. Git uses 8 > bits. This substantially cuts down on the number of chunk files in the same > directory and gives increased performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3292: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) I have committed this. Thanks, Arpit! > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Fix For: 3.0.0 > > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2815) Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.
[ https://issues.apache.org/jira/browse/HDFS-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2815: -- Target Version/s: 1.1.0, 2.0.0, 3.0.0 (was: 0.23.2, 1.1.0, 0.24.0) > Namenode is not coming out of safemode when we perform ( NN crash + restart ) > . Also FSCK report shows blocks missed. > -- > > Key: HDFS-2815 > URL: https://issues.apache.org/jira/browse/HDFS-2815 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0, 0.24.0, 0.23.1, 1.0.0, 1.1.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Critical > Fix For: 0.24.0, 0.23.2 > > Attachments: HDFS-2815-22-branch.patch, HDFS-2815-Branch-1.patch, > HDFS-2815.patch, HDFS-2815.patch > > > When tested the HA(internal) with continuous switch with some 5mins gap, > found some *blocks missed* and namenode went into safemode after next switch. > >After the analysis, i found that this files already deleted by clients. > But i don't see any delete commands logs namenode log files. But namenode > added that blocks to invalidateSets and DNs deleted the blocks. >When restart of the namenode, it went into safemode and expecting some > more blocks to come out of safemode. >Here the reason could be that, file has been deleted in memory and added > into invalidates after this it is trying to sync the edits into editlog file. > By that time NN asked DNs to delete that blocks. Now namenode shuts down > before persisting to editlogs.( log behind) >Due to this reason, we may not get the INFO logs about delete, and when we > restart the Namenode (in my scenario it is again switch), Namenode expects > this deleted blocks also, as delete request is not persisted into editlog > before. >I reproduced this scenario with bedug points. *I feel, We should not add > the blocks to invalidates before persisting into Editlog*. > Note: for switch, we used kill -9 (force kill) > I am currently in 0.20.2 version. Same verified in 0.23 as well in normal > crash + restart scenario. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3133) Add support for DFS upgrade with HA enabled
[ https://issues.apache.org/jira/browse/HDFS-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3133: -- Target Version/s: 2.0.0 (was: 0.24.0) Affects Version/s: (was: 0.24.0) 2.0.0 > Add support for DFS upgrade with HA enabled > --- > > Key: HDFS-3133 > URL: https://issues.apache.org/jira/browse/HDFS-3133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers > > For the first implementation of HA NameNode, we punted on allowing DFS > upgrade with HA enabled, which makes doing a DFS upgrade on an HA-enabled > cluster quite cumbersome and error-prone. We should add better support for > this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3096) dfs.datanode.data.dir.perm is set to 755 instead of 700
[ https://issues.apache.org/jira/browse/HDFS-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-3096: -- Target Version/s: 1.0.3, 2.0.0 (was: 1.0.3, 0.23.2) > dfs.datanode.data.dir.perm is set to 755 instead of 700 > --- > > Key: HDFS-3096 > URL: https://issues.apache.org/jira/browse/HDFS-3096 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.23.0, 1.0.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > > dfs.datanode.data.dir.perm is used by the datanode to set the permissions of > it data directories. This is set by default to 755 which gives read > permissions to everyone to that directory, opening up possibility of reading > the data blocks by anyone in a secure cluster. Admins can over-ride this > config but its sub-optimal practice for the default to be weak. IMO, the > default should be strong and the admins can relax it if necessary. > The fix is to change default permissions to 700. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3297) Update free space in the DataBlockScanner rather than using du
Update free space in the DataBlockScanner rather than using du -- Key: HDFS-3297 URL: https://issues.apache.org/jira/browse/HDFS-3297 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor As the DataNode adds new blocks to a BlockPool, it keeps track of how much space that block pool consumes. This information gets sent to the NameNode so we can track statistics and so forth. Periodically, we check what's actually on the disk to make sure that the counts we are keeping are accurate. The DataNode currently kicks off a "du -s" process through the shell every few minutes and takes the result as the new used space number. We should do this in the DataBlockScanner, rather than using a separate du process. The main reason to do this is so that we don't cause a lot of random I/O operations on the disk. Since du has to visit every file in the BlockPool, it is essentially re-doing the work of the block scanner, for no reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3292: - Hadoop Flags: Reviewed +1 patch looks good. > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2617) Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
[ https://issues.apache.org/jira/browse/HDFS-2617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-2617: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3) > Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution > -- > > Key: HDFS-2617 > URL: https://issues.apache.org/jira/browse/HDFS-2617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: HDFS-2617-a.patch > > > The current approach to secure and authenticate nn web services is based on > Kerberized SSL and was developed when a SPNEGO solution wasn't available. Now > that we have one, we can get rid of the non-standard KSSL and use SPNEGO > throughout. This will simplify setup and configuration. Also, Kerberized > SSL is a non-standard approach with its own quirks and dark corners > (HDFS-2386). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3025) Automatic log sync shouldn't happen inside logEdit path
[ https://issues.apache.org/jira/browse/HDFS-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-3025: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3) > Automatic log sync shouldn't happen inside logEdit path > --- > > Key: HDFS-3025 > URL: https://issues.apache.org/jira/browse/HDFS-3025 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node, performance >Affects Versions: 0.23.3 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-3025.txt, hdfs-3025.txt, hdfs-3025.txt > > > HDFS-3020 fixes the "automatic log sync" functionality so that, when logEdits > is called without log sync, it eventually triggers a sync. That sync ends up > being inline, though, which means the FSN lock is usually held during it. > This causes a bunch of threads to pile up. > Instead, we should have it just set a "syncNeeded" flag and trigger a sync > from another thread which isn't holding the lock (or from the same thread > using a "logSyncIfNeeded" call). > (credit to the FB branch for this idea) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3141) The NN should log "missing" blocks
[ https://issues.apache.org/jira/browse/HDFS-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-3141: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3, 1.1.0) > The NN should log "missing" blocks > -- > > Key: HDFS-3141 > URL: https://issues.apache.org/jira/browse/HDFS-3141 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3141-b1.txt > > > It would be help debugging if the NN logged at info level and "missing" > blocks. In v1 missing means there are no live / decommissioned replicas (ie > they're all excess or corrupt), in trunk it means all replicas of the block > are corrupt. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3123) BNN is getting Nullpointer execption and shuttingdown When NameNode got formatted
[
https://issues.apache.org/jira/browse/HDFS-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-3123:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3, 0.24.0)
> BNN is getting Nullpointer execption and shuttingdown When NameNode got
> formatted
> --
>
> Key: HDFS-3123
> URL: https://issues.apache.org/jira/browse/HDFS-3123
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.24.0, 0.23.4
>Reporter: Brahma Reddy Battula
>Assignee: Uma Maheswara Rao G
>
> Scenario 1
> ==
> Start NN and BNN
> stop NN and BNN
> Format NN and start only BNN
> Then BNN as getting Nullpointer and getting shutdown
> {noformat}
> 12/03/20 21:26:05 ERROR ipc.RPC: Tried to call RPC.stopProxy on an object
> that is not a proxy.
> java.lang.IllegalArgumentException: not a proxy instance
> at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:591)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:547)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:847)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:908)
> 12/03/20 21:26:05 ERROR ipc.RPC: Could not get invocation handler null for
> proxy class class
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB, or invocation
> handler is not closeable.
> 12/03/20 21:26:05 ERROR namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getFSImage(NameNode.java:609)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:547)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:847)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:908)
> 12/03/20 21:26:05 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at HOST-10-18-40-233/10.18.40.233
> /
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3128) TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp
[ https://issues.apache.org/jira/browse/HDFS-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HDFS-3128: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3) > TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp > -- > > Key: HDFS-3128 > URL: https://issues.apache.org/jira/browse/HDFS-3128 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Eli Collins >Priority: Minor > > Saw this on jenkins, TestResolveHdfsSymlink#testFcResolveAfs creates > /tmp/alpha which interferes with other executors on the same machine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3286) When the threshold value for balancer is 0(zero) ,unexpected output is displayed
[
https://issues.apache.org/jira/browse/HDFS-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-3286:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3, 0.24.0)
> When the threshold value for balancer is 0(zero) ,unexpected output is
> displayed
>
>
> Key: HDFS-3286
> URL: https://issues.apache.org/jira/browse/HDFS-3286
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
>Affects Versions: 0.23.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 0.24.0
>
>
> Replication factor =1
> Step 1: Start NN,DN1.write 4 GB of data
> Step 2: Start DN2
> Step 3: issue the balancer command(./hdfs balancer -threshold 0)
> The threshold parameter is a fraction in the range of (0%, 100%) with a
> default value of 10%
> When the above scenario is executed the Source DN and Target DN is choosen
> and the number of bytes to be moved from source to target DN is also
> calculated .
> Then the balancer is exiting with the following message "No block can be
> moved. Exiting..." which is not expected.
> {noformat}
> HOST-xx-xx-xx-xx:/home/Andreina/APril10/install/hadoop/namenode/bin # ./hdfs
> balancer -threshold 0
> 12/04/16 16:22:07 INFO balancer.Balancer: Using a threshold of 0.0
> 12/04/16 16:22:07 INFO balancer.Balancer: namenodes =
> [hdfs://HOST-xx-xx-xx-xx:9000]
> 12/04/16 16:22:07 INFO balancer.Balancer: p =
> Balancer.Parameters[BalancingPolicy.Node, threshold=0.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> 12/04/16 16:22:10 INFO net.NetworkTopology: Adding a new node:
> /default-rack/yy.yy.yy.yy:50176
> 12/04/16 16:22:10 INFO net.NetworkTopology: Adding a new node:
> /default-rack/xx.xx.xx.xx:50010
> 12/04/16 16:22:10 INFO balancer.Balancer: 1 over-utilized:
> [Source[xx.xx.xx.xx:50010, utilization=7.212458091389678]]
> 12/04/16 16:22:10 INFO balancer.Balancer: 1 underutilized:
> [BalancerDatanode[yy.yy.yy.yy:50176, utilization=4.650670324367203E-5]]
> 12/04/16 16:22:10 INFO balancer.Balancer: Need to move 1.77 GB to make the
> cluster balanced.
> No block can be moved. Exiting...
> Balancing took 5.142 seconds
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3041) DFSOutputStream.close doesn't properly handle interruption
[
https://issues.apache.org/jira/browse/HDFS-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-3041:
--
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.3, 0.24.0)
> DFSOutputStream.close doesn't properly handle interruption
> --
>
> Key: HDFS-3041
> URL: https://issues.apache.org/jira/browse/HDFS-3041
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: test.txt
>
>
> TestHFlush.testHFlushInterrupted can fail occasionally due to a race: if a
> thread is interrupted while calling close(), then the {{finally}} clause of
> the {{close}} function sets {{closed = true}}. At this point it has enqueued
> the "end of block" packet to the DNs, but hasn't called {{completeFile}}.
> Then, if {{close}} is called again (as in the test case), it will be
> short-circuited since {{closed}} is already true. Thus {{completeFile}} never
> ends up getting called. This also means that the test can fail if the
> pipeline is running slowly, since the assertion that the file is the correct
> length won't see the last packet or two.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256896#comment-13256896 ] Hadoop QA commented on HDFS-3282: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523204/HDFS-3282.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 41 javac compiler warnings (more than the trunk's current 20 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2300//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2300//console This message is automatically generated. > Expose getFileLength API. > - > > Key: HDFS-3282 > URL: https://issues.apache.org/jira/browse/HDFS-3282 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-3282.patch, HDFS-3282.patch > > > This JIRA is to expose the getFileLength API through a new public > DistributedFileSystemInfo class. > I would appreciate if someone suggest good name for this public class. > Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3275) Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
[
https://issues.apache.org/jira/browse/HDFS-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256891#comment-13256891
]
Uma Maheswara Rao G commented on HDFS-3275:
---
Amith, thanks a lot for working on this issue.
I just reviewed your patch! Some comments:
1){quote} File base_dir = new File(System.getProperty("test.build.data",
+"build/test/data"), "dfs/");
{quote}
can't we use getBaseDirectory from minidfs cluster?
2) {quote}
NameNode.format(conf); // Namenode should not format dummy or any other
+ // non file schemes
{quote}
instead of wrapping the comment into two lines, can we add it above to the
foamt call?
3){code}
System.err
+.println("Storage directory "
++ dirUri
++ " is not in file scheme currently formatting is not
supported for this scheme");
{code}
can you please format this correctly?
ex:
{code}
System.err.println("Storage directory "
+ " is not in file scheme currently "
+ "formatting is not supported for this scheme");
{code}
4) File curDir = new File(dirUri.getPath());
File will take uri also, so need not cnvert it to string right?
5) Also message can be like, 'Formatting supported only for file based storage
directories. Current directory scheme is "+scheme+" . So, ignoring it for
format"
6) HATestUtil#setFailoverConfigurations would do almost similar setup as in
test. is it possible to use it by passing mock cluster or slightly changed
HATestUtil#setFailoverConfigurations?
7)you mean "Could not delete hdfs directory '" -> "Could not delete namespace
directory '"
8) testOnlyFileSchemeDirsAreFormatted ->
testFormatShouldBeIgnoredForNonFileBasedDirs ?
> Format command overwrites contents of non-empty shared edits dir if name dirs
> are empty without any prompting
> -
>
> Key: HDFS-3275
> URL: https://issues.apache.org/jira/browse/HDFS-3275
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, name-node
>Affects Versions: 2.0.0
>Reporter: Vinithra Varadharajan
>Assignee: amith
> Attachments: HDFS-3275.patch
>
>
> To reproduce:
> # Configure a NameNode with namedirs and a shared edits dir, all of which are
> empty.
> # Run hdfs namenode -format. Namedirs and shared edits dir gets populated.
> # Delete the contents of the namedirs. Leave the shared edits dir as is.
> Check the timestamps of the shared edits dir contents.
> # Run format again. The namedirs as well as the shared edits dir get
> formatted. The shared edits dir's contents have been replaced without any
> prompting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256870#comment-13256870 ] Arpit Gupta commented on HDFS-3292: --- no tests added as only deprecated class and methods were removed. > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256868#comment-13256868 ] Hudson commented on HDFS-3263: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2114 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2114/]) HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu) (Revision 1327627) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327627 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebApp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/FileSystemAccess.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/FileSystemAccessException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/ServerSetup.apt.vm * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/lib/service/hadoop/TestFileSystemAccessService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/HadoopUsersConfTestHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > HttpFS should read HDFS config from Hadoop site.xml files > - > > Key: HDFS-3263 > URL: https://issues.apache.org/jira/browse/HDFS-3263 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.0 > > Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch, > HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch > > > Currently HttpFS reads HDFS client configuration from the httfs-site.xml from > any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' > This is a bit inconvenient. > Instead we should support a single property 'httpfs.hadoop.configuration.dir' > that can be pointed to HADOOP conf/ dir and the core-site.xml and > hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
[ https://issues.apache.org/jira/browse/HDFS-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256865#comment-13256865 ] Hadoop QA commented on HDFS-3292: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523230/HDFS-3292.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2299//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2299//console This message is automatically generated. > Remove the deprecated DistributedFileSystem.DiskStatus and the related methods > -- > > Key: HDFS-3292 > URL: https://issues.apache.org/jira/browse/HDFS-3292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: 3.0.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Arpit Gupta > Attachments: HDFS-3292.patch, HDFS-3292.patch > > > DistributedFileSystem.DiskStatus and the related methods were deprecated in > 0.21 by HADOOP-4368. They can be removed in 0.23, i.e. 2.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256855#comment-13256855 ] Hudson commented on HDFS-3263: -- Integrated in Hadoop-Hdfs-trunk-Commit #2170 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2170/]) HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu) (Revision 1327627) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327627 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebApp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/FileSystemAccess.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/FileSystemAccessException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/ServerSetup.apt.vm * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/lib/service/hadoop/TestFileSystemAccessService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/HadoopUsersConfTestHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > HttpFS should read HDFS config from Hadoop site.xml files > - > > Key: HDFS-3263 > URL: https://issues.apache.org/jira/browse/HDFS-3263 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.0 > > Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch, > HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch > > > Currently HttpFS reads HDFS client configuration from the httfs-site.xml from > any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' > This is a bit inconvenient. > Instead we should support a single property 'httpfs.hadoop.configuration.dir' > that can be pointed to HADOOP conf/ dir and the core-site.xml and > hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256844#comment-13256844 ] Hudson commented on HDFS-3263: -- Integrated in Hadoop-Common-trunk-Commit #2097 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2097/]) HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu) (Revision 1327627) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1327627 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServerWebApp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/FileSystemAccess.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/FileSystemAccessException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/service/hadoop/FileSystemAccessService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/httpfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/apt/ServerSetup.apt.vm * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/TestHttpFSFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/lib/service/hadoop/TestFileSystemAccessService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/test/HadoopUsersConfTestHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > HttpFS should read HDFS config from Hadoop site.xml files > - > > Key: HDFS-3263 > URL: https://issues.apache.org/jira/browse/HDFS-3263 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.0 > > Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch, > HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch > > > Currently HttpFS reads HDFS client configuration from the httfs-site.xml from > any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' > This is a bit inconvenient. > Instead we should support a single property 'httpfs.hadoop.configuration.dir' > that can be pointed to HADOOP conf/ dir and the core-site.xml and > hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2962) jsvc needs to be removed from hdfs packaging
[
https://issues.apache.org/jira/browse/HDFS-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins resolved HDFS-2962.
---
Resolution: Duplicate
Fix Version/s: (was: 0.23.3)
Dupe of HDFS-2303.
> jsvc needs to be removed from hdfs packaging
>
>
> Key: HDFS-2962
> URL: https://issues.apache.org/jira/browse/HDFS-2962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, security
>Affects Versions: 0.23.0
>Reporter: Mingjie Lai
>
> As we discussed at HDFS-2303:
> ... hit the same problem today when I started to try secured DN of 0.23. The
> shipped jsvc (commons-daemon-1.0.3) with 0.23 doesn't work on my local ubuntu
> 11.10 (as described by atm, with libcap.2 instead of 1), nor our cluster
> centos 5 (without any libcap).
> The problem doesn't happen to 0.20.205 (what we use now) since
> commons-daemon-1.0.2 jsvc binary doesn't link to any libcap
> (http://archive.apache.org/dist/commons/daemon/binaries/1.0.2/linux/commons-daemon-1.0.2-bin-linux-${os-arch}.tar.gz)
> So the current way - downloading jsvc binary - won't work for 0.23+ secured
> DN.
> ...
> jsvc shouldn't be bundled with hadoop. Instead, it should be installed by
> user.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256830#comment-13256830 ] Hadoop QA commented on HDFS-3194: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523214/HDFS-3194_1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestSetrepDecreasing +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2298//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2298//console This message is automatically generated. > Continuous block scanning at DN side > > > Key: HDFS-3194 > URL: https://issues.apache.org/jira/browse/HDFS-3194 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: suja s >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3194.patch, HDFS-3194_1.patch > > > Block scanning interval by default should be taken as 21 days(3 weeks) and > each block scanning should happen once in 21 days. > Here the block is being scanned continuosly. > 2012-04-03 10:44:47,056 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:02,064 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:17,071 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:32,079 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2607) Use named daemon threads for the directory scanner
[ https://issues.apache.org/jira/browse/HDFS-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2607: -- Target Version/s: 2.0.0 Affects Version/s: (was: 0.22.0) (was: 0.21.0) Fix Version/s: (was: 0.23.3) > Use named daemon threads for the directory scanner > -- > > Key: HDFS-2607 > URL: https://issues.apache.org/jira/browse/HDFS-2607 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Eli Collins > > HDFS-854 added a thread pool for block scanners. It would be better to use a > factory that names the threads and daemonizes them so they don't block > shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2833) Add GETMERGE operation to httpfs
[ https://issues.apache.org/jira/browse/HDFS-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2833: -- Resolution: Won't Fix Fix Version/s: (was: 0.23.3) Status: Resolved (was: Patch Available) Closing as won't fix, Tucu, feel free to re-open if you disagree. > Add GETMERGE operation to httpfs > > > Key: HDFS-2833 > URL: https://issues.apache.org/jira/browse/HDFS-2833 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFS-2833.patch > > > Add to a convenience operation GETMERGE to httpfs. > This will simplify for external system accessing over HTTP to consume the > output of an MR job a single stream. > It would have the same semantics as the 'hadoop fs -getmerge' command. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2761) Improve Hadoop subcomponent integration in Hadoop 0.23
[ https://issues.apache.org/jira/browse/HDFS-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-2761. --- Resolution: Fixed Fix Version/s: (was: 0.23.3) 0.23.1 > Improve Hadoop subcomponent integration in Hadoop 0.23 > -- > > Key: HDFS-2761 > URL: https://issues.apache.org/jira/browse/HDFS-2761 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, hdfs client, scripts >Affects Versions: 0.23.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 0.23.1 > > > Please see HADOOP-7939 for a complete description and discussion. This JIRA > is for patch tracking purposes only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-3263: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk and branch-2. > HttpFS should read HDFS config from Hadoop site.xml files > - > > Key: HDFS-3263 > URL: https://issues.apache.org/jira/browse/HDFS-3263 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.0 > > Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch, > HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch > > > Currently HttpFS reads HDFS client configuration from the httfs-site.xml from > any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' > This is a bit inconvenient. > Instead we should support a single property 'httpfs.hadoop.configuration.dir' > that can be pointed to HADOOP conf/ dir and the core-site.xml and > hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3290) Use a better local directory layout for the datanode
[ https://issues.apache.org/jira/browse/HDFS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256812#comment-13256812 ] Matt Foley commented on HDFS-3290: -- Also same class name in hadoop-2, at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java. > Use a better local directory layout for the datanode > > > Key: HDFS-3290 > URL: https://issues.apache.org/jira/browse/HDFS-3290 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > When the HDFS DataNode stores chunks in a local directory, it currently puts > all of the chunk files into one big directory. As the number of files > increases, this does not work well at all. Local filesystems are not > optimized for the case where there are hundreds of thousands of files in the > same directory. It also makes inspecting directories with standard UNIX > tools difficult. > Similar to the git version control system, HDFS should create a few different > top level directories keyed off of a few bits in the chunk ID. Git uses 8 > bits. This substantially cuts down on the number of chunk files in the same > directory and gives increased performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-3263: - Attachment: HDFS-3263.patch Thanks Eli. Attaching patch addressing your second NIT. The first NIT is not a NIT, the property name is 'httpfs.hadoop.config.dir', there is name aggregation of server/service. > HttpFS should read HDFS config from Hadoop site.xml files > - > > Key: HDFS-3263 > URL: https://issues.apache.org/jira/browse/HDFS-3263 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.0 > > Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch, > HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch > > > Currently HttpFS reads HDFS client configuration from the httfs-site.xml from > any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' > This is a bit inconvenient. > Instead we should support a single property 'httpfs.hadoop.configuration.dir' > that can be pointed to HADOOP conf/ dir and the core-site.xml and > hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
[ https://issues.apache.org/jira/browse/HDFS-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256802#comment-13256802 ] Todd Lipcon commented on HDFS-3271: --- It's not a bug in the library, but rather a bug in one of the nss backends (sssd). Plus, that would require re-building the native libs on every different version of EL6, whereas right now a single binary works against any EL6 release. > src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc > -- > > Key: HDFS-3271 > URL: https://issues.apache.org/jira/browse/HDFS-3271 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3275) Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
[ https://issues.apache.org/jira/browse/HDFS-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256805#comment-13256805 ] Hadoop QA commented on HDFS-3275: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523215/HDFS-3275.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2297//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2297//console This message is automatically generated. > Format command overwrites contents of non-empty shared edits dir if name dirs > are empty without any prompting > - > > Key: HDFS-3275 > URL: https://issues.apache.org/jira/browse/HDFS-3275 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Vinithra Varadharajan >Assignee: amith > Attachments: HDFS-3275.patch > > > To reproduce: > # Configure a NameNode with namedirs and a shared edits dir, all of which are > empty. > # Run hdfs namenode -format. Namedirs and shared edits dir gets populated. > # Delete the contents of the namedirs. Leave the shared edits dir as is. > Check the timestamps of the shared edits dir contents. > # Run format again. The namedirs as well as the shared edits dir get > formatted. The shared edits dir's contents have been replaced without any > prompting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
