[jira] [Commented] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs
[ https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175686#comment-13175686 ] Uma Maheswara Rao G commented on HDFS-2720: --- Updated a patch, First it will format one NN and copy the dirs to remaining other nodes. After this step it will start all NNs. > HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 > nameSpaceDirs to NN2 nameSpaceDirs > > > Key: HDFS-2720 > URL: https://issues.apache.org/jira/browse/HDFS-2720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2720.patch > > > To maintain the clusterID same , we are copying the namespaceDirs from 1st NN > to other NNs. > While copying this files, in_use.lock file may not allow to copy in all the > OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs
[ https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2720: -- Attachment: HDFS-2720.patch > HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 > nameSpaceDirs to NN2 nameSpaceDirs > > > Key: HDFS-2720 > URL: https://issues.apache.org/jira/browse/HDFS-2720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2720.patch > > > To maintain the clusterID same , we are copying the namespaceDirs from 1st NN > to other NNs. > While copying this files, in_use.lock file may not allow to copy in all the > OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1314) dfs.block.size accepts only absolute value
[ https://issues.apache.org/jira/browse/HDFS-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175682#comment-13175682 ] Harsh J commented on HDFS-1314: --- Where possible, lets use FileSystem.getDefaultBlockSize(); and have that do the work underneath. > dfs.block.size accepts only absolute value > -- > > Key: HDFS-1314 > URL: https://issues.apache.org/jira/browse/HDFS-1314 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Karim Saadah >Assignee: Sho Shimauchi >Priority: Minor > Labels: newbie > Attachments: hdfs-1314.txt > > > Using "dfs.block.size=8388608" works > but "dfs.block.size=8mb" does not. > Using "dfs.block.size=8mb" should throw some WARNING on NumberFormatException. > (http://pastebin.corp.yahoo.com/56129) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2722) HttpFs shouldn't be using an int for block size
[
https://issues.apache.org/jira/browse/HDFS-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175681#comment-13175681
]
Harsh J commented on HDFS-2722:
---
We should use {{fs.getDefaultBlockSize();}} here.
> HttpFs shouldn't be using an int for block size
> ---
>
> Key: HDFS-2722
> URL: https://issues.apache.org/jira/browse/HDFS-2722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: Harsh J
>Assignee: Sho Shimauchi
>
> {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
> blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
> Should instead be using dfs.blocksize and should instead be long.
> I'll post a patch for this after HDFS-1314 is resolved -- which changes the
> internal behavior a bit (should be getLongBytes, and not just getLong, to
> gain formatting advantages).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1314) dfs.block.size accepts only absolute value
[
https://issues.apache.org/jira/browse/HDFS-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175676#comment-13175676
]
Harsh J commented on HDFS-1314:
---
Some comments:
- {{./hadoop-hdfs-project/hadoop-hdfs/src/main/native/hdfs.c}} needs an update
as well -- both to dfs.blocksize rename and the method signature switch.
- Found multiple issues with HttpFs and opened HDFS-2722 -- as it could get
more involved than this one's scope, need a separate patch for this.
- Please update dfs.blocksize documentations (and also hdfs-default.xml) to
indicate that users may supply k/m/g characters to indicate a block size.
> dfs.block.size accepts only absolute value
> --
>
> Key: HDFS-1314
> URL: https://issues.apache.org/jira/browse/HDFS-1314
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Karim Saadah
>Assignee: Sho Shimauchi
>Priority: Minor
> Labels: newbie
> Attachments: hdfs-1314.txt
>
>
> Using "dfs.block.size=8388608" works
> but "dfs.block.size=8mb" does not.
> Using "dfs.block.size=8mb" should throw some WARNING on NumberFormatException.
> (http://pastebin.corp.yahoo.com/56129)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2722) HttpFs shouldn't be using an int for block size
HttpFs shouldn't be using an int for block size
---
Key: HDFS-2722
URL: https://issues.apache.org/jira/browse/HDFS-2722
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Reporter: Harsh J
{{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
Should instead be using dfs.blocksize.
I'll post a patch for this after HDFS-1314 is resolved -- which changes the
internal behavior a bit (should be getLongBytes, and not just getLong, to gain
formatting advantages).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2722) HttpFs shouldn't be using an int for block size
[
https://issues.apache.org/jira/browse/HDFS-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J reassigned HDFS-2722:
-
Assignee: Harsh J
> HttpFs shouldn't be using an int for block size
> ---
>
> Key: HDFS-2722
> URL: https://issues.apache.org/jira/browse/HDFS-2722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: Harsh J
>Assignee: Harsh J
>
> {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
> blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
> Should instead be using dfs.blocksize and should instead be long.
> I'll post a patch for this after HDFS-1314 is resolved -- which changes the
> internal behavior a bit (should be getLongBytes, and not just getLong, to
> gain formatting advantages).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2722) HttpFs shouldn't be using an int for block size
[
https://issues.apache.org/jira/browse/HDFS-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J reassigned HDFS-2722:
-
Assignee: Sho Shimauchi (was: Harsh J)
> HttpFs shouldn't be using an int for block size
> ---
>
> Key: HDFS-2722
> URL: https://issues.apache.org/jira/browse/HDFS-2722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: Harsh J
>Assignee: Sho Shimauchi
>
> {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
> blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
> Should instead be using dfs.blocksize and should instead be long.
> I'll post a patch for this after HDFS-1314 is resolved -- which changes the
> internal behavior a bit (should be getLongBytes, and not just getLong, to
> gain formatting advantages).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2722) HttpFs shouldn't be using an int for block size
[
https://issues.apache.org/jira/browse/HDFS-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-2722:
--
Description:
{{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
Should instead be using dfs.blocksize and should instead be long.
I'll post a patch for this after HDFS-1314 is resolved -- which changes the
internal behavior a bit (should be getLongBytes, and not just getLong, to gain
formatting advantages).
was:
{{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
Should instead be using dfs.blocksize.
I'll post a patch for this after HDFS-1314 is resolved -- which changes the
internal behavior a bit (should be getLongBytes, and not just getLong, to gain
formatting advantages).
> HttpFs shouldn't be using an int for block size
> ---
>
> Key: HDFS-2722
> URL: https://issues.apache.org/jira/browse/HDFS-2722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: Harsh J
>
> {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java:
> blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
> Should instead be using dfs.blocksize and should instead be long.
> I'll post a patch for this after HDFS-1314 is resolved -- which changes the
> internal behavior a bit (should be getLongBytes, and not just getLong, to
> gain formatting advantages).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2721) Balancer tests failing in trunk
[ https://issues.apache.org/jira/browse/HDFS-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175671#comment-13175671 ] Uma Maheswara Rao G commented on HDFS-2721: --- Looks TestBalancer is timing out. But it is passing in my local Box! > Balancer tests failing in trunk > --- > > Key: HDFS-2721 > URL: https://issues.apache.org/jira/browse/HDFS-2721 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G > > Looks Balancer tests started failing. > https://builds.apache.org/job/Hadoop-Hdfs-trunk/lastCompletedBuild/testReport/ > Also seen timing out of TestBalancer in precommit build > https://builds.apache.org/job/PreCommit-HDFS-Build/1737//console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[
https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175664#comment-13175664
]
Uma Maheswara Rao G commented on HDFS-2712:
---
Failures looks to be due to Balancer tests time out
{quote}
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.302 sec
Running org.apache.hadoop.hdfs.server.balancer.TestBalancer
Running org.apache.hadoop.hdfs.web.TestOffsetUrlInputStream
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.567 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.199 sec <<<
FAILURE!
Running org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.274 sec <<<
FAILURE!
{quote}
In recent build balancer tests has broken in Jenkins.
https://builds.apache.org/job/Hadoop-Hdfs-trunk/lastCompletedBuild/testReport/
Filed a JIRA for Balancer tests HDFS-2721
findbugs also unrelated.
> setTimes should support only for files and move the atime field down to
> iNodeFile.
> --
>
> Key: HDFS-2712
> URL: https://issues.apache.org/jira/browse/HDFS-2712
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2712.patch
>
>
> After the dicussion in HDFS-2436, unsupported behaviour for setTimes was
> intentional (HADOOP-1869).
> But current INode structure hierarchy seems, it supports atime for
> directories also. But as per HADOOP-1869, we are supporting only for files.
> To avoid the confusions, we can move the atime fields to INodeFile as we
> planned to support setTimes only for files. And also restrict the support for
> setTimes on directories ( which is implemented with HDFS-2436 ).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2721) Balancer tests failing in trunk
Balancer tests failing in trunk --- Key: HDFS-2721 URL: https://issues.apache.org/jira/browse/HDFS-2721 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.24.0 Reporter: Uma Maheswara Rao G Looks Balancer tests started failing. https://builds.apache.org/job/Hadoop-Hdfs-trunk/lastCompletedBuild/testReport/ Also seen timing out of TestBalancer in precommit build https://builds.apache.org/job/PreCommit-HDFS-Build/1737//console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[ https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175586#comment-13175586 ] Hadoop QA commented on HDFS-2712: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508558/HDFS-2712.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1737//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1737//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1737//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1737//console This message is automatically generated. > setTimes should support only for files and move the atime field down to > iNodeFile. > -- > > Key: HDFS-2712 > URL: https://issues.apache.org/jira/browse/HDFS-2712 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2712.patch > > > After the dicussion in HDFS-2436, unsupported behaviour for setTimes was > intentional (HADOOP-1869). > But current INode structure hierarchy seems, it supports atime for > directories also. But as per HADOOP-1869, we are supporting only for files. > To avoid the confusions, we can move the atime fields to INodeFile as we > planned to support setTimes only for files. And also restrict the support for > setTimes on directories ( which is implemented with HDFS-2436 ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[ https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2712: -- Status: Patch Available (was: Open) > setTimes should support only for files and move the atime field down to > iNodeFile. > -- > > Key: HDFS-2712 > URL: https://issues.apache.org/jira/browse/HDFS-2712 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2712.patch > > > After the dicussion in HDFS-2436, unsupported behaviour for setTimes was > intentional (HADOOP-1869). > But current INode structure hierarchy seems, it supports atime for > directories also. But as per HADOOP-1869, we are supporting only for files. > To avoid the confusions, we can move the atime fields to INodeFile as we > planned to support setTimes only for files. And also restrict the support for > setTimes on directories ( which is implemented with HDFS-2436 ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs
[
https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175542#comment-13175542
]
Uma Maheswara Rao G commented on HDFS-2720:
---
Thanks a lot,Todd for the suggestions.
{quote}
For testing on actual clusters, I've done this by shutting down the active NN,
then just rsyncing the storage dir to the standby, then starting the standby.
{quote}
I feel we should automate this once we built the automatic HA right. We were
depending on zookeeper to store namespaceID(in our internal HA). But i am not
sure how we can handle it in linux HA case.
Shall i file one JIRA for it?
Thanks
Uma
> HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1
> nameSpaceDirs to NN2 nameSpaceDirs
>
>
> Key: HDFS-2720
> URL: https://issues.apache.org/jira/browse/HDFS-2720
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> To maintain the clusterID same , we are copying the namespaceDirs from 1st NN
> to other NNs.
> While copying this files, in_use.lock file may not allow to copy in all the
> OSs since it has aquired the lock on it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[ https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175537#comment-13175537 ] Uma Maheswara Rao G commented on HDFS-2712: --- 1st Patch : HDFS-2712 1) UnsupportedActionException will be thrown if we try to setTimes on directories 2) Moved the atime field to iNodeFile obj. 3) also moved atime down to iNodeSymlink, i am not sure whether really a time required on iNodeSymlink. But not to break any functionality, i just added atime to it as well. Thanks Uma > setTimes should support only for files and move the atime field down to > iNodeFile. > -- > > Key: HDFS-2712 > URL: https://issues.apache.org/jira/browse/HDFS-2712 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2712.patch > > > After the dicussion in HDFS-2436, unsupported behaviour for setTimes was > intentional (HADOOP-1869). > But current INode structure hierarchy seems, it supports atime for > directories also. But as per HADOOP-1869, we are supporting only for files. > To avoid the confusions, we can move the atime fields to INodeFile as we > planned to support setTimes only for files. And also restrict the support for > setTimes on directories ( which is implemented with HDFS-2436 ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[ https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2712: -- Attachment: HDFS-2712.patch > setTimes should support only for files and move the atime field down to > iNodeFile. > -- > > Key: HDFS-2712 > URL: https://issues.apache.org/jira/browse/HDFS-2712 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2712.patch > > > After the dicussion in HDFS-2436, unsupported behaviour for setTimes was > intentional (HADOOP-1869). > But current INode structure hierarchy seems, it supports atime for > directories also. But as per HADOOP-1869, we are supporting only for files. > To avoid the confusions, we can move the atime fields to INodeFile as we > planned to support setTimes only for files. And also restrict the support for > setTimes on directories ( which is implemented with HDFS-2436 ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[ https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2712: -- Attachment: (was: HDFS-2712.patch) > setTimes should support only for files and move the atime field down to > iNodeFile. > -- > > Key: HDFS-2712 > URL: https://issues.apache.org/jira/browse/HDFS-2712 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2712.patch > > > After the dicussion in HDFS-2436, unsupported behaviour for setTimes was > intentional (HADOOP-1869). > But current INode structure hierarchy seems, it supports atime for > directories also. But as per HADOOP-1869, we are supporting only for files. > To avoid the confusions, we can move the atime fields to INodeFile as we > planned to support setTimes only for files. And also restrict the support for > setTimes on directories ( which is implemented with HDFS-2436 ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2712) setTimes should support only for files and move the atime field down to iNodeFile.
[ https://issues.apache.org/jira/browse/HDFS-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2712: -- Attachment: HDFS-2712.patch > setTimes should support only for files and move the atime field down to > iNodeFile. > -- > > Key: HDFS-2712 > URL: https://issues.apache.org/jira/browse/HDFS-2712 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2712.patch > > > After the dicussion in HDFS-2436, unsupported behaviour for setTimes was > intentional (HADOOP-1869). > But current INode structure hierarchy seems, it supports atime for > directories also. But as per HADOOP-1869, we are supporting only for files. > To avoid the confusions, we can move the atime fields to INodeFile as we > planned to support setTimes only for files. And also restrict the support for > setTimes on directories ( which is implemented with HDFS-2436 ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs
[ https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175532#comment-13175532 ] Todd Lipcon commented on HDFS-2720: --- For testing on actual clusters, I've done this by shutting down the active NN, then just rsyncing the storage dir to the standby, then starting the standby. Your idea of skipping in_use.lock is one solution for MiniDFSCluster. The other solution would be to copy the storage dir to all the standbys before starting the first NN. But that might break "addNameNode" support in HA - maybe not a big deal since I don't think we use that in HA cluster tests at the moment. > HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 > nameSpaceDirs to NN2 nameSpaceDirs > > > Key: HDFS-2720 > URL: https://issues.apache.org/jira/browse/HDFS-2720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > To maintain the clusterID same , we are copying the namespaceDirs from 1st NN > to other NNs. > While copying this files, in_use.lock file may not allow to copy in all the > OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2698) BackupNode is downloading image from NameNode for every checkpoint
[ https://issues.apache.org/jira/browse/HDFS-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175529#comment-13175529 ] Konstantin Shvachko commented on HDFS-2698: --- This is specific for 0.22 as trunk doesn't have rollFSImage() and no checkpoint time. > BackupNode is downloading image from NameNode for every checkpoint > -- > > Key: HDFS-2698 > URL: https://issues.apache.org/jira/browse/HDFS-2698 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Attachments: rollFSImage.patch, rollFSImage.patch > > > BackupNode can make periodic checkpoints without downloading image and edits > files from the NameNode, but with just saving the namespace to local disks. > This is not happening because NN renews checkpoint time after every > checkpoint, thus making its image ahead of the BN's even though they are in > sync. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs
[ https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2720: -- Summary: HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 nameSpaceDirs to NN2 nameSpaceDirs (was: HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to NN2 ) > HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 > nameSpaceDirs to NN2 nameSpaceDirs > > > Key: HDFS-2720 > URL: https://issues.apache.org/jira/browse/HDFS-2720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > To maintain the clusterID same , we are copying the namespaceDirs from 1st NN > to other NNs. > While copying this files, in_use.lock file may not allow to copy in all the > OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to NN2
[ https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175375#comment-13175375 ] Uma Maheswara Rao G commented on HDFS-2720: --- One way to fix this is, we can just skip the in_use.lock file from copying the NamespaceDirs. Other way could be, manage the clusterID to be insync between active and format all as initial as real clusters does. @Todd, i want to know how we planned to sync the cluster id between active and standby nodes here. > HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to > NN2 > > > Key: HDFS-2720 > URL: https://issues.apache.org/jira/browse/HDFS-2720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > To maintain the clusterID same , we are copying the namespaceDirs from 1st NN > to other NNs. > While copying this files, in_use.lock file may not allow to copy in all the > OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to NN2
[ https://issues.apache.org/jira/browse/HDFS-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175371#comment-13175371 ] Uma Maheswara Rao G commented on HDFS-2720: --- Trace: java.io.IOException: The process cannot access the file because another process has locked a portion of the file at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:177) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:75) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:49) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:95) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329) at org.apache.hadoop.hdfs.MiniDFSCluster.copyNameDirs(MiniDFSCluster.java:671) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:616) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:542) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:274) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:263) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:256) at org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyIsHot.testStandbyIsHot Other HA related tests also fails with the same reason. > HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to > NN2 > > > Key: HDFS-2720 > URL: https://issues.apache.org/jira/browse/HDFS-2720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > To maintain the clusterID same , we are copying the namespaceDirs from 1st NN > to other NNs. > While copying this files, in_use.lock file may not allow to copy in all the > OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2720) HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to NN2
HA : TestStandbyIsHot is failing while copying in_use.lock file from NN1 to NN2 Key: HDFS-2720 URL: https://issues.apache.org/jira/browse/HDFS-2720 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G To maintain the clusterID same , we are copying the namespaceDirs from 1st NN to other NNs. While copying this files, in_use.lock file may not allow to copy in all the OSs since it has aquired the lock on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
