[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302777#comment-14302777 ] sam liu commented on HDFS-7630: --- Hi Arpit, they pass on power platform so far, but it will be better if we could consider the differences of the page size of operating systems in regarding tests. > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294898#comment-14294898 ] sam liu commented on HDFS-7630: --- Arpit, Not all such hard-coding will cause failure and the patch is mainly to remove the hard-coding. But sometimes the hard-coding could cause failure. For example, without patch HDFS-7585, test TestEnhancedByteBufferAccess will fail on power platform. In the tests, the size of BLOCK_SIZE is usually set as 4096 and it just equals to the default page size of x86 Linux operating system, but on power Linux operating system the default page size is 65536. As HDFS is based on operating system, it might be better the unit tests could consider the differences of operating systems. Thanks! > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293539#comment-14293539 ] sam liu commented on HDFS-7630: --- Hi Arpit, According to your comments, I combined all fixes into HDFS-7630.002.patch. All the unit tests passed excepting TestBlockScanner, however the failure is not caused by current patch. Could you please help take a review? Thanks! > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7630: -- Attachment: HDFS-7630.002.patch Combined the patches of HDFS-7624 through HDFS-7630 into one(HDFS-7630.002.patch). > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281363#comment-14281363 ] sam liu commented on HDFS-7624: --- The failure of TestRetryCacheWithHA should not be caused by the patch, because TestRetryCacheWithHA does not invoke any API of current test TestFileAppendRestart. > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch, HDFS-7624.002.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7624: -- Attachment: HDFS-7624.002.patch > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch, HDFS-7624.002.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281120#comment-14281120 ] sam liu commented on HDFS-7624: --- Hi Konstantin, The "power platform" means IBM power system/server and the Linux OS for power has some differences from Linux OS for x86 system. In the test, 'BLOCK_SIZE = 4096' does not mean the HDFS block size is 4096, but mean the operating system page size is 4096. The page size of operating systems are different, so we should not hardcode the page size in the test. The correct way is to get the page size using 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7624: -- Status: Open (was: Patch Available) > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280270#comment-14280270 ] sam liu commented on HDFS-7630: --- The failure of TestFileTruncate should not be caused by the patch, because: - TestFileTruncate does not invoke any API of current test TestConnCache - Actually, in my test env, TestFileTruncate also failed without applying any patch > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7624: -- Status: Patch Available (was: Open) > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7625) TestPersistBlocks hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279980#comment-14279980 ] sam liu commented on HDFS-7625: --- The failure of TestBalancer should not be caused by the patch, because TestBalancer does not invoke any API of current test TestPersistBlocks. In my test env, TestBalancer passed with applying current patch. > TestPersistBlocks hardcode block size without considering native OS > --- > > Key: HDFS-7625 > URL: https://issues.apache.org/jira/browse/HDFS-7625 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7625.001.patch > > > TestPersistBlocks hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7628) TestNameEditsConfigs hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279968#comment-14279968 ] sam liu commented on HDFS-7628: --- The failure of TestFileTruncate should not be caused by the patch, because: - TestFileTruncate does not invoke any API of current test TestNameEditsConfigs - Actually, in my test env, TestFileTruncate also failed without applying any patch > TestNameEditsConfigs hardcode block size without considering native OS > -- > > Key: HDFS-7628 > URL: https://issues.apache.org/jira/browse/HDFS-7628 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7628.001.patch > > > TestNameEditsConfigs hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7630: -- Status: Patch Available (was: Open) > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7630) TestConnCache hardcode block size without considering native OS
sam liu created HDFS-7630: - Summary: TestConnCache hardcode block size without considering native OS Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7630: -- Attachment: HDFS-7630.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestConnCache hardcode block size without considering native OS > --- > > Key: HDFS-7630 > URL: https://issues.apache.org/jira/browse/HDFS-7630 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7630.001.patch > > > TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7629) TestDisableConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7629: -- Status: Patch Available (was: Open) > TestDisableConnCache hardcode block size without considering native OS > -- > > Key: HDFS-7629 > URL: https://issues.apache.org/jira/browse/HDFS-7629 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7629.001.patch > > > TestDisableConnCache hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7629) TestDisableConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7629: -- Attachment: HDFS-7629.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestDisableConnCache hardcode block size without considering native OS > -- > > Key: HDFS-7629 > URL: https://issues.apache.org/jira/browse/HDFS-7629 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7629.001.patch > > > TestDisableConnCache hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7629) TestDisableConnCache hardcode block size without considering native OS
sam liu created HDFS-7629: - Summary: TestDisableConnCache hardcode block size without considering native OS Key: HDFS-7629 URL: https://issues.apache.org/jira/browse/HDFS-7629 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestDisableConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7628) TestNameEditsConfigs hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7628: -- Status: Patch Available (was: Open) > TestNameEditsConfigs hardcode block size without considering native OS > -- > > Key: HDFS-7628 > URL: https://issues.apache.org/jira/browse/HDFS-7628 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7628.001.patch > > > TestNameEditsConfigs hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7628) TestNameEditsConfigs hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7628: -- Attachment: HDFS-7628.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestNameEditsConfigs hardcode block size without considering native OS > -- > > Key: HDFS-7628 > URL: https://issues.apache.org/jira/browse/HDFS-7628 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7628.001.patch > > > TestNameEditsConfigs hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7628) TestNameEditsConfigs hardcode block size without considering native OS
sam liu created HDFS-7628: - Summary: TestNameEditsConfigs hardcode block size without considering native OS Key: HDFS-7628 URL: https://issues.apache.org/jira/browse/HDFS-7628 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestNameEditsConfigs hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7627) TestCacheDirectives hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7627: -- Status: Patch Available (was: Open) > TestCacheDirectives hardcode block size without considering native OS > - > > Key: HDFS-7627 > URL: https://issues.apache.org/jira/browse/HDFS-7627 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7627.001.patch > > > TestCacheDirectives hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7627) TestCacheDirectives hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7627: -- Attachment: HDFS-7627.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestCacheDirectives hardcode block size without considering native OS > - > > Key: HDFS-7627 > URL: https://issues.apache.org/jira/browse/HDFS-7627 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7627.001.patch > > > TestCacheDirectives hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7627) TestCacheDirectives hardcode block size without considering native OS
sam liu created HDFS-7627: - Summary: TestCacheDirectives hardcode block size without considering native OS Key: HDFS-7627 URL: https://issues.apache.org/jira/browse/HDFS-7627 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestCacheDirectives hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7626) TestPipelinesFailover hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7626: -- Status: Patch Available (was: Open) > TestPipelinesFailover hardcode block size without considering native OS > --- > > Key: HDFS-7626 > URL: https://issues.apache.org/jira/browse/HDFS-7626 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7626.001.patch > > > TestPipelinesFailover hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7626) TestPipelinesFailover hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7626: -- Attachment: HDFS-7626.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestPipelinesFailover hardcode block size without considering native OS > --- > > Key: HDFS-7626 > URL: https://issues.apache.org/jira/browse/HDFS-7626 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7626.001.patch > > > TestPipelinesFailover hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7626) TestPipelinesFailover hardcode block size without considering native OS
sam liu created HDFS-7626: - Summary: TestPipelinesFailover hardcode block size without considering native OS Key: HDFS-7626 URL: https://issues.apache.org/jira/browse/HDFS-7626 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestPipelinesFailover hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7625) TestPersistBlocks hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7625: -- Status: Patch Available (was: Open) > TestPersistBlocks hardcode block size without considering native OS > --- > > Key: HDFS-7625 > URL: https://issues.apache.org/jira/browse/HDFS-7625 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7625.001.patch > > > TestPersistBlocks hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7625) TestPersistBlocks hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7625: -- Attachment: HDFS-7625.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestPersistBlocks hardcode block size without considering native OS > --- > > Key: HDFS-7625 > URL: https://issues.apache.org/jira/browse/HDFS-7625 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7625.001.patch > > > TestPersistBlocks hardcode block size with 'BLOCK_SIZE = 4096', however it's > incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7625) TestPersistBlocks hardcode block size without considering native OS
sam liu created HDFS-7625: - Summary: TestPersistBlocks hardcode block size without considering native OS Key: HDFS-7625 URL: https://issues.apache.org/jira/browse/HDFS-7625 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestPersistBlocks hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7624: -- Status: Patch Available (was: Open) > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7624: -- Attachment: HDFS-7624.001.patch Remove the hardcode on block size, and use 'NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize()' instead. > TestFileAppendRestart hardcode block size without considering native OS > --- > > Key: HDFS-7624 > URL: https://issues.apache.org/jira/browse/HDFS-7624 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7624.001.patch > > > TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however > it's incorrect on some platforms. For example, on power platform, the correct > value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
sam liu created HDFS-7624: - Summary: TestFileAppendRestart hardcode block size without considering native OS Key: HDFS-7624 URL: https://issues.apache.org/jira/browse/HDFS-7624 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[ https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273575#comment-14273575 ] sam liu commented on HDFS-7585: --- Hi Colin, updated the patch and could you please help take a review? Thanks! > TestEnhancedByteBufferAccess hard code the block size > - > > Key: HDFS-7585 > URL: https://issues.apache.org/jira/browse/HDFS-7585 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu > Attachments: HDFS-7585.001.patch, HDFS-7585.002.patch > > > The test TestEnhancedByteBufferAccess hard code the block size, and it fails > with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[
https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sam liu updated HDFS-7585:
--
Attachment: HDFS-7585.002.patch
According to Colin's suggestion, replaced
{code}
private static final int BLOCK_SIZE = 4096;
{code}
with
{code}
private static final int BLOCK_SIZE =
NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize();
{code}
Furthermore, made some other regarding modifications:
- Replaced all block size hard codes. For example, replace '4096' with
'BLOCK_SIZE'
- Updated some other codes accordingly
> TestEnhancedByteBufferAccess hard code the block size
> -
>
> Key: HDFS-7585
> URL: https://issues.apache.org/jira/browse/HDFS-7585
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: sam liu
>Assignee: sam liu
> Attachments: HDFS-7585.001.patch, HDFS-7585.002.patch
>
>
> The test TestEnhancedByteBufferAccess hard code the block size, and it fails
> with exceptions on power linux.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[ https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267151#comment-14267151 ] sam liu commented on HDFS-7585: --- Could please help review this patch? Thanks! > TestEnhancedByteBufferAccess hard code the block size > - > > Key: HDFS-7585 > URL: https://issues.apache.org/jira/browse/HDFS-7585 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Blocker > Attachments: HDFS-7585.001.patch > > > The test TestEnhancedByteBufferAccess hard code the block size, and it fails > with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[ https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7585: -- Status: Open (was: Patch Available) > TestEnhancedByteBufferAccess hard code the block size > - > > Key: HDFS-7585 > URL: https://issues.apache.org/jira/browse/HDFS-7585 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Blocker > Attachments: HDFS-7585.001.patch > > > The test TestEnhancedByteBufferAccess hard code the block size, and it fails > with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[ https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7585: -- Status: Patch Available (was: Open) > TestEnhancedByteBufferAccess hard code the block size > - > > Key: HDFS-7585 > URL: https://issues.apache.org/jira/browse/HDFS-7585 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Blocker > Attachments: HDFS-7585.001.patch > > > The test TestEnhancedByteBufferAccess hard code the block size, and it fails > with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[ https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7585: -- Attachment: HDFS-7585.001.patch > TestEnhancedByteBufferAccess hard code the block size > - > > Key: HDFS-7585 > URL: https://issues.apache.org/jira/browse/HDFS-7585 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Blocker > Attachments: HDFS-7585.001.patch > > > The test TestEnhancedByteBufferAccess hard code the block size, and it fails > with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
[ https://issues.apache.org/jira/browse/HDFS-7585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-7585: -- Status: Patch Available (was: Open) The solution is to remove the hard-code of block size and use native OS page size instead. In this way, this test could pass on both x86 platform and power platform. > TestEnhancedByteBufferAccess hard code the block size > - > > Key: HDFS-7585 > URL: https://issues.apache.org/jira/browse/HDFS-7585 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Blocker > Attachments: HDFS-7585.001.patch > > > The test TestEnhancedByteBufferAccess hard code the block size, and it fails > with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
sam liu created HDFS-7585: - Summary: TestEnhancedByteBufferAccess hard code the block size Key: HDFS-7585 URL: https://issues.apache.org/jira/browse/HDFS-7585 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Blocker The test TestEnhancedByteBufferAccess hard code the block size, and it fails with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5526) Datanode cannot roll back to previous layout version
[ https://issues.apache.org/jira/browse/HDFS-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142992#comment-14142992 ] sam liu commented on HDFS-5526: --- I also encountered similar error on the secondary namenode and opened another jira HDFS-7114 > Datanode cannot roll back to previous layout version > > > Key: HDFS-5526 > URL: https://issues.apache.org/jira/browse/HDFS-5526 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Tsz Wo Nicholas Sze >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 0.23.10, 2.3.0 > > Attachments: HDFS-5526.patch, HDFS-5526.patch > > > Current trunk layout version is -48. > Hadoop v2.2.0 layout version is -47. > If a cluster is upgraded from v2.2.0 (-47) to trunk (-48), the datanodes > cannot start with -rollback. It will fail with IncorrectVersionException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7114) Secondary NameNode failed to rollback from 2.4.1 to 2.2.0
[ https://issues.apache.org/jira/browse/HDFS-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142863#comment-14142863 ] sam liu commented on HDFS-7114: --- Similar issue encountered on datanode as well, and that issue is resolved by jira HDFS-5526. > Secondary NameNode failed to rollback from 2.4.1 to 2.2.0 > - > > Key: HDFS-7114 > URL: https://issues.apache.org/jira/browse/HDFS-7114 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: sam liu >Priority: Blocker > > Can upgrade from 2.2.0 to 2.4.1, but failed to rollback the secondary > namenode with following issue. > 2014-09-22 10:41:28,358 FATAL > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start > secondary namenode > org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected > version of storage directory /var/hadoop/tmp/hdfs/dfs/namesecondary. > Reported: -56. Expecting = -47. > at > org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1082) > at > org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:890) > at > org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:585) > at > org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:921) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.recoverCreate(SecondaryNameNode.java:913) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:249) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.(SecondaryNameNode.java:199) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:652) > 2014-09-22 10:41:28,360 INFO org.apache.hadoop.util.ExitUtil: Exiting with > status 1 > 2014-09-22 10:41:28,363 INFO > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7114) Secondary NameNode failed to rollback from 2.4.1 to 2.2.0
sam liu created HDFS-7114: - Summary: Secondary NameNode failed to rollback from 2.4.1 to 2.2.0 Key: HDFS-7114 URL: https://issues.apache.org/jira/browse/HDFS-7114 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: sam liu Priority: Blocker Can upgrade from 2.2.0 to 2.4.1, but failed to rollback the secondary namenode with following issue. 2014-09-22 10:41:28,358 FATAL org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start secondary namenode org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /var/hadoop/tmp/hdfs/dfs/namesecondary. Reported: -56. Expecting = -47. at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1082) at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:890) at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:585) at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:921) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.recoverCreate(SecondaryNameNode.java:913) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:249) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.(SecondaryNameNode.java:199) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:652) 2014-09-22 10:41:28,360 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2014-09-22 10:41:28,363 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7053) Failed to rollback hdfs version from 2.4.1 to 2.2.0
[ https://issues.apache.org/jira/browse/HDFS-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136871#comment-14136871 ] sam liu commented on HDFS-7053: --- Hi Jing, Thanks for your comments! However, I still failed to rollback hadoop version on a non HA cluster. I doubt this is a bug or did I miss any step? Below are my steps: [A] Upgrade 1. Install Hadoop 2.2.0 cluster 2. Stop Hadoop services 3. Replace 2.2.0 binaries with 2.4.1 binaries 4. Start datanodes: $HADOOP_HOME/sbin/hadoop-daemon.sh start datanode 5. Start namenode with option upgrade: $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode -upgrade 6. Start secondary namenode, tasktracker and jobtracker Result: - Whole upgrade process could be completed successfully. [B] Rollback 1. Stop all hadoop services 2. Replace 2.4.1 binaries with 2.2.0 binaries 3. Start datanodes: $HADOOP_HOME/sbin/hadoop-daemon.sh start datanode 4. Start namenode with option upgrade: $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode -rollback Result: - Namenode service could be started - Datanodes failed with exception: Issue: DataNode failed with following exception 2014-09-17 11:04:51,416 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /hadoop/hdfs/data/in_use.lock acquired by nodename 817443@shihc071-public 2014-09-17 11:04:51,418 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-977402492-9.181.64.185-1410497086460 (storage id ) service to hostname/ip:9000 org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /hadoop/hdfs/data. Reported: -55. Expecting = -47. at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataStorage.setFieldsFromProperties(DataStorage.java:302) at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:921) Thanks! > Failed to rollback hdfs version from 2.4.1 to 2.2.0 > --- > > Key: HDFS-7053 > URL: https://issues.apache.org/jira/browse/HDFS-7053 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.4.1 >Reporter: sam liu >Priority: Blocker > > I can successfully upgrade from 2.2.0 to 2.4.1 with QJM HA enabled and with > downtime, but failed to rollback from 2.4.1 to 2.2.0. The error message: > 2014-09-10 16:50:29,599 FATAL > org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join > org.apache.hadoop.HadoopIllegalArgumentException: Invalid startup option. > Cannot perform DFS upgrade with HA enabled. > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1207) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320) > 2014-09-10 16:50:29,601 INFO org.apache.hadoop.util.ExitUtil: Exiting with > status 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7053) Failed to rollback hdfs version from 2.4.1 to 2.2.0
sam liu created HDFS-7053: - Summary: Failed to rollback hdfs version from 2.4.1 to 2.2.0 Key: HDFS-7053 URL: https://issues.apache.org/jira/browse/HDFS-7053 Project: Hadoop HDFS Issue Type: Bug Components: ha, namenode Affects Versions: 2.4.1 Reporter: sam liu Priority: Blocker I can successfully upgrade from 2.2.0 to 2.4.1 with QJM HA enabled and with downtime, but failed to rollback from 2.4.1 to 2.2.0. The error message: 2014-09-10 16:50:29,599 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join org.apache.hadoop.HadoopIllegalArgumentException: Invalid startup option. Cannot perform DFS upgrade with HA enabled. at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1207) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320) 2014-09-10 16:50:29,601 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7002) Failed to rolling upgrade hdfs from 2.2.0 to 2.4.1
[ https://issues.apache.org/jira/browse/HDFS-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14122697#comment-14122697 ] sam liu commented on HDFS-7002: --- Failed to rolling upgrade hdfs from 2.2.0 to 2.4.1, according to http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Upgrading_Non-Federated_Clusters. [A] Cluster Info: Name Nodes: host2(active), host3(standby) QJM Nodes: host1, host4, host5 Data Nodes: host4, host5 [B] Issue Reproduce Steps: 1. stop standy namenode on host3 2. replace hadoop-2.2.0 binary with hadoop-2.4.1 binary 3. failed to execute command 'hdfs namenode -initializeSharedEdits' with following exception, accordint to last comment of HDFS-5550. 14/09/05 16:31:49 INFO namenode.AclConfigFlag: ACLs enabled? false 14/09/05 16:31:49 INFO common.Storage: Lock on /hadoop/hdfs/name/in_use.lock acquired by nodename 155436@host3 14/09/05 16:31:49 WARN namenode.FSNamesystem: Encountered exception loading fsimage java.io.IOException: File system image contains an old layout version -47. An upgrade to version -56 is required. Please restart NameNode with the "-rollingUpgrade started" option if a rolling upgraded is already started; or restart NameNode with the "-upgrade" option to start a new upgrade. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:224) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638) at org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:980) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1340) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1424) 14/09/05 16:31:49 ERROR namenode.NameNode: Could not initialize shared edits dir 4. failed to execute command 'hdfs namenode -rollingUpgrade started' 14/09/05 16:32:22 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://host4:8480/getJournal?jid=BICluster&segmentTxId=2570&storageInfo=-56%3A1686370894%3A0%3ACID-70b084bc-5e08-402e-b2e5-f4461d7dc69c' to transaction ID 2570 14/09/05 16:32:22 ERROR namenode.EditLogInputStream: caught exception initializing http://host4:8480/getJournal?jid=BICluster&segmentTxId=2570&storageInfo=-56%3A1686370894%3A0%3ACID-70b084bc-5e08-402e-b2e5-f4461d7dc69c org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://host4:8480/getJournal?jid=BICluster&segmentTxId=2570&storageInfo=-56%3A1686370894%3A0%3ACID-70b084bc-5e08-402e-b2e5-f4461d7dc69c failed with status code 403 Response message: This node has storage info '-47:1686370894:0:CID-70b084bc-5e08-402e-b2e5-f4461d7dc69c' but the requesting node expected '-56:1686370894:0:CID-70b084bc-5e08-402e-b2e5-f4461d7dc69c' at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:472) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:460) at java.security.AccessController.doPrivileged(AccessController.java:366) at javax.security.auth.Subject.doAs(Subject.java:572) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1558) at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448) at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:459) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:145) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:196) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:254) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords
[jira] [Created] (HDFS-7002) Failed to rolling upgrade hdfs from 2.2.0 to 2.4.1
sam liu created HDFS-7002: - Summary: Failed to rolling upgrade hdfs from 2.2.0 to 2.4.1 Key: HDFS-7002 URL: https://issues.apache.org/jira/browse/HDFS-7002 Project: Hadoop HDFS Issue Type: Bug Components: journal-node, namenode, qjm Affects Versions: 2.4.1, 2.2.0 Reporter: sam liu Priority: Blocker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-5046) Hang when add/remove a datanode into/from a 2 datanode cluster
sam liu created HDFS-5046: - Summary: Hang when add/remove a datanode into/from a 2 datanode cluster Key: HDFS-5046 URL: https://issues.apache.org/jira/browse/HDFS-5046 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 1.1.1 Environment: Red Hat Enterprise Linux Server release 5.3, 64 bit Reporter: sam liu 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in hdfs-site.xml, set the 'dfs.replication' to 2 2. Add node dn3 into the cluster as a new datanode, and did not change the 'dfs.replication' value in hdfs-site.xml and keep it as 2 note: step 2 passed 3. Decommission dn3 from the cluster Expected result: dn3 could be decommissioned successfully Actual result: a). decommission progress hangs and the status always be 'Waiting DataNode status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the decommission continues and will be completed finally. b). However, if the initial cluster includes >= 3 datanodes, this issue won't be encountered when add/remove another datanode. For example, if I setup a cluster with 3 datanodes, and then I can successfully add the 4th datanode into it, and then also can successfully remove the 4th datanode from the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4527) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/HDFS-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585297#comment-13585297 ] sam liu commented on HDFS-4527: --- Added a draft patch for reference > For shortening the time of TaskTracker heartbeat, decouple the statics > collection operations > > > Key: HDFS-4527 > URL: https://issues.apache.org/jira/browse/HDFS-4527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 1.1.1 >Reporter: sam liu > Labels: patch > Fix For: 1.1.1 > > Attachments: HDFS-4527.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In each heartbeat of TaskTracker, it will calculate some system statics, like > the free disk space, available virtual/physical memory, cpu usage, etc. > However, it's not necessary to calculate all the statics in every heartbeat, > and this will consume many system resource and impace the performance of > TaskTracker heartbeat. Furthermore, the characteristics of system > properties(disk, memory, cpu) are different and it's better to collect their > statics in different intervals. > To reduce the latency of TaskTracker heartbeat, one solution is to decouple > all the system statics collection operations from it, and issue separate > threads to do the statics collection works when the TaskTracker starts. The > threads could be three: the first one is to collect cpu related statics in a > short interval; the second one is to collect memory related statics in a > normal interval; the third one is to collect disk related statics in a long > interval. And all the interval could be customized by the parameter > "mapred.stats.collection.interval" in the mapred-site.xml. At last, the > heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4527) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/HDFS-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-4527: -- Attachment: HDFS-4527.patch > For shortening the time of TaskTracker heartbeat, decouple the statics > collection operations > > > Key: HDFS-4527 > URL: https://issues.apache.org/jira/browse/HDFS-4527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 1.1.1 >Reporter: sam liu > Labels: patch > Fix For: 1.1.1 > > Attachments: HDFS-4527.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In each heartbeat of TaskTracker, it will calculate some system statics, like > the free disk space, available virtual/physical memory, cpu usage, etc. > However, it's not necessary to calculate all the statics in every heartbeat, > and this will consume many system resource and impace the performance of > TaskTracker heartbeat. Furthermore, the characteristics of system > properties(disk, memory, cpu) are different and it's better to collect their > statics in different intervals. > To reduce the latency of TaskTracker heartbeat, one solution is to decouple > all the system statics collection operations from it, and issue separate > threads to do the statics collection works when the TaskTracker starts. The > threads could be three: the first one is to collect cpu related statics in a > short interval; the second one is to collect memory related statics in a > normal interval; the third one is to collect disk related statics in a long > interval. And all the interval could be customized by the parameter > "mapred.stats.collection.interval" in the mapred-site.xml. At last, the > heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4527) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/HDFS-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-4527: -- Status: Open (was: Patch Available) > For shortening the time of TaskTracker heartbeat, decouple the statics > collection operations > > > Key: HDFS-4527 > URL: https://issues.apache.org/jira/browse/HDFS-4527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 1.1.1 >Reporter: sam liu > Labels: patch > Fix For: 1.1.1 > > Attachments: HDFS-4527.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In each heartbeat of TaskTracker, it will calculate some system statics, like > the free disk space, available virtual/physical memory, cpu usage, etc. > However, it's not necessary to calculate all the statics in every heartbeat, > and this will consume many system resource and impace the performance of > TaskTracker heartbeat. Furthermore, the characteristics of system > properties(disk, memory, cpu) are different and it's better to collect their > statics in different intervals. > To reduce the latency of TaskTracker heartbeat, one solution is to decouple > all the system statics collection operations from it, and issue separate > threads to do the statics collection works when the TaskTracker starts. The > threads could be three: the first one is to collect cpu related statics in a > short interval; the second one is to collect memory related statics in a > normal interval; the third one is to collect disk related statics in a long > interval. And all the interval could be customized by the parameter > "mapred.stats.collection.interval" in the mapred-site.xml. At last, the > heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4527) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
[ https://issues.apache.org/jira/browse/HDFS-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated HDFS-4527: -- Fix Version/s: 1.1.1 Status: Patch Available (was: Open) > For shortening the time of TaskTracker heartbeat, decouple the statics > collection operations > > > Key: HDFS-4527 > URL: https://issues.apache.org/jira/browse/HDFS-4527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: performance >Affects Versions: 1.1.1 >Reporter: sam liu > Labels: patch > Fix For: 1.1.1 > > Original Estimate: 24h > Remaining Estimate: 24h > > In each heartbeat of TaskTracker, it will calculate some system statics, like > the free disk space, available virtual/physical memory, cpu usage, etc. > However, it's not necessary to calculate all the statics in every heartbeat, > and this will consume many system resource and impace the performance of > TaskTracker heartbeat. Furthermore, the characteristics of system > properties(disk, memory, cpu) are different and it's better to collect their > statics in different intervals. > To reduce the latency of TaskTracker heartbeat, one solution is to decouple > all the system statics collection operations from it, and issue separate > threads to do the statics collection works when the TaskTracker starts. The > threads could be three: the first one is to collect cpu related statics in a > short interval; the second one is to collect memory related statics in a > normal interval; the third one is to collect disk related statics in a long > interval. And all the interval could be customized by the parameter > "mapred.stats.collection.interval" in the mapred-site.xml. At last, the > heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4527) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
sam liu created HDFS-4527: - Summary: For shortening the time of TaskTracker heartbeat, decouple the statics collection operations Key: HDFS-4527 URL: https://issues.apache.org/jira/browse/HDFS-4527 Project: Hadoop HDFS Issue Type: Improvement Components: performance Affects Versions: 1.1.1 Reporter: sam liu In each heartbeat of TaskTracker, it will calculate some system statics, like the free disk space, available virtual/physical memory, cpu usage, etc. However, it's not necessary to calculate all the statics in every heartbeat, and this will consume many system resource and impace the performance of TaskTracker heartbeat. Furthermore, the characteristics of system properties(disk, memory, cpu) are different and it's better to collect their statics in different intervals. To reduce the latency of TaskTracker heartbeat, one solution is to decouple all the system statics collection operations from it, and issue separate threads to do the statics collection works when the TaskTracker starts. The threads could be three: the first one is to collect cpu related statics in a short interval; the second one is to collect memory related statics in a normal interval; the third one is to collect disk related statics in a long interval. And all the interval could be customized by the parameter "mapred.stats.collection.interval" in the mapred-site.xml. At last, the heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
