[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302674#comment-14302674 ] Arpit Agarwal commented on HDFS-7630: - Hi Sam, thank you for the clarification. Let's fix only those tests which are confirmed to fail due to OS page size assumptions. Do any of the tests in this patch fail on power platform without the change? TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302777#comment-14302777 ] sam liu commented on HDFS-7630: --- Hi Arpit, they pass on power platform so far, but it will be better if we could consider the differences of the page size of operating systems in regarding tests. TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302799#comment-14302799 ] Arpit Agarwal commented on HDFS-7630: - Hi Sam, I understand your concern but I think for these tests it is not an issue. The tests should have no dependency on OS page size. I believe the choice of 4096 as the block size was arbitrary and it being the same as the default page size on x86/x64 is just a coincidence. TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294898#comment-14294898 ] sam liu commented on HDFS-7630: --- Arpit, Not all such hard-coding will cause failure and the patch is mainly to remove the hard-coding. But sometimes the hard-coding could cause failure. For example, without patch HDFS-7585, test TestEnhancedByteBufferAccess will fail on power platform. In the tests, the size of BLOCK_SIZE is usually set as 4096 and it just equals to the default page size of x86 Linux operating system, but on power Linux operating system the default page size is 65536. As HDFS is based on operating system, it might be better the unit tests could consider the differences of operating systems. Thanks! TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293539#comment-14293539 ] sam liu commented on HDFS-7630: --- Hi Arpit, According to your comments, I combined all fixes into HDFS-7630.002.patch. All the unit tests passed excepting TestBlockScanner, however the failure is not caused by current patch. Could you please help take a review? Thanks! TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[
https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293183#comment-14293183
]
Hadoop QA commented on HDFS-7630:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694711/HDFS-7630.002.patch
against trunk revision 6f9fe76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9338//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9338//console
This message is automatically generated.
TestConnCache hardcode block size without considering native OS
---
Key: HDFS-7630
URL: https://issues.apache.org/jira/browse/HDFS-7630
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: sam liu
Assignee: sam liu
Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch
TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's
incorrect on some platforms. For example, on power platform, the correct
value is 65536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294212#comment-14294212 ] Arpit Agarwal commented on HDFS-7630: - Thanks [~sam liu]. The modified tests pass on Windows, Linux and OS X so I'd be fine with committing it. However I not sure why the BLOCK_SIZE needs to be the same as the OS page size. Could you please share some more details about the failure? e.g. on Linux TestConnCache passes even if BLOCK_SIZE is set to 2048. TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch, HDFS-7630.002.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292353#comment-14292353 ] Arpit Agarwal commented on HDFS-7630: - HI [~sam liu], it is hard to review and commit many trivial patches. It would really help if HDFS-7624 through HDFS-7630 could be combined into a single Jira with a consolidated patch. I think the change looks fine, I'd like to verify it on Windows before I +1. TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[
https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280011#comment-14280011
]
Hadoop QA commented on HDFS-7630:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692698/HDFS-7630.001.patch
against trunk revision c4ccbe6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9243//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9243//console
This message is automatically generated.
TestConnCache hardcode block size without considering native OS
---
Key: HDFS-7630
URL: https://issues.apache.org/jira/browse/HDFS-7630
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: sam liu
Assignee: sam liu
Attachments: HDFS-7630.001.patch
TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's
incorrect on some platforms. For example, on power platform, the correct
value is 65536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280270#comment-14280270 ] sam liu commented on HDFS-7630: --- The failure of TestFileTruncate should not be caused by the patch, because: - TestFileTruncate does not invoke any API of current test TestConnCache - Actually, in my test env, TestFileTruncate also failed without applying any patch TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7630) TestConnCache hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14279849#comment-14279849 ] Yi Liu commented on HDFS-7630: -- [~sam liu], you can do the same small change for all test cases in one JIRA, you don't need to create JIRA for each test case, that doesn't make sense. TestConnCache hardcode block size without considering native OS --- Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
