[
https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710968#comment-14710968
]
Tony Reix commented on HDFS-8520:
---------------------------------
Hi Andrew
Sorry of answering so late. I was busy with other staff.
My understanding of this defect is:
- I had analyzed the code and had found hardcoded 4096 values for PageSize in
many places, which does cover mainly basic x86 case but not other architectures
- on Linux on Power, PageSize is different
- I had proposed to use the getOperatingSystemPageSize() routine instead of
hardcoded 4096 value in order to make the test generic (work on all targets
with no change)
- You proposed instead to hardcode 4096 in the existing test and to add a new
test with 64KB as hardcoded value (which would cover x86 Linux Large Page case)
I guess that this would mean to add some code to check, before running
these 2 tests, which ones are applicable on the targeted architecture.
What about other architectures that would not use 4096 nor 64KB ?
What is the status now for Hadoop 2.7.2 ?
Are you planning to make the code more homogeneous and to enable it to handle
cases (like Linux on Power) where basic PageSize is NOT 4096 ?
It is very difficult for me to propose a patch since:
1) there are a lot of places where 4096 and PageSize appear, with no
clear clue if that deals with my issue
2) your approach is very different from mine: hardcode values instead
of making code generic
Moreover, the way the tests "fail" is a pain, since they end with TimeOut.
In my opinion, a huge work of rewriting test code so that it deals by default
with very different architectures (including x86 4KB and 64KB cases) would be
the best approach.
I also would suggest to add a Linux/PPC64 machine in the development/test
environment in order to handle different architectures at soon as possible.
> Patch for PPC64 block size
> --------------------------
>
> Key: HDFS-8520
> URL: https://issues.apache.org/jira/browse/HDFS-8520
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.7.1
> Environment: RHEL 7.1 /PPC64
> Reporter: Tony Reix
> Assignee: Tony Reix
> Attachments: HDFS-8520-2.patch, HDFS-8520.patch
>
>
> The attached patch enables Hadoop to work on PPC64.
> That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64.
> There are changes in 3 files:
> -
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
> -
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java
> -
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
> where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE
> The patch has been built on branch-2.7 .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
