This is an automated email from the ASF dual-hosted git repository.
rakeshr pushed a change to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 54d3e21 HDFS-14923. Remove dead code from HealthMonitor. Contributed
by Fei Hui.
new 3f6f095 HDFS-14354: Refactor MappableBlock to align with the
implementation of SCM cache. Contributed by Feilong He.
new 0c68241 HDFS-14393. Refactor FsDatasetCache for SCM cache
implementation. Contributed by Rakesh R
new 6bc73a9 HDFS-14355 : Implement HDFS cache on SCM by using pure java
mapped byte buffer. Contributed by Feilong He.
new f3571c1 HDFS-14401. Refine the implementation for HDFS cache on SCM.
Contributed by Feilong He.
new 0fe720d HDFS-14402. Use FileChannel.transferTo() method for
transferring block to SCM cache. Contributed by Feilong He.
new 6eb5fb5 HDFS-14356. Implement HDFS cache on SCM with native PMDK
libs. Contributed by Feilong He.
new 75c66bf HDFS-14458. Report pmem stats to namenode. Contributed by
Feilong He.
new c6d59e6 HDFS-14357. Update documentation for HDFS cache on SCM
support. Contributed by Feilong He.
new dc2fad4 HDFS-14700. Clean up pmem cache before setting pmem cache
capacity. Contributed by Feilong He.
new a6cdcf6 HDFS-14818. Check native pmdk lib by 'hadoop checknative'
command. Contributed by Feilong He.
The 10 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
BUILDING.txt | 28 ++
dev-support/bin/dist-copynativelibs | 8 +
hadoop-common-project/hadoop-common/pom.xml | 2 +
.../hadoop-common/src/CMakeLists.txt | 21 ++
.../hadoop-common/src/config.h.cmake | 1 +
.../org/apache/hadoop/io/nativeio/NativeIO.java | 151 +++++++-
.../apache/hadoop/util/NativeLibraryChecker.java | 10 +
.../src/org/apache/hadoop/io/nativeio/NativeIO.c | 264 ++++++++++++++
.../src/org/apache/hadoop/io/nativeio/pmdk_load.c | 106 ++++++
.../src/org/apache/hadoop/io/nativeio/pmdk_load.h | 90 +++++
.../apache/hadoop/io/nativeio/TestNativeIO.java | 153 ++++++++
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 5 +
.../apache/hadoop/hdfs/server/datanode/DNConf.java | 9 +
.../server/datanode/fsdataset/impl/CacheStats.java | 212 +++++++++++
.../datanode/fsdataset/impl/FsDatasetCache.java | 231 ++++++------
.../datanode/fsdataset/impl/FsDatasetImpl.java | 34 +-
.../datanode/fsdataset/impl/FsDatasetUtil.java | 49 +++
.../datanode/fsdataset/impl/MappableBlock.java | 155 +-------
...MappableBlock.java => MappableBlockLoader.java} | 162 +++++----
.../fsdataset/impl/MappableBlockLoaderFactory.java | 51 +++
.../fsdataset/impl/MemoryMappableBlockLoader.java | 125 +++++++
.../datanode/fsdataset/impl/MemoryMappedBlock.java | 59 +++
.../impl/NativePmemMappableBlockLoader.java | 192 ++++++++++
.../fsdataset/impl/NativePmemMappedBlock.java | 85 +++++
.../fsdataset/impl/PmemMappableBlockLoader.java | 149 ++++++++
.../datanode/fsdataset/impl/PmemMappedBlock.java | 69 ++++
.../datanode/fsdataset/impl/PmemVolumeManager.java | 398 +++++++++++++++++++++
.../src/main/resources/hdfs-default.xml | 10 +
.../site/markdown/CentralizedCacheManagement.md | 18 +-
.../datanode/TestFsDatasetCacheRevocation.java | 31 +-
.../impl/TestCacheByPmemMappableBlockLoader.java | 325 +++++++++++++++++
.../{ => fsdataset/impl}/TestFsDatasetCache.java | 15 +-
32 files changed, 2845 insertions(+), 373 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.c
create mode 100644
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/pmdk_load.h
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/CacheStats.java
copy
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/{MappableBlock.java
=> MappableBlockLoader.java} (56%)
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlockLoaderFactory.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappableBlockLoader.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MemoryMappedBlock.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappableBlockLoader.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/NativePmemMappedBlock.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemMappableBlockLoader.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemMappedBlock.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/PmemVolumeManager.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestCacheByPmemMappableBlockLoader.java
rename
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/{
=> fsdataset/impl}/TestFsDatasetCache.java (97%)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
