The RDMA for Apache Hadoop team is pleased to announce the release of
0.9.9 package (technology preview) with the following features.
* RDMA for Apache Hadoop 0.9.9 Features
- Based on Apache Hadoop 1.2.1
- High performance design with native InfiniBand and RoCE support
at the verbs-level for HDFS, MapReduce, and RPC components
- Compliant with Apache Hadoop 1.2.1 APIs and applications
- Easily configurable for native InfiniBand, RoCE, and the
traditional sockets-based support (Ethernet and InfiniBand
with IPoIB)
- On-demand connection setup
- HDFS over native InfiniBand and RoCE
- RDMA-based write
- RDMA-based replication
- Parallel replication support
- MapReduce over native InfiniBand and RoCE
- RDMA-based shuffle
- Prefetching and caching of map outputs
- In-memory merge
- Advanced optimization in overlapping
- map, shuffle, and merge
- shuffle, merge, and reduce
- RPC over native InfiniBand and RoCE
- JVM-bypassed buffer management
- RDMA or send/recv based adaptive communication
- Intelligent buffer allocation and adjustment for serialization
- Tested with
- Mellanox InfiniBand adapters (DDR, QDR, and FDR)
- RoCE
- Various multi-core platforms
- Different file systems with disks and SSDs
* Bug Fixes (since 0.9.8)
- Fix an issue in framework startup
- Fix for configurable port selection
- MapReduce over native InfiniBand and RoCE
- Fix for progress calculation in Reduce tasks
- HDFS over native InfiniBand and RoCE
- Fix an issue in DataNode initialization
For downloading RDMA for Apache Hadoop 0.9.9 package and the
associated user guide, please visit the following URL:
http://hadoop-rdma.cse.ohio-state.edu
Sample performance numbers for benchmarks using RDMA for Apache Hadoop
0.9.9 version can be viewed by visiting the `Performance' tab of the
above website.
All questions, feedbacks and bug reports are welcome. Please post it
to the hadoop-rdma-discuss mailing list (hadoop-rdma-discuss at
cse.ohio-state.edu).
Thanks,
The RDMA for Apache Hadoop Team
