This is an automated email from the ASF dual-hosted git repository.
nicholasjiang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 67f7c4b59 [CELEBORN-2185] Support native kqueue transport on BSD/MacOS
67f7c4b59 is described below
commit 67f7c4b59154c7822fec7c300e93b1231f92f365
Author: SteNicholas <[email protected]>
AuthorDate: Wed Oct 29 14:07:52 2025 +0800
[CELEBORN-2185] Support native kqueue transport on BSD/MacOS
### What changes were proposed in this pull request?
Support native kqueue transport on BSD/MacOS for
`celeborn.<module>.io.mode`.
Backport:
- https://github.com/apache/spark/pull/52703
- https://github.com/apache/spark/pull/52724
### Why are the changes needed?
Netty provides the following platform specific JNI transports for [native
transports](https://netty.io/wiki/native-transports.html):
- Linux (since 4.0.16)
- MacOS/BSD (since 4.1.11)
These JNI transports add features specific to a particular platform,
generate less garbage, and generally improve performance when compared to the
NIO based transport.
### Does this PR introduce _any_ user-facing change?
Change the default value of `celeborn.<module>.io.mode` from `NIO` to
`EPOLL` if epoll mode is available, from `NIO` to `KQUEUE` if kqueue mode is
available, falling back to `NIO` otherwise.
### How was this patch tested?
CI.
Closes #3518 from SteNicholas/CELEBORN-2185.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: SteNicholas <[email protected]>
---
client-flink/flink-1.16-shaded/pom.xml | 23 +++++++++++++++++-----
client-flink/flink-1.17-shaded/pom.xml | 23 +++++++++++++++++-----
client-flink/flink-1.18-shaded/pom.xml | 23 +++++++++++++++++-----
client-flink/flink-1.19-shaded/pom.xml | 23 +++++++++++++++++-----
client-flink/flink-1.20-shaded/pom.xml | 23 +++++++++++++++++-----
client-flink/flink-2.0-shaded/pom.xml | 23 +++++++++++++++++-----
client-flink/flink-2.1-shaded/pom.xml | 23 +++++++++++++++++-----
client-mr/mr-shaded/pom.xml | 23 +++++++++++++++++-----
client-spark/spark-2-shaded/pom.xml | 23 +++++++++++++++++-----
client-spark/spark-3-shaded/pom.xml | 23 +++++++++++++++++-----
client-spark/spark-4-shaded/pom.xml | 23 +++++++++++++++++-----
client-tez/tez-shaded/pom.xml | 23 +++++++++++++++++-----
.../celeborn/common/network/util/IOMode.java | 11 ++++++-----
.../celeborn/common/network/util/NettyUtils.java | 14 ++++++++++++-
.../common/network/util/TransportConf.java | 7 +++++--
.../org/apache/celeborn/common/CelebornConf.scala | 11 ++++++++---
docs/configuration/network.md | 2 +-
docs/migration.md | 2 ++
18 files changed, 251 insertions(+), 72 deletions(-)
diff --git a/client-flink/flink-1.16-shaded/pom.xml
b/client-flink/flink-1.16-shaded/pom.xml
index 4eb0fbfb5..c856b1108 100644
--- a/client-flink/flink-1.16-shaded/pom.xml
+++ b/client-flink/flink-1.16-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-flink/flink-1.17-shaded/pom.xml
b/client-flink/flink-1.17-shaded/pom.xml
index 83223ae60..fd8512fe7 100644
--- a/client-flink/flink-1.17-shaded/pom.xml
+++ b/client-flink/flink-1.17-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-flink/flink-1.18-shaded/pom.xml
b/client-flink/flink-1.18-shaded/pom.xml
index 4dbb4b9a6..04d5b9651 100644
--- a/client-flink/flink-1.18-shaded/pom.xml
+++ b/client-flink/flink-1.18-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-flink/flink-1.19-shaded/pom.xml
b/client-flink/flink-1.19-shaded/pom.xml
index 7f1070c41..060ce1644 100644
--- a/client-flink/flink-1.19-shaded/pom.xml
+++ b/client-flink/flink-1.19-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-flink/flink-1.20-shaded/pom.xml
b/client-flink/flink-1.20-shaded/pom.xml
index d215d6eef..3b004cbed 100644
--- a/client-flink/flink-1.20-shaded/pom.xml
+++ b/client-flink/flink-1.20-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-flink/flink-2.0-shaded/pom.xml
b/client-flink/flink-2.0-shaded/pom.xml
index 7be10544b..020728018 100644
--- a/client-flink/flink-2.0-shaded/pom.xml
+++ b/client-flink/flink-2.0-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-flink/flink-2.1-shaded/pom.xml
b/client-flink/flink-2.1-shaded/pom.xml
index 0faa1f651..d2c874ba5 100644
--- a/client-flink/flink-2.1-shaded/pom.xml
+++ b/client-flink/flink-2.1-shaded/pom.xml
@@ -120,11 +120,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-mr/mr-shaded/pom.xml b/client-mr/mr-shaded/pom.xml
index 089d83019..2ffa40e1a 100644
--- a/client-mr/mr-shaded/pom.xml
+++ b/client-mr/mr-shaded/pom.xml
@@ -131,11 +131,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-spark/spark-2-shaded/pom.xml
b/client-spark/spark-2-shaded/pom.xml
index cfbc02079..9db62b423 100644
--- a/client-spark/spark-2-shaded/pom.xml
+++ b/client-spark/spark-2-shaded/pom.xml
@@ -121,11 +121,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git a/client-spark/spark-3-shaded/pom.xml
b/client-spark/spark-3-shaded/pom.xml
index 8cce8577a..bc8c2065e 100644
--- a/client-spark/spark-3-shaded/pom.xml
+++ b/client-spark/spark-3-shaded/pom.xml
@@ -125,11 +125,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="deleting META-INF/native-image folder"></echo>
<delete
dir="${project.build.directory}/unpacked/META-INF/native-image"></delete>
<echo message="repackaging netty jar"></echo>
diff --git a/client-spark/spark-4-shaded/pom.xml
b/client-spark/spark-4-shaded/pom.xml
index 633eabbec..5e741d601 100644
--- a/client-spark/spark-4-shaded/pom.xml
+++ b/client-spark/spark-4-shaded/pom.xml
@@ -125,11 +125,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="deleting META-INF/native-image folder"></echo>
<delete
dir="${project.build.directory}/unpacked/META-INF/native-image"></delete>
<echo message="repackaging netty jar"></echo>
diff --git a/client-tez/tez-shaded/pom.xml b/client-tez/tez-shaded/pom.xml
index e192d3bdb..e8060d95a 100644
--- a/client-tez/tez-shaded/pom.xml
+++ b/client-tez/tez-shaded/pom.xml
@@ -145,11 +145,24 @@
<fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
<mapper from="libnetty_transport_native_epoll_aarch_64.so"
to="liborg_apache_celeborn_shaded_netty_transport_native_epoll_aarch_64.so"
type="glob"></mapper>
</move>
- <echo message="deleting native kqueue library"></echo>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib"></delete>
- <delete
file="${project.build.directory}/unpacked/META-INF/native/libnetty_resolver_dns_native_macos_x86_64.jnilib"></delete>
+ <echo message="renaming native kqueue library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_x86_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_transport_native_kqueue_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_transport_native_kqueue_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <echo message="renaming native dns resolver library"></echo>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_aarch_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_aarch_64.jnilib"
type="glob"></mapper>
+ </move>
+ <move includeemptydirs="false"
todir="${project.build.directory}/unpacked/META-INF/native">
+ <fileset
dir="${project.build.directory}/unpacked/META-INF/native"></fileset>
+ <mapper
from="libnetty_resolver_dns_native_macos_x86_64.jnilib"
to="liborg_apache_celeborn_shaded_netty_resolver_dns_native_macos_x86_64.jnilib"
type="glob"></mapper>
+ </move>
<echo message="repackaging netty jar"></echo>
<jar basedir="${project.build.directory}/unpacked"
destfile="${project.build.directory}/${artifactId}-${version}.jar"></jar>
</target>
diff --git
a/common/src/main/java/org/apache/celeborn/common/network/util/IOMode.java
b/common/src/main/java/org/apache/celeborn/common/network/util/IOMode.java
index 7d2ddd895..ed66464c6 100644
--- a/common/src/main/java/org/apache/celeborn/common/network/util/IOMode.java
+++ b/common/src/main/java/org/apache/celeborn/common/network/util/IOMode.java
@@ -17,11 +17,12 @@
package org.apache.celeborn.common.network.util;
-/**
- * Selector for which form of low-level IO we should use. NIO is always
available, while EPOLL is
- * only available on Linux. AUTO is used to select EPOLL if it's available, or
NIO otherwise.
- */
+/** Selector for which form of low-level IO we should use. */
public enum IOMode {
+ /** Java NIO (Selector), cross-platform portable */
NIO,
- EPOLL
+ /** Native EPOLL via JNI, Linux only */
+ EPOLL,
+ /** Native KQUEUE via JNI, MacOS/BSD only */
+ KQUEUE
}
diff --git
a/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
b/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
index 2f04179d7..109328233 100644
---
a/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
+++
b/common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java
@@ -36,6 +36,9 @@ import io.netty.channel.ServerChannel;
import io.netty.channel.epoll.EpollEventLoopGroup;
import io.netty.channel.epoll.EpollServerSocketChannel;
import io.netty.channel.epoll.EpollSocketChannel;
+import io.netty.channel.kqueue.KQueueEventLoopGroup;
+import io.netty.channel.kqueue.KQueueServerSocketChannel;
+import io.netty.channel.kqueue.KQueueSocketChannel;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioServerSocketChannel;
import io.netty.channel.socket.nio.NioSocketChannel;
@@ -47,7 +50,10 @@ import org.apache.celeborn.common.CelebornConf;
import org.apache.celeborn.common.metrics.source.AbstractSource;
import org.apache.celeborn.common.util.JavaUtils;
-/** Utilities for creating various Netty constructs based on whether we're
using EPOLL or NIO. */
+/**
+ * Utilities for creating various Netty constructs based on whether we're
using NIO, EPOLL, or
+ * KQUEUE.
+ */
public class NettyUtils {
private static final ByteBufAllocator[] _sharedByteBufAllocator = new
ByteBufAllocator[2];
private static final ConcurrentHashMap<String, Integer> allocatorsIndex =
@@ -80,6 +86,8 @@ public class NettyUtils {
: new NioEventLoopGroup(numThreads, threadFactory);
case EPOLL:
return new EpollEventLoopGroup(numThreads, threadFactory);
+ case KQUEUE:
+ return new KQueueEventLoopGroup(numThreads, threadFactory);
default:
throw new IllegalArgumentException("Unknown io mode: " + mode);
}
@@ -92,6 +100,8 @@ public class NettyUtils {
return NioSocketChannel.class;
case EPOLL:
return EpollSocketChannel.class;
+ case KQUEUE:
+ return KQueueSocketChannel.class;
default:
throw new IllegalArgumentException("Unknown io mode: " + mode);
}
@@ -104,6 +114,8 @@ public class NettyUtils {
return NioServerSocketChannel.class;
case EPOLL:
return EpollServerSocketChannel.class;
+ case KQUEUE:
+ return KQueueServerSocketChannel.class;
default:
throw new IllegalArgumentException("Unknown io mode: " + mode);
}
diff --git
a/common/src/main/java/org/apache/celeborn/common/network/util/TransportConf.java
b/common/src/main/java/org/apache/celeborn/common/network/util/TransportConf.java
index 55b22306d..26a0ce25e 100644
---
a/common/src/main/java/org/apache/celeborn/common/network/util/TransportConf.java
+++
b/common/src/main/java/org/apache/celeborn/common/network/util/TransportConf.java
@@ -20,6 +20,7 @@ package org.apache.celeborn.common.network.util;
import java.io.File;
import io.netty.channel.epoll.Epoll;
+import io.netty.channel.kqueue.KQueue;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -44,9 +45,11 @@ public class TransportConf {
return module;
}
- /** IO mode: nio or epoll */
+ /** IO mode: NIO, EPOLL, or KQUEUE */
public String ioMode() {
- return Epoll.isAvailable() ? celebornConf.networkIoMode(module) :
IOMode.NIO.name();
+ return Epoll.isAvailable() || KQueue.isAvailable()
+ ? celebornConf.networkIoMode(module)
+ : IOMode.NIO.name();
}
/** If true, we will prefer allocating off-heap byte buffers within Netty. */
diff --git
a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
index 2fa19f5e3..010cd57a4 100644
--- a/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
+++ b/common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala
@@ -28,6 +28,7 @@ import scala.util.Try
import scala.util.matching.Regex
import io.netty.channel.epoll.Epoll
+import io.netty.channel.kqueue.KQueue
import
org.apache.celeborn.common.authentication.AnonymousAuthenticationProviderImpl
import org.apache.celeborn.common.client.{ApplicationInfoProvider,
DefaultApplicationInfoProvider}
@@ -543,7 +544,9 @@ class CelebornConf(loadDefaults: Boolean) extends Cloneable
with Logging with Se
def networkIoMode(module: String): String = {
get(
NETWORK_IO_MODE.key.replace("<module>", module),
- if (Epoll.isAvailable) IOMode.EPOLL.name() else IOMode.NIO.name())
+ if (Epoll.isAvailable) { IOMode.EPOLL.name() }
+ else if (KQueue.isAvailable) { IOMode.KQUEUE.name() }
+ else { IOMode.NIO.name() })
}
def networkIoPreferDirectBufs(module: String): Boolean = {
@@ -2089,10 +2092,12 @@ object CelebornConf extends Logging {
val NETWORK_IO_MODE: OptionalConfigEntry[String] =
buildConf("celeborn.<module>.io.mode")
.categories("network")
- .doc("Netty EventLoopGroup backend, available options: NIO, EPOLL. If
epoll mode is available, the default IO mode is EPOLL; otherwise, the default
is NIO.")
+ .doc("Netty EventLoopGroup backend, available options: NIO, EPOLL,
KQUEUE. " +
+ "For Linux environments, EPOLL is used if available before using NIO.
" +
+ "For MacOS/BSD environments, KQUEUE is used if available before using
NIO.")
.stringConf
.transform(_.toUpperCase)
- .checkValues(Set(IOMode.NIO.name(), IOMode.EPOLL.name()))
+ .checkValues(Set(IOMode.NIO.name(), IOMode.EPOLL.name(),
IOMode.KQUEUE.name()))
.createOptional
val NETWORK_IO_PREFER_DIRECT_BUFS: ConfigEntry[Boolean] =
diff --git a/docs/configuration/network.md b/docs/configuration/network.md
index ec809f57d..2cdad5aa0 100644
--- a/docs/configuration/network.md
+++ b/docs/configuration/network.md
@@ -29,7 +29,7 @@ license: |
| celeborn.<module>.io.connectionTimeout | <value of
celeborn.network.timeout> | false | Connection active timeout. If setting
<module> to `rpc_app`, works for shuffle client. If setting <module> to
`rpc_service`, works for master or worker. If setting <module> to `data`, it
works for shuffle client push and fetch data. If setting <module> to `push`, it
works for worker receiving push data. If setting <module> to `replicate`, it
works for replicate server or client of worker [...]
| celeborn.<module>.io.lazyFD | true | false | Whether to initialize
FileDescriptor lazily or not. If true, file descriptors are created only when
data is going to be transferred. This can reduce the number of open files. If
setting <module> to `fetch`, it works for worker fetch server. | | |
| celeborn.<module>.io.maxRetries | 3 | false | Max number of times we
will try IO exceptions (such as connection timeouts) per request. If set to 0,
we will not do any retries. If setting <module> to `data`, it works for shuffle
client push and fetch data. If setting <module> to `replicate`, it works for
replicate client of worker replicating data to peer worker. If setting <module>
to `push`, it works for Flink shuffle client push data. | | |
-| celeborn.<module>.io.mode | <undefined> | false | Netty
EventLoopGroup backend, available options: NIO, EPOLL. If epoll mode is
available, the default IO mode is EPOLL; otherwise, the default is NIO. | | |
+| celeborn.<module>.io.mode | <undefined> | false | Netty
EventLoopGroup backend, available options: NIO, EPOLL, KQUEUE. For Linux
environments, EPOLL is used if available before using NIO. For MacOS/BSD
environments, KQUEUE is used if available before using NIO. | | |
| celeborn.<module>.io.numConnectionsPerPeer | 1 | false | Number of
concurrent connections between two nodes. If setting <module> to `rpc_app`,
works for shuffle client. If setting <module> to `rpc_service`, works for
master or worker. If setting <module> to `data`, it works for shuffle client
push and fetch data. If setting <module> to `replicate`, it works for replicate
client of worker replicating data to peer worker. | | |
| celeborn.<module>.io.preferDirectBufs | true | false | If true, we
will prefer allocating off-heap byte buffers within Netty. If setting <module>
to `rpc_app`, works for shuffle client. If setting <module> to `rpc_service`,
works for master or worker. If setting <module> to `data`, it works for shuffle
client push and fetch data. If setting <module> to `push`, it works for worker
receiving push data. If setting <module> to `replicate`, it works for replicate
server or client of w [...]
| celeborn.<module>.io.receiveBuffer | 0b | false | Receive buffer size
(SO_RCVBUF). Note: the optimal size for receive buffer and send buffer should
be latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth =
10Gbps buffer size should be ~ 1.25MB. If setting <module> to `rpc_app`, works
for shuffle client. If setting <module> to `rpc_service`, works for master or
worker. If setting <module> to `data`, it works for shuffle client push and
fetch data. If setting <mod [...]
diff --git a/docs/migration.md b/docs/migration.md
index 81babe0ab..2c302a534 100644
--- a/docs/migration.md
+++ b/docs/migration.md
@@ -29,6 +29,8 @@ license: |
- Since 0.7.0, Celeborn worker metrics `FlushDataTime` is renamed as
`FlushLocalDataTime`.
+- Since 0.7.0, Celeborn changed the default value of
`celeborn.<module>.io.mode` from `NIO` to `KQUEUE` if kqueue mode is available,
falling back to `NIO` otherwise.
+
# Upgrading from 0.5 to 0.6
- Since 0.6.0, Celeborn deprecate
`celeborn.client.spark.fetch.throwsFetchFailure`. Please use
`celeborn.client.spark.stageRerun.enabled` instead.
