HyukjinKwon commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6076275217
@pan3793 Good question. Yes, but FFM cannot be the only bridge yet because of the Java baseline. Spark compiles with `--release 17` and supports Java 17, 21 and 25, while the FFM API is final only from Java 22 ([JEP 454](https://openjdk.org/jeps/454)): it is a preview API in Java 21 and an incubator module in Java 17. So an FFM bridge would need a separate source set for Java 22+, and we would still need the JNI path for Java 17 and 21, with two interfaces for library authors to choose from. FFM would fit this design well, though, and the design leaves room for it: - With FFM, a library could export a plain C ABI, without `JNIEnv` or the `jni` crate, and report errors with return codes instead of Java exceptions. Spark could also open each library with `SymbolLookup.libraryLookup` instead of defining a copy of `NativeBridge` in a class loader of its own for each library. - Performance does not decide it either way: the calls are coarse-grained (per scan, partition and batch), and the data crosses the boundary as Arrow C Data Interface structs, without copying. - The interface is versioned (`NativeBridge.abiVersion`, and `abiVersion` in the manifest of a package), and only `NativeBridge` is public: loading and calling the libraries is internal. Once the minimum Java version of Spark allows it, a later version of the interface can be a plain C ABI called through FFM, while Spark keeps loading the libraries of version 1. - Spark already runs with `--enable-native-access=ALL-UNNAMED`, without which recent JDKs warn about both JNI ([JEP 472](https://openjdk.org/jeps/472)) and the restricted methods of FFM, so neither needs new flags. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
