Hi all, I have an open PR (https://github.com/apache/spark/pull/59313) and I would like others to take a look especially the interface part.
*TL;DR:* - It is JNI binding. There are example implementations of Data Sources in Rust and C++ in PR description. - It will look up the PATH with spark*datasource** native library (like how we do in Python Data Source), and register it automatically. Otherwise, users can manually add it via spark.addArtifact - The performance, from my benchmark, can improve: read side by around 2x: write side 10%. - Downstream Data Sources would be implemented in pure C++ or pure Rust, etc. *There might be some other aspects to consider e.g.,:* - There could be some design options, e.g., the downstream Data Sources could be mixed with Java. In that case, we could provide a better interface, or, we wouldn’t even need this PR at all because downstream Data Sources could already have JNI binding itself on their own. - Using FFM instead of JNI could be an option too. For now, 1. it is using JNI to avoid having a header file in diff languages and 2. to support lower JDK. I plan to release version 2 with FFM in Spark 5. Please review this PR - I plan to open this awhile.
