LuciferYang commented on PR #59424: URL: https://github.com/apache/spark/pull/59424#issuecomment-6184757389
Thanks for fixing the data writer so quickly, with a test, and for running the native suites on Java 25 in the CI of pull requests. The function table also makes the header easier to build against than JNI: the compiler checks every function a library provides. Speaking as one of the Lance maintainers, I'd like to share how this looks from lance-spark, since Lance is one of the motivating examples in the description, and hear your thoughts. For reads and writes, lance-spark already goes through the Rust core of Lance: the Lance Java bindings fill an `ArrowArrayStream` at an address that the JVM allocates (`LanceFragmentScanner.exportArrowStream`), which is the same exchange as the `read` function of `SparkDsLibrary`. So a native data source would not remove a layer or a copy for Lance. Most of what users get from lance-spark is beyond reads and writes: the `VECTOR_SEARCH`, `SEARCH` and `HYBRID_SEARCH` table functions, `CREATE INDEX` and the other index commands, `UPDATE`, `DELETE` and `MERGE INTO` through `_rowid` and `_rowaddr`, fragment-aware joins, `count(*)` and top-N pushdown, and branches and tags. Version 1 of the interface covers reads and writes with filter, column and limit pushdown, so a native Lance data source would give users less than lance-spark does today. This is also where our cost of supporting several Spark releases is. lance-spark has a module for each Spark version from 3.4 to 4.2, but the read and write code is shared. What differs between versions is mainly the SQL extensions (parser and optimizer rules) and the row-level operations. For example, Spark 4.2 plans an insert-only `MERGE INTO` as `InsertOnlyMerge` instead of `AppendData`, and we needed a new rule so that blob columns are still copied. A stable ABI would not help with that part. Where native data sources look like a clear win to me is a source that only has a native client and only needs reads and writes, such as an in-house storage system in C++ or a vendor's C SDK. Could the description and the docs lead with that kind of source, and with a concrete one if this work started from one? Your reply to @tdcmeehan lists what could be appended to the table later, such as statistics, aggregate and top-N pushdown, and overwrite by filter. For a source like Lance, table functions and row-level operations matter as much. Do you see them in scope for the table later, or out of scope on purpose? Saying so in the docs would help library authors tell early whether it fits their source. None of this blocks the implementation; it is only about how the feature is positioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
