tokoko commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6146218315
@HyukjinKwon this is probably longer-term, but since the plan above is to move to a plain C ABI, have you considered building on ADBC (with extensions where needed) rather than defining something Spark-specific? As part of #54603 I’m already working on a distributed ingest API for ADBC (apache/arrow-adbc#4317). Streaming offsets and Spark’s pushdown semantics (maybe via Sql or Substrait) would need work too, but I don’t see a fundamental blocker unless the Arrow folks object. A Spark-specific interface is certainly simpler to evolve, but being able to hook into the independent and growing set of ADBC drivers, and letting things like iceberg (https://github.com/tokoko/adbc-iceberg), delta or Lance write one driver that serves every engine seems like a big win to me. cc @lidavidm @ianmcook @holdenk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
