benbellick commented on PR #1399:
URL: https://github.com/apache/arrow-go/pull/1399#issuecomment-6926707877

   Hi @zeroshade! I agree that removing `substrait-go` types from Arrow's 
public API would be ideal. But that seems difficult for functions like 
`ExecuteScalarExpression`, which lets libraries like Iceberg build Substrait 
expressions and pass them to Arrow for execution.
   
   First, I think we should reduce the exposure of `substrait-go`'s API where 
we can. For example, Arrow probably doesn't need to expose function/type anchor 
bookkeeping. I put together [a draft exploring that 
cleanup](https://github.com/benbellick/arrow-go/pull/1).
   
   Beyond that, I see a few options:
   
   1. Keep existing execution functions and add new ones for each major 
version. For example, keep `ExecuteScalarExpression` accepting v8 expressions 
and add `ExecuteScalarExpressionV9` for v9. We'd maintain some duplicate code, 
but users could upgrade when they're ready. Iceberg already builds its own 
expressions, so we wouldn't need to duplicate Arrow's builders for this use 
case.
   
   2. Move the Arrow/Substrait code into its own independently versioned 
module. We'd still have breaking changes, but they wouldn't force a major 
release of all of Arrow.
   
   3. Remove Substrait types from the public API entirely. I'm not sure how to 
do this cleanly without moving execution elsewhere or giving Arrow its own 
expression API, which goes against the idea of sharing Substrait's model.
   
   We'd want to make any necessary breaking changes together in one major 
release, with a plan to avoid repeating this in the future. After that, adding 
execution support for newer Substrait versions wouldn't need to break existing 
callers.
   
   What do you think? Any preference between those three? I think 1 is the 
simplest, though I'm not sure how much code we'd need to duplicate for each new 
major version. It would also mean keeping dependencies on multiple 
`substrait-go` major versions rather than replacing the old one with the new 
one.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to