findepi commented on issue #16622: URL: https://github.com/apache/datafusion/issues/16622#issuecomment-3019111118
If we look at the problem through the lens of project consumers, the expectation to never break anything is natural. If we look at the problem of feature implementers, the exception to be able to introduce changes, even those that are unfortunately breaking, is natural. arrow-rs has 2.6x less code and conceptually much simpler API surface. DataFusion is a library for building query engines and data processing tools. It is in active development and is not yet in "mostly complete, mostly maintenance" stage, so internal API changes are often unavoidable. DataFusion being a library, it has enormous API surface, so most of "internal API changes" are actually potentially breaking changes for downstream consumers. Being able to do such changes once a quarter (4 times a year), looks too rare to me. > Also, while it could work for icerberg and delta-rs potentially, I am not sure we could plausibly bring all the integrations into the same repo Agreed. At the same time, this issue is not on context of _any integrations_. It's in a context of real-life problems we (some of us) have today. What are these problems, and what integrations they pertain? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
