jorgecarleitao commented on issue #80: URL: https://github.com/apache/arrow-datafusion/issues/80#issuecomment-826900252
I think that the general problem is that we pin arrow (and many others) in datafusion; datafusion is a library and it should thus avoid pinning dependencies. Instead, it should bracket them, via e.g. `^3.0.0`, so that consumers of the library may use a different version of any of its dependencies, for as long as they are compatible, and have cargo find a valid dependency version between what the consumer wants and what datafusion requires. As it stands, consumers must use the exact same version of arrow that datafusion uses or cargo will pick two different arrow versions. This happens because Cargo cannot guarantee that the two different arrow versions (what datafusion demands and what the consumer wants) are ABI compatible. Consumers can't pass structs from a version of arrow (that they use) to another version of arrow (that datafusion uses). Note that in this context a different feature set corresponds to a different version, as cargo has no way of knowing whether a feature will retain ABI compatibility. So, I think the ask here is: 1. datafusion writes something like `^4.0.0` on its `cargo.toml` 2. we release arrow more often and strictly according to semver Is this the idea, @tustvold ? I am not sure whether we could get away with the [multiple paths](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations) approach, e.g. ``` arrow = { git = "https://github.com/arrow-rs/arrow";, version = "^3.0.0" } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
