Hi, imo the time-scale of changes in the format is too large to justify the complexity. I also think that we should not force users to clone or submodule the repo to even compile the crate.
What if we just do not have the format files there at all, and instead just keep the generated code? Updates to the format are manually performed (i.e. instead of copying the file, we run the build against latest). This is what we already do in practice, anyways: whenever the format changed, tonic would auto-generate new rust code that we would commit before any change. This avoids having copies of files between repos, which imo confuses people into which one is the official one (even when they are all equal, people will not do diffs in their heads). Best, Jorge On Wed, Apr 28, 2021 at 12:13 PM Andrew Lamb <al...@influxdata.com> wrote: > I also think manually copying the format .fbs files to arrow-rs is probably > ok for the time being. > > Once Arrow gets to the point where many implementations that need > format.fbs live in many different repos, pulling out the format files into > their own repo might be worth reconsidering. > > Andrew > > On Tue, Apr 27, 2021 at 5:45 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > I wouldn't be too excited about this. Here are my thoughts: > > > > 1. Having the format/ directory in apache/arrow be a submodule would be > > cumbersome and error-prone for developers. The only submodules we have > > right now are optional testing dependencies — not having these > initialized > > and updated does not result in a broken project, whereas this change > would. > > We have a copy of parquet.thrift from apache/parquet-format for similar > > reasons. > > > > 2. So based on #1, we would want to maintain a copy of the format files > in > > apache/arrow, even if there were a separate apache/arrow-format > repository. > > The format files are slow-moving enough that I don't think it's > burdensome > > to mirror these into satellite repositories like arrow/arrow-rs. > > > > On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale <nevilled...@gmail.com> > > wrote: > > > > > Hi Arrow devs, > > > > > > Andy noticed that we carry a copy of the format directory in arrow-rs, > > > which > > > is bound to get outdated in the future. > > > > > > We would like to propose creating an arrow-format repository, similar > to > > > parquet-format, so that arrow-rs and other future separate repositories > > > could > > > add this as a submodule. > > > > > > What are your thoughts? > > > > > > Regards > > > Neville > > > > > >
