I think the one place where it might break is for Union types (I seem to recall a breaking change just prior to 1.0).
On Wed, Jun 2, 2021 at 1:00 PM Antoine Pitrou <anto...@python.org> wrote: > > Le 02/06/2021 à 21:57, Rares Vernica a écrit : > > Thanks for the pointers! The migration is going well. > > > > We have been using Arrow 0.16.0 RecordBatchStreamWriter > > < > https://github.com/Paradigm4/bridge/blob/master/src/PhysicalXSave.cpp#L450 > > > > with & without CompressedOutputStream and wrote the resulting Arrow > Buffer > > data to S3 > > <https://github.com/Paradigm4/bridge/blob/master/src/S3Driver.cpp#L168> > or file > > system > > <https://github.com/Paradigm4/bridge/blob/master/src/FSDriver.cpp#L156>. > We > > have a sizable amount of data saved this way. > > > > Once we upgrade our C++ code to use Arrow 3.0.0 or 4.0.0, will it be > > possible to read the Arrow steam files written with Arrow 0.16.0? > > It definitely should, unless it's a bug. > That said, for extra safety, I suggest you test loading the files before > doing the final migration. > > By the way, for saving lots of data to S3, it may be more efficient to > use Parquet. It will be more CPU-intensive but will result in > significant space savings. > > Regards > > Antoine. >
