Hi Micah, Yes, those files are read correctly. We test against them. I was trying to generate gold files based on 0.17.1, so I could debug against those, I'll work on that in the coming days.
On Sat, 11 Jul 2020, 05:58 Micah Kornfield, <emkornfi...@gmail.com> wrote: > Hi Neville, > Thanks for the update. One question, we have "gold" files for 0.14.0 > checked into the test-data repo and run integration tests on those to > ensure we can read them in a few implementations. Does Rust at least read > those correctly? > > Thanks, > Micah > > On Fri, Jul 10, 2020 at 1:03 PM Neville Dipale <nevilled...@gmail.com> > wrote: > > > Good day Arrow devs, > > > > I've spent a few evenings looking into the issues that we're experiencing > > with Rust integration testing. > > In summary, none of our tests pass (zero batch doesn't count :) ). > > This is mainly because of changes from the legacy padding in the 0.15.0 > > release, which we never made in Rust ( > > https://issues.apache.org/jira/browse/ARROW-6313). > > I honestly didn't see this at the time. > > > > Anyways, the implication is that Rust is using the 'legacy' format of > > 4-byte alignment, and fails to parse and read the ipc::Message into > > RecordBatches. > > > > I'm so far struggling with the fixes in Rust, so this work might take > long > > and likely won't make it to 1.0.0. I'll open JIRAs for whatever work is > > necessary. > > > > The Rust implementation appears to be behind with quite some > implementation > > details, so I'll try catching up by the next release. > > > > Regards > > Neville > > >
