Hi Wes, Thanks for the update. It would be interesting to add a centralized plan for tensors in Arrow. It would allow sharing data between packages like numpy, ndarray, pytorch, tensorflow really easy. Don't you think so? Let me have a look at how the integration tests are created in Archery so I can add some to start testing IPC in rust.
Thanks On Tue, Mar 16, 2021 at 3:16 PM Wes McKinney <wesmck...@gmail.com> wrote: > hi Fernando — for clarity, there is no centralized planning in this > project. If a volunteer wants to do something and there are no > objections from other people, then they are free to go ahead and do > it. If there aren't any Jira issues about adding integration tests, it > would make sense to go ahead and open some and clarify the scope of > what you would like to see get developed. > > On Tue, Mar 16, 2021 at 3:25 AM Antoine Pitrou <anto...@python.org> wrote: > > > > > > Hi Fernando, > > > > Currently there are no explicit plans to do it, but that would be > > certainly useful if other implementation start implementing tensor IPC > > support. > > > > One should start by defining a reference format (probably JSON) such as > > exists for other IPC types: > > https://arrow.apache.org/docs/format/Integration.html > > > > Regards > > > > Antoine. > > > > > > Le 16/03/2021 à 10:02, Fernando Herrera a écrit : > > > Are there any plans to include integration testing for tensors in the > > > pipeline? > > > > > > Thanks, > > > Fernando > > > > > > On Mon, Mar 15, 2021 at 8:16 PM Antoine Pitrou <anto...@python.org> > wrote: > > > > > >> On Mon, 15 Mar 2021 19:48:22 +0000 > > >> Fernando Herrera <fernando.j.herr...@gmail.com> wrote: > > >>> Hi Neal, > > >>> > > >>> Thanks for the update and the link. > > >>> > > >>> I found that the project has these files for tensor checking > > >>> > > >> > https://github.com/apache/arrow-testing/tree/e8ce32338f2dfeca3a5126f7677bdee159604000/data/arrow-ipc-tensor-stream > > >>> > > >>> So, if I understand correctly, for any application to be compatible > > >>> with C++ tensors it should be able to read these files. Am I correct? > > >> > > >> No, these are invalid files found by fuzz testing, that used to crash > > >> the C++ IPC reader. More information here: > > >> https://arrow.apache.org/docs/developers/cpp/fuzzing.html > > >> > > >> We don't have any reference files for integration testing of tensors > > >> and sparse tensors currently. > > >> > > >> Regards > > >> > > >> Antoine. > > >> > > >> > > >> > > > >
