Oh, thank you! It is true that I thought arrow2 was the one that was incorrect, but when I saw the low activity on recent issues, I assumed it was a project with lower maintenance capacities. Thank you for pointing out [3]!
On Wed, 2 Aug 2023 at 10:09, Raphael Taylor-Davies <r.taylordav...@googlemail.com.invalid> wrote: > Hi Anja, > > I believe this was clarified in [1] with the addition of the following > text to the standard > > > The buffer pointers MAY be null only in two situations: > > > > 1. for the null bitmap buffer, if :c:member:`ArrowArray.null_count` > is 0; > > 2. for any buffer, if the size in bytes of the corresponding buffer > would be 0. > > This is supported by arrow-rs [2], and I believe arrow2 is incorrect in > not supporting this. I would recommend taking this up with the arrow2 > maintainers [3]. > > Kind Regards, > > Raphael Taylor-Davies > > [1]: https://github.com/apache/arrow/pull/14808 > [2]: https://github.com/apache/arrow-rs/pull/3276 > [3]: https://github.com/jorgecarleitao/arrow2/pull/1476 > > On 02/08/2023 17:43, Anja wrote: > > Hi! > > > > I opened two issues recently on a spec incompatibility between arrow2 and > > arrow-C++: https://github.com/apache/arrow/issues/36960 and > > . > > > > Namely, there are situations where arrow-C++ will legally create a null > > buffer, and arrow2 does not allow null buffers. > > > > I am happy to provide code-effort, but I am looking for consensus from > the > > community on this issue. I think an interoperability issue like this is > > important to resolve, especially with the plans to unify arrow2 and > > arrow-rs: https://github.com/apache/arrow-rs/issues/1176 > > > > The `kNonNullFiller` ( > > > https://github.com/apache/arrow/blob/a06b2618420ef89431373a9e8f07a5da64d546a5/cpp/src/arrow/util/ubsan.h#L33 > ) > > exists for being used in cases where null buffers can cause issues. I am > > proposing changes such that it gets pointed to for representing the > reading > > of an empty table. > > > > Are there any thoughts? I welcome engagement on either of the issues. > This > > problem can, in theory, be solved within either project. > > > > ~Anja > > >
