Micah, you rock. That was the issue - thanks. On Sat, Feb 19, 2022 at 11:07 AM Micah Kornfield <[email protected]> wrote:
> Hi Chris, > It isn't not clear from you example, but I believe in the flatbuffer > schema description the second parameter is a bit-width [1] and not a > byte-width (so I would expect it to read "(MICROSECOND, 64)"). Could this > be the issue? > > Cheers, > Micah > > [1] https://github.com/apache/arrow/blob/master/format/Schema.fbs#L230 > > On Sat, Feb 19, 2022 at 8:43 AM Chris Nuernberger <[email protected]> > wrote: > >> I have an arrow IPC I wrote out that uses the 64 bit microsecond TIME >> datatype. >> >> When trying to read it in via python I get: >> >> pyarrow.lib.ArrowInvalid: Time is 64 bits for micro/nano unit >> >> The schema when printed prints: >> >> Schema<uints: Int(32, false) not null, longs: Int(64, true) not null, >> ubytes: Int(8, false) not null, strings: Utf8[dictionary: -1774527980], >> doubles: FloatingPoint(DOUBLE) not null, ushorts: Int(16, false) not null, >> local_times: Time(MICROSECOND, 8) not null, local_dates: Date(DAY) not >> null, ints: Int(32, true) not null, instants: Timestamp(MICROSECOND, ) not >> null, shorts: Int(16, true) not null, bytes: Int(8, true) not null, >> boolean: Bool not null, floats: FloatingPoint(SINGLE) not null, text: Utf8> >> >> I have narrowed it down to the local_times field which is '(MICROSECOND, >> 8) not null'. >> >> From my reading of the spec, micro and nano second times should be 64 >> bit, not 32 bit. >> >>
