Somewhat related issue: https://issues.apache.org/jira/browse/ARROW-10406
On Wed, Mar 17, 2021 at 11:22 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > BTW, this nuance always felt a little strange to me, but would have > required adding additional information to the file format, to disambiguate > when exactly a dictionary was intended to be replaced. > > On Wed, Mar 17, 2021 at 11:19 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > Hi Jacob, > > There is nuance. The file format does not support dictionary > replacement, > > the specification [1] why that is currently the case. Only the "stream > > format" supports replacement (i.e. no magic number, only schema followed > by > > one or more dictionary/record-batch messages). > > > > -Micah > > > > [1] https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format > > > > On Wed, Mar 17, 2021 at 11:04 PM Jacob Quinn <quinn.jac...@gmail.com> > > wrote: > > > >> Had an issue come up here: > >> https://github.com/JuliaData/Arrow.jl/issues/129#issuecomment-777350450 > . > >> From the implementation status page, it says C++ supports replacement > >> dictionaries and that python tracks the C++ implementation. Is this > just a > >> pyarrow issue where it specifically doesn't support replacement > >> dictionaries? Or it's not "hooked in" properly? > >> > >> -Jacob > >> > > >
