I don't have a strong opinion since, I'm not sure what use-case requires the nested dictionary types, but it seems like we should keep the specification but it seems more natural to writers to write the outer dictionary first. Keeping this as a limitation of the C++ implementation seems reasonable for the time being until other languages have had the opportunity to support this.
Antoine are you actively working on something that will require nested dictionaries or is this simply for completeness? Thanks, Micah On Thu, May 21, 2020 at 6:08 PM Wes McKinney <[email protected]> wrote: > It is true that refactoring the IPC reader code to defer dictionary > reassembly given out-of-order dictionaries would be some work. The > worst case scenario in the short term is that this part of the C++ > implementation is not implemented, and demonstrating that it works > when they appear in depth-first/bottom-up order may be good enough for > the 1.0 release. > > On Thu, May 21, 2020 at 1:12 PM Antoine Pitrou <[email protected]> wrote: > > > > > > Le 21/05/2020 à 19:46, Micah Kornfield a écrit : > > > Hi Antoine, > > > Can you expand on why that restriction is necessary/makes things > easier? > > > It seems a little strange since each dictionary batch has the ID > attached, > > > I wouldn't think it would be hard for the reader to track their > arrival in > > > any order. > > > > The problem is, you can't really instantiate the outer dictionary (as a > > C++ Array) without having the inner dictionary already. We could work > > around that by keeping the DictionaryBatch around and decoding it > > lazily, but I don't know the cost of that refactor (nor the runtime > > consequences, e.g. react later to an error in the stream). > > > > Regards > > > > Antoine. >
