It is true that refactoring the IPC reader code to defer dictionary reassembly given out-of-order dictionaries would be some work. The worst case scenario in the short term is that this part of the C++ implementation is not implemented, and demonstrating that it works when they appear in depth-first/bottom-up order may be good enough for the 1.0 release.
On Thu, May 21, 2020 at 1:12 PM Antoine Pitrou <anto...@python.org> wrote: > > > Le 21/05/2020 à 19:46, Micah Kornfield a écrit : > > Hi Antoine, > > Can you expand on why that restriction is necessary/makes things easier? > > It seems a little strange since each dictionary batch has the ID attached, > > I wouldn't think it would be hard for the reader to track their arrival in > > any order. > > The problem is, you can't really instantiate the outer dictionary (as a > C++ Array) without having the inner dictionary already. We could work > around that by keeping the DictionaryBatch around and decoding it > lazily, but I don't know the cost of that refactor (nor the runtime > consequences, e.g. react later to an error in the stream). > > Regards > > Antoine.
