I think it is too late in the game to make this fundamental change. It would be very hard to assess whether it is no op or has massive implications to existing datasets. Just among Dremio customers in the 30 days we stored more than 100mm datasets that leveraged the current format.
I'm supportive of enforcing non nulls on the write side but I don't think we should change the current read behavior. On Sat, Jan 18, 2020, 7:28 AM Antoine Pitrou <anto...@python.org> wrote: > > Ok, I've asked on the flatbuffers ML: > https://groups.google.com/forum/#!topic/flatbuffers/PU2NG7ksWHU > > If we decide to do this, I agree it should probably be a 1.0.0 blocker. > > Best Regards > > Antoine. > > > Le 17/01/2020 à 07:28, Micah Kornfield a écrit : > > I too, couldn't find anything that says this would break backwards > > compatibility for the binary format. But it probably pays to open an > issue > > with the flatbuffer team just to be safe. > > > > Two points: > > 1. I'd like to make sure we are conservative in choosing "definitely > > required" > > 2. Before committing to the change, it would be good to get a sense of > how > > much this affects other language bindings (e.g. scope of work). > > 3. If we decide to this it seems like it should be a 1.0.0 blocker? >
