Maarten, I don't expect regressions for flat cases (I'm going to try to run benchmarks comparison tonight).
In terms of the flag, I'm more concerned about some corner case I didn't think of in testing or a workload that for some reason is better with the prior code. If either of these arise I would like to give users a way to mitigate issues so a patch release isn't an imperative. I'm happy to cleanup the existing code as soon as the next release happens though. Thoughts? -Micah On Thu, Mar 12, 2020 at 9:11 AM Wes McKinney <wesmck...@gmail.com> wrote: > Maarten -- AFAIK Micah's work only affects nested / non-flat column > paths, so flat data should not be impacted. Since we have a partial > implementation of writes for nested data (lists-of-lists and > structs-of-structs, but no mix of the two) that was the performance > difference I was referencing. > > On Thu, Mar 12, 2020 at 10:43 AM Maarten Ballintijn <maart...@xs4all.nl> > wrote: > > > > Hi Micah, > > > > How does the performance change for “flat” schemas? > > (particularly in the case of a large number of columns) > > > > Thanks, > > Maarten > > > > > > > > > On Mar 11, 2020, at 11:53 PM, Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > > > Another status update. I've integrated the level generation code with > the > > > parquet writing code [1]. > > > > > > After that PR is merged I'll add bindings in Python to control > versions of > > > the level generation algorithm and plan on moving on to the read side. > > > > > > Thanks, > > > Micah > > > > > > [1] https://github.com/apache/arrow/pull/6586 > > > > > > On Tue, Mar 3, 2020 at 9:07 PM Micah Kornfield <emkornfi...@gmail.com> > > > wrote: > > > > > >> Hi Igor, > > >> If you have the time https://issues.apache.org/jira/browse/ARROW-7960 > might > > >> be a good task to pick up for this I think it should be a relatively > small > > >> amount of code, so it is probably a good contribution to the > project. Once > > >> that is wrapped up we can see were we both are. > > >> > > >> Cheers, > > >> Micah > > >> > > >> On Tue, Mar 3, 2020 at 8:25 AM Igor Calabria <igor.calab...@gmail.com > > > > >> wrote: > > >> > > >>> Hi Micah, I actually got involved with another personal project and > had > > >>> to postpone my contribution to arrow a bit. The good news is that I'm > > >>> almost done with it, so I could help you with the read side very > soon. Any > > >>> ideas how we could coordinate this? > > >>> > > >>> Em qua., 26 de fev. de 2020 às 21:06, Wes McKinney < > wesmck...@gmail.com> > > >>> escreveu: > > >>> > > >>>> hi Micah -- great news on the level generation PR. I'll try to carve > > >>>> out some time for reviewing over the coming week. > > >>>> > > >>>> On Wed, Feb 26, 2020 at 3:10 AM Micah Kornfield < > emkornfi...@gmail.com> > > >>>> wrote: > > >>>>> > > >>>>> Hi Igor, > > >>>>> I was wondering if you have made any progress on this? > > >>>>> > > >>>>> I posted a new PR [1] which I believe handles the difficult > algorithmic > > >>>>> part of writing. There will be some follow-ups but I think this PR > > >>>> might > > >>>>> take a while to review, so I was thinking of starting to take a > look > > >>>> at the > > >>>>> read side if you haven't started yet, and circle back to the final > > >>>>> integration for the write side once the PR is checked in. > > >>>>> > > >>>>> Thanks, > > >>>>> Micah > > >>>>> > > >>>>> [1] https://github.com/apache/arrow/pull/6490 > > >>>>> > > >>>>> On Mon, Feb 3, 2020 at 4:08 PM Igor Calabria < > igor.calab...@gmail.com> > > >>>>> wrote: > > >>>>> > > >>>>>> Hi, I would love to help with this issue. I'm aware that this is a > > >>>> huge > > >>>>>> task for a first contribution to arrow, but I feel that I could > help > > >>>> with > > >>>>>> the read path. > > >>>>>> Reading parquet seems like a extremely complex task since both > > >>>> hive[0] and > > >>>>>> spark[1] tried to implement a "vectorized" version and they all > > >>>> stopped > > >>>>>> short of supporting complex types. > > >>>>>> I wanted to at least give it a try and find out where the > challenge > > >>>> lies. > > >>>>>> > > >>>>>> Since you guys are much more familiar with the current code base, > I > > >>>> could > > >>>>>> use some starting tips so I don't fall in common pitfalls and > > >>>> whatnot. > > >>>>>> > > >>>>>> [0] https://issues.apache.org/jira/browse/HIVE-18576 > > >>>>>> [1] > > >>>>>> > > >>>>>> > > >>>> > https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java#L45 > > >>>>>> > > >>>>>> On 2020/02/03 06:01:25, Micah Kornfield <e...@gmail.com> wrote: > > >>>>>>> Just to give an update. I've been a little bit delayed, but my > > >>>> progress > > >>>>>> is> > > >>>>>>> as follows:> > > >>>>>>> 1. Had 1 PR merged that will exercise basic end-to-end tests.> > > >>>>>>> 2. Have another PR open that allows a configuration option in > C++ > > >>>> to> > > >>>>>>> determine which algorithm version to use for reading/writing, the > > >>>>>> existing> > > >>>>>>> version and the new version supported complex-nested arrays. I > > >>>> think a> > > >>>>>>> large amount of code will be reused/delegated to but I will err > on > > >>>> the > > >>>>>> side> > > >>>>>>> of not touching the existing code/algorithms so that any errors > in > > >>>> the> > > >>>>>>> implementation or performance regressions can hopefully be > > >>>> mitigated at> > > >>>>>>> runtime. I expect in later releases (once the code has "baked") > > >>>> will> > > >>>>>>> become a no-op.> > > >>>>>>> 3. Started coding the write path.> > > >>>>>>> > > >>>>>>> Which leaves:> > > >>>>>>> 1. Finishing the write path (I estimate 2-3 weeks) to be code > > >>>> complete> > > >>>>>>> 2. Implementing the read path.> > > >>>>>>> > > >>>>>>> Again, I'm happy to collaborate if people have bandwidth and want > > >>>> to> > > >>>>>>> contribute.> > > >>>>>>> > > >>>>>>> Thanks,> > > >>>>>>> Micah> > > >>>>>>> > > >>>>>>> On Thu, Jan 9, 2020 at 10:31 PM Micah Kornfield <em...@gmail.com > >> > > >>>>>>> wrote:> > > >>>>>>> > > >>>>>>>> Hi Wes,> > > >>>>>>>> I'm still interested in doing the work. But don't to hold > > >>>> anybody up > > >>>>>> if> > > >>>>>>>> they have bandwidth.> > > >>>>>>>>> > > >>>>>>>> In order to actually make progress on this, my plan will be to:> > > >>>>>>>> 1. Help with the current Java review backlog through early next > > >>>> week > > >>>>>> or> > > >>>>>>>> so (this has been taking the majority of my time allocated for > > >>>> Arrow> > > >>>>>>>> contributions for the last 6 months or so).> > > >>>>>>>> 2. Shift all my attention to trying to get this done (this > > >>>> means no> > > >>>>>>>> reviews other then closing out existing ones that I've started > > >>>> until it > > >>>>>> is> > > >>>>>>>> done). Hopefully, other Java committers can help shrink the > > >>>> backlog> > > >>>>>>>> further (Jacques thanks for you recent efforts here).> > > >>>>>>>>> > > >>>>>>>> Thanks,> > > >>>>>>>> Micah> > > >>>>>>>>> > > >>>>>>>> On Thu, Jan 9, 2020 at 8:16 AM Wes McKinney <we...@gmail.com> > > >>>> wrote:> > > >>>>>>>>> > > >>>>>>>>> hi folks,> > > >>>>>>>>>> > > >>>>>>>>> I think we have reached a point where the incomplete C++ > > >>>> Parquet> > > >>>>>>>>> nested data assembly/disassembly is harming the value of > > >>>> several> > > >>>>>>>>> others parts of the project, for example the Datasets API. As > > >>>> another> > > >>>>>>>>> example, it's possible to ingest nested data from JSON but not > > >>>> write> > > >>>>>>>>> it to Parquet in general.> > > >>>>>>>>>> > > >>>>>>>>> Implementing the nested data read and write path completely is > > >>>> a> > > >>>>>>>>> difficult project requiring at least several weeks of dedicated > > >>>> work,> > > >>>>>>>>> so it's not so surprising that it hasn't been accomplished yet. > > >>>> I > > >>>>>> know> > > >>>>>>>>> that several people have expressed interest in working on it, > > >>>> but I> > > >>>>>>>>> would like to see if anyone would be able to volunteer a > > >>>> commitment > > >>>>>> of> > > >>>>>>>>> time and guess on a rough timeline when this work could be > > >>>> done. It> > > >>>>>>>>> seems to me if this slips beyond 2020 it will significant > > >>>> diminish > > >>>>>> the> > > >>>>>>>>> value being created by other parts of the project.> > > >>>>>>>>>> > > >>>>>>>>> Since I'm pretty familiar with all the Parquet code I'm one > > >>>> candidate> > > >>>>>>>>> person to take on this project (and I can dedicate the time, > > >>>> but it> > > >>>>>>>>> would come at the expense of other projects where I can also > be> > > >>>>>>>>> useful). But Micah and others expressed interest in working on > > >>>> it, so> > > >>>>>>>>> I wanted to have a discussion about it to see what others > > >>>> think.> > > >>>>>>>>>> > > >>>>>>>>> Thanks> > > >>>>>>>>> Wes> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>>>> > > >>>> > > >>> > > >
