Re: Parquet, Arrow, and Drill Roadmap

2017-05-02 Thread Padma Penumarthy
One thing I want to add is use_new_reader uses reader from parquet-mr library, where as default one is drill’s native reader which is supposed to be better, performance wise. But, it does not support complex types and we automatically switch to use reader from parquet library when we have to rea

Re: Parquet, Arrow, and Drill Roadmap

2017-05-02 Thread Jinfeng Ni
> > - What the two readers are (is one a special drill thing, is the other a > standard reader from the parquet project?) > - What is the eventual goal here... to be able to use and switch between > both? To provide the option? To have code parity with another project? Both readers were for readi

Parquet, Arrow, and Drill Roadmap

2017-05-01 Thread John Omernik
Hey all - I posted this to both dev and user as I could mentally make the argument for both, Sorry if this is answered somewhere already. I know in the past, there have been discussions around using two different readers for Parquet, and performance gains/losses, issues. etc. Right now, the store