Re: Delta Lake support for DataFusion

2021-06-10 Thread Jorge Cardoso Leitão
and whatever else was needed to integrate DataFusion > with > > > > DeltaLake > > > > > > > > This structure could also start a pattern of publishing plugins for > > > > DataFusion separately from the core. > > > > > > > > Andrew >

Re: Delta Lake support for DataFusion

2021-06-10 Thread QP Hou
> > This structure could also start a pattern of publishing plugins for > > > DataFusion separately from the core. > > > > > > Andrew > > > p.s. now that Arrow is publishing more incrementally (e.g. 4.1.0, 4.2.0, > > > etc), I think delta-rs[1] and

Re: Delta Lake support for DataFusion

2021-06-09 Thread Andrew Lamb
now that Arrow is publishing more incrementally (e.g. 4.1.0, > 4.2.0, > > > etc), I think delta-rs[1] and datafusion both only specify `4.x` so > they > > > should work together nicely > > > > > > https://github.com/delta-io/delta-rs/blame/main/rust

Re: Delta Lake support for DataFusion

2021-06-09 Thread Daniël Heres
tps://github.com/delta-io/delta-rs/blame/main/rust/Cargo.toml > > > > On Wed, Jun 9, 2021 at 2:29 AM Daniël Heres > wrote: > > > > > Hi all, > > > > > > I would like to receive some feedback about adding Delta Lake support > to > >

Re: Delta Lake support for DataFusion

2021-06-09 Thread Neville Dipale
her nicely > > https://github.com/delta-io/delta-rs/blame/main/rust/Cargo.toml > > On Wed, Jun 9, 2021 at 2:29 AM Daniël Heres wrote: > > > Hi all, > > > > I would like to receive some feedback about adding Delta Lake support to > > DataFusion (https://github.

Re: Delta Lake support for DataFusion

2021-06-09 Thread Andrew Lamb
Wed, Jun 9, 2021 at 2:29 AM Daniël Heres wrote: > Hi all, > > I would like to receive some feedback about adding Delta Lake support to > DataFusion (https://github.com/apache/arrow-datafusion/issues/525). > As you might know, Delta Lake <https://delta.io/> is a format adding &g

Re: Delta Lake support for DataFusion

2021-06-09 Thread Jorge Cardoso Leitão
Hi, Some questions that come to mind: 1. If we add vendor X to datafusion, will we be open to other vendor Y? How do we compare vendors? How do we draw the line of "not sufficiently relevant"? 2. How do we ensure that we do not distort the same level playing field that some people expect from

Delta Lake support for DataFusion

2021-06-09 Thread Daniël Heres
Hi all, I would like to receive some feedback about adding Delta Lake support to DataFusion (https://github.com/apache/arrow-datafusion/issues/525). As you might know, Delta Lake <https://delta.io/> is a format adding features like ACID transactions, statistics, and storage optimi