Re: Re: [DISCUSS][DataFusion] Table time travel support

2023-08-21 Thread Akshara Uke
Thanks Marko for the detailed answer and the references. Cheers! On Mon, Aug 21, 2023 at 1:57 PM Marko Grujic wrote: > Hi Akshara, > > > Just for my understanding - the proposal assumes that writes will result > in > > a new table version correct? > > Actually, the implementation I had in mind

RE: Re: [DISCUSS][DataFusion] Table time travel support

2023-08-21 Thread Marko Grujic
Hi Akshara, > Just for my understanding - the proposal assumes that writes will result in > a new table version correct? Actually, the implementation I had in mind does not make any assumptions about the behaviour of writes, it only accounts for the fact that there may be different versions of

Re: [DISCUSS][DataFusion] Table time travel support

2023-08-19 Thread Akshara Uke
Hi Marko, Indeed most databases do support time travel/stale reads (specially distributed databases) , hence an important feature,IMHO. Just for my understanding - the proposal assumes that writes will result in a new table version correct? Asking since, some databases provide stale read support

[DISCUSS][DataFusion] Table time travel support

2023-08-17 Thread Marko Grujic
Hi all! I'm wondering what people think of a possibility to extend DataFusion so as to accommodate time-travel querying? This would work well with the new table formats, particularly Iceberg and Delta Lake, where table versioning is at the core of the protocol. You can see some details in the iss