Hi Ravindra,
I have some questions regarding the feature:
1. *What would be the behaviour if the user just fires a 'select * from
table'(non-transaction query)?*
Would we still read the transaction file to get the latest tablestatus
file name or would we keep the latest transaction in cache.
My concern is that this may impact the query performance as the
transaction file grows.
2. *Would the user be able to create child datamaps like 'preaggregate',
'mv', 'bloom' with some transaction id?*
ex: create datamap dm1 using 'mv' as select * from maintable where
'somefilter' @YYYYmmDDHHmmSS
*I think this scenario can be blocked.*
3.* Would the user be able to clean the retention files using 'clean files'
or some new DDL would be exposed for transaction cleanup?*
4. *Impacted areas should include index server as the transaction details
have to be send to the Server for pruning.*
5. *Impacted Area Point 8: 'Alter table operations should open and close
the transaction'*
Does this mean that for each alter operation a transaction entry would
be maintained and the user can query the old schema by specifying the
transaction id before that operation? If yes then would multiple
versions of schema files be maintained?
6. *Can the user travel both ways in the revert command?*
First reset to an old transaction id and then come back to the latest
ID?
+1 for not removing the compacted segments immediately to maintain
transaction history.
Thanks
Kunal Kapoor
On Fri, Aug 23, 2019 at 6:08 PM Ravindra Pesala <[email protected]>
wrote:
> Hi All,
>
> CarbonData allows to store the data incrementally and do the Update/Delete
> operations on the stored data. But the user always can access the latest
> state of data at that point of time.
>
> In the current system, it is not possible to access the old version of
> data. And it is not possible to rollback to the old version in case of some
> issues in current version data.
>
> This proposal adds the automatic versioning of data that we store and we
> can access any historical version of that data.
>
>
> The design is attached on the jira
> https://issues.apache.org/jira/browse/CARBONDATA-3500 Please check it.
>
> --
> Thanks & Regards,
> Ravindra.
>
