Hi Ravindra, I have some doubts and suggestion,
1. Since for compaction, you are suggesting to keep the compacted segments as it is, it will be applicable for delete segment by id or date operation also right? 2. Since there is a proposal for moving delete delta file data to segment file, and since we have one segment file for one segment, if we do a multiple delete operation on a segment, then for each transaction multiple segment file is generated or in the same segment file, the transaction id of each delete operation is maintained? So, (i). if there will be multiple segment files for each delete operation transaction, any store structure gonna change? (ii). I suggest to keep may be a transaction id of delete operation as a key in segment file, which has multiple values which are basically multiple delete delta values. Which can help in reduce IO on segment file. 3. Select * from table1@tyyyyMMDDHHmmSS, will give the data at this point if transaction right? and will there be any support to get data between two transactions? basically it will be like getting delta data between two transaction. may be there won't be any use case, just a doubt. 4. Currently we have MAX_QUERY_EXECUTION_TIME which will clean the segments after the timeout, so since we are deciding to keep the compacted segments as it is without cleaning, mauy be we need to eliminate this property and introduce the transaction timeout for clean up, else exiting default may be increased. Regards, Akash R Nilugal On 2019/08/23 12:37:30, Ravindra Pesala <[email protected]> wrote: > Hi All, > > CarbonData allows to store the data incrementally and do the Update/Delete > operations on the stored data. But the user always can access the latest > state of data at that point of time. > > In the current system, it is not possible to access the old version of > data. And it is not possible to rollback to the old version in case of some > issues in current version data. > > This proposal adds the automatic versioning of data that we store and we > can access any historical version of that data. > > > The design is attached on the jira > https://issues.apache.org/jira/browse/CARBONDATA-3500 Please check it. > > -- > Thanks & Regards, > Ravindra. >
