Re: [discuss]CarbonData update operation enhance

2020-09-22 Thread David CaiQiang
hi Linwood, 1. better to implement "Update feature enhancement" at first, it will create a new segment to store new files. http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Update-feature-enhancement-td99769.html 2. clean deletedelta files now carbon need

Re: [discuss]CarbonData update operation enhance

2020-09-22 Thread Liang Chen
Hi Thank you started this discussion. This proposal is for improving data updation performance, right ? Regards Liang Linwood wrote > *[Background]* > Update operation will clean up delta files before update( see > cleanUpDeltaFiles(carbonTable, false)), It's loop traversal metadata path > and

[discuss]CarbonData update operation enhance

2020-09-07 Thread Linwood
*[Background]* Update operation will clean up delta files before update( see cleanUpDeltaFiles(carbonTable, false)), It's loop traversal metadata path and segment path many times. When there are too many files, the overhead will increase and update time will be longer. *[Motivation & Goal]* During