Hi Community, Carbondata supports update and delete using spark. So basically update is delete + Insert, and delete is just delete But we use spark APIs or actions on collections that use spark jobs to do them, like map, partition etc So Spark adds overhead of task serialization cost, total job execution in remote nodes, shuffle etc So even just for simple updates, Carbon takes a lot of time, and the same for delete as well due to these overheads.
Carbondata 2.1.0 supports update and delete for SDK. This is implemented at the carbon file format level so we can reuse the same for simple updates and deletes and avoid spark completely and can perform simple update and delete on transactional tables using simple java code. This helps to avoid all the overhead of spark and make updates and deletes faster. I have added an initial V1 design document, please check and give comments/inputs/suggestions. https://docs.google.com/document/d/1-M6xPKZG8l6yAu0c9qo3jdUKhpXHWgUR-h8HeUUmk8M/edit?usp=sharing Thanks, Regards, Akash R Nilugal
