+1 I am looking forward to this feature as most of the update/delete operations are simple and it can simplify and improve the performance as well. Thank you.
On Thu, 19 Nov 2020 at 19:41, Akash Nilugal <[email protected]> wrote: > Hi Community, > > Carbondata supports update and delete using spark. So basically update is > delete + Insert, and delete is just delete > But we use spark APIs or actions on collections that use spark jobs to do > them, like map, partition etc > So Spark adds overhead of task serialization cost, total job execution in > remote nodes, shuffle etc > So even just for simple updates, Carbon takes a lot of time, and the same > for delete as well due to these overheads. > > Carbondata 2.1.0 supports update and delete for SDK. This is implemented at > the carbon file format level > > so we can reuse the same for simple updates and deletes and avoid spark > completely and can perform simple update > > and delete on transactional tables using simple java code. This helps to > avoid all the overhead of spark and make > > updates and deletes faster. > > I have added an initial V1 design document, please check and give > comments/inputs/suggestions. > > > https://docs.google.com/document/d/1-M6xPKZG8l6yAu0c9qo3jdUKhpXHWgUR-h8HeUUmk8M/edit?usp=sharing > > Thanks, > > Regards, > Akash R Nilugal > -- Thanks & Regards, Ravi
