Re: Spark :- Update record in partition.

2020-06-07 Thread ayan guha
Hi Please look at delta.io which is a companion open source project. It addresses the exact use case you are after. On Mon, Jun 8, 2020 at 2:35 AM Sunil Kalra wrote: > Hi All, > > If i have to update a record in partition using spark, do i have to read > the whole partition and update the row

Spark :- Update record in partition.

2020-06-07 Thread Sunil Kalra
Hi All, If i have to update a record in partition using spark, do i have to read the whole partition and update the row and overwrite the partition? Is there a way to only update 1 row like DBMS. Otherwise 1 row update takes a long time to rewrite the whole partition ? Thanks Sunil