Re: Spark support for update/delete operations on Hive ORC transactional tables

Thanks for the confirmation Mich! On Wednesday, June 22, 2016, Mich Talebzadeh wrote: > Hi Ajay, > > I am afraid for now transaction heart beat do not work through Spark, so I > have no other solution. > > This is interesting point as with Hive running on Spark engine

Re: Spark support for update/delete operations on Hive ORC transactional tables

Hi Ajay, I am afraid for now transaction heart beat do not work through Spark, so I have no other solution. This is interesting point as with Hive running on Spark engine there is no issue with this as Hive handles the transactions, I gather in simplest form Hive has to deal with its metadata

Re: Spark support for update/delete operations on Hive ORC transactional tables

Hi Mich, Right now I have a similar usecase where I have to delete some rows from a hive table. My hive table is of type ORC, Bucketed and included transactional property. I can delete from hive shell but not from my spark-shell or spark app. Were you able to find any work around? Thank you.

Re: Spark support for update/delete operations on Hive ORC transactional tables

thanks for that. I will have a look Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 2 June 2016 at

Re: Spark support for update/delete operations on Hive ORC transactional tables

Related to this, there exists an API in Hive to simplify the integrations of other frameworks with Hive's ACID feature: See: https://cwiki.apache.org/confluence/display/Hive/HCatalog+Streaming+Mutation+API It contains code for maintaining heartbeats, handling locks and transactions, and

Spark support for update/delete operations on Hive ORC transactional tables

Hi, Spark does not support transactions because as I understand there is a piece in the execution side that needs to send heartbeats to Hive metastore saying a transaction is still alive". That has not been implemented in Spark yet to my knowledge." Any idea on the timelines when we are going to