Re: Hive on spark

Thanks Guys for the help! Thanks, Mudit From: Mich Talebzadeh Reply-To: Date: Thursday, July 28, 2016 at 9:43 AM To: user Subject: Re: Hive on spark Hi, I made a presentation in London on 20th July on this subject:. In that I explained how to make Spark work as an execution engine for

Re: Hive compaction didn't launch

I made some research on that issue. The problem is in ValidCompactorTxnList::isTxnRangeValid method. Here's code: @Override public RangeResponse isTxnRangeValid(l

Fwd: Building Spark 2 from source that does not include the Hive jars

Anyone in Hive forum knows about this? Thanks This has worked before including 1.6.1 etc Build Spark without Hive jars. The idea being to use Spark as Hive execution engine. There is some notes on Hive on Spark: Getting Started

Re: Hive compaction didn't launch

Hive is doing the right thing there, as it cannot compact the deltas into a base file while there are still open transactions in the delta. Storm should be committing on some frequency even if it doesn’t have enough data to commit. Alan. > On Jul 28, 2016, at 05:36, Igor Kuzmenko wrote: > >

Re: Hive compaction didn't launch

But this *minOpenTxn* value isn't from from delta I want to compact. *minOpenTxn* can point on transaction in partition *A *while in partition *B *there's deltas ready for compaction. If *minOpenTxn* is less than txnIds in partition *B *deltas, compaction won't happen. So open transaction in partit

Re: Hive compaction didn't launch

But until those transactions are closed you don’t know that they won’t write to partition B. After they write to A they may choose to write to B and then commit. The compactor can not make any assumptions about what sessions with open transactions will do in the future. Alan. > On Jul 28, 20

Re: Hive compaction didn't launch

I think Storm has some timeout parameter that will close the transaction if there are no events for a certain amount of time. How many transactions do you per transaction batch? Perhaps making the batches smaller will make them close sooner. Eugene On 7/28/16, 3:59 PM, "Alan Gates" wrote: >Bu