Partition by directory

2012-12-10 Thread Erik Thorson
Hello All, I have been using the AWS setup for EMR for some time now and I am currently in the process of implementing spark/shark on my own cluster. I am installing from https://github.com/downloads/mesos/spark/spark-0.6.0-sources.tar.gz. Which includes hive0.9.0. I am using this with s3 and a

Re: Locking in HIVE : How to use locking/unlocking features using hive java API ?

2012-12-10 Thread Manish Malhotra
Thanks Ruslan, Please see my inline comments, Why do you need metadata backup? Can't you just store all the table create statements in an init file? MM: Because I don't want to depend on the init script that will have all the entries for all the tables. And this backup tool should be independent

Re: FROM INSERT after ADD COLUMN

2012-12-10 Thread yaboulna
Is there an index in the RC File to avoid a complete pass on the record "keys" for matching old and new records. Also, wouldn't the RCFile need to be rebuilt anyway, since the file actually stores blocks of n rows by m column achieving a certain block size? I haven't carefully read the RCFi

Re: FROM INSERT after ADD COLUMN

2012-12-10 Thread Shreepadma Venugopalan
Sorry hit the send too soon :) While storing data in a column major format such as RCFile would help with adding new column data after executing an alter table...add columns statement, Hive doesn't provide a way to do it today. It is possible to do so outside of Hive today, but we would need to en

Re: FROM INSERT after ADD COLUMN

2012-12-10 Thread Shreepadma Venugopalan
On Sun, Dec 9, 2012 at 10:32 PM, Bertrand Dechoux wrote: > I will reopen the subject a bit. > > I don't know the details of the RCFile implementation in Hive but if the > data were stored that way it is theoretically possible to add the column > data even without append and without rewriting the w

RE: PK violation during Hive add partition

2012-12-10 Thread Karlen Lie
Thanks! Looks like I've missed enabling the concurrency flag. -karlen From: Edward Capriolo [mailto:edlinuxg...@gmail.com] Sent: Monday, December 10, 2012 7:19 AM To: user@hive.apache.org Subject: Re: PK violation during Hive add partition This also could be an issue with datanucleas and m$ sql

Re: Running commands at hive cli or hive thirft startup

2012-12-10 Thread विनोद सिंह
I did not tried thrift, so not sure whether this will work or not. Thanks, Vinod On Mon, Dec 10, 2012 at 9:31 PM, John Omernik wrote: > Will that work for my thrift server connections? > > > On Sun, Dec 9, 2012 at 7:56 PM, विनोद सिंह wrote: > >> Put a .hiverc file in your home directory contai

Re: Running commands at hive cli or hive thirft startup

2012-12-10 Thread John Omernik
Will that work for my thrift server connections? On Sun, Dec 9, 2012 at 7:56 PM, विनोद सिंह wrote: > Put a .hiverc file in your home directory containing commands, Hive CLI > will execute all of them at startup. > > Thanks, > Vinod > > On Sun, Dec 9, 2012 at 10:25 PM, John Omernik wrote: > >> I

Re: PK violation during Hive add partition

2012-12-10 Thread Edward Capriolo
This also could be an issue with datanucleas and m$ sql server. The project only officially supports derby and MySQL. Only tests using derby. Everything else is at your own risk. On Mon, Dec 10, 2012 at 10:08 AM, Ruslan Al-Fakikh wrote: > Hi! > > Have you enabled Hive concurrency? Hive should not

Re: Locking in HIVE : How to use locking/unlocking features using hive java API ?

2012-12-10 Thread Ruslan Al-Fakikh
Hi Manish! Why do you need metadata backup? Can't you just store all the table create statements in an init file? If you care about Partitions that have been created dynamically then you can restore them from data by RECOVER PARTITIONS (if using Amazon EMR) or an analog check command for a regular

Re: PK violation during Hive add partition

2012-12-10 Thread Ruslan Al-Fakikh
Hi! Have you enabled Hive concurrency? Hive should not be accessed concurrently if the appropriate property is not enabled. Ruslan On Sat, Dec 8, 2012 at 6:01 AM, Karlen Lie wrote: > nal table, and the query below is run concurrently by multiple oo

Re: Locking in HIVE : How to use locking/unlocking features using hive java API ?

2012-12-10 Thread Manish Malhotra
Sending again, as got no response. Can somebody from Hive dev group please review my approach and reply? Cheers, Manish On Thu, Dec 6, 2012 at 11:17 PM, Manish Malhotra < manish.hadoop.w...@gmail.com> wrote: > Hi, > > I'm building / designing a back-up and restore tool for hive data for > Disa