Matias, Hive partitions map to subdirectories in HDFS. You can do a 'mv' if you're lucky enough to have each partition in a distinct HDFS file that could be moved to the right partition subdirectory. Otherwise, you can run a MapReduce job to collate your data into separate files per partition. You can use MultipleOutputs to do this for you. Hive trunk also supports dynamic partitions which would allow you to do this with a Hive statement instead of writing Java MR code.
Regards, - Patrick On Mon, May 10, 2010 at 7:55 PM, Matias Silva <[email protected]>wrote: > Hi Everyone, thanks for your time. What's the best way to repartition one > table into 3 partitions using a > replication factor of 3? We have anywhere between 100-150 TB in this > table. I would like to avoid > copying data over. Any suggestions? > > From what I understand that Hadoop/Hive is file based and is there any > chance to define the new table > with the new partitions and then perform a mv instead of cp on the data? > Am I dreaming? > > Thanks, > Matt > > > > > > > >
