Re: better partitioning strategy in hive

2012-03-02 Thread Ravikumar MAV
.com > > "Best Trading Platform" - World Finance's Forex Awards 2009. > "The One to Watch" - Treasury Today's Adam Smith Awards 2009. > > > - Original Message - > From: "rk vishu" > To: cdh-u...@cloudera.org, common-user@hadoop.

Re: better partitioning strategy in hive

2012-03-02 Thread Mark Grover
Awards 2009. "The One to Watch" - Treasury Today's Adam Smith Awards 2009. - Original Message - From: "rk vishu" To: cdh-u...@cloudera.org, common-user@hadoop.apache.org, u...@hive.apache.org Sent: Saturday, February 18, 2012 4:39:48 AM Subject: Re: better partit

Re: better partitioning strategy in hive

2012-02-18 Thread rk vishu
> Hello All, > > We have a hive table partitioned by date and hour(330 columns). We have 5 > years worth of data for the table. Each hourly partition have around 800MB. > So total 43,800 partitions with one file per partition. > > When we run select count(*) from table, hive is taking for ever to s

better partitioning strategy in hive

2012-02-18 Thread rk vishu
Hello All, We have a hive table partitioned by date and hour(330 columns). We have 5 years worth of data for the table. Each hourly partition have around 800MB. So total 43,800 partitions with one file per partition. When we run select count(*) from table, hive is taking for ever to submit the jo