Hi, We use Hive to store user click logs and dig out interesting trends, doing a lot of big table joins. We mostly partition by date, and inside each partition data is bucketed by some hashed-id.
We are wondering how many buckets to use? If I understand correctly, Hive DB stores each bucket in one file. One file could be break down to several buckets based on bucket size. Is this one of the criteria to determine bucket size s.t. each file in one block? What else should be considered? Is there a rule-of-thumb in selecting how many buckets to use? Any insight/comment welcome! Thanks, Xun
