Per Seffensen, I would urge you to step away from the keyboard and rethink your design. It sounds like you want to replicate a date partition model similar to what you would do if you were attempting this with HBase.
HBase is not a relational database and you have a different way of doing things. You could put the date/time stamp in the key such that your data is sorted by date. However, this would cause hot spots. Think about how you access the data. It sounds like you access the more recent data more frequently than historical data. This is a bad idea in HBase. (note: it may still make sense to do this ... You have to think more about the data and consider alternatives.) I personally would hash the key for even distribution, again depending on the data access pattern. (hashed data means you can't do range queries but again, it depends on what you are doing...) You also have to think about how you purge the data. You don't just drop a region. Doing a full table scan once a month to delete may not be a bad thing. Again it depends on what you are doing... Just my opinion. Others will have their own... Now I'm stepping away from the keyboard to get my morning coffee... :-) Sent from a remote device. Please excuse any typos... Mike Segel On Dec 8, 2011, at 7:13 AM, Per Steffensen <st...@designware.dk> wrote: > Hi > > The system we are going to work on will receive 50mio+ new datarecords every > day. We need to keep a history of 2 years of data (thats 35+ billion > datarecords in the storage all in all), and that basically means that we also > need to delete 50mio+ datarecords every day, or e.g. 1,5 billion every month. > We plan to store the datarecords in HBase. > > Is it somehow possible to tell HBase to put (route) all datarecords belonging > to a specific date or month to a designated set of regions (and route nothing > else there), so that deleting all data belonging to that day/month i > basically deleting those regions entirely? And is explicit deletion of entire > regions possible at all? > > The reason I want to do this is that I expect it to be much faster than doing > explicit deletion record by record of 50mio+ records every day. > > Regards, Per Steffensen > > >
