So, will it be ok if we have 80 data nodes (8TB on each node) and only one namenode? Will it works for the messaging system? We will have 2x replication so there are 320 TB of data (per year) (640 TB with replication). 130000 R+W ops/sec. Each message 100 bytes or 1024 bytes. Is it possible to handle such load with hbase?
Sincerely, Alexandr On Sat, Aug 20, 2016 at 8:44 AM, Dima Spivak <dspi...@cloudera.com> wrote: > You can easily store that much data as long as you don't have small files, > which is typically why people turn to federation. > > -Dima > > On Friday, August 19, 2016, Alexandr Porunov <alexandr.poru...@gmail.com> > wrote: > > > We are talking about facebook. So, there are 25 TB per month. 15 billion > > messages with 1024 bytes and 120 billion messages with 100 bytes per > month. > > > > I thought that they used only hbase to handle such a huge data If they > used > > their own implementation of hbase then I haven't questions. > > > > Sincerely, > > Alexandr > > > > On Sat, Aug 20, 2016 at 1:39 AM, Dima Spivak <dspi...@cloudera.com > > <javascript:;>> wrote: > > > > > I'd +1 what Vladimir says. How much data (in TBs/PBs) and how many > files > > > are we talking about here? I'd say that use cases that benefit from > HBase > > > don't tend to hit the kind of HDFS file limits that federation seeks to > > > address. > > > > > > -Dima > > > > > > On Fri, Aug 19, 2016 at 2:19 PM, Vladimir Rodionov < > > vladrodio...@gmail.com <javascript:;> > > > > > > > wrote: > > > > > > > FB has its own "federation". It is a proprietary code, I presume. > > > > > > > > -Vladimir > > > > > > > > > > > > On Fri, Aug 19, 2016 at 1:22 PM, Alexandr Porunov < > > > > alexandr.poru...@gmail.com <javascript:;>> wrote: > > > > > > > > > No. There isn't. But I want to figure out how to configure that > type > > of > > > > > cluster in the case if there is particular reason. How facebook can > > > > handle > > > > > such a huge amount of ops without federation? I don't think that > they > > > > just > > > > > have one namenode server and one standby namenode server. It isn't > > > > > possible. I am sure that they use federation. > > > > > > > > > > On Fri, Aug 19, 2016 at 10:08 PM, Vladimir Rodionov < > > > > > vladrodio...@gmail.com <javascript:;>> > > > > > wrote: > > > > > > > > > > > >> I am not sure how to do it but I have to configure federated > > > cluster > > > > > > with > > > > > > >> hbase to store huge amount of messages (client to client) (40% > > > > writes, > > > > > > 60% > > > > > > >> reads). > > > > > > > > > > > > Any particular reason for federated cluster? How huge is huge > > amount > > > > and > > > > > > what is the message size? > > > > > > > > > > > > -Vladimir > > > > > > > > > > > > On Fri, Aug 19, 2016 at 11:57 AM, Dima Spivak < > > dspi...@cloudera.com <javascript:;>> > > > > > > wrote: > > > > > > > > > > > > > As far as I know, HBase doesn't support spreading tables across > > > > > > namespaces; > > > > > > > you'd have to point it at one namenode at a time. I've heard of > > > > people > > > > > > > trying to run multiple HBase instances in order to get access > to > > > all > > > > > > their > > > > > > > HDFS data, but it doesn't tend to be much fun. > > > > > > > > > > > > > > -Dima > > > > > > > > > > > > > > On Fri, Aug 19, 2016 at 11:51 AM, Alexandr Porunov < > > > > > > > alexandr.poru...@gmail.com <javascript:;>> wrote: > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > I am not sure how to do it but I have to configure federated > > > > cluster > > > > > > with > > > > > > > > hbase to store huge amount of messages (client to client) > (40% > > > > > writes, > > > > > > > 60% > > > > > > > > reads). Does somebody have any idea or examples how to > > configure > > > > it? > > > > > > > > > > > > > > > > Of course we can configure hdfs in a federated mode but as > for > > me > > > > it > > > > > > > isn't > > > > > > > > suitable for hbase. If we want to save message from client 1 > to > > > > > client > > > > > > 2 > > > > > > > in > > > > > > > > the hbase cluster then how hbase know in which namespace it > > have > > > to > > > > > > save > > > > > > > > it? Which namenode will be responsible for that message? How > we > > > can > > > > > > read > > > > > > > > client messages? > > > > > > > > > > > > > > > > Give me any ideas, please > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > Alexandr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -Dima > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > -Dima > > > > > > > > -- > -Dima >
