5B logs a day? => Yes, 5B/day You store a single log into a single cell? => No, I store more than 50 cells because with extra information.
The log in the hbase cell is compressed or you have enabled compression on the hbase column family? => enabled compression on the hbase column family What is your test environment like? Is it a single server? How are you running your compare? => about 100 region server, and I tested hive sql vs hbase scan from same origin data. (origin data -> put hbase, convert orc file -> test hive sql vs hbase scan) Best regards, Minwoo Kang ________________________________________ 보낸 사람: Stack <st...@duboce.net> 대신 saint....@gmail.com <saint....@gmail.com> 보낸 날짜: 2018년 5월 24일 목요일 01:33 받는 사람: Hbase-User 제목: Re: How to improve HBase read performance. On Wed, May 16, 2018 at 7:30 PM, Kang Minwoo <minwoo.k...@outlook.com> wrote: > Here is information. > > store about 5 billion a day. > 5B logs a day? > size of each log is about 4k. > You store a single log into a single cell? > compression by snappy. > > The log in the hbase cell is compressed or you have enabled compression on the hbase column family? What is your test environment like? Is it a single server? How are you running your compare? Thanks, M > Best regards, > Minwoo Kang > > ________________________________________ > 보낸 사람: Ted Yu <yuzhih...@gmail.com> > 보낸 날짜: 2018년 5월 17일 목요일 00:01 > 받는 사람: user@hbase.apache.org > 제목: Re: How to improve HBase read performance. > > bq. store a lot of logs in HBase > > Kang: > Can you tell us a bit more about how you store (and access) the log(s) - > size of each log, whether log is encoded in hbase ? > > ORC is columnar format while hbase uses different format. > > Thanks > > On Wed, May 16, 2018 at 6:41 AM, Marcell Ortutay <mortu...@23andme.com> > wrote: > > > This ticket: https://issues.apache.org/jira/browse/HBASE-20459 was fixed > > in > > the latest version of HBase, upgrading to latest may help with > performance > > > > On Wed, May 16, 2018 at 3:55 AM, Kang Minwoo <minwoo.k...@outlook.com> > > wrote: > > > > > Hi, Users. > > > > > > I store a lot of logs in HBase. > > > However, the reading speed of the log is too slow compared to the Hive > > ORC > > > file. > > > I know that HBase is slow compared to the Hive ORC file. > > > The problem is that it is too slow. > > > HBase is about 6 times slower. > > > > > > Is there a good way to speed up HBase's reading speed? > > > Should I put a lot of servers? > > > > > > I am using HBase 1.2.6. > > > > > > Best regards, > > > Minwoo Kang > > > > > >
