> I hope we could strengthen our faith in HBase capability Us too. Would you be interested in taking the metrics and discussion of them that came out in this thread into a post for the HBase project blog (https://blogs.apache.org/hbase)? As you can see from the other blog entries details about the use case does not need to reveal proprietary information, readers would be most interested in the metrics you observed/achieved on 11/11 followed by a technical discussion of how (roughly) to replicate them. You have good command of the English language so that won't be a problem and anyway I offer my services as editor should you like to try. Think about it. This would be a great post. I am sure, very popular.
> On Nov 22, 2016, at 12:51 AM, Yu Li <car...@gmail.com> wrote: > > bq. If it were not "confidential" might you mention why there is such a > large (several orders of magnitude) explosion of end user queries to > backend ones? > For index building and online machine learning system, there're more > information recorded after each visit/trade, such as user query/click > history, item stock updates, etc., and multiple user-specific feature data > will be read/updated for better recommendation. The flow is pretty much > like: > user visit some items > -> put them into shopping cart > -> checkout/removing item from shopping cart > -> item stock update/recommend new items to user > -> user visit new items > Not that much details could be supplied but I believe we could imagine how > many queries/updates there will be at backend for such loops, right? (smile) > > Thanks again for the interest and questions although a little bit derail of > the thread, and I hope we could strengthen our faith in HBase capability > after these discussions. :-) > > Best Regards, > Yu > >> On 21 November 2016 at 01:26, Stephen Boesch <java...@gmail.com> wrote: >> >> Thanks Yu - given your apparent direct knowledge of the data that is >> helpful (my response earlier had been to 张铎) . It is important so as to >> ensure informing colleagues of numbers that are "real". >> >> If it were not "confidential" might you mention why there is such a large >> (several orders of magnitude) explosion of end user queries to backend >> ones? >> >> >> >> 2016-11-20 7:51 GMT-08:00 Yu Li <car...@gmail.com>: >> >>> Thanks everyone for the feedback/comments, glad this data means something >>> and have drawn your interesting. Let me answer the questions (and sorry >> for >>> the lag) >>> >>> For the backport patches, ours are based on a customized 1.1.2 version >> and >>> cannot apply directly for any 1.x branches. It would be easy for us to >>> upload existing patches somewhere but obviously not that useful... so >> maybe >>> we still should get them in branch-1 and officially support read-path >>> offheap in future 1.x release? Let me create one JIRA about this and >> let's >>> discuss in the JIRA system. And to be very clear, it's a big YES to share >>> our patches with all rather than only numbers, just which way is better >>> (smile). >>> >>> And answers for @Stephen Boesch: >>> >>> bq. In any case the data is marked as 9/25/16 not 11/11/16 >>> It's specially noted that the data on 9/25 are from our online A/B test >>> cluster, and not showing fully online data because we published offheap >>> together with NettyRpcServer for online thus no standalone comparison >> data >>> for offheap. Please check my original email more carefully (smile). >>> >>> bq. Repeating my earlier question: 20*Meg* queries per second?? Just >>> checked and *google* does 40*K* queries per second. >>> As you already noticed, the 20M QPS is number from A/B testing cluster >> (450 >>> nodes), and there're much more on 11/11 online cluster (1600+ nodes). >>> Please note that this is NOT some cluster directly serves queries from >> end >>> user, but serving the index building and online machine learning system. >>> Refer to our talk on hbasecon2016 (slides >>> <http://www.slideshare.net/HBaseCon/improvements-to- >> apache-hbase-and-its- >>> applications-in-alibaba-search> >>> /recording >>> <https://www.youtube.com/watch?v=UVGDd2JeIMg&list=PLe-h9HrA9qfDVOeNh1l_ >>> T5HvwvkO9raWy&index=10>) >>> for more details, if you're interested. And different from google, >> there's >>> an obvious "hot spot" for us, so I don't think the QPS of these two >>> different systems are comparable. >>> >>> bq. So maybe please check your numbers again. >>> The numbers are got from online monitoring system and all real not fake, >> so >>> no need to check. Maybe just need some more time to take and understand? >>> (smile) >>> >>> Best Regards, >>> Yu >>> >>>> On 20 November 2016 at 23:03, Stephen Boesch <java...@gmail.com> wrote: >>>> >>>> Your arguments do not reflect direct knowledge of the numbers. (a) >> There >>>> is no super-spikiness int he graphs in the data (b) In any case the >> data >>> is >>>> marked as 9/25/16 not 11/11/16. (c) The number of internet users says >>>> little about the number of *concurrent* users. >>>> >>>> Overall it would be helpful for those who actually collected the data >> to >>>> comment - not just speculation from someone who does not. As I had >>>> mentioned already there *may* be a huge fanout from number of >>>> user/application queries to the backend: but *huge* it would seemingly >>> need >>>> to be to generate the numbers shown. >>>> >>>> 2016-11-19 22:39 GMT-08:00 张铎 <palomino...@gmail.com>: >>>> >>>>> 11.11 is something like the Black Friday. Almost every item on >> Alibaba >>>> will >>>>> discount a lot at 11.11. Alibaba earned a 1 billion revenue within 1 >>>>> minute(52 seconds) and 10 billion revenue within 7 minutes(6 minutes >> 58 >>>>> seconds) at 11.11. The Chinese people had payed more 120 billion >>> Chinese >>>>> yuan to alibaba at 11.11. And I remember that Jeff Dean used to give >> a >>>>> slides that for google the amplification from user queries to the >>> storage >>>>> system queries is also very large(I can not remember the exact >> number. >>>> The >>>>> slides is used to explain that hedge read is very useful for reducing >>>>> latency). So I think the peak throughput is true. >>>>> >>>>> There are more than 600 million people in China that use internet. So >>> if >>>>> they decide to do something to your system at the same time, it looks >>>> like >>>>> a DDOS to your system... >>>>> >>>>> Thanks. >>>>> >>>>> 2016-11-20 12:56 GMT+08:00 Stephen Boesch <java...@gmail.com>: >>>>> >>>>>> Repeating my earlier question: 20*Meg* queries per second?? Just >>>>> checked >>>>>> and *google* does 40*K* queries per second. Now maybe the "queries" >>>> are a >>>>>> decomposition of far fewer end-user queries that cause a fanout of >>>>> backend >>>>>> queries. *But still .. * >>>>>> >>>>>> So maybe please check your numbers again. >>>>>> >>>>>> 2016-11-19 17:05 GMT-08:00 Heng Chen <heng.chen.1...@gmail.com>: >>>>>> >>>>>>> The performance looks great! >>>>>>> >>>>>>> 2016-11-19 18:03 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: >>>>>>>> Opening a JIRA would be fine. >>>>>>>> This makes it easier for people to obtain the patch(es). >>>>>>>> >>>>>>>> Cheers >>>>>>>> >>>>>>>>> On Nov 18, 2016, at 11:35 PM, Anoop John < >> anoop.hb...@gmail.com >>>> >>>>>> wrote: >>>>>>>>> >>>>>>>>> Because of some compatibility issues, we decide that this will >>> be >>>>> done >>>>>>>>> in 2.0 only.. Ya as Andy said, it would be great to share the >>> 1.x >>>>>>>>> backported patches. Is it a mega patch at ur end? Or issue >> by >>>>> issue >>>>>>>>> patches? Latter would be best. Pls share patches in some >> place >>>>> and a >>>>>>>>> list of issues backported. I can help with verifying the >> issues >>>> once >>>>>>>>> so as to make sure we dont miss any... >>>>>>>>> >>>>>>>>> -Anoop- >>>>>>>>> >>>>>>>>>> On Sat, Nov 19, 2016 at 12:32 AM, Enis Söztutar < >>>>> enis....@gmail.com> >>>>>>> wrote: >>>>>>>>>> Thanks for sharing this. Great work. >>>>>>>>>> >>>>>>>>>> I don't see any reason why we cannot backport to branch-1. >>>>>>>>>> >>>>>>>>>> Enis >>>>>>>>>> >>>>>>>>>> On Fri, Nov 18, 2016 at 9:37 AM, Andrew Purtell < >>>>>>> andrew.purt...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Yes, please, the patches will be useful to the community >> even >>> if >>>>> we >>>>>>> decide >>>>>>>>>>> not to backport into an official 1.x release. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> On Nov 18, 2016, at 12:25 PM, Bryan Beaudreault < >>>>>>>>>>>> bbeaudrea...@hubspot.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Is the backported patch available anywhere? Not seeing it >> on >>>> the >>>>>>>>>>> referenced >>>>>>>>>>>> JIRA. If it ends up not getting officially backported to >>>> branch-1 >>>>>>> due to >>>>>>>>>>>> 2.0 around the corner, some of us who build our own deploy >>> may >>>>> want >>>>>>> to >>>>>>>>>>>> integrate into our builds. Thanks! These numbers look great >>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Nov 18, 2016 at 12:20 PM Anoop John < >>>>>> anoop.hb...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Yu Li >>>>>>>>>>>>> Good to see that the off heap work help you.. >>>> The >>>>>> perf >>>>>>>>>>>>> numbers looks great. So this is a compare of on heap L1 >>> cache >>>>> vs >>>>>>> off >>>>>>>>>>> heap >>>>>>>>>>>>> L2 cache(HBASE-11425 enabled). So for 2.0 we should make >>> L2 >>>>> off >>>>>>> heap >>>>>>>>>>>>> cache ON by default I believe. Will raise a jira for that >>> we >>>>> can >>>>>>>>>>> discuss >>>>>>>>>>>>> under that. Seems like L2 off heap cache for data blocks >>> and >>>>> L1 >>>>>>> cache >>>>>>>>>>> for >>>>>>>>>>>>> index blocks seems a right choice. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the backport and the help in testing the >>> feature.. >>>>> You >>>>>>> were >>>>>>>>>>>>> able to find some corner case bugs and helped community to >>> fix >>>>>>> them.. >>>>>>>>>>>>> Thanks goes to ur whole team. >>>>>>>>>>>>> >>>>>>>>>>>>> -Anoop- >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Nov 18, 2016 at 10:14 PM, Yu Li < >> car...@gmail.com> >>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry guys, let me retry the inline images: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Performance w/o offheap: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Performance w/ offheap: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Peak Get QPS of one single RS during Singles' Day >> (11/11): >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> And attach the files in case inline still not working: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Performance_without_offheap.png >>>>>>>>>>>>>> < >>>>>>>>>>>>> https://drive.google.com/file/d/0B017Q40_ >>>>>>> F5uwbWEzUGktYVIya3JkcXVjRkFvVG >>>>>>>>>>> NtM0VxWC1n/view?usp=drive_web >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Performance_with_offheap.png >>>>>>>>>>>>>> < >>>>>>>>>>>>> https://drive.google.com/file/d/0B017Q40_ >>>>>>> F5uweGR2cnJEU0M1MWwtRFJ5YkxUeF >>>>>>>>>>> VrcUdPc2ww/view?usp=drive_web >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Peak_Get_QPS_of_Single_RS.png >>>>>>>>>>>>>> < >>>>>>>>>>>>> https://drive.google.com/file/d/0B017Q40_ >>>>>>> F5uwQ2FkR2k0ZmEtRVNGSFp5RUxHM3 >>>>>>>>>>> F6bHpNYnJz/view?usp=drive_web >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Yu >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 18 November 2016 at 19:29, Ted Yu < >> yuzhih...@gmail.com >>>> >>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yu: >>>>>>>>>>>>>>> With positive results, more hbase users would be asking >>> for >>>>> the >>>>>>>>>>> backport >>>>>>>>>>>>>>> of offheap read path patches. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you think you or your coworker has the bandwidth to >>>> publish >>>>>>>>>>> backport >>>>>>>>>>>>>>> for branch-1 ? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Nov 18, 2016, at 12:11 AM, Yu Li <car...@gmail.com> >>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dear all, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We have backported read path offheap (HBASE-11425) to >> our >>>>>>> customized >>>>>>>>>>>>>>> hbase-1.1.2 (thanks @Anoop for the help/support) and run >>> it >>>>>>> online for >>>>>>>>>>>>> more >>>>>>>>>>>>>>> than a month, and would like to share our experience, >> for >>>> what >>>>>>> it's >>>>>>>>>>>>> worth >>>>>>>>>>>>>>> (smile). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Generally speaking, we gained a better and more stable >>>>>>>>>>>>>>> throughput/performance with offheap, and below are some >>>>> details: >>>>>>>>>>>>>>>> 1. QPS become more stable with offheap >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Performance w/o offheap: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Performance w/ offheap: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> These data come from our online A/B test cluster (with >>> 450 >>>>>>> physical >>>>>>>>>>>>>>> machines, and each with 256G memory + 64 core) with real >>>> world >>>>>>>>>>>>> workloads, >>>>>>>>>>>>>>> it shows using offheap we could gain a more stable >>>> throughput >>>>> as >>>>>>> well >>>>>>>>>>> as >>>>>>>>>>>>>>> better performance >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Not showing fully online data here because for online >> we >>>>>>> published >>>>>>>>>>> the >>>>>>>>>>>>>>> version with both offheap and NettyRpcServer together, >> so >>> no >>>>>>>>>>> standalone >>>>>>>>>>>>>>> comparison data for offheap >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Full GC frequency and cost >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Average Full GC STW time reduce from 11s to 7s with >>>> offheap. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. Young GC frequency and cost >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> No performance degradation observed with offheap. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4. Peak throughput of one single RS >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Singles Day (11/11), peak throughput of one single >> RS >>>>>> reached >>>>>>>>>>> 100K, >>>>>>>>>>>>>>> among which 90K from Get. Plus internet in/out data we >>> could >>>>>> know >>>>>>> the >>>>>>>>>>>>>>> average result size of get request is ~1KB >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Offheap are used on all online machines (more than 1600 >>>>> nodes) >>>>>>>>>>> instead >>>>>>>>>>>>>>> of LruCache, so the above QPS is gained from offheap >>>>>> bucketcache, >>>>>>>>>>> along >>>>>>>>>>>>>>> with NettyRpcServer(HBASE-15756). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Just let us know if any comments. Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>> Yu >>>>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
