On Tue, Aug 8, 2017 at 12:01 PM, Yu Li <car...@gmail.com> wrote: ... > Maybe we could open several umbrellas in JIRA to follow up the > implement-able topics and do some prioritizing? Thanks. >
Agree. Let me start up the DISCUSS suggested by Ashish; it just came up again on the tail of an issue.... St.Ack > Best Regards, > Yu > > On 8 August 2017 at 10:54, ashish singhi <ashish.sin...@huawei.com> wrote: > > > Great write up, Stack. Covering everything what we all discussed. > > It was very nice meeting you all and hope we can continue this HBaseCon > > Asia. > > > > Regards, > > Ashish > > > > From: saint....@gmail.com [mailto:saint....@gmail.com] On Behalf Of > Stack > > Sent: 08 August 2017 00:07 > > To: HBase Dev List <dev@hbase.apache.org> > > Subject: Notes from dev meetup in Shenzhen, August 5th, 2017 > > > > At fancy Huawei headquarters, 10:00-12:00AM or so (with nice coffee and > > fancy little cake squares provided about half way through the session). > > > > For list of attendees, see picture at end of this email. > > > > Discussion was mostly in Chinese with about 25% in English plus some > > gracious sideline translation so the below is patchy. Hopefully you get > the > > gist. > > > > For client-side scanner going against hfiles directly; is there a means > of > > being able to pass the permissions from hbase to hdfs? > > > > Issues w/ the hbase 99th percentile were brought up. "DynamoDB can do > > 10ms". How to do better? > > > > SSD is not enough. > > > > GC messes us up. > > > > Will the Distributed Log Replay come back to help improve MTTR? We could > > redo on new ProcedureV2 basis. ZK timeout is the biggest issue. Do as we > > used to and just rely on the regionserver heartbeating... > > > > Read replica helps w/ MTTR. > > > > Ratis incubator project to do a quorum based hbase? > > > > Digression on licensing issues around fb wangle and folly. > > > > Redo of hbase but quorum based would be another project altogether. > > > > Decided to go around the table to talk about concerns and what people are > > working on. > > > > Jieshan wondered what could be done to improve OLAP over hbase. > > > > Client side scanner was brought up again as means of skipping RS overhead > > and doing better OLAP. > > > > Have HBase compact to parquet files. Query parquet and hbase. > > > > At Huawei, they are using 1.0 hbase. Most problems are assignment. They > > have .5M regions. RIT is a killer. Double assignment issues. And RIT. > They > > run their own services. Suggested they upgrade to get fixes at least. > Then > > 2.0. > > > > Will HBase federate like HDFS? Can Master handle load at large scale? It > > needs to do federation too? > > > > Anyone using Bulk loaded replication? (Yes, it just works so no one talks > > about it...) > > > > Request that fixes be backported to all active branches, not just most > > current. > > > > Andrew was good at backporting... not all RMs are. > > > > Too many branches. What should we do? > > > > Proliferation of branches makes for too much work. > > > > Need to cleanup bugs in 1.3. Make it stable release now. > > > > Lets do more active EOL'ing of branches. 1.1?. > > > > Hubert asked if we can have clusters where RS are differently capable? > > i.e. several generations of HW all running in the same cluster. > > > > What if fat server goes down. > > > > Balancer could take of it all. RS Capacity. Balancer can take it into > > account. > > Regionserver labels like YARN labels. Characteristics. > > > > Or run it all in docker when heterogeneous cluster. The K8 talk from day > > before was mentioned; we should all look at being able to deploy in k8 > and > > docker. > > > > Lets put out kubernetes blog...(Doing). > > > > Alibaba looking at HBase as native YARN app. > > > > i/o is hard even when containers. > > > > Use autoscaler of K8 when heavy user. > > > > Limit i/o use w/ CP. Throttle. > > > > Spark and client-side scanner came up again. > > > > Snapshot input format in spark. > > > > HBase federation came up again. jd.com<http://jd.com> talking of 3k to > 4k > > nodes in a cluster. Millions of regions. Region assignment is messing > them > > up. > > > > Maybe federation is good idea? Argument that it is too much operational > > conplexity. Can we fix master load w/ splittable meta, etc? > > > > Was brought up that even w/ 100s of RS there is scale issue, nvm > thousands. > > > > Alibaba talked about disaster recovery. Described issue where HDFS has > > fencing problem during an upgrade. There was no active NN. All RS went > down. > > ZK is another POF. If ZK is not available. Operators were being asked how > > much longer the cluster was going to be down but they could not answer > the > > question. No indicators from HBase on how much longer it will be down or > > how many WALs its processed and how many more to go. Operator unable to > > tell his org how long it would be before it all came back on line. Should > > say how many regions are online and how many more to do. > > > > Alibaba use SQL to lower cost. HBase API is low-level. Row-key > > construction is tricky. New users make common mistakes. If you don't do > > schema right, high-performance is difficult. > > > > Alibaba are using a subset of Phoenix... simple sql only; throws > > exceptions if user tries to do joins, etc.., anything but basic ops. > > > > HareQL is using hive for meta store. Don't have data typing in hbase. > > > > HareQL could perhaps contribute some piece... or a module in hbase to > > sql... From phoenix? > > > > Secondary index. > > > > Client is complicated in phoenix. Was suggested thin client just does > > parse... and then offload to server for optimization and execution. > > > > Then secondary index. Need transaction engine. Consistency of secondary > > index. > > > > We adjourned. > > > > Your dodgy secretary, > > St.Ack > > P.S. Please add to this base set of notes if I missed anything. > > > > > > > > >
