Ryan, The working plan is to use Avro to replace the current DataNode streaming protocol, in addition to the HTTP-based protocol for fetching data during the shuffle and the standard Hadoop IPC. Having a unified serialization and RPC strategy should make enforcing things like security constraints and versions far more straightforward. It would be great to see the HBase community express their specific needs to the Avro folks so that we could have unified serialization and RPC throughout the stack (not you, Michael).
Later, Jeff On Mon, Jun 22, 2009 at 9:36 PM, Ryan Rawson <[email protected]> wrote: > HBase has performance requirements on par with the datanode streaming > protocol - we want to provide the data in our systems as fast as we can > read > and stream them. > > -ryan > > On Mon, Jun 22, 2009 at 9:19 PM, Amr Awadallah <[email protected]> wrote: > > > CCing avro-dev@ to comment, but my understanding is that it is currently > > functioning and way superior to the alternatives you mention below :) > > > > -- amr > > > > > > Andrew Purtell wrote: > > > >> I my opinion, we should not bother to wait for Avro. I've been hearing > >> about it on and off for three months now. If it is ready the day we > >> start work on 1015 for 0.21, and it fits the bill, fine, but otherwise > >> we should just use Thrift or pbufs. Thrift may be preferable as its > >> compact binary protocol is competitive with pbufs plus it has a fully > >> implemented async rpc stack. I think this applies for both 1015 and > >> 1295. Also I'm skeptical that something to supplant RMI won't have > >> overheads related to that we don't need, e.g. transmitting class and > >> method names as strings, etc. > >> - Andy > >> > >> > >> > >> > >> ________________________________ > >> From: Jim Kellerman (POWERSET) <[email protected]> > >> To: "[email protected]" <[email protected]> > >> Sent: Saturday, June 20, 2009 3:26:58 PM > >> Subject: RE: who's doing what for 0.21? > >> > >> I am also interested in 1295 (I have quite a bit of experience > >> with cross data center replication), but more interested in > >> getting more of the master into zookeeper. > >> > >> As for 1556, I might wait a bit. At the Cloudera off-site, one > >> of the things talked about was doing something similar for > >> Hadoop which we might leverage. > >> > >> What really needs to get done around builds is when you mark > >> a Jira as patch available, we should do a patch build and test > >> like Hadoop does. Noone has had time to do it to date, but if > >> you are taking on the build, that would be a "nice to have". > >> > >> For 1015, should you wait for Avro? > >> > >> And if you missed it, here are the notes from the Cloudera off-site: > >> http://wiki.apache.org/hadoop/DeveloperOffsite20090612 > >> > >> --- > >> Jim Kellerman, Powerset (Live Search, Microsoft Corporation) > >> > >> > >> > >> > >>> -----Original Message----- > >>> From: Andrew Purtell [mailto:[email protected]] > >>> Sent: Saturday, June 20, 2009 12:11 PM > >>> To: [email protected] > >>> Subject: who's doing what for 0.21? > >>> > >>> http://tinyurl.com/m7nt72 > >>> > >>> I have an interest in these: > >>> > >>> https://issues.apache.org/jira/browse/HBASE-1015 > >>> https://issues.apache.org/jira/browse/HBASE-1295 > >>> https://issues.apache.org/jira/browse/HBASE-1556 > >>> > >>> I think for 1015 and 1295, there is interest on the part of at least > >>> myself, > >>> dj_ryan, and jgray. dj_ryan was saying something about su executives > >>> making > >>> 1295 a priority for him. We should figure out how to divide up and > >>> assign > >>> out the work. > >>> > >>> Also, probably I'll end up taking on the grunt work of 1556, because > >>> it > >>> needs to be done. > >>> > >>> Have we set a time and place for the next dev meeting? > >>> > >>> - Andy > >>> > >>> > >>> > >> > >> > >> > >> > > >
