Hey Michael, Wow, very cool. I am a bit embarrassed that I didn't get pyhbase in a better state before you got to hacking on it, but super happy to have you join the fun.
I've opened a ticket at https://issues.apache.org/jira/browse/AVRO-721 to discuss the inclusion of your Tornado client in Avro proper. I think that would be the right direction for pyhbase. Let's continue the discussion over there! Regards, Jeff On Mon, Dec 27, 2010 at 5:58 PM, Michael Russo <mjru...@gmail.com> wrote: > Jeff, thanks very much for the response and for pushing forward with the > gateway. > > On the Python front, I just wrote an asynchronous client (Tornado-based). > Commit: > > https://github.com/mjrusso/pyhbase/commit/07ed39527bd6752158b1293e8d2df528d649a613 > > Let me know if you have any feedback on this. If you don't feel that it is > a fit for pyhbase, I'm happy to maintain a fork. (There's also the > question > of if the TornadoRequestor and TornadoHTTPTransceiver should be included in > the Avro package itself.) > > Best, > Michael > > On Sun, Dec 26, 2010 at 3:16 PM, Jeff Hammerbacher <ham...@cloudera.com > >wrote: > > > Okay, I felt guilty about not cleaning this code up in a while. > > > > I opened https://issues.apache.org/jira/browse/HBASE-3393 to track > > updating > > the server to use the new join() method rather than sleep()ing for a long > > time. The tests are failing for a very strange reason right now (claiming > > that a field is missing on an Avro record that's clearly present in the > > .java file; I'm at a loss), but I'll put my patch up in case you want to > > try > > to make progress. > > > > Also, Maven can't seem to find the Netty jar needed by Avro. I've got an > > email out to the Avro lists to figure out what's up with that. For now, > you > > can just add the JBoss repository to the HBase pom.xml as directed at > > http://www.jboss.org/netty/downloads.html. > > > > I also fixed a bug where the server wasn't using the port number passed > at > > the command line. > > > > I also opened https://issues.apache.org/jira/browse/HBASE-3394 to change > > the > > build process to generate the Java code at compile time rather than > forcing > > us to check in the compiled Java code. That might not be preferred for > some > > who prefer source code to be checked in. Happy to discuss on that ticket. > > > > Hope that helps, > > Jeff > > > > On Sun, Dec 26, 2010 at 10:50 AM, Jeff Hammerbacher <ham...@cloudera.com > > >wrote: > > > > > Hey Michael, > > > > > > Actually, it was something more basic: AVRO-544 changed the way servers > > are > > > started so that the gateway class could just join the Avro server > thread > > and > > > run forever, rather than having to run for a finite amount of time. I > > still > > > need to get an HBase-side change in that uses this new feature of > Avro--I > > > just haven't had much time for coding of late. > > > > > > Other than that, I have been lax in keeping up with some of the new > APIs > > > introduced in 0.90 (the multi* stuff, for example). There are a few > other > > > cleanup tasks at https://issues.apache.org/jira/browse/HBASE-2400 that > > are > > > worth checking out. > > > > > > Thanks for sending the patches my way for the Python client. I'll do my > > > best to get to some of these updates soon, and would be more than happy > > to > > > review patches for HBase, Avro, or the Python client on Github (thanks > > for > > > the patches you've already sent my way). > > > > > > Regards, > > > Jeff > > > > > > > > > > > > On Sun, Dec 26, 2010 at 7:16 AM, Michael Russo <mjru...@gmail.com> > > wrote: > > > > > >> Jeff Hammerbacher stated on Quora [1] that "the Avro gateway is not > > >> production quality" and made a reference to the improvements that an > > >> upgrade > > >> to Avro 1.4 would bring. > > >> > > >> I would like to understand a little better what this means. > > >> > > >> I reviewed the Avro changelog [2] for 1.4+, and very quickly > > >> scanned o.a.h.h.avro.AvroServer.java [3] and the only Avro change that > > >> really stood out in the context of significantly improving the > > production > > >> quality of the gateway is AVRO-405 (Netty-based Java RPC server) [4]. > > >> > > >> Is this a reasonable analysis or are there any other important > elements > > >> that > > >> I'm missing? > > >> > > >> [1]: > > >> > > > http://www.quora.com/What-is-the-current-status-for-using-Avro-with-HBase > > >> [2]: https://github.com/apache/avro/blob/trunk/CHANGES.txt > > >> [3]: > > >> > > >> > > > http://svn.apache.org/viewvc/hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/AvroServer.java?view=markup > > >> [4]: https://issues.apache.org/jira/browse/AVRO-405 > > >> > > >> Thanks, > > >> Michael > > >> > > > > > > > > >
