.Net library preferences
Hello all, I'm working away at the .Net version of Avro and would like the opinion of anyone that is listening. I'm trying to do the port using a .Net 2.0 feature spec to make mono support a bit easier. .Net 2.0 does not have a native json parser so I've been using JayRock (http://jayrock.berlios.de/) which has a LGPLv3 license. My questions are: 1.) Excuse my ignorance but is LGPLv3 compatible with an apache license? 2.) Is JayRock a preferred library for json parsing? 3.) Does anyone have issues if I bring in Apache Log4net (http://logging.apache.org/log4net/) ? I have an abstracted logger that I use that allows anyone who compiles from source to disable logging using compiler directives so they would not have to use log4net. For public releases my personal preference would be to build against the latest stable version of log4net. What do you guys think? If you are interested in what I have done so far you can look at my repo on github. I'm pushing there pretty frequently now that I've finished my move to Austin. http://github.com/jcustenborder/avro/tree/dotnet-port J
Re: .Net library preferences
Boo on the LGPLv3. That means the patch I posted to jira is invalid. I'll refactor the json code to use another library. Heh this time I'll actually make sure that the library is compatible license wise before I start using it. On Fri, May 7, 2010 at 2:01 AM, Jeff Hammerbacher ham...@cloudera.com wrote: 1.) Excuse my ignorance but is LGPLv3 compatible with an apache license? Nope, sadly. 2.) Is JayRock a preferred library for json parsing? No idea, maybe ask Stack Overflow? 3.) Does anyone have issues if I bring in Apache Log4net (http://logging.apache.org/log4net/) ? Go for it!
[jira] Updated: (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Custenborder updated AVRO-533: - Attachment: dotnet.patch Here is a patch of what I have done so far. Just to make things nice and confusing I changed my development environment to follow http://wiki.apache.org/hadoop/GitAndHadoop so now I'm working out of a branch instead of trunk. This changes my github url for anyone who is interested. The correct github url is http://github.com/jcustenborder/avro/tree/dotnet-port I'm not too familiar with Apache patch procedures so please let me know if you need anything modified. .NET implementation of Avro --- Key: AVRO-533 URL: https://issues.apache.org/jira/browse/AVRO-533 Project: Avro Issue Type: New Feature Reporter: Jeff Hammerbacher Attachments: dotnet.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-533) .NET implementation of Avro
[ https://issues.apache.org/jira/browse/AVRO-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12862298#action_12862298 ] Jeremy Custenborder commented on AVRO-533: -- I am happy to accept this issue. I've already been working on a port of avro to .net. I've currently on the middle of moving to Austin, tx so progress will be minimal over the next couple weeks. What I have done so far is available at http://github.com/jcustenborder/avro. My plans is to get things up and running with code generation support by mid summer. Hopefully sooner. The cool part is that I have a need for cross platform RPC at my new job meaning this will be one of my early development tasks. If you want to take what is in my github as an initial patch you are more than welcome. So far it passes a lot of the schema tests and would be a great starting point for anyone that wants to help. My longterm goals is to have a fast client and server implementation that uses reflection.emit under the hood for schema serialization. .NET implementation of Avro --- Key: AVRO-533 URL: https://issues.apache.org/jira/browse/AVRO-533 Project: Avro Issue Type: New Feature Reporter: Jeff Hammerbacher -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Thoughts on an RPC protocol
I really like the model that Voldemort uses for their protocol buffers
implementation. I ported their client to .net and it was really
simple. The framing is really simple as it prefixes an integer value
before binary data. The binary data is a request made with a protocol
buffer, and so is the response. The response is a protocol buffer
package specific to the method called. They used enums for the method
names. For streaming results it prefixes a 1 if there is another
record and either a 0 or -1 (I forget and need to look at the code.
You get the idea) if it's the end of the stream. I like the idea of
keeping things as simple as possible because it makes it easier for
additional languages to be added quickly. Personally I would prefer to
have an easy request then response model that blocks on the client
side. For simplicity a client could just block and wait for the data
to return or the client could be more advanced and use callbacks or io
completion ports. This would allow a client to use things like
connection pooling to handle concurrency of multiple requests.
As for routing I typically don't like using a proxy because that
limits you to the interface bandwidth of the proxy appliance. For
example if you use something like a netscaler as your proxy in front
of your back end servers. You will have 50 back end servers with gig
connections trying to feed a couple netscalers that can constrain the
bandwidth. Adding more bandwidth means adding more proxy boxes which
gets expensive fast especially with netscalers. Voldemort uses the
concept of node banning. If a node doesn't respond quickly enough or
has errors it will get banned for a period of time. Couple this with
something like SRV records in DNS and you can easily manipulate the
direction of your traffic without using a proxy saving you some cash.
I'm currently working on the .net port of avro and the rpc
implementation is one of my priorities. My goal is to get something to
the point where I can have a utility that connects to the rpc server,
gets protocol handshake, then builds out strongly typed code that a
developer can work against.
{
namespace: com.acme,
protocol: HelloWorld,
doc: Protocol Greetings,
types: [
{name: Greeting, type: record, fields: [
{name: message, type: string}]},
{name: Curse, type: error, fields: [
{name: message, type: string}]}
],
messages: {
hello: {
doc: Say hello.,
request: [{name: greeting, type: Greeting }],
response: Greeting,
errors: [Curse]
}
}
}
would generate
namespace com.acme
{
public class HelloWorld:Avro.Protocol
{
public Greeting hello(Greeting greeting)
}
}
On Thu, Apr 8, 2010 at 3:43 PM, Doug Cutting cutt...@apache.org wrote:
Bruce,
Overall this looks like a good approach to me.
How do you anticipate allocating channels? I'm guessing this would be one
per client object, that a pool of open connections to servers would be
maintained, and creating a new client object would allocate a new channel.
Currently we perform a handshake per request. This is fairly cheap and
permits things like routing through proxy servers. Different requests over
the same connection can talk to different backend servers running different
versions of the protocol. Also consider the case where, between calls on an
object, the connection times out, and a new session is established and a new
handshake must take place.
That said, having a session where the handshake can be assumed vastly
simplifies one-way messages. Without a response or error on which to prefix
a handshake response, a one-way client has no means to know that the server
was able to even parse its request. Yet we'd still like a handshake for
one-way messages, so that clients and servers need not be versioned in
lockstep. So the handshake-per-request model doesn't serve one-way messages
well.
How can we address both of these needs: to permit flexible payload routing
and efficient one-way messaging?
Doug
Bruce Mitchener wrote:
* Think about adding something for true one-way messages, but an empty
reply frame is probably sufficient, since that still allows reporting
errors
if needed (or desired).
C# port
I've started putting together a c# port of Avro based on the python implementation. It currently has support for Schema classes and partial Binary Encoding support. Patches and comments are appreciated. :) The code is available at http://github.com/jcustenborder/avro
