Ok, I have looked over the source and it is making a little more sense. I think what I ultimately want to do is this:
source A -> channel A -> sink A ———> source B -> channel B -> sink B source A will be looking at a log file. Each line of the log file will have a certain format/schema. I would write Source A such that it could write the schema/line as an event into the channel and pass that through the system all the way ultimately to sink B so that it would know the schema also. I was thinking Avro would be a good format for source A to use when writing into it’s channel. If Sink A is an existing Avro Sink and Source B is an exiting Avro source, would this still work? Does this mean I would have 2 Avro headers (one encapsulating the other) which wasteful or can the existing Avro source and sink deal with this unmodified? Is there a better way to accomplish what I want to do? Just looking for some guidance. Thanks, Ed On Sep 4, 2014, at 4:44 AM, Ashish <[email protected]> wrote: > Avro records shall have the schema embedded with them. Have a look at source, > that shall help a bit > > > On Wed, Sep 3, 2014 at 10:30 PM, Ed Judge <[email protected]> wrote: > That’s helpful but isn’t there some type of Avro schema negotiation that > occurs? > > -Ed > > On Sep 3, 2014, at 12:02 AM, Jeff Lord <[email protected]> wrote: > >> Ed, >> >> Did you take a look at the javadoc in the source? >> Basically the source uses netty as a server and the sink is just an rpc >> client. >> If you read over the doc which is in the two links below and take a look at >> the developer guide and still have questions just ask away and someone will >> help to answer. >> >> https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java >> >> https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java >> >> https://flume.apache.org/FlumeDeveloperGuide.html#transaction-interface >> >> -Jeff >> >> >> >> >> >> >> On Tue, Sep 2, 2014 at 6:36 PM, Ed Judge <[email protected]> wrote: >> Does anyone know of any good documentation that talks about the >> protocol/negotiation used between an Avro source and sink? >> >> Thanks, >> Ed >> >> > > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal
