On Mon, Feb 3, 2014 at 11:27 AM, Rob Miller <[email protected]> wrote:
> Right. You can do the deserialization directly in the input, but if you > pass it on to a decoder (which will be running in its own separate > goroutine) that frees up the input to go back to the network to grab the > next record, keeping the pipeline full. Won't matter much when maxprocs=1, > but might allow for higher throughput if you're using multiple cores. > > As always, YMMV, it's better to measure than guess. :) > > I like this approache since it not only allows for decoding to happen in one or many goroutines, but also for cleaner way to for allowing other decoders to be specified via config. Do you have any advice on why copying into pack.Msgbytes and sending to a DecodeRunner would cause errors like: 2014/02/08 07:18:42 Decoder 'ProtobufDecoder' error: proto: can't skip unknown wire type 6 for message.Message and 2014/02/08 07:18:41 Decoder 'ProtobufDecoder' error: proto: field/encoding mismatch: wrong type for field code: https://gist.github.com/davidbirdsong/8877708 while a direct call to Unmarshal works like a charm: https://gist.github.com/davidbirdsong/8877719 ...for background, the writer to kafka that creates the message objects that I'm trying to decode strips the header just like this input module does: https://github.com/mozilla-services/heka/blob/dev/plugins/file/logfile_input.go#L403 > -r > > > > On Mon 03 Feb 2014 10:08:17 AM PST, Michael Trinkala wrote: > >> For use in an input plugin 'b' should be written into >> pipeline.PipelinePack.MsgBytes and the pack should be sent to a >> ProtobufDecoder (which will perform the de-serialization below) >> >> Trink >> >> >> ------------------------------------------------------------------------ >> >> *From: *"David Birdsong" <[email protected]> >> *To: *"Michael Trinkala" <[email protected]> >> *Cc: *"heka" <[email protected]> >> *Sent: *Friday, January 31, 2014 11:35:05 PM >> *Subject: *Re: [heka] initializing msg from a Message.proto dumped >> >> string >> >> That is a quick way, thanks! >> >> On Jan 31, 2014 10:27 PM, "Michael Trinkala" <[email protected] >> <mailto:[email protected]>> wrote: >> >> import "code.google.com/p/goprotobuf/proto < >> http://code.google.com/p/goprotobuf/proto>" >> import "github.com/mozilla-services/heka/message < >> http://github.com/mozilla-services/heka/message>" >> >> >> b := // []byte value from your code >> msg := new(message.Message) >> err := proto.Unmarshal(b, msg) >> >> >> Trink >> >> >> >> ------------------------------------------------------------ >> ------------ >> >> *From: *"David Birdsong" <[email protected] >> <mailto:[email protected]>> >> *To: *"heka" <[email protected] <mailto:[email protected]>> >> *Sent: *Friday, January 31, 2014 7:07:19 PM >> *Subject: *[heka] initializing msg from a Message.proto >> >> dumped string >> >> What's the easiest way to initialize a message that's been >> previously serialized? I'm writing an input module from >> kafka that takes care of message framing. So I've got a >> []byte value that could be directly instantiated into a >> message object. I'm looking at other input modules for an >> example, but so many of them deal with some sort of fd and >> parsing the input stream to fully build up a message object. >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> >> https://mail.mozilla.org/listinfo/heka >> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/heka >> >> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/heka >> > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
