Re: [heka] Binary TCP format?

Fri, 03 Jul 2015 12:15:58 -0700 

Hi Rob,
Thank you for your guidance. I'm trying to use heka-inject, but being a go-noob, I might be doing something wrong. 

I'm trying this:

~  go get github.com/mozilla-services/heka/cmd/heka-inject
# github.com/mozilla-services/heka/message

dev/gocode/src/github.com/mozilla-services/heka/message/message_matcher.go:21: 
undefined: tree



This is the same error as I get when trying to import the client 
https://github.com/mozilla-services/heka/issues/1599 which I filed an 
issue for.



I'm probably doing something wrong, or perhaps something funky was 
committed not too long ago to the repo?


Regards,
Henrik


[GPG Pub 
Key](https://countermail.com/process.php?pubview=aGVucmlrQGxvZ2liaXQuc2U=)

On 3 Jul 2015, at 21:02, Rob Miller wrote:


Sorry you're having trouble here. The docs you linked to are indeed 
correct. I'm afraid I don't have the cycles to dig in deeply to your 
code or the binary output that it's generating, but I can give you a 
couple of additional tips that might help with your debugging.



My recommendation is that you start from a known good binary encoding 
of a Heka message stream and use that to compare your own output. 
Generating a binary stream is easy; if you set up a Heka instance with 
a FileOutput using a ProtobufEncoder then the generated file will 
contain exactly such a stream. The following steps should get you to 
something you can use as a reference:



* Set up a testing Heka config w/ a TcpInput using a ProtobufDecoder, 
and a FileOutput using a ProtobufEncoder.
* Use heka-inject (see 
https://hekad.readthedocs.org/en/latest/developing/testing.html#heka-inject) 
to generate a single Heka message and send it to the listening 
TcpInput.

* Shut down Heka.


As long as the message_matcher on the FileOutput was correctly set up 
to catch the message you sent, you should now have a file containing 
exactly one Heka message, along with the appropriate stream header. 
Then you can use the code that you're writing to generate a binary 
encoding for a message that is identical to the one you specified with 
the heka-inject call (i.e. matching payload, hostname, severity, type, 
etc. etc. etc.). Compare what your code is emitting with the binary 
stream in the file generated by Heka and see how they match up.



Also, it might be useful to test your encoding logic separately from 
the TCP connection stuff, so bugs from the one aren't mistaken as 
problems with the other. To do this, I'd recommend starting out by 
having your code write the Heka message stream to disk. If this is 
done correctly, you will be able to see the contents of the message 
stream using the heka-cat tool (see 
https://hekad.readthedocs.org/en/latest/developing/testing.html#heka-cat). 
If your encoder is generating files that heka-cat can read, but you're 
still seeing I/O related errors, then you know the issue isn't with 
the stream format but is instead related to the I/O code itself.



Sorry I can't be more help, hopefully this gives you enough to be able 
to debug and resolve your issue.


-r


On 07/03/2015 09:18 AM, Henrik Feldt wrote:

Greetings,

What is the binary format to send to Heka with?


I'm trying to implement a client in F# for sending natively to Heka, 
but

I'm failing.

This is a trace:

D 2015-07-03T15:22:12.9760700+00:00: running: writing to heka
[Logary.Targets.Heka]
write record separator
write header length
BAM!! System.IO.IOException: Write failure --->
System.Net.Sockets.SocketException: The socket has been shut down
at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset,
Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0

at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, 
Int32

offset, Int32 size) [0x00000] in <filename unknown>:0
//snip

Here's the code:

|     Choice1Of2 (async {
   try
     use msgMs = msgMs
     use hdrMs = hdrMs
     printfn "write record separator"
     do s.WriteByte Constants.RecordSeparator
     do! Async.Sleep 200
     printfn "write header length"
     do s.WriteByte (byte (hdrMs.Length))
     do! Async.Sleep 200
     printfn "write header"
     do! fromZero hdrMs (fun ms -> ms.CopyToAsync s)
     do! Async.Sleep 200
     printfn "write unit separator"
     do s.WriteByte Constants.UnitSeparator
     do! Async.Sleep 200
     printfn "write unit message"
     do! fromZero msgMs (fun ms -> ms.CopyToAsync s)
   with e -> printfn "BAM!! %A" e
 })
|


Repro: 
https://github.com/logary/logary/tree/feature/heka/examples/heka

or
https://github.com/logary/logary/blob/feature/heka/src/targets/Logary.Targets.Heka/Script.fsx

For the second repro above (.fsx), which contains more data, it looks
like this:

write record separator
write header length
write header
BAM!! System.IO.IOException: EndWrite failure --->
System.Net.Sockets.SocketException: The socket has been shut down

at System.Net.Sockets.Socket.EndSend (IAsyncResult result) [0x00000] 
in

<filename unknown>:0

I have unit tests that mimick what your tests seem to have:


* 
https://github.com/mozilla-services/heka/blob/dev/client/encoders_test.go
* 
https://github.com/logary/logary/blob/feature/heka/src/tests/Logary.Targets.Heka.Tests/Program.fs#L28-L39


And I'm following this framing format:

* https://hekad.readthedocs.org/en/latest/message/index.html


So at this point I'm at my wits end, and can't find another way of 
doing

it... Here's a sample byte array that I try to send to the socket
(including the full message), on the wire:

1E02083E1F0A105C18DA1111A14D4DBAF1F8116DCC76CE1080C48C9491A9E8D4131A0B68656B612E6C6F676172792200320040BA8F024A0B636F696E64756374696F6E

Here's the Heka config:

|[hekad]
maxprocs = 2

[TcpInput]
address = ":5565"

[PayloadEncoder]
append_newlines = false

[LogOutput]
message_matcher = "TRUE"
encoder = "PayloadEncoder"
|

The logs:

2015/07/03 09:10:06 Pre-loading: [DashboardOutput]
2015/07/03 09:10:06 Pre-loading: [ProtobufEncoder]
2015/07/03 09:10:06 Loading: [ProtobufEncoder]
2015/07/03 09:10:06 Pre-loading: [TokenSplitter]
2015/07/03 09:10:06 Loading: [TokenSplitter]
2015/07/03 09:10:06 Pre-loading: [HekaFramingSplitter]
2015/07/03 09:10:06 Loading: [HekaFramingSplitter]
2015/07/03 09:10:06 Pre-loading: [NullSplitter]
2015/07/03 09:10:06 Loading: [NullSplitter]
2015/07/03 09:10:06 Pre-loading: [ProtobufDecoder]
2015/07/03 09:10:06 Loading: [ProtobufDecoder]
2015/07/03 09:10:06 Loading: [DashboardOutput]
2015/07/03 09:10:06 Starting hekad...
2015/07/03 09:10:06 Output started: DashboardOutput
2015/07/03 09:10:06 MessageRouter started.

I've verified I can connect using telnet to 5565.

You can netcat it to Heka to test my code. Or you can use the

|prepare.sh| script to test it live (you'll need to install mono 
first
http://www.mono-project.com/download/ (or apt-get install 
mono-complete)


Regards,
Henrik

https://countermail.com/process.php?pubview=aGVucmlrQGxvZ2liaXQuc2U=


_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka


_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka






















					Reply via email to