Sorry you're having trouble here. The docs you linked to are indeed correct.
I'm afraid I don't have the cycles to dig in deeply to your code or the binary
output that it's generating, but I can give you a couple of additional tips
that might help with your debugging.
My recommendation is that you start from a known good binary encoding of a Heka
message stream and use that to compare your own output. Generating a binary
stream is easy; if you set up a Heka instance with a FileOutput using a
ProtobufEncoder then the generated file will contain exactly such a stream. The
following steps should get you to something you can use as a reference:
* Set up a testing Heka config w/ a TcpInput using a ProtobufDecoder, and a
FileOutput using a ProtobufEncoder.
* Use heka-inject (see
https://hekad.readthedocs.org/en/latest/developing/testing.html#heka-inject) to
generate a single Heka message and send it to the listening TcpInput.
* Shut down Heka.
As long as the message_matcher on the FileOutput was correctly set up to catch
the message you sent, you should now have a file containing exactly one Heka
message, along with the appropriate stream header. Then you can use the code
that you're writing to generate a binary encoding for a message that is
identical to the one you specified with the heka-inject call (i.e. matching
payload, hostname, severity, type, etc. etc. etc.). Compare what your code is
emitting with the binary stream in the file generated by Heka and see how they
match up.
Also, it might be useful to test your encoding logic separately from the TCP
connection stuff, so bugs from the one aren't mistaken as problems with the
other. To do this, I'd recommend starting out by having your code write the
Heka message stream to disk. If this is done correctly, you will be able to see
the contents of the message stream using the heka-cat tool (see
https://hekad.readthedocs.org/en/latest/developing/testing.html#heka-cat). If
your encoder is generating files that heka-cat can read, but you're still
seeing I/O related errors, then you know the issue isn't with the stream format
but is instead related to the I/O code itself.
Sorry I can't be more help, hopefully this gives you enough to be able to debug
and resolve your issue.
-r
On 07/03/2015 09:18 AM, Henrik Feldt wrote:
Greetings,
What is the binary format to send to Heka with?
I'm trying to implement a client in F# for sending natively to Heka, but
I'm failing.
This is a trace:
D 2015-07-03T15:22:12.9760700+00:00: running: writing to heka
[Logary.Targets.Heka]
write record separator
write header length
BAM!! System.IO.IOException: Write failure --->
System.Net.Sockets.SocketException: The socket has been shut down
at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset,
Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32
offset, Int32 size) [0x00000] in <filename unknown>:0
//snip
Here's the code:
| Choice1Of2 (async {
try
use msgMs = msgMs
use hdrMs = hdrMs
printfn "write record separator"
do s.WriteByte Constants.RecordSeparator
do! Async.Sleep 200
printfn "write header length"
do s.WriteByte (byte (hdrMs.Length))
do! Async.Sleep 200
printfn "write header"
do! fromZero hdrMs (fun ms -> ms.CopyToAsync s)
do! Async.Sleep 200
printfn "write unit separator"
do s.WriteByte Constants.UnitSeparator
do! Async.Sleep 200
printfn "write unit message"
do! fromZero msgMs (fun ms -> ms.CopyToAsync s)
with e -> printfn "BAM!! %A" e
})
|
Repro: https://github.com/logary/logary/tree/feature/heka/examples/heka
or
https://github.com/logary/logary/blob/feature/heka/src/targets/Logary.Targets.Heka/Script.fsx
For the second repro above (.fsx), which contains more data, it looks
like this:
write record separator
write header length
write header
BAM!! System.IO.IOException: EndWrite failure --->
System.Net.Sockets.SocketException: The socket has been shut down
at System.Net.Sockets.Socket.EndSend (IAsyncResult result) [0x00000] in
<filename unknown>:0
I have unit tests that mimick what your tests seem to have:
* https://github.com/mozilla-services/heka/blob/dev/client/encoders_test.go
*
https://github.com/logary/logary/blob/feature/heka/src/tests/Logary.Targets.Heka.Tests/Program.fs#L28-L39
And I'm following this framing format:
* https://hekad.readthedocs.org/en/latest/message/index.html
So at this point I'm at my wits end, and can't find another way of doing
it... Here's a sample byte array that I try to send to the socket
(including the full message), on the wire:
1E02083E1F0A105C18DA1111A14D4DBAF1F8116DCC76CE1080C48C9491A9E8D4131A0B68656B612E6C6F676172792200320040BA8F024A0B636F696E64756374696F6E
Here's the Heka config:
|[hekad]
maxprocs = 2
[TcpInput]
address = ":5565"
[PayloadEncoder]
append_newlines = false
[LogOutput]
message_matcher = "TRUE"
encoder = "PayloadEncoder"
|
The logs:
2015/07/03 09:10:06 Pre-loading: [DashboardOutput]
2015/07/03 09:10:06 Pre-loading: [ProtobufEncoder]
2015/07/03 09:10:06 Loading: [ProtobufEncoder]
2015/07/03 09:10:06 Pre-loading: [TokenSplitter]
2015/07/03 09:10:06 Loading: [TokenSplitter]
2015/07/03 09:10:06 Pre-loading: [HekaFramingSplitter]
2015/07/03 09:10:06 Loading: [HekaFramingSplitter]
2015/07/03 09:10:06 Pre-loading: [NullSplitter]
2015/07/03 09:10:06 Loading: [NullSplitter]
2015/07/03 09:10:06 Pre-loading: [ProtobufDecoder]
2015/07/03 09:10:06 Loading: [ProtobufDecoder]
2015/07/03 09:10:06 Loading: [DashboardOutput]
2015/07/03 09:10:06 Starting hekad...
2015/07/03 09:10:06 Output started: DashboardOutput
2015/07/03 09:10:06 MessageRouter started.
I've verified I can connect using telnet to 5565.
You can netcat it to Heka to test my code. Or you can use the
|prepare.sh| script to test it live (you'll need to install mono first
http://www.mono-project.com/download/ (or apt-get install mono-complete)
Regards,
Henrik
https://countermail.com/process.php?pubview=aGVucmlrQGxvZ2liaXQuc2U=
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
