Greetings,
What is the binary format to send to Heka with?
I'm trying to implement a client in F# for sending natively to Heka, but
I'm failing.
This is a trace:
D 2015-07-03T15:22:12.9760700+00:00: running: writing to heka
[Logary.Targets.Heka]
write record separator
write header length
BAM!! System.IO.IOException: Write failure --->
System.Net.Sockets.SocketException: The socket has been shut down
at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset,
Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer,
Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
//snip
Here's the code:
Choice1Of2 (async {
try
use msgMs = msgMs
use hdrMs = hdrMs
printfn "write record separator"
do s.WriteByte Constants.RecordSeparator
do! Async.Sleep 200
printfn "write header length"
do s.WriteByte (byte (hdrMs.Length))
do! Async.Sleep 200
printfn "write header"
do! fromZero hdrMs (fun ms -> ms.CopyToAsync s)
do! Async.Sleep 200
printfn "write unit separator"
do s.WriteByte Constants.UnitSeparator
do! Async.Sleep 200
printfn "write unit message"
do! fromZero msgMs (fun ms -> ms.CopyToAsync s)
with e -> printfn "BAM!! %A" e
})
Repro: https://github.com/logary/logary/tree/feature/heka/examples/heka
or
https://github.com/logary/logary/blob/feature/heka/src/targets/Logary.Targets.Heka/Script.fsx
For the second repro above (.fsx), which contains more data, it looks
like this:
write record separator
write header length
write header
BAM!! System.IO.IOException: EndWrite failure --->
System.Net.Sockets.SocketException: The socket has been shut down
at System.Net.Sockets.Socket.EndSend (IAsyncResult result) [0x00000]
in <filename unknown>:0
I have unit tests that mimick what your tests seem to have:
-
https://github.com/mozilla-services/heka/blob/dev/client/encoders_test.go
-
https://github.com/logary/logary/blob/feature/heka/src/tests/Logary.Targets.Heka.Tests/Program.fs#L28-L39
And I'm following this framing format:
- https://hekad.readthedocs.org/en/latest/message/index.html
So at this point I'm at my wits end, and can't find another way of doing
it... Here's a sample byte array that I try to send to the socket
(including the full message), on the wire:
1E02083E1F0A105C18DA1111A14D4DBAF1F8116DCC76CE1080C48C9491A9E8D4131A0B68656B612E6C6F676172792200320040BA8F024A0B636F696E64756374696F6E
Here's the Heka config:
[hekad]
maxprocs = 2
[TcpInput]
address = ":5565"
[PayloadEncoder]
append_newlines = false
[LogOutput]
message_matcher = "TRUE"
encoder = "PayloadEncoder"
The logs:
2015/07/03 09:10:06 Pre-loading: [DashboardOutput]
2015/07/03 09:10:06 Pre-loading: [ProtobufEncoder]
2015/07/03 09:10:06 Loading: [ProtobufEncoder]
2015/07/03 09:10:06 Pre-loading: [TokenSplitter]
2015/07/03 09:10:06 Loading: [TokenSplitter]
2015/07/03 09:10:06 Pre-loading: [HekaFramingSplitter]
2015/07/03 09:10:06 Loading: [HekaFramingSplitter]
2015/07/03 09:10:06 Pre-loading: [NullSplitter]
2015/07/03 09:10:06 Loading: [NullSplitter]
2015/07/03 09:10:06 Pre-loading: [ProtobufDecoder]
2015/07/03 09:10:06 Loading: [ProtobufDecoder]
2015/07/03 09:10:06 Loading: [DashboardOutput]
2015/07/03 09:10:06 Starting hekad...
2015/07/03 09:10:06 Output started: DashboardOutput
2015/07/03 09:10:06 MessageRouter started.
I've verified I can connect using telnet to 5565.
You can netcat it to Heka to test my code. Or you can use the
`prepare.sh` script to test it live (you'll need to install mono first
http://www.mono-project.com/download/ (or apt-get install mono-complete)
Regards,
Henrik
[GPG Pub
Key](https://countermail.com/process.php?pubview=aGVucmlrQGxvZ2liaXQuc2U=)_______________________________________________
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
