Re: Typed Message Passing between D Processes
On Thursday, 30 July 2015 at 05:53:48 UTC, yawniek wrote: On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote: LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atila cool. what are the advantages of cereald over msgpack? AFAIK, features. The kind of features I need/use to write networking code and reduce the boilerplate to an absolute minimum. can you stream in packets with cereald too? I don't know exactly what you mean. I've only used it to go from network packets structs to bytes and vice-versa. cool thing about msgpack is that there exist libraries for many language. so we use it do actually store logs and then process them with other tools too. I wrote cerealed for networking. You _can_ use it to convert whatever else to binary, but its focus is easy networking. Atila
Re: Typed Message Passing between D Processes
On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote: On Monday, 29 June 2015 at 08:45:15 UTC, Atila Neves wrote: On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote: On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote: I'd have to benchmark it against something, but I'm pretty sure cerealed is fast. Faster than msgpack? I guess I'm going to have benchmark this now... :) Atila In release builds, _nearly_. Here's the benchmark program: http://dpaste.dzfl.pl/17b0ed9c0204. Results below. msgpack is... fast. From the docs, cerealed seems to have more features that I actually use and need though. Results: DMD debug: Cerealed: 4 secs, 987 ms, 700 μs, and 5 hnsecs MsgPack: 1 sec, 771 ms, 713 μs, and 7 hnsecs DMD release: Cerealed: 2 secs, 556 ms, 714 μs, and 6 hnsecs MsgPack: 1 sec, 89 ms, 561 μs, and 3 hnsecs GDC debug: Cerealed: 4 secs, 863 ms, 501 μs, and 1 hnsec MsgPack: 2 secs, 32 ms, 53 μs, and 1 hnsec GDC release: Cerealed: 1 sec, 740 ms, 726 μs, and 4 hnsecs MsgPack: 1 sec, 20 ms, 287 μs, and 3 hnsecs LDC debug: Cerealed: 7 secs, 711 ms, 154 μs, and 4 hnsecs MsgPack: 3 secs, 694 ms, 566 μs, and 2 hnsecs LDC release: Cerealed: 1 sec, 795 ms, 380 μs, and 7 hnsecs MsgPack: 931 ms, 355 μs, and 5 hnsecs Before I had time to look into it, somebody graciously optimised cerealed for me in this PR: https://github.com/atilaneves/cerealed/pull/6 And now then benchmarks are (dub build --build=release, fastest of 10 runs for each compiler): DMD: Cerealed: 1 sec, 199 ms, 118 μs, and 1 hnsec MsgPack: 1 sec, 85 ms, and 516 μs GDC: Cerealed: 927 ms, 677 μs, and 4 hnsecs MsgPack: 1 sec, 32 ms, 736 μs, and 5 hnsecs LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atila
Re: Typed Message Passing between D Processes
On Monday, 29 June 2015 at 13:59:37 UTC, Nordlöw wrote: On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote: I guess I'm going to have benchmark this now... :) What about doing a memory profiling using DMD fresh builtin profiler of http://dpaste.dzfl.pl/17b0ed9c0204 ? I'm guessing the GC might give misguiding results as your testStruct returns a relatively small data structure. I would rather like to see a larger (randomized) structure being tested on. You could make use of my https://github.com/nordlow/justd/blob/master/random_ex.d for random instance generation :) I tried your code and added it to the benchmark. It didn't make much of a difference. Atila
Re: Typed Message Passing between D Processes
On Wednesday, 29 July 2015 at 16:36:41 UTC, Atila Neves wrote: LDC: Cerealed: 970 ms, 482 μs, and 6 hnsecs MsgPack: 896 ms, 591 μs, and 2 hnsecs Not too shabby! Atila cool. what are the advantages of cereald over msgpack? can you stream in packets with cereald too? cool thing about msgpack is that there exist libraries for many language. so we use it do actually store logs and then process them with other tools too.
Re: Typed Message Passing between D Processes
On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote: I'd have to benchmark it against something, but I'm pretty sure cerealed is fast. Faster than msgpack?
Re: Typed Message Passing between D Processes
On Monday, 29 June 2015 at 08:45:15 UTC, Atila Neves wrote: On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote: On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote: I'd have to benchmark it against something, but I'm pretty sure cerealed is fast. Faster than msgpack? I guess I'm going to have benchmark this now... :) Atila In release builds, _nearly_. Here's the benchmark program: http://dpaste.dzfl.pl/17b0ed9c0204. Results below. msgpack is... fast. From the docs, cerealed seems to have more features that I actually use and need though. Results: DMD debug: Cerealed: 4 secs, 987 ms, 700 μs, and 5 hnsecs MsgPack: 1 sec, 771 ms, 713 μs, and 7 hnsecs DMD release: Cerealed: 2 secs, 556 ms, 714 μs, and 6 hnsecs MsgPack: 1 sec, 89 ms, 561 μs, and 3 hnsecs GDC debug: Cerealed: 4 secs, 863 ms, 501 μs, and 1 hnsec MsgPack: 2 secs, 32 ms, 53 μs, and 1 hnsec GDC release: Cerealed: 1 sec, 740 ms, 726 μs, and 4 hnsecs MsgPack: 1 sec, 20 ms, 287 μs, and 3 hnsecs LDC debug: Cerealed: 7 secs, 711 ms, 154 μs, and 4 hnsecs MsgPack: 3 secs, 694 ms, 566 μs, and 2 hnsecs LDC release: Cerealed: 1 sec, 795 ms, 380 μs, and 7 hnsecs MsgPack: 931 ms, 355 μs, and 5 hnsecs
Re: Typed Message Passing between D Processes
On Sunday, 28 June 2015 at 17:02:42 UTC, Nordlöw wrote: On Friday, 26 June 2015 at 21:40:49 UTC, Atila Neves wrote: I'd have to benchmark it against something, but I'm pretty sure cerealed is fast. Faster than msgpack? I guess I'm going to have benchmark this now... :) Atila
Re: Typed Message Passing between D Processes
On Monday, 29 June 2015 at 10:22:10 UTC, Atila Neves wrote: I guess I'm going to have benchmark this now... :) What about doing a memory profiling using DMD fresh builtin profiler of http://dpaste.dzfl.pl/17b0ed9c0204 ? I'm guessing the GC might give misguiding results as your testStruct returns a relatively small data structure. I would rather like to see a larger (randomized) structure being tested on. You could make use of my https://github.com/nordlow/justd/blob/master/random_ex.d for random instance generation :)
Re: Typed Message Passing between D Processes
On Friday, 26 June 2015 at 11:13:11 UTC, Per Nordlöw wrote: On Friday, 26 June 2015 at 10:23:26 UTC, Laeeth Isharc wrote: On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote: On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote: not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ? Why is a lower-level pipe needed, when we have pipe in std.process? well, that is the sort of lower level pipe I was referring to. Ok, thanks. It should be easy enough to write so you can switch out the transport protocol. You could use something like nanomsg if you are prepared to use a beta product. I wrote some D bindings, but they are pretty raw and haven't been used much except for tiny applications. Fantastic if you feel like giving them a try and improving them. However if you don't like them you can just call nanomsg C api directly. But nanomsg itself is still early stage. http://nanomsg.org/index.html http://java.dzone.com/articles/look-nanomsg-and-scalability https://github.com/Laeeth/d-nanomsg
Re: Typed Message Passing between D Processes
On Friday, 26 June 2015 at 12:31:04 UTC, Dicebot wrote: std.concurrency was supposed to be able to handle that by design but it is impossible to do without any sort of standard serialization utility in Phobos (and, ideally, very fast binary serialization utility) I'd have to benchmark it against something, but I'm pretty sure cerealed is fast.
Re: Typed Message Passing between D Processes
On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote: not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ? Why is a lower-level pipe needed, when we have pipe in std.process?
Re: Typed Message Passing between D Processes
On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote: On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote: not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ? Why is a lower-level pipe needed, when we have pipe in std.process? well, that is the sort of lower level pipe I was referring to.
Re: Typed Message Passing between D Processes
std.concurrency was supposed to be able to handle that by design but it is impossible to do without any sort of standard serialization utility in Phobos (and, ideally, very fast binary serialization utility)
Re: Typed Message Passing between D Processes
On Friday, 26 June 2015 at 10:23:26 UTC, Laeeth Isharc wrote: On Friday, 26 June 2015 at 09:06:18 UTC, Per Nordlöw wrote: On Thursday, 25 June 2015 at 23:23:01 UTC, Laeeth Isharc wrote: not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ? Why is a lower-level pipe needed, when we have pipe in std.process? well, that is the sort of lower level pipe I was referring to. Ok, thanks.
Re: Typed Message Passing between D Processes
On Thursday, 25 June 2015 at 14:04:23 UTC, Per Nordlöw wrote: Is there an alternative to http://dlang.org/phobos/std_process.html#.pipe that can be used to do _typed_ _message_ _passing_ between two D processes with the same convenience as `send` and `receive` in std.concurrency ? Either in Phobos or in a third party library? You'd have to implement your own IPC for that to work. You can always use TCP and serialise everything. Atila
Re: Typed Message Passing between D Processes
On Thursday, 25 June 2015 at 14:04:23 UTC, Per Nordlöw wrote: Is there an alternative to http://dlang.org/phobos/std_process.html#.pipe that can be used to do _typed_ _message_ _passing_ between two D processes with the same convenience as `send` and `receive` in std.concurrency ? Either in Phobos or in a third party library? not sure if this is quite what you are looking for, or if the performance overhead is acceptable but have you looked at msgpack to go on top of a lower level pipe ?
