Re: obsolete D libraries/modules
Jacob Carlborg wrote: > Well, dsource.org is dead. How would one know? I certainly didn't when as a complete D newbie I tried to find some libraries a few months ago. > The projects are basically there for > reference only. Most project these days are at Github. I guess there > isn't an obvious way to find all of them though. I'd say that's a pity and there should be some way to find them :) -- Marek Janukowicz
Decrease DMD memory usage
How can I decrease DMD memory usage? My program does now use about 3.5GB during compilation, which is horrible, as even machines with 4GB RAM sometimes fail to compile successfully :( -- Marek Janukowicz
Re: std.serialization: pre-voting review / discussion
Andrej Mitrovic wrote: >> I recently needed some way to serialize a data structure (in order by >> save the state of the app and restore it later) and was quite >> disappointed there is nothing like that in Phobos. > > FWIW you could try out msgpack-d: > https://github.com/msgpack/msgpack-d#usage > > It's a very tiny and a fast library. That's what I ended up using, but I would be much more happy to have something like this in Phobos. -- Marek Janukowicz
Re: std.serialization: pre-voting review / discussion
ilya-stromberg wrote: >>> The serialized data is returned as an array, so that is >>> compatible with >>> the range interface, it just won't be lazy. >> >> This seems like a major limitation. (Disclaimer: I haven't read >> the documentation yet.) >> >> Andrei > > Shall we fix it before accept the std.serialization? > > For example, if I have 10GB of data and 16GB operating memory, I > can't use std.serialization. It saves all my data into string > into operating memory, so I haven't got enough memory to save > data in file. It's currently limited by std.xml. > > In other hand, std.serialization can help in many other cases if > I have enough memory to store copy of my data. > > As I can see, we have a few options: > - accept std.serialization as is. If users can't use > std.serialization due memory limitation, they should find another > way. > - hold std.serialization until we will have new std.xml module > with support of range/file input/output. Users should use Orange > if they need std.serialization right now. > - hold std.serialization until we will have binary archive for > serialization with support of range/file input/output. Users > should use Orange if they need std.serialization right now. > - use another xml library, for example from Tango. My opinion is - accept it as it is (if it's not completely broken). I recently needed some way to serialize a data structure (in order by save the state of the app and restore it later) and was quite disappointed there is nothing like that in Phobos. Although XML is not necessarily well suited to my particular use case, it's still better than nothing. Binary archive would be a great plus, but allow me to point out that current state of affairs (std.serialization being in a pre-accepted state for a long time AFAIK) is probably the worst state we might have - on the one hand I would not use third party libs, because std.serialization is just around the corner, on the other I don't have std.serialization distributed with the compiler yet. Also binary archive is an extension, not a change, so I don't see any reason why it could not be added later (because it would be backward compatible). -- Marek Janukowicz
Re: Ideas for a brand new widget toolkit
Andrei Alexandrescu wrote: > On 8/15/13 12:42 PM, Adam D. Ruppe wrote: >> On Thursday, 15 August 2013 at 18:35:22 UTC, Joakim wrote: >>> I've only done X11 forwarding over ssh, both WAN and LAN, it was >>> incredibly laggy in both cases. >> >> You're probably using bloated apps! I'm using my crappygui.d on remote X >> and it works quite well. Of course it is mostly rectangles of solid >> colors :) >> >> I also just tried creating an OpenGL window remotely, and it performed >> reasonably well too, drawing some colorful triangles. I think I'll add >> some minimal support for creating opengl contexts to simpledisplay.d, >> that's useful stuff. > > The X Window System is slow by design, particularly when used over a > network (regardless how fast). There are various proxies and alternate > remoting protocols that avoid its inherent latencies. At work we use > OpenNX. If we are discussing X's shortcomings: why don't we use something like http://wayland.freedesktop.org/ which will (hopefully) become next big standard and which addresses many of X's flaws? -- Marek Janukowicz
Re: Network server design question
Johannes Pfau wrote: >> > This is a bug in std.socket BTW. Blocking calls will get >> > interrupted by >> > the GC - there's no way to avoid that - but std.socket should >> > handle >> > this internally and just retry the interrupted operation. >> > Please file a >> > bug report about this. >> >> I'm not sure whether we can do anything about Socket.select >> itself at this point, as it would be a breaking API change – >> interrupted calls returning a negative value is even mentioned >> explicitly in the docs. >> >> There should, however, be a way to implement this in a >> platform-independent manner in client code, or even a second >> version that handles signal interruptions internally. >> >> > (Partial writes is another issue that could/should be handled in >> > std.socket so the user doesn't have to care about it) >> >> I don't think that would be possible – std.socket by design is a >> thin wrapper around BSD sockets (whether that's a good idea or >> not is another question), and how to handle partial writes >> depends entirely on the environment the socket is used in (think >> event-based architecture using fibers vs. other designs). >> >> In general, I wonder what the best way for going forward with >> std.socket is. Sure, we could try to slowly morph it into a >> "modern" networking implementation, but the current state also >> has its merits, as it allows people to use the familiar BSD >> sockets API without having to worry about all the trivial >> differences between the platforms (e.g. in symbol names). >> >> We should definitely add a note to std.socket though that it is a >> low-level API and that there might be a better choice for most >> applications (e.g. vibe.d, Thrift, …). >> >> David > > You're right, I somehow thought std.socket was supposed to offer a high > level API. But as it was designed as a low level wrapper we probably > can't do much without breaking API compatibility. But - as I mentioned in another post - it looks like "interrupted system call" problem happens only with select and not eg. with blocking read. This means that current behaviour is inconsistent between std.socket functions. Also it was possible to make this work for read (I believe this bug & fix address that - http://d.puremagic.com/issues/show_bug.cgi?id=2242) and I don't think anyone considered it as "compatibility breaking", so why not take the same route for select? -- Marek Janukowicz
Re: Network server design question
Marek Janukowicz wrote: > I'm writing a network server with some specific requirements: > - 5-50 clients connected (almost) permanently (maybe a bit more, but > definitely not hundreds of them) > - possibly thousands of requests per seconds > - responses need to be returned within 5 seconds or the client will > disconnect and complain > > Currently I have a Master thread (which is basically the main thread) > which is handling connections/disconnections, socket operations, sends > parsed requests for processing to single Worker thread, sends responses to > clients. Interaction with Worker is done via message passing. I'd like to thank anyone for valuable input. For now I chose Dmitry's suggestion (which was an extension of my idea to go with thread per client), so I have multiple receivers, single worker and multiple senders. That works quite well, although I didn't really test that with many clients. One nice thing is that "interrupted system call" problem magically went away - it looks like it occurred with Socket.select (which I don't use after architectural changes anymore) only and socket.send/receive is apparently not affected. -- Marek Janukowicz
Re: Network server design question
Dmitry Olshansky wrote:
> 04-Aug-2013 23:38, Marek Janukowicz пишет:
>> I'm writing a network server with some specific requirements:
>> - 5-50 clients connected (almost) permanently (maybe a bit more, but
>> definitely not hundreds of them)
>> - possibly thousands of requests per seconds
>> - responses need to be returned within 5 seconds or the client will
>> disconnect and complain
>>
>> Currently I have a Master thread (which is basically the main thread)
>> which is handling connections/disconnections, socket operations, sends
>> parsed requests for processing to single Worker thread, sends responses
>> to clients. Interaction with Worker is done via message passing.
>
> Typical approach would be to separate responsibilities even more and
> make a pool of threads per each stage.
>
> You may want to make a Master thread only handle new connections
> selecting over an "accept socket" (or a few if multiple end-points).
> Then it may distribute connected clients over I/O worker threads.
>
> A pool of I/O workers would then only send/receive data passing parsed
> request to "real" workers and responses back. They handle disconnects
> and closing though.
This is basically approach "2." I mentioned in my original post, I'm glad
you agree it makes sense :)
> The real workers could be again pooled to be more responsive (or e.g.
> just one per each I/O thread).
There are more things specific to this particular application that would
play a role here. One is that such "real workers" would operate on a common
data structure and I would have to introduce some synchronization. Single
worker thread was not my first approach, but after some woes with other
solutions I decided to take it, because the problem is really not in
processing (where a single thread does just fine so far), but in socket
read/write operations.
>> The problem with my approach is that I read as much data as possible from
>> each ready client in order. As there are many requests this read phase
>> might take a few seconds making the clients disconnect. Now I see 2
>> possible solutions:
>>
>> 1. Stay with the design I have, but change the workflow somewhat -
>> instead of reading all the data from clients just read some requests and
>> then send responses that are ready and repeat; the downside is that it's
>> more complicated than current design, might be slower (more loop
>> iterations with less work done in each iteration) and might require quite
>> a lot of tweaking when it comes to how many requests/responses handle
>> each time etc.
>
> Or split the clients across a group of threads to reduce maximum
> latency. See above, just determine the amount of clients per thread your
> system can sustain in time. A better way would be to dynamically
> load-balance clients between threads but it's far more complicated.
Yeah, both approaches seem to be somewhat more complicated and I'd like to
aovid this if possible. So one client per thread makes sense to me.
>> 2. Create separate thread per each client connection. I think this could
>> result in a nice, clean setup, but I see some problems:
>> - I'm not sure how ~50 threads will do resource-wise (although they will
>> probably be mostly waiting on Socket.select)
>
> 50 threads is not that big a problem. Around 100+ could be, 1000+ is a
> killer.
Thanks for those numbers, it's great to know at least the ranges here.
> The benefit with thread per client is that you don't even need
> Socket.select, just use blocking I/O and do the work per each parsed
> request in the same thread.
Not really. This is something that Go (the language I also originally
considered for the project) has solved in much better way - you can "select"
on a number of "channels" and have both I/O and message passing covered by
those. In D I must react both to network data or message from worker
incoming, which means either self-pipe trick (which leads to Socket.select
again) or some quirky stuff with timeouts on socket read and message receive
(but this is basically a busy loop).
>> - I can't initialize threads created via std.concurrency.spawn with a
>> Socket object ("Aliases to mutable thread-local data not allowed.")
>
> This can be hacked with casts to shared void* and back. Not pretty but
> workable.
I'm using this trick elsewhere, was a bit reluctant to try it here. Btw.
would it work if I pass a socket to 2 threads - reader and writer (by
working I mean - not running into race conditions and other scary concurrent
stuff)?
Also I'm really puzzled by the fact this common idiom doesn't work in some
eleg
Re: Network server design question
John Colvin wrote: > Take a look at how vibe.d approaches the problem: > http://vibed.org/ Vibe.d uses fibers, which I don't find feasible for my particular application for a number of reasons: - I have constant number of ever-connected clients, not an ever-changing number of random clients - after I read and parse a request there is not much room for yielding during processing (I don't do I/O or database calls, I have an in-memory "database" for performance reasons) - event-based programming generally looks complicated to me and (for the reason mentioned above) I don't see much point in utilizing it in this case -- Marek Janukowicz
Network server design question
I'm writing a network server with some specific requirements:
- 5-50 clients connected (almost) permanently (maybe a bit more, but
definitely not hundreds of them)
- possibly thousands of requests per seconds
- responses need to be returned within 5 seconds or the client will
disconnect and complain
Currently I have a Master thread (which is basically the main thread) which
is handling connections/disconnections, socket operations, sends parsed
requests for processing to single Worker thread, sends responses to clients.
Interaction with Worker is done via message passing.
The problem with my approach is that I read as much data as possible from
each ready client in order. As there are many requests this read phase might
take a few seconds making the clients disconnect. Now I see 2 possible
solutions:
1. Stay with the design I have, but change the workflow somewhat - instead
of reading all the data from clients just read some requests and then send
responses that are ready and repeat; the downside is that it's more
complicated than current design, might be slower (more loop iterations with
less work done in each iteration) and might require quite a lot of tweaking
when it comes to how many requests/responses handle each time etc.
2. Create separate thread per each client connection. I think this could
result in a nice, clean setup, but I see some problems:
- I'm not sure how ~50 threads will do resource-wise (although they will
probably be mostly waiting on Socket.select)
- I can't initialize threads created via std.concurrency.spawn with a Socket
object ("Aliases to mutable thread-local data not allowed.")
- I already have problems with "interrupted system call" on Socket.select
due to GC kicking in; I'm restarting the call manually, but TBH it sucks I
have to do anything about that and would suck even more to do that with 50
or so threads
If anyone has any idea how to handle the problems I mentioned or has any
idea for more suitable design I would be happy to hear it. It's also
possible I'm approaching the issue from completely wrong direction, so you
can correct me on that as well.
--
Marek Janukowicz
