I built a PostgreSQL Java client from scratch recently. Since it was not a goal to support JDBC ever I thought about writing it with non-blocking async IO. I played with some ideas but ultimately I decided not to. The reason is quite simple. Async IO works well if you do enough IO to keep one a more cores busy by just doing IO, which you get with something like a server with many (100k+) connections for example. Threaded IO works equally well or better if you only have a very few connections.
You can't have many thousands connections to a postgresql server since a connection is quite expensive and if you really stress the server you can overload it with a handful of connections already. More connections won't make anything any faster since they will basically slow each other down more. So if you limit the connections anyway its alot simpler to do blocking IO and API. That does not mean you can't interact with the database through core.async channels, it's just not required to do in at the socket level. Async IO is not a magical solution to scalable servers, in fact many servers probably won't benefit at all. Of course there are going to be some that would benefit. Re-inventing the wheel is probably a huge waste of time. Stuff like Netty already works pretty well and can be used from Clojure quite easily. Just my 2 cents, /thomas On Monday, January 5, 2015 11:18:42 PM UTC+1, Robin Heggelund Hansen wrote: > > I guess this post is mostly going to be a question, but one that could > shape up to be a long open source project and contribution on my part, if > it is warranted. > > The Clojure community has been blessed with good language interoperability > with Java, which has made it easy to use and wrap Java-libraries, which > again I suspect is why Clojure is where it is today, a tool for > professional development. > > This has also, to some extent, been a curse, as we've relied on libraries > made for a different language, instead of "creating the world in our > image", so to speak. I assume this is why core.async isn't as integrated in > the Clojure ecosystem as I would like, because we already have libraries > that works, and taking the time to make sure they scale well simply isn't > worth it. > > So I thought I would re-invent the wheel a little, but it depends on my > premise being correct. > > From what I understand, core.async basically creates state machines, that > are run on a threadpool. Once you do something that blocks (like IO), you > are kinda ruining the idea behind core.async, which is efficient > concurrency at a large scale. I also assume, that having more than one > threadpool, isn't really what you want. You want core.async to have the > only threadpool running, and you want to run most things as go-blocks. > > Today, way to many things block, like reading a file or reading from a > database. Things that are async, mostly uses it's own threadpool. If I got > this correctly, a standard web-app today will usually perform a blocking > action for most DB-ops, requests will run in a http-server-specific > threadpool, while agents or go-blocks has their own threadpool again. > > Would I be correct that a clojure web-server, would be more efficient (at > scale) if DB-ops and general request handling, ran entirely as go-blocks on > the core.async threadpool alone? > > I was thinking of creating a async.io library (core.async + NIO for file > and socket ops), and after that perhaps create a socket-pool library before > creating a core.async friendly SQL interface. Is there a point to this, or > would I just be doing a lot of work for very little gain? > > Thanks! > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
