I was just starting to use Sente [1] (which relies on httpkit [2]) and this
conversation is a real eye opener. Unless a person uses a library that
supports backpressure, as mentioned earlier, your
transport-concern-made-opaque-because-of-core-async must become an
application-level concern. The
A slightly more straightforward implementation can be found
at https://gist.github.com/ztellman/fb64e81d1d7f0b261ccd. I'm fairly sure
it's equivalent to Cristophe's, but I may be missing some nuance. At any
rate, I've found using the async/put! and callback mechanism to be a much
more
Hi Zach,
Thanks for the clarity of thought that went into this post.
Perhaps it is obvious to everyone but me, but I saw this post by Christophe
Grande yesterday that appears to address these concerns:
Back-pressurized interop for
core.async
The reason the thread-per-connection approach is nice is because it
correctly propagates backpressure. If we're copying data from a source to
a sink (let's say reading it in from the network and writing to a file),
it's possible that the production of data may outstrip the consumption. If
This is something that should be configurable extendable in core.async
really... but given how long it took to have this possibility with agents I
am not holding my breath (not to mention it was never even considered for
c.c/future).
On Tuesday, October 7, 2014 8:48:23 PM UTC+2, Brian Guthrie
One way how to handle this elegantly in Java is to add support for
java.nio.channels.SelectableChannel in core.async buffer. That way you
could select on any combination of core.async channels and io
resources, waiting for core.async channels to free and resources to
have data ready.
Jozef
On
Check out https://github.com/halgari/com.tbaldridge.hermod for an
interesting take on this.
On Wednesday, October 8, 2014 1:17:11 AM UTC-4, Sun Ning wrote:
BTW, is there any network based core.async channel available now?
On 10/08/2014 04:36 AM, adrian...@mail.yu.edu javascript: wrote:
I wasn't aware of hermod, that's interesting. I would still characterize
its approach to backpressure as broken, though, since when the queues get
full it silently drops messages on the ground. In fairness, this is very
clearly documented, so it's less pernicious than some of the other cases
out
If you want to handle multiple TCP connections and async channels in one
thread, you need a way how to block on both connections (wait for new input
to arrive) and channels (wait for a free space in a buffer). Blocking only
on connections will get you a busy loop if channels are full. If you
The documentation for Manifold can explain the API better than I can here.
The point where that interacts with Netty w.r.t. backpressure is here:
https://github.com/ztellman/aleph/blob/0.4.0/src/aleph/netty.clj#L109.
Here the stream represents data coming off the wire, and if the put onto
the
Sorry, didn't cover converse case. That's handled by the ChannelSink
directly underneath. Note that each write returns a Netty ChannelFuture
representing completion of the write, which is transformed into a Manifold
deferred. Any time a Manifold put returns an unrealized deferred, that
creates
Thank you! Using put! callback to control backpressure is a very elegant
solution. BTW there always has to be blocking somewhere. Netty uses
selector to block at [1] and .setAutoRead causes respective channel to
deregister itself from a selector [2], until put! completes.
Regarding the other
Yes, I didn't mean to imply that there are no blocking operations anywhere,
only that nothing in the worker threads (where the put! occurs) will block,
and that backpressure isn't exerted by causing an active thread to hang.
As to the second point, I'm not sure why you'd think that. These aren't
It was just my ignorance of manifold's concepts :). I was thinking in terms
of I have this thread which should connect these channels to those
sockets, when to do what. Manifold's async approach with its connectable
streams and deferreds with callbacks seems to abstract away such
bookkeeping.
On Sun, Oct 5, 2014 at 11:57 PM, Zach Tellman ztell...@gmail.com wrote:
If I'm reading this correctly, you're using non-blocking thread pools for
blocking operations on the sockets. Given more than N connections (last
time I looked the thread pool's size was 42), you risk deadlock or at the
On Mon, Oct 6, 2014 at 12:10 AM, adrian.med...@mail.yu.edu wrote:
Zach makes an excellent point; I've used AsyncSocketChannels and its irk (
http://docs.oracle.com/javase/8/docs/api/java/nio/channels/AsynchronousServerSocketChannel.html),
with core.async in the past. Perhaps replacing your
It's not about 'safety' (depending on what that means in this context), but
as Zach pointed out, if you aren't careful about backpressure you can run
into performance bottlenecks with unrestrained async IO operations because
although they let you code as if you could handle an unlimited amount
BTW, is there any network based core.async channel available now?
On 10/08/2014 04:36 AM, adrian.med...@mail.yu.edu wrote:
It's not about 'safety' (depending on what that means in this
context), but as Zach pointed out, if you aren't careful about
backpressure you can run into performance
Could you please expand on this, Zach?
Please note that if you use core.async with java.nio, you need to make
sure backpressure is properly propagated (this happens automatically with
java.io, assuming you have a thread per connection)
Thanks!
--
You received this message because you
Hi all,
I'm releasing a little library for working with sockets. Feedback and pull
requests gratefully appreciated.
The skinny
---
This library allows you to create socket servers and socket clients and
interact with them asynchronously using channels. Servers return a record
with a
If I'm reading this correctly, you're using non-blocking thread pools for
blocking operations on the sockets. Given more than N connections (last
time I looked the thread pool's size was 42), you risk deadlock or at the
very least poor average throughput.
On Sunday, October 5, 2014 7:06:56 PM
Zach makes an excellent point; I've used AsyncSocketChannels and its irk
(http://docs.oracle.com/javase/8/docs/api/java/nio/channels/AsynchronousServerSocketChannel.html),
with core.async in the past. Perhaps replacing your direct java.net.Sockets
with nio classes that can be given
Please note that if you use core.async with java.nio, you need to make sure
backpressure is properly propagated (this happens automatically with
java.io, assuming you have a thread per connection).
On Sunday, October 5, 2014 9:10:24 PM UTC-7, adrian...@mail.yu.edu wrote:
Zach makes an
23 matches
Mail list logo
