Given the nature of the problem I would wonder why the cpu usage is high.
restQL seems to be mostly a passthrough entity completely dependent upon IO
so there may be something spinning or a pmap somewhere there should not be.
- jstacktrace may offer insight here.
If nothing is out of whack in terms of what the machine is actually doing
then you could implement a timed go macro that skipped its internals if it
took more than Y seconds to execute.
(defmacro timed-go
[timeout & args]
`(let [time# (get-time)]
(go
(when (< (- (get-time) time#) ~timeout)
~@args))))
I wouldn’t start there though. Here are some things I would check before I
implemented the above macro.
- Potentially something is using the same execution pool that clojure’s
async library uses for go. This would also cause go starvation.
- Or, something go calls is blocking and go can’t see it because of
function abstraction (to my knowledge go doesn’t trace into function calls)
or blocking on something other than a channel.
- Potentially something in the go pathway is doing too much cpu work.
Move heavy computation *out* of the go pathway itself to a separate
executor, potentially lowering the thread priority. Use the go executor
for blocking io only. Somewhere in there is a computational bandwidth vs.
latency tradeoff.
Hope this is solved and you can enjoy your weekend!
Chris
On Fri, Apr 12, 2019 at 12:26 PM Ricardo Mayerhofer <[email protected]>
wrote:
> We're investigating a starvation issue in restQL (
> https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy
> use of core.async, all I/O is non-blocking, we use Aleph as http server and
> client.
>
> The symptom is that it takes a long time for the first line of go routine
> to execute. e.g.:
>
> (defn a []
> (f1)
> (f2)
> (f3)
> (go
> (f4)
> )
> )
>
> f1, f2, f3 are executed very fast in sequence. f4 takes more than 10
> seconds to execute.
>
> The CPU usage is high, so I guess the core.async thread pool is full so it
> takes a long time for the dispatched block to execute in the thread pool.
> This causes every request to timeout because no one able is able to execute
> under the allowed time.
>
> So my questions is:
> - Is there a way to cancel a go execution block when it's queued for a
> certain amount of time, such as max queue time? When the starvation process
> start this would cancel the go routines and release resources.
>
> Or any idea? Thanks in advance.
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to [email protected]
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
