Hi Chris, Thank you very much for the thoughtful answer! We've been profiling CPU usage, most time currently is used in hard to optimize operations such as into (which in turn calls reduce) e.g.: https://github.com/B2W-BIT/restQL-clojure/blob/f473ddb1dcd30704764bf64b7f88eff27391aa76/src/main/restql/core/runner/request.clj#L215
But we haven't give up and will keep working on it. On the other hand we'd like when high CPU usage happens to protect restQL, which means process some requests and drop others, keep running. I don't think the thread pool is used anywhere else. I could only find reference of clojure.core.async.impl.dispatch (who stores the pool) in go, take and put. I think the timed-go will help, gonna try it. Em sexta-feira, 12 de abril de 2019 16:30:21 UTC-3, Chris Nuernberger escreveu: > > Given the nature of the problem I would wonder why the cpu usage is high. > restQL seems to be mostly a passthrough entity completely dependent upon IO > so there may be something spinning or a pmap somewhere there should not be. > > - jstacktrace may offer insight here. > > If nothing is out of whack in terms of what the machine is actually doing > then you could implement a timed go macro that skipped its internals if it > took more than Y seconds to execute. > > (defmacro timed-go > [timeout & args] > `(let [time# (get-time)] > (go > (when (< (- (get-time) time#) ~timeout) > ~@args)))) > > I wouldn’t start there though. Here are some things I would check before > I implemented the above macro. > > - Potentially something is using the same execution pool that > clojure’s async library uses for go. This would also cause go starvation. > - Or, something go calls is blocking and go can’t see it because of > function abstraction (to my knowledge go doesn’t trace into function > calls) > or blocking on something other than a channel. > - Potentially something in the go pathway is doing too much cpu work. > Move heavy computation *out* of the go pathway itself to a separate > executor, potentially lowering the thread priority. Use the go executor > for blocking io only. Somewhere in there is a computational bandwidth vs. > latency tradeoff. > > Hope this is solved and you can enjoy your weekend! > > Chris > > > > > On Fri, Apr 12, 2019 at 12:26 PM Ricardo Mayerhofer <ricar...@gmail.com > <javascript:>> wrote: > >> We're investigating a starvation issue in restQL ( >> https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy >> use of core.async, all I/O is non-blocking, we use Aleph as http server and >> client. >> >> The symptom is that it takes a long time for the first line of go routine >> to execute. e.g.: >> >> (defn a [] >> (f1) >> (f2) >> (f3) >> (go >> (f4) >> ) >> ) >> >> f1, f2, f3 are executed very fast in sequence. f4 takes more than 10 >> seconds to execute. >> >> The CPU usage is high, so I guess the core.async thread pool is full so >> it takes a long time for the dispatched block to execute in the thread >> pool. This causes every request to timeout because no one able is able to >> execute under the allowed time. >> >> So my questions is: >> - Is there a way to cancel a go execution block when it's queued for a >> certain amount of time, such as max queue time? When the starvation process >> start this would cancel the go routines and release resources. >> >> Or any idea? Thanks in advance. >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> <javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clo...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clo...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
