[ANN] Cognitect Labs' aws-api 0.8.292
Cognitect Labs' aws-api 0.8.292 is now available! CHANGES - improved support for apigatewaymanagementapi :PostToConnection NOTE: this represents a breaking change for anybody using :PostToConnection. See https://github.com/cognitect-labs/aws-api/#posttoconnection for the correct way to set up your client and request. README: https://github.com/cognitect-labs/aws-api/ API Docs: https://cognitect-labs.github.io/aws-api/ Changelog: https://github.com/cognitect-labs/aws-api/blob/master/CHANGES.md Latest Releases of api, endpoints, and all services: https://github.com/cognitect-labs/aws-api/blob/master/latest-releases.edn -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Go block starvation issue
Hi Chris, Thank you very much for the thoughtful answer! We've been profiling CPU usage, most time currently is used in hard to optimize operations such as into (which in turn calls reduce) e.g.: https://github.com/B2W-BIT/restQL-clojure/blob/f473ddb1dcd30704764bf64b7f88eff27391aa76/src/main/restql/core/runner/request.clj#L215 But we haven't give up and will keep working on it. On the other hand we'd like when high CPU usage happens to protect restQL, which means process some requests and drop others, keep running. I don't think the thread pool is used anywhere else. I could only find reference of clojure.core.async.impl.dispatch (who stores the pool) in go, take and put. I think the timed-go will help, gonna try it. Em sexta-feira, 12 de abril de 2019 16:30:21 UTC-3, Chris Nuernberger escreveu: > > Given the nature of the problem I would wonder why the cpu usage is high. > restQL seems to be mostly a passthrough entity completely dependent upon IO > so there may be something spinning or a pmap somewhere there should not be. > >- jstacktrace may offer insight here. > > If nothing is out of whack in terms of what the machine is actually doing > then you could implement a timed go macro that skipped its internals if it > took more than Y seconds to execute. > > (defmacro timed-go > [timeout & args] > `(let [time# (get-time)] > (go >(when (< (- (get-time) time#) ~timeout) > ~@args > > I wouldn’t start there though. Here are some things I would check before > I implemented the above macro. > >- Potentially something is using the same execution pool that >clojure’s async library uses for go. This would also cause go starvation. >- Or, something go calls is blocking and go can’t see it because of >function abstraction (to my knowledge go doesn’t trace into function > calls) >or blocking on something other than a channel. >- Potentially something in the go pathway is doing too much cpu work. >Move heavy computation *out* of the go pathway itself to a separate >executor, potentially lowering the thread priority. Use the go executor >for blocking io only. Somewhere in there is a computational bandwidth vs. >latency tradeoff. > > Hope this is solved and you can enjoy your weekend! > > Chris > > > > > On Fri, Apr 12, 2019 at 12:26 PM Ricardo Mayerhofer > wrote: > >> We're investigating a starvation issue in restQL ( >> https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy >> use of core.async, all I/O is non-blocking, we use Aleph as http server and >> client. >> >> The symptom is that it takes a long time for the first line of go routine >> to execute. e.g.: >> >> (defn a [] >> (f1) >> (f2) >> (f3) >> (go >> (f4) >> ) >> ) >> >> f1, f2, f3 are executed very fast in sequence. f4 takes more than 10 >> seconds to execute. >> >> The CPU usage is high, so I guess the core.async thread pool is full so >> it takes a long time for the dispatched block to execute in the thread >> pool. This causes every request to timeout because no one able is able to >> execute under the allowed time. >> >> So my questions is: >> - Is there a way to cancel a go execution block when it's queued for a >> certain amount of time, such as max queue time? When the starvation process >> start this would cancel the go routines and release resources. >> >> Or any idea? Thanks in advance. >> >> >> >> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clo...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clo...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Go block starvation issue
Given the nature of the problem I would wonder why the cpu usage is high. restQL seems to be mostly a passthrough entity completely dependent upon IO so there may be something spinning or a pmap somewhere there should not be. - jstacktrace may offer insight here. If nothing is out of whack in terms of what the machine is actually doing then you could implement a timed go macro that skipped its internals if it took more than Y seconds to execute. (defmacro timed-go [timeout & args] `(let [time# (get-time)] (go (when (< (- (get-time) time#) ~timeout) ~@args I wouldn’t start there though. Here are some things I would check before I implemented the above macro. - Potentially something is using the same execution pool that clojure’s async library uses for go. This would also cause go starvation. - Or, something go calls is blocking and go can’t see it because of function abstraction (to my knowledge go doesn’t trace into function calls) or blocking on something other than a channel. - Potentially something in the go pathway is doing too much cpu work. Move heavy computation *out* of the go pathway itself to a separate executor, potentially lowering the thread priority. Use the go executor for blocking io only. Somewhere in there is a computational bandwidth vs. latency tradeoff. Hope this is solved and you can enjoy your weekend! Chris On Fri, Apr 12, 2019 at 12:26 PM Ricardo Mayerhofer wrote: > We're investigating a starvation issue in restQL ( > https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy > use of core.async, all I/O is non-blocking, we use Aleph as http server and > client. > > The symptom is that it takes a long time for the first line of go routine > to execute. e.g.: > > (defn a [] > (f1) > (f2) > (f3) > (go > (f4) > ) > ) > > f1, f2, f3 are executed very fast in sequence. f4 takes more than 10 > seconds to execute. > > The CPU usage is high, so I guess the core.async thread pool is full so it > takes a long time for the dispatched block to execute in the thread pool. > This causes every request to timeout because no one able is able to execute > under the allowed time. > > So my questions is: > - Is there a way to cancel a go execution block when it's queued for a > certain amount of time, such as max queue time? When the starvation process > start this would cancel the go routines and release resources. > > Or any idea? Thanks in advance. > > > > > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Go block starvation issue
We're investigating a starvation issue in restQL ( https://github.com/B2W-BIT/restQL-http) under heavy load. We make heavy use of core.async, all I/O is non-blocking, we use Aleph as http server and client. The symptom is that it takes a long time for the first line of go routine to execute. e.g.: (defn a [] (f1) (f2) (f3) (go (f4) ) ) f1, f2, f3 are executed very fast in sequence. f4 takes more than 10 seconds to execute. The CPU usage is high, so I guess the core.async thread pool is full so it takes a long time for the dispatched block to execute in the thread pool. This causes every request to timeout because no one able is able to execute under the allowed time. So my questions is: - Is there a way to cancel a go execution block when it's queued for a certain amount of time, such as max queue time? When the starvation process start this would cancel the go routines and release resources. Or any idea? Thanks in advance. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Add support for seq on Java streams?
We have been looking at how to provide default interop for some of the function APIs, and this seems equally interesting now. I don’t recall a ticket for this in particular so feel free to make one. Needs some assessment of course. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Add support for seq on Java streams?
Since now the minimum required Java version is 8 (I believe) is there any plan to implement this in core? On Tuesday, 23 February 2016 18:06:48 UTC+1, Alex Miller wrote: > > It may, however keep in mind that Clojure supports Java 1.6+ and Stream > was added in 1.8. That's not an impossible hurdle, but it might make sense > to make longer before hurdling it. > > On Tuesday, February 23, 2016 at 11:03:55 AM UTC-6, 676c...@gmail.com > wrote: >> >> Hello! >> >> For interoperation with Java, Clojure’s seq supports the Iterable >> interface directly, which means that all Java collections are >> automatically seqable. seq also supports the CharSequence and >> java.util.Map interfaces, and arrays too. >> >> Would it make sense to have seq also support java.util.stream.Stream? >> Streams are just as important an abstraction as collections, and I >> believe that stream-producing APIs are becoming more and more common. It >> would be nice if Clojure supported it out of the box. >> >> An example (context: http://stackoverflow.com/q/35574155): >> >> (->> (.splitAsStream #"\s+" "one two three") >> (remove #(clojure.string/includes? % "o")) >> (map clojure.string/upper-case) >> first) >> >> Thanks, >> >> >> -- >> David >> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
