What is the ‘fetch’ function here? Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN An Architect's View -- http://corfield.org/
"If you're not annoying somebody, you're not really alive." -- Margaret Atwood From: lawrence.krub...@gmail.com<mailto:lawrence.krub...@gmail.com> Sent: Tuesday, July 11, 2017 1:18 PM To: Clojure<mailto:clojure@googlegroups.com> Subject: Re: I can only get the first item of a lazyseq via a Manifold stream, and I can't get/find an Exception Justin, thanks. I'd like to ask a follow up question. To be clear, if I go into (doseq) then this works fine: (defn start [] (let [ config (get-config) mysql-db { :dbtype "mysql" :dbname (get-in config [:database :config :connection :database]) :user (get-in config [:database :config :connection :user] ) :password (get-in config [:database :config :connection :password] ) :host (get-in config [:database :config :connection :host]) } data (fetch mysql-db) ] (slingshot/try+ (doseq [row data] (queue/enqueue row)) (catch Object o (errors/error o "" "error in query_database/start: "))))) but if I do this instead: ;;;;;;; (doseq [row data] (queue/enqueue data)) and then enqueue does this: (->> (ms/->source data) (ms/onto executor) (ms/map api/query)) The first row comes through fine, but then the second row is never processed. So are you suggesting that simply passing "data" from one function to the next is enough to lose the database context? But only after the first row has been pulled? On Tuesday, July 11, 2017 at 1:44:15 PM UTC-4, Justin Smith wrote: My first suspicion would be that by the time you access the second element, you have exited the context of your database transaction, so there's no data stream available to get it from. Lazyness doesn't tend to mix well with stateful resources and contexts. On Mon, Jul 10, 2017 at 9:45 PM <lawrence...@gmail.com<javascript:>> wrote: Okay, that was a deadend. After going through line by line, I could pretty well rule out any kind of Exception or Error. The first row from the database seems to go through all of the functions perfectly, without any errors. But the second row never gets called. Which takes me back to where I started. Why can't I give a lazy-seq to a Manifold Stream and simply have a function map over the stream? On Monday, July 10, 2017 at 11:25:43 PM UTC-4, lawrence...@gmail.com wrote: Once again, my lack of knowledge of Java trips me up. Manifold relies on Dirigiste, which relies on Java's Executor Service. I see a bit here: http://www.nurkiewicz.com/2014/11/executorservice-10-tips-and-tricks.html Nurkiewicz writes: "I got bitten by that too many times: it won't print anything. No sign of java.lang.ArithmeticException: / by zero, nothing. Thread pool just swallows this exception, as if it never happened. If it was a good'ol java.lang.Thread created from scratch, UncaughtExceptionHandler<https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.UncaughtExceptionHandler.html> could work. " I suspect I'm facing something like that. On Monday, July 10, 2017 at 8:28:03 PM UTC-4, lawrence...@gmail.com wrote: By the way, this code works fine if I go into a (doseq) a level above enqueue, and then put each individual row onto the stream. Then the code loops over all of the rows. But that seems to defeat the whole point of using something like Manifold. I want to be able to put the whole lazy-seq on the stream. That is supposed to work, yes? On Monday, July 10, 2017 at 8:18:07 PM UTC-4, lawrence...@gmail.com wrote: I'm using Zach Tellman's excellent Manifold library, though I admit I don't fully understand it. My code queries a MySQL database and then needs to do some processing on each row retrieved. I copy-and-pasted some code from the documentation for Manifold: ;; 2017-07-10 -- we want a thread pool. I'm arbitrarily choosing 200 threads. ;; query_database/start will pull data from the database and dump it onto a ;; stream below, at which point each row should be assigned to one of the rows ;; on our thread pool. (def executor (me/fixed-thread-executor 200)) (defn enqueue [sequence-from-database] (slingshot/try+ (println "the type of the object from the database: " (type sequence-from-database)) (->> (ms/->source sequence-from-database) (ms/onto executor) (ms/map api/query)) (catch Object o (println " message queue is not happy about the message we were given") (errors/error o "" " we tried to put something on the message queue, but we got an error ")))) The line where I print out the type assures that I'm dealing with a LazySeq. The code prints out the type and the first row: the type of the object from the database: clojure.lang.LazySeq in query_api/query {:profile_id 2, :profile_name Mike Shaw Automotive Group, :headquarters_addr1 90 Madison St., :headquarters_city Denver, :headquarters_state_code CO, :headquarters_country_code US, :url mikeshawauto.com<http://mikeshawauto.com>} and then it stops. I assume there must be an Exception or Error happening, but I can't find it. I've added as many general Catch clauses as I could: (defn query [row & args] (println " in query_api/query " row) (let [config (if args (first args)) ] ;; 2017-03-30 -- the API is overwhelmed and all I get is Socket Timeout errors (Thread/sleep 300) (slingshot/try+ (call-api row) (catch Object o (println " error : " o) ;;(errors/error o row " we tried to call-api, but we got this error ") ) (catch Error e (println " there Error: " e)) ))) So if I do: java -jar scan-database-find-similar-items-standalone.jar The code runs till it prints out the first row from the database, and then it stops. Nothing else happens. There are no error messages. What did I miss? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clo...@googlegroups.com<javascript:> Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com<javascript:> For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com<javascript:>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com<mailto:clojure+unsubscr...@googlegroups.com>. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
