By the way, this code works fine if I go into a (doseq) a level above
enqueue, and then put each individual row onto the stream. Then the code
loops over all of the rows. But that seems to defeat the whole point of
using something like Manifold. I want to be able to put the whole lazy-seq
on the stream. That is supposed to work, yes?
On Monday, July 10, 2017 at 8:18:07 PM UTC-4, lawrence...@gmail.com wrote:
>
> I'm using Zach Tellman's excellent Manifold library, though I admit I
> don't fully understand it.
>
> My code queries a MySQL database and then needs to do some processing on
> each row retrieved. I copy-and-pasted some code from the documentation for
> Manifold:
>
>
> ;; 2017-07-10 -- we want a thread pool. I'm arbitrarily choosing 200
> threads.
> ;; query_database/start will pull data from the database and dump it onto a
> ;; stream below, at which point each row should be assigned to one of the
> rows
> ;; on our thread pool.
> (def executor (me/fixed-thread-executor 200))
>
>
> (defn enqueue
> [sequence-from-database]
> (slingshot/try+
> (println "the type of the object from the database: " (type
> sequence-from-database))
> (->> (ms/->source sequence-from-database)
> (ms/onto executor)
> (ms/map api/query))
> (catch Object o
> (println " message queue is not happy about the message we were
> given")
> (errors/error o "" " we tried to put something on the message queue,
> but we got an error "))))
>
> The line where I print out the type assures that I'm dealing with a
> LazySeq.
>
> The code prints out the type and the first row:
>
>
> the type of the object from the database: clojure.lang.LazySeq
>
> in query_api/query {:profile_id 2, :profile_name Mike Shaw Automotive
> Group, :headquarters_addr1 90 Madison St., :headquarters_city Denver,
> :headquarters_state_code CO, :headquarters_country_code US, :url
> mikeshawauto.com}
>
> and then it stops. I assume there must be an Exception or Error happening,
> but I can't find it. I've added as many general Catch clauses as I could:
>
>
> (defn query
> [row & args]
>
> (println " in query_api/query " row)
>
> (let [config (if args
> (first args))
> ]
> ;; 2017-03-30 -- the API is overwhelmed and all I get is Socket
> Timeout errors
> (Thread/sleep 300)
>
> (slingshot/try+
> (call-api row)
> (catch Object o
> (println " error : " o)
>
> ;;(errors/error o row " we tried to call-api, but we got this error
> ")
>
>
> )
>
> (catch Error e
> (println " there Error: " e))
> )))
>
> So if I do:
>
> java -jar scan-database-find-similar-items-standalone.jar
>
>
> The code runs till it prints out the first row from the database, and then
> it stops. Nothing else happens. There are no error messages.
>
> What did I miss?
>
>
>
>
>
>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
