Justin, thanks. I'd like to ask a follow up question. To be clear, if I go
into (doseq) then this works fine:
(defn start []
(let [
config (get-config)
mysql-db {
:dbtype "mysql"
:dbname (get-in config [:database :config :connection
:database])
:user (get-in config [:database :config :connection
:user] )
:password (get-in config [:database :config :connection
:password] )
:host (get-in config [:database :config :connection
:host])
}
data (fetch mysql-db)
]
(slingshot/try+
(doseq [row data]
(queue/enqueue row))
(catch Object o
(errors/error o "" "error in query_database/start: ")))))
but if I do this instead:
;;;;;;; (doseq [row data]
(queue/enqueue data))
and then enqueue does this:
(->> (ms/->source data)
(ms/onto executor)
(ms/map api/query))
The first row comes through fine, but then the second row is never
processed.
So are you suggesting that simply passing "data" from one function to the
next is enough to lose the database context? But only after the first row
has been pulled?
On Tuesday, July 11, 2017 at 1:44:15 PM UTC-4, Justin Smith wrote:
>
> My first suspicion would be that by the time you access the second
> element, you have exited the context of your database transaction, so
> there's no data stream available to get it from. Lazyness doesn't tend to
> mix well with stateful resources and contexts.
>
> On Mon, Jul 10, 2017 at 9:45 PM <lawrence...@gmail.com <javascript:>>
> wrote:
>
>> Okay, that was a deadend. After going through line by line, I could
>> pretty well rule out any kind of Exception or Error. The first row from the
>> database seems to go through all of the functions perfectly, without any
>> errors. But the second row never gets called.
>>
>> Which takes me back to where I started. Why can't I give a lazy-seq to a
>> Manifold Stream and simply have a function map over the stream?
>>
>>
>> On Monday, July 10, 2017 at 11:25:43 PM UTC-4, lawrence...@gmail.com
>> wrote:
>>>
>>>
>>> Once again, my lack of knowledge of Java trips me up. Manifold relies on
>>> Dirigiste, which relies on Java's Executor Service. I see a bit here:
>>>
>>> http://www.nurkiewicz.com/2014/11/executorservice-10-tips-and-tricks.html
>>>
>>> Nurkiewicz writes:
>>> "I got bitten by that too many times: it won't print *anything*. No
>>> sign of java.lang.ArithmeticException: / by zero, nothing. Thread pool
>>> just swallows this exception, as if it never happened. If it was a good'ol
>>> java.lang.Thread created from scratch, UncaughtExceptionHandler
>>> <https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.UncaughtExceptionHandler.html>
>>> could
>>> work. "
>>>
>>> I suspect I'm facing something like that.
>>>
>>>
>>>
>>> On Monday, July 10, 2017 at 8:28:03 PM UTC-4, lawrence...@gmail.com
>>> wrote:
>>>>
>>>> By the way, this code works fine if I go into a (doseq) a level above
>>>> enqueue, and then put each individual row onto the stream. Then the code
>>>> loops over all of the rows. But that seems to defeat the whole point of
>>>> using something like Manifold. I want to be able to put the whole lazy-seq
>>>> on the stream. That is supposed to work, yes?
>>>>
>>>>
>>>> On Monday, July 10, 2017 at 8:18:07 PM UTC-4, lawrence...@gmail.com
>>>> wrote:
>>>>>
>>>>> I'm using Zach Tellman's excellent Manifold library, though I admit I
>>>>> don't fully understand it.
>>>>>
>>>>> My code queries a MySQL database and then needs to do some processing
>>>>> on each row retrieved. I copy-and-pasted some code from the documentation
>>>>> for Manifold:
>>>>>
>>>>>
>>>>> ;; 2017-07-10 -- we want a thread pool. I'm arbitrarily choosing 200
>>>>> threads.
>>>>> ;; query_database/start will pull data from the database and dump it
>>>>> onto a
>>>>> ;; stream below, at which point each row should be assigned to one of
>>>>> the rows
>>>>> ;; on our thread pool.
>>>>> (def executor (me/fixed-thread-executor 200))
>>>>>
>>>>>
>>>>> (defn enqueue
>>>>> [sequence-from-database]
>>>>> (slingshot/try+
>>>>> (println "the type of the object from the database: " (type
>>>>> sequence-from-database))
>>>>> (->> (ms/->source sequence-from-database)
>>>>> (ms/onto executor)
>>>>> (ms/map api/query))
>>>>> (catch Object o
>>>>> (println " message queue is not happy about the message we were
>>>>> given")
>>>>> (errors/error o "" " we tried to put something on the message
>>>>> queue, but we got an error "))))
>>>>>
>>>>> The line where I print out the type assures that I'm dealing with a
>>>>> LazySeq.
>>>>>
>>>>> The code prints out the type and the first row:
>>>>>
>>>>>
>>>>> the type of the object from the database: clojure.lang.LazySeq
>>>>>
>>>>> in query_api/query {:profile_id 2, :profile_name Mike Shaw Automotive
>>>>> Group, :headquarters_addr1 90 Madison St., :headquarters_city Denver,
>>>>> :headquarters_state_code CO, :headquarters_country_code US, :url
>>>>> mikeshawauto.com}
>>>>>
>>>>> and then it stops. I assume there must be an Exception or Error
>>>>> happening, but I can't find it. I've added as many general Catch clauses
>>>>> as
>>>>> I could:
>>>>>
>>>>>
>>>>> (defn query
>>>>> [row & args]
>>>>>
>>>>> (println " in query_api/query " row)
>>>>>
>>>>> (let [config (if args
>>>>> (first args))
>>>>> ]
>>>>> ;; 2017-03-30 -- the API is overwhelmed and all I get is Socket
>>>>> Timeout errors
>>>>> (Thread/sleep 300)
>>>>>
>>>>> (slingshot/try+
>>>>> (call-api row)
>>>>> (catch Object o
>>>>> (println " error : " o)
>>>>>
>>>>> ;;(errors/error o row " we tried to call-api, but we got this
>>>>> error ")
>>>>>
>>>>>
>>>>> )
>>>>>
>>>>> (catch Error e
>>>>> (println " there Error: " e))
>>>>> )))
>>>>>
>>>>> So if I do:
>>>>>
>>>>> java -jar scan-database-find-similar-items-standalone.jar
>>>>>
>>>>>
>>>>> The code runs till it prints out the first row from the database, and
>>>>> then it stops. Nothing else happens. There are no error messages.
>>>>>
>>>>> What did I miss?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
