Oh - this appears to be a separate issue that isn't just about leaving the
context of the db transaction, I don't know enough about those parts of
manifold to address the issue, sorry.
On Tue, Jul 11, 2017 at 1:18 PM <lawrence.krub...@gmail.com> wrote:
> Justin, thanks. I'd like to ask a follow up question. To be clear, if I go
> into (doseq) then this works fine:
>
> (defn start []
> (let [
> config (get-config)
> mysql-db {
> :dbtype "mysql"
> :dbname (get-in config [:database :config :connection
> :database])
> :user (get-in config [:database :config :connection
> :user] )
> :password (get-in config [:database :config :connection
> :password] )
> :host (get-in config [:database :config :connection
> :host])
> }
> data (fetch mysql-db)
> ]
> (slingshot/try+
> (doseq [row data]
> (queue/enqueue row))
> (catch Object o
> (errors/error o "" "error in query_database/start: ")))))
>
>
> but if I do this instead:
>
> ;;;;;;; (doseq [row data]
> (queue/enqueue data))
>
> and then enqueue does this:
>
> (->> (ms/->source data)
> (ms/onto executor)
> (ms/map api/query))
>
>
> The first row comes through fine, but then the second row is never
> processed.
>
> So are you suggesting that simply passing "data" from one function to the
> next is enough to lose the database context? But only after the first row
> has been pulled?
>
>
>
>
>
>
> On Tuesday, July 11, 2017 at 1:44:15 PM UTC-4, Justin Smith wrote:
>
>> My first suspicion would be that by the time you access the second
>> element, you have exited the context of your database transaction, so
>> there's no data stream available to get it from. Lazyness doesn't tend to
>> mix well with stateful resources and contexts.
>>
>> On Mon, Jul 10, 2017 at 9:45 PM <lawrence...@gmail.com> wrote:
>>
> Okay, that was a deadend. After going through line by line, I could pretty
>>> well rule out any kind of Exception or Error. The first row from the
>>> database seems to go through all of the functions perfectly, without any
>>> errors. But the second row never gets called.
>>>
>>> Which takes me back to where I started. Why can't I give a lazy-seq to a
>>> Manifold Stream and simply have a function map over the stream?
>>>
>>>
>>> On Monday, July 10, 2017 at 11:25:43 PM UTC-4, lawrence...@gmail.com
>>> wrote:
>>>>
>>>>
>>>> Once again, my lack of knowledge of Java trips me up. Manifold relies
>>>> on Dirigiste, which relies on Java's Executor Service. I see a bit here:
>>>>
>>>>
>>>> http://www.nurkiewicz.com/2014/11/executorservice-10-tips-and-tricks.html
>>>>
>>>> Nurkiewicz writes:
>>>> "I got bitten by that too many times: it won't print *anything*. No
>>>> sign of java.lang.ArithmeticException: / by zero, nothing. Thread pool
>>>> just swallows this exception, as if it never happened. If it was a good'ol
>>>> java.lang.Thread created from scratch, UncaughtExceptionHandler
>>>> <https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.UncaughtExceptionHandler.html>
>>>> could
>>>> work. "
>>>>
>>>> I suspect I'm facing something like that.
>>>>
>>>>
>>>>
>>>> On Monday, July 10, 2017 at 8:28:03 PM UTC-4, lawrence...@gmail.com
>>>> wrote:
>>>>>
>>>>> By the way, this code works fine if I go into a (doseq) a level above
>>>>> enqueue, and then put each individual row onto the stream. Then the code
>>>>> loops over all of the rows. But that seems to defeat the whole point of
>>>>> using something like Manifold. I want to be able to put the whole lazy-seq
>>>>> on the stream. That is supposed to work, yes?
>>>>>
>>>>>
>>>>> On Monday, July 10, 2017 at 8:18:07 PM UTC-4, lawrence...@gmail.com
>>>>> wrote:
>>>>>>
>>>>>> I'm using Zach Tellman's excellent Manifold library, though I admit I
>>>>>> don't fully understand it.
>>>>>>
>>>>>> My code queries a MySQL database and then needs to do some processing
>>>>>> on each row retrieved. I copy-and-pasted some code from the documentation
>>>>>> for Manifold:
>>>>>>
>>>>>>
>>>>>> ;; 2017-07-10 -- we want a thread pool. I'm arbitrarily choosing 200
>>>>>> threads.
>>>>>> ;; query_database/start will pull data from the database and dump it
>>>>>> onto a
>>>>>> ;; stream below, at which point each row should be assigned to one of
>>>>>> the rows
>>>>>> ;; on our thread pool.
>>>>>> (def executor (me/fixed-thread-executor 200))
>>>>>>
>>>>>>
>>>>>> (defn enqueue
>>>>>> [sequence-from-database]
>>>>>> (slingshot/try+
>>>>>> (println "the type of the object from the database: " (type
>>>>>> sequence-from-database))
>>>>>> (->> (ms/->source sequence-from-database)
>>>>>> (ms/onto executor)
>>>>>> (ms/map api/query))
>>>>>> (catch Object o
>>>>>> (println " message queue is not happy about the message we were
>>>>>> given")
>>>>>> (errors/error o "" " we tried to put something on the message
>>>>>> queue, but we got an error "))))
>>>>>>
>>>>>> The line where I print out the type assures that I'm dealing with a
>>>>>> LazySeq.
>>>>>>
>>>>>> The code prints out the type and the first row:
>>>>>>
>>>>>>
>>>>>> the type of the object from the database: clojure.lang.LazySeq
>>>>>>
>>>>>> in query_api/query {:profile_id 2, :profile_name Mike Shaw
>>>>>> Automotive Group, :headquarters_addr1 90 Madison St., :headquarters_city
>>>>>> Denver, :headquarters_state_code CO, :headquarters_country_code US, :url
>>>>>> mikeshawauto.com}
>>>>>>
>>>>>> and then it stops. I assume there must be an Exception or Error
>>>>>> happening, but I can't find it. I've added as many general Catch clauses
>>>>>> as
>>>>>> I could:
>>>>>>
>>>>>>
>>>>>> (defn query
>>>>>> [row & args]
>>>>>>
>>>>>> (println " in query_api/query " row)
>>>>>>
>>>>>> (let [config (if args
>>>>>> (first args))
>>>>>> ]
>>>>>> ;; 2017-03-30 -- the API is overwhelmed and all I get is Socket
>>>>>> Timeout errors
>>>>>> (Thread/sleep 300)
>>>>>>
>>>>>> (slingshot/try+
>>>>>> (call-api row)
>>>>>> (catch Object o
>>>>>> (println " error : " o)
>>>>>>
>>>>>> ;;(errors/error o row " we tried to call-api, but we got this
>>>>>> error ")
>>>>>>
>>>>>>
>>>>>> )
>>>>>>
>>>>>> (catch Error e
>>>>>> (println " there Error: " e))
>>>>>> )))
>>>>>>
>>>>>> So if I do:
>>>>>>
>>>>>> java -jar scan-database-find-similar-items-standalone.jar
>>>>>>
>>>>>>
>>>>>> The code runs till it prints out the first row from the database, and
>>>>>> then it stops. Nothing else happens. There are no error messages.
>>>>>>
>>>>>> What did I miss?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>>
>> To post to this group, send email to clo...@googlegroups.com
>>
>>
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>>
>> clojure+u...@googlegroups.com
>>
>>
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to clojure+u...@googlegroups.com.
>>
>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
