The plan is to run it in a thread, yeah. The process-fn I'm planning on running it with does some stuff to the tweet and then uploads the results to a local couchdb instance. I've been periodically checking /var/log/couchdb/couch.log to verify it's actually doing stuff.
I *think* this could benefit from some core.async magic, but I don't understand that lib yet, and haven't had the time to watch Rich's talk on it. Someday... On Wednesday, November 12, 2014 5:34:14 PM UTC-5, Francis Avila wrote: > > Your loop pattern should work. (I've used this pattern before.) > > Just a sanity check: you *are* running this function in a different > thread, right? Because whatever thread calls this function *will* block > forever, whether the queue is empty or not. And unless you provide some > side-effecting process-fn there will be no evidence that the thread is > doing anything. (I.e. your first function will hang forever and appear to > do nothing.) > > > On Wednesday, November 12, 2014 10:43:57 AM UTC-6, Sam Raker wrote: >> >> I'm using Twitter's HBC library to read from Twitter's public stream. HBC >> stores results in a LinkedBlockingQueue, from which you can then `.take` >> tweets and do stuff to them (in my case, doing some >> processing/normalization, then storing them in CouchDB). I've been >> struggling with how exactly best to do this, though. I tried `doseq`, but >> it stops when the queue is empty, which isn't what I want. Since my code is >> basically entirely IO, `map` and other lazy stuff causes me problems. Next, >> I reached for `loop`: >> >> (defn process-stream-nores >> ([in-queue] >> (process-stream-nores in-queue identity)) >> ([in-queue process-fn] >> (loop [res (.take in-queue)] >> (process-fn (parse-string res true)) >> (recur (.take in-queue))))) >> >> >> (where `in-queue` is a LinkedBlockingQueue). Unfortunately, this just >> hangs, even when the LinkedBlockingQueue isn't empty (I'd expect it to hang >> when the queue is empty, since the queue blocks until it gets something). >> >> I've also tried >> >> ... >> (while true (process-fn (parse-string (.take in-queue) true)) >> ... >> >> but that 1) also hangs, and 2) seems profoundly un-idiomatic. >> >> I want to continuously take from the queue, process the tweet, and then >> put it in CouchDB. (I'm planning on putting this in a Thread that I can >> stop when I have enough tweets.) I feel like loop is the right way to go, >> but I don't understand it very well, and can't get it to work. Any help >> would be greatly appreciated. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
