Thanks very much for the reply. I've considered creating namespaces on the fly and interning vars in them for each thread pool. Thread-local binding that follows the stack discipline is a beautiful concept (if I call you, accept and pass on my bindings), and thread-pool binding seemed like it would be a natural counterpart (if I spawn you, accept and intern my bindings to your spawn). But I haven't come across Clojure code that attempts to do this, so I hesitated.
Using closures seems natural enough. I guess I could wrap functions for their Executor with a simple macro like (defmacro with-args-enclosed [f & args] `(fn [] (~f ~...@args))) and keep it simple. Thanks again, Dan On Tue, Jul 21, 2009 at 10:13 AM, Stuart Sierra <the.stuart.sie...@gmail.com > wrote: > > Hi Dan, > > I'm not sure, but closures actually sound like the way to go here. > That would be the traditional functional-programming solution to this > problem. It's true the Java equivalent is ugly, but that's because > Java doesn't have real closures. :) > > -SS > > On Jul 20, 3:30 pm, Dan Fichter <daniel.fich...@gmail.com> wrote: > > I want to set up Executor thread pools spawned by threads in other > Executor > > thread pools and to scope certain variables to these nested pools. I > wish > > something I would call "thread-pool scope" were easily achieved in > Clojure. > > Is it? > > > > I'm writing a Web crawler (let's pretend) that has three top-level > > functions: > > > > - *start-crawlers* will spawn several crawlers by invoking (crawl) > several > > times in an Executor pool, and it'll give each crawler its own http > > user-agent to crawl with > > > > - *crawl, *each time it's invoked, will construct a queue of hyperlinks > and > > a new Executor thread pool; it'll then invoke (visit) on several threads > in > > this pool, and it'll expect them to use this queue and the user-agent > that > > was supplied when (crawl) was invoked > > > > - *visit* will consume its pool's queue, using its pool's user-agent to > > visit Web pages > > > > So there are nested thread pools, and the question is how to scope the > data > > structures holding user-agents and queues to them. How does > start-crawlers > > tell each (crawl) invocation which user-agent to use? And how does crawl > > pass this value to each (visit) invocation in the thread pool it's > spawned > > and share its queue with these visitors? > > > > These are the options I've come up with (three "can't"s and one lonely > > "can"): > > > > 1. Reference passing: why can't start-crawlers simply pass a user-agent > when > > it invokes (crawl)? and why can't crawl pass this user-agent and a > reference > > to a queue when it invokes (visit)? Because Executors can execute > > no-arguments functions only. If they want to be threaded, crawl and > visit > > cannot take arguments. (In the land of broken threading interfaces, > Python: > > 1; Java: 0.) > > > > 2. Dynamic var bindings: these won't work, because var bindings are > > thread-local. If start-crawlers created a dynamic binding for > user-agent, > > it wouldn't be visible to the (crawl)-invoking threads that needed it. > > > > 3. Global vars and atoms: these would work only if my thread pools > weren't > > nested. But I can't have a single global var called user-agent or a > single > > global atom called queue. Each crawl-spawned thread pool needs its own > > queue and its own user-agent, so globals won't work. > > > > 4. Closures: these will work. Rather than have start-crawlers evaluate > > (.execute pool crawl), it can do (.execute pool (get-crawl-fn > user-agent)), > > where (get-crawl-fn) returns a zero-arguments crawl-like function that > > closes over the user-agent supplied. Likewise, crawl can do (.execute > pool > > (get-visit-fn user-agent queue)), where get-visit-fn serves up a > > zero-arguments visitor function. > > > > But I wish I didn't have to use closures. It's more abstraction than I > > think this problem should require. Or the wrong kind of abstraction. > It's > > equivalent to what I imagine the Java solution would be: to have a > Runnable > > implementor capture some arguments in its constructor call and then > access > > these in its no-arguments run() method. Conceptually, it isn't what I > want. > > > > I don't want lexical, thread-local, or global scope; my program consists > of > > nested thread pools, and I want thread-pool scope. Is there a nice > Clojure > > way to get this? Would you use dynamically-generated namespaces or > > something else I haven't dug into yet? > > > > Thanks for your help, and thanks for Clojure, > > > > Dan > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
