Re: ANN: 6pm Sat, Jan 9th: Wraith Scheme, Paralell Distributed Clojure at the Hacker Dojo
Whoops, yes, I meant Jan 9, thanks On Jan 6, 8:02 pm, ajay gopalakrishnan ajgop...@gmail.com wrote: It's Jan 9 I guess On Wed, Jan 6, 2010 at 10:53 PM, nallen05 nalle...@gmail.com wrote: Two very cool presentations this Saturday at the Hacker Dojo in Mountain View: 1. An introduction to Wraith Scheme by Jay Reynolds Freeman 2. An introduction to parallel and distributed programming with Clojure by Amit Rathore Go here for more info:http://www.meetup.com/balisp/calendar/12248048/ Hope to see you there! Nick -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en- Hide quoted text - - Show quoted text - -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
ANN: 6pm Sat, Dec 9th: Wraith Scheme, Paralell Distributed Clojure at the Hacker Dojo
Two very cool presentations this Saturday at the Hacker Dojo in Mountain View: 1. An introduction to Wraith Scheme by Jay Reynolds Freeman 2. An introduction to parallel and distributed programming with Clojure by Amit Rathore Go here for more info: http://www.meetup.com/balisp/calendar/12248048/ Hope to see you there! Nick -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: 6pm Sat, Dec 9th: Wraith Scheme, Paralell Distributed Clojure at the Hacker Dojo
It's Jan 9 I guess On Wed, Jan 6, 2010 at 10:53 PM, nallen05 nalle...@gmail.com wrote: Two very cool presentations this Saturday at the Hacker Dojo in Mountain View: 1. An introduction to Wraith Scheme by Jay Reynolds Freeman 2. An introduction to parallel and distributed programming with Clojure by Amit Rathore Go here for more info: http://www.meetup.com/balisp/calendar/12248048/ Hope to see you there! Nick -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Distributed Clojure
On Thu, Jan 29, 2009 at 7:28 PM, Greg Harman ghar...@gmail.com wrote: Hank: I have looked at TC in the past, and took another look today at your suggestion. Terracotta certainly seems to have promise feature-wise, but I have to admit it's a heavier solution than I had been thinking of, and there are probably all sorts of gotchas (and reviewing old threads on the topic, a few are hinted at) when it comes to distributing some of Clojure's concurrency data structures in the shared-memory approach. Of course until someone tries we won't know where the limitations are for sure. But my thought is that even if terracotta turns out not to be ideal as a mechanism to implement everything, as I see it, the terracotta feature set (or hazelcast) is critical to anything like this, and I can't see re-engineering all that work because you need/want certain things that may need to be done outside it. Regarding it being heavier, my feeling is that for what is really needed, it really is a heavy problem. Hank Kevin: I hadn't heard of Hadoop before, but at first glance it's exactly what I'm looking for. (Thanks!) The problem I am working on is a processing pipeline for massive data sets, and that seems to be the advertised use case for Hadoop. On Jan 29, 7:14 pm, Kevin Downey redc...@gmail.com wrote: have you looked at the available java frameworks like hadoop? there is also some kind of java interface to erlang instead of reinventing the wheel again... On Thu, Jan 29, 2009 at 6:15 AM, Greg Harman ghar...@gmail.com wrote: One of Clojure's big selling points (obviously) is the support for concurrent programming. However, the performance gains you get with this concurrency hits a scalability wall once you're fully utilizing all the cores available in your server. The next step, of course, is to add additional servers. So, I've been mulling over the idea of putting together a framework for distributed applications. I think it should deal with issues such as: * Locating, assessing, and registering CPUs * Division of large jobs into smaller sub-jobs * Dispatching of jobs, including optimization planning (send more jobs to faster CPUs) * Failure recovery * Seamless code integration (so that this can be retrofit as an optimization) both horizontally and vertically * Horizontal = take one time-consuming task and parallelize it * Vertical = take two sequential tasks and pipeline them across different servers Is anybody already working on something similar? Is this already on the Clojure language roadmap as something to be added after 1.0? Any design advice or feature requests if I were to put something like this together? -Greg -- And what is good, Phaedrus, And what is not good— Need we ask anyone to tell us these things? -- blog: whydoeseverythingsuck.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Distributed Clojure
One of Clojure's big selling points (obviously) is the support for concurrent programming. However, the performance gains you get with this concurrency hits a scalability wall once you're fully utilizing all the cores available in your server. The next step, of course, is to add additional servers. So, I've been mulling over the idea of putting together a framework for distributed applications. I think it should deal with issues such as: * Locating, assessing, and registering CPUs * Division of large jobs into smaller sub-jobs * Dispatching of jobs, including optimization planning (send more jobs to faster CPUs) * Failure recovery * Seamless code integration (so that this can be retrofit as an optimization) both horizontally and vertically * Horizontal = take one time-consuming task and parallelize it * Vertical = take two sequential tasks and pipeline them across different servers Is anybody already working on something similar? Is this already on the Clojure language roadmap as something to be added after 1.0? Any design advice or feature requests if I were to put something like this together? -Greg --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
On Jan 29, 2009, at 15:15, Greg Harman wrote: So, I've been mulling over the idea of putting together a framework for distributed applications. I think it should deal with issues such as: ... I think the very first step should be to implement the basics of distributed computing: - communication between nodes (including serialization of data) - synchronization - remote function calls Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
As has been discussed on this list before, it seems to me the basis for this should be terracotta, which handles much (most?) of the heavy lifiting required for this kind of task. Have you looked at it? On Thu, Jan 29, 2009 at 9:15 AM, Greg Harman ghar...@gmail.com wrote: One of Clojure's big selling points (obviously) is the support for concurrent programming. However, the performance gains you get with this concurrency hits a scalability wall once you're fully utilizing all the cores available in your server. The next step, of course, is to add additional servers. So, I've been mulling over the idea of putting together a framework for distributed applications. I think it should deal with issues such as: * Locating, assessing, and registering CPUs * Division of large jobs into smaller sub-jobs * Dispatching of jobs, including optimization planning (send more jobs to faster CPUs) * Failure recovery * Seamless code integration (so that this can be retrofit as an optimization) both horizontally and vertically * Horizontal = take one time-consuming task and parallelize it * Vertical = take two sequential tasks and pipeline them across different servers Is anybody already working on something similar? Is this already on the Clojure language roadmap as something to be added after 1.0? Any design advice or feature requests if I were to put something like this together? -Greg -- blog: whydoeseverythingsuck.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
Agreed; the communication layer needs to come first. Regarding serialization, specifically, I think we get that for free with s- exps (there may be some under-the-hood evaluation time necessary for remoted expressions, but [de]serialization is rarely a lightweight process). On Jan 29, 10:03 am, Konrad Hinsen konrad.hin...@laposte.net wrote: I think the very first step should be to implement the basics of distributed computing: - communication between nodes (including serialization of data) - synchronization - remote function calls --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
have you looked at the available java frameworks like hadoop? there is also some kind of java interface to erlang instead of reinventing the wheel again... On Thu, Jan 29, 2009 at 6:15 AM, Greg Harman ghar...@gmail.com wrote: One of Clojure's big selling points (obviously) is the support for concurrent programming. However, the performance gains you get with this concurrency hits a scalability wall once you're fully utilizing all the cores available in your server. The next step, of course, is to add additional servers. So, I've been mulling over the idea of putting together a framework for distributed applications. I think it should deal with issues such as: * Locating, assessing, and registering CPUs * Division of large jobs into smaller sub-jobs * Dispatching of jobs, including optimization planning (send more jobs to faster CPUs) * Failure recovery * Seamless code integration (so that this can be retrofit as an optimization) both horizontally and vertically * Horizontal = take one time-consuming task and parallelize it * Vertical = take two sequential tasks and pipeline them across different servers Is anybody already working on something similar? Is this already on the Clojure language roadmap as something to be added after 1.0? Any design advice or feature requests if I were to put something like this together? -Greg -- And what is good, Phaedrus, And what is not good— Need we ask anyone to tell us these things? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
Hank: I have looked at TC in the past, and took another look today at your suggestion. Terracotta certainly seems to have promise feature-wise, but I have to admit it's a heavier solution than I had been thinking of, and there are probably all sorts of gotchas (and reviewing old threads on the topic, a few are hinted at) when it comes to distributing some of Clojure's concurrency data structures in the shared-memory approach. Kevin: I hadn't heard of Hadoop before, but at first glance it's exactly what I'm looking for. (Thanks!) The problem I am working on is a processing pipeline for massive data sets, and that seems to be the advertised use case for Hadoop. On Jan 29, 7:14 pm, Kevin Downey redc...@gmail.com wrote: have you looked at the available java frameworks like hadoop? there is also some kind of java interface to erlang instead of reinventing the wheel again... On Thu, Jan 29, 2009 at 6:15 AM, Greg Harman ghar...@gmail.com wrote: One of Clojure's big selling points (obviously) is the support for concurrent programming. However, the performance gains you get with this concurrency hits a scalability wall once you're fully utilizing all the cores available in your server. The next step, of course, is to add additional servers. So, I've been mulling over the idea of putting together a framework for distributed applications. I think it should deal with issues such as: * Locating, assessing, and registering CPUs * Division of large jobs into smaller sub-jobs * Dispatching of jobs, including optimization planning (send more jobs to faster CPUs) * Failure recovery * Seamless code integration (so that this can be retrofit as an optimization) both horizontally and vertically * Horizontal = take one time-consuming task and parallelize it * Vertical = take two sequential tasks and pipeline them across different servers Is anybody already working on something similar? Is this already on the Clojure language roadmap as something to be added after 1.0? Any design advice or feature requests if I were to put something like this together? -Greg -- And what is good, Phaedrus, And what is not good— Need we ask anyone to tell us these things? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
How about Hazelcast? [1] Its much lighter weight than TC. [1] http://www.hazelcast.com On Fri, Jan 30, 2009 at 1:28 PM, Greg Harman ghar...@gmail.com wrote: Hank: I have looked at TC in the past, and took another look today at your suggestion. Terracotta certainly seems to have promise feature-wise, but I have to admit it's a heavier solution than I had been thinking of, and there are probably all sorts of gotchas (and reviewing old --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
Another thing to look at (there's always another thing to look at) this worries me... it is sorta like the discussions i see about how some new db engine is FAST, oh but it totally fails on occasion and trashes the data, oopsy, oh well. how do we know any given lib is not going to just end up introducing bugs into our system? sure, it is an age-old question, but one has to pay more heed to it when looking at such risky-hard features as concurrency or distribution, no? (this goes back to an old thread in clojure where i was asking about how clojure's stm is tested. i personally can't believe that only inspection and no testing is the right way to get things right, but ymmv.) sincerely. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Distributed Clojure
On 30.01.2009, at 00:59, Greg Harman wrote: Agreed; the communication layer needs to come first. Regarding serialization, specifically, I think we get that for free with s- exps (there may be some under-the-hood evaluation time necessary for remoted expressions, but [de]serialization is rarely a lightweight process). Unfortunately it is not that simple. The fundamental Clojure data structures have an obvious serialization, but Clojure code can use arbitrary Java objects, and it has to deal with its own functions that are present in compiled form. Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
