Most stochastic solutions are very amenable to multi-core.
On Sun, Nov 24, 2013 at 5:58 PM, Pascal Jasmin <[email protected]>wrote: > Probably not news to anyone here, but any algorithm that can be expressed > in n ^ x time (polynomial including cases where x is 1 or less) can more > imporantly be expressed as a multi-core/thread/processor algorithm (of k > cores) if the data can be segmented into k parts. n/k ^ x can be a > significant performance improvement if x >1 , but even if x = 1 or less, a > data partitioned algorithms is valuable considering the reality that there > are more affordable quad core 3 ghz processors, than 12 ghz single core > processors. > > So simple search usually has n/a run time, and is usually partitionable, > and it can benefit from multi-core approaches, but there are costs to > coordinating the threads and accumulating results. > > The point, if you are looking for ideas to apply multi-core solutions to, > is that you can do simple search as an example, or focus on any other > problem that has a data partitionable solution. > > > > ________________________________ > From: Joe Bogner <[email protected]> > To: [email protected] > Sent: Saturday, November 23, 2013 9:46:24 PM > Subject: Re: [Jprogramming] wiki: task scheduler and zeromq intro > > > Can anyone share specific examples where it was needed to scale out to > multiple cores and machines? I am interested in learning about the types of > problems this would be applied to. I have read some examples while > researching but haven't ran into anyone who has. > > > > > For example, last week I had to create a database of the best 100,000 > solutions out 56 billion combinations as part of a work deliverable. I am > sure there may have been more elegant solutions however brute forcing with > 4 instances of R and 32 gig of ram took 3 hours, which was fine. > > > > It might be worthwhile to create a small reproducible example of a problem > that would benefit from multiple cores and machines. I could make one up or > borrow from somewhere else but does anyone have any examples that come to > mind? > > > > > > > > > > > > > > > > On Sat, Nov 23, 2013 at 7:09 PM, Scott Locklin <[email protected] > ="mailto:[email protected]";>> wrote: > Dang, thanks for working on this, Pascal. This is one of the use cases I > had in mind for J-ZMQ, though it was taking me some time to figure out how > to do it in my clumsy way. I had been looking at joebo's fork thing for a > little inspiration, but I got busy with a half dozen other things. I will > try to play with this later next week. FWIIW, one way to prevent the "miss > the first message" problem is to fire up the SUB instances before you send > them anything. > > > > For what it is worth; I originally wrote these zeromq hooks for use in a > ticker plant for my own P/L. I had a better idea while working on a > consulting job recently. "Big data" and "the cloud" is in the news all the > time now. Mostly, this is hype, but there are some real business needs > involving extracting meaning from data sets which do not fit into core. The > existing solutions are generally not real impressive (Hadoop) or not > designed for weakly coupled "clouds" (MPI based solutions). Almost none of > these "solutions" use ideas which are actually appropriate to big data > (Vowpal Wabbit being a rare exception), and I have no inclination to > contribute to these tools. > > > > J has several fast and flexible data stores (still learning about Jd; very > impressed so far). J is also extremely memory efficient, and can do out of > core calculations. I have not done any tests to see if ZMQ can get the data > across the pipes well enough to do anything useful on AWS and other such > weakly coupled "clouds" favored by business, but I think I can write some > useful coarse grained parallel ML tools using ZMQ and J. This is perhaps > arrogant of me to think about: my J skills are weak, this is my first ZMQ > project, and it's rare I have time to think about big projects like this, > but I do know how to build fast and scalable machine learning algorithms. I > think this could be a way forward which solves important business problems. > > > Stuff I don't know yet: > > 0) Stability: as Pascal noticed, J and ZMQ are sometimes unstable when > used together. I haven't fired up GDB to find out why yet. It's probably a > buffer allocation thing. It's possible there is a show stopper here: I > don't know. It seems to work with Kx at least: > https://github.com/jaeheum/qzmq > > 1) Fault tolerance: things are going to crash or blow up memory. Maybe > Pascal's task manager is enough for now. > > 2) Data provisioning: I'm guessing I'll need a framework for provisioning > each server with "owned" data, using Jd or JDB. I have to look at how other > frameworks do this. > > 3) Software provisioning: J is pretty simple to set up, but if this is > going to scale out to more than a couple of machines, some kind of tool > will be needed to accomplish this. I know such tools exist, but I have no > way of picking the "right one" at present (suggestions?). > > 4) Security: many of the existing parallel analytics tools have none. > > CZMQ seems to provide some, but it looks unpleasant to use compared to the > rest > > of ZMQ. This is low priority, since nobody else bothers with it. > > > > Will I ever actually accomplish this? Probably not real quickly (too many > day jobs), but I think it is an exciting potential use for J. > > > Oddly, this thread does not show up in Nabble, where I usually read the > J-lists. > > > > -Scott > > > > > > http://www.jsoftware.com/jwiki/PascalJasmin/OOP%20scheduler%20and%20ZeroMQ > > > > > > Thanks to Scott for getting this started. Its a work in progress, but > its > > > probably more helpful to see the simplest core version first, than just > the > > > bloated version. > > > > > > > > > The scheduler is a framework for multitasking several polling (endless) > > > loops within a > > single J instance. The simplest multithreading > > > synchronization library is avoiding multithreading altogether, and an in > > > process scheduler allows what are semantically seperate processes to work > > > together without concerning yourself about the possibilities of one > process > > > writting to a variable that is being read or written to by another > process. > > > > > > It can integrate with other "real" multiprocessing setups by grouping > > > together tasks that need tight cooperation. The canonical usefulness is > for > > > socket programing, which typically involve polling loops for each client > > > and server that add testing tedium even with just a single client and > > > server. The scheduler eases development and testing of several clients > and > > > servers all in a single application, and simplifies testing/learning of > > > frameworks like ZeroMQ and its J implementation. > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
