Re: [MacRuby-devel] Fibers and Enumerators
Because of the way MacRuby is implemented, fibers would need to unwind the stack most of the time you switch continuations, which has poor runtime performance. We will eventually implement continuations which can serve to implement fibers at some point (which is what MRI does I believe), but I wouldn't recommend to use them for anything significant because of the performance impact.My concern for the feature is focused not so much on the performance impact of using it, as it focuses on the fact that Fibers have been added to the 1.9.x language. They appear to be an important part of the External Enumerators feature and that feature seems to have a lot of people very excited.Any external libraries that make use of Fibers or External Enumerators stand a good chance of not working in MacRuby. In that case MacRuby becomes "Mac-MostOfRuby" which makes it harder to adopt... etc. I'd rather that didn't happen.I personally believe the GCD model is much more appealing to achieve multitasking, especially now that we enter the multicore age.With all due respect to Shakespeare, however, there are more types of multitasking than are dreamt of in GCD's philosophy.Having said that, though, I really love GCD. I'd really love to use GCD. I really, really, really ENVY folks that can rely on GCD and (C-language) blocks to help them solve their problems.As for myself, I've never had the luxury of working on a system where I can count on GCD being available. The last time I did desktop development we were supporting systems back to Leopard. About the time GCD was announced, I switched to the iOS realm and to this day my code has to support operating systems back to iPhone OS 3.1.3.I look forward to the day I can throw some blocks around in Objective-C, but (sadly) that day is not today. :-(My true concern is the marginalization of MacRuby. Ruby developers have to be concerned not only with portability to old operating systems, but portability across operating systems. That MacRuby can rely on GCD is really great for MacRuby but does nothing whatsoever for Ruby at large.I've worked my way up through the days of System 6 through to Mac OS X, I've seen the Macintosh platform marginalized entirely too often because developers in the wider world have to cater to the Least Common Denominator of technology... invariably it was an LCD that the Mac didn't implement because the platform had a better solution. I'd hate to see that happen to MacRuby as well.Scott___ MacRuby-devel mailing list [email protected] http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Fibers and Enumerators
On Aug 12, 2010, at 03:33 PM, [email protected] wrote:While I am certain everyone would agree that 100% compatibility should be the goal, I am not convinced how "critical" this addition is: in particular, how difficult is it to do a quick code change? Unlike basic data types, there are quite a few reasons I don't see Fibers being a large percentage of any code. I hope that folks involved with this discussion don't misinterpret the reason I originally posted. I'm not agitating, or pushing, or cajoling, or even suggesting that we _NEED_ Fibers and must have them ASAP.Instead I was curious to know if there was "a plan" for implementing Fibers and if so, I was curious to know what operating system technology they would be built on. It appears that the problem has not been looked at, in-depth, yet and I am satisfied with that. Nor does it appear that there is any OS level technology that would particularly support the creation of Fibers (i.e. Mac OS X doesn't really have any built-in support for cooperatively scheduled multitasking... outside of deprecated technologies like the Carbon Thread Manager and makecontext/swapcontext).It seems that Mac OS X has made a fairly clean break from the cooperative multitasking roots of the "classic" Mac OS. In some ways that's a really Good Thing(tm) but I'm not convinced that it is true in all cases. I'm working off the assumption (with all that implies) that Languages like Python, and now Ruby have technologies like Fibers for a reason.Does anyone have any truly compelling examples where Fibers are truly the only/right way to go, vs. an alternate design? I don't see them scaling well (see ruby-doc.org) - meaning, their advantage is restricted in scope.I can't answer your question other than to suggest that the language feature must have been added for a reason. (and, Yes, that is a cheap cop-out of a response).I can tell you the particular context that kicked off this whole line of inquiry for me. I will do so here, but be forewarned that it will be a long explanation and my pursuit is purely academic. :-) That is to say, it's something I've been studying mostly for the sake of learning something new. If that kind of thing doesn't interest you... if you're more of a practical applications kind of person, then you might want to skip the explanation.What got me started on the topic was a data structure from a beginning computer science class called "Streams" (not I/O streams... but something different with a confusingly homogenous name). In Clojure they are represented using a structure called a "lazy sequence".Basically the idea is that you can express a computation as a series of streams and processing elements. We see this all the time in Ruby where the stream is the sequence of objects yielded up by and enumerator, one processing element is a block, and another processing element is a method. For example:[1, 2, 3, 4, 5].collect { |x| x* x }Looking at this from the Streams perspective, we are taking a Stream of numbers (the sequence of the elements in the array), passing each element through a processor (represented by the block) that computes the square of the elements fed into it (creating a stream of the squares), and then collecting all the elements of that stream into an array.This technique gets interesting when you start adding the concept of infinitely long streams. Consider a problem where you want to compute the first 10 prime numbers that are also palindrome number (i.e. which are "spelled" the same backward and forward). If you had a stream of the whole numbers, a filter for picking out just the primes, and a filter for picking out just the palindrome numbers you could express the whole solution as:whole_numbers.filtered_by(:prime).filtered_by(:palindrome).take(10)whole_numbers is an "infinite stream" which we filter out only primes (creating an infinite stream of prime numbers), then filter out only palindrome numbers (an infinite stream of primes that are also palindrome numbers), of which we only take the first 10 elements.There's a lot of implementation detail left out (and my syntax here is only designed to illustrate the concepts, not to suggest an implementation) but at the heart of this kind of system is the need to represent an infinite stream of the whole numbers. I've seen this called a "Generator". In Ruby 1.9 we can create a Generator for the whole numbers as:whole_numbers = Enumerator.new do |yielder| number = 0; loop do yielder.yield number number = number + 1 endendIn other words, the sequence of whole numbers HAS to be a process, not a data structure, because it is infinite (and infinitely large data structures have a hard time squeezing into finite memory). Because it is a process, though, we also need some means of interrupting the process to retrieve the elements from it. (and in practice we don't want the process itself to run without pause because it would consume an infinite amount of processor time).This lead me to
Re: [MacRuby-devel] Fibers and Enumerators
I said:>> (i.e. Mac OS X doesn't really have any built-in support for cooperatively scheduled multitasking... outside of >> deprecated technologies like the Carbon Thread Manager and makecontext/swapcontext). Dr. Prabhakar said:Actually, I'm confused by that statement. My understanding is that cooperatively scheduled threads is a just a subset of the system-scheduled model. In particular, I'm pretty sure you can emulate them on top of GCD using semaphores. Should I explain further, or did you already consider that and discover it was unworkable? You're right it is a very poorly worded statement and a convoluted mess conceptually. I do understand that you can emulate cooperative multitasking using threads and semaphores. I didn't really reject it as unworkable. I rejected it because it wasn't the solution I was looking for. :-)I am working under the impression that pthreads are a "heavy-weight" construct (an impression bolstered by watching too many GCD sessions, I imagine). By that I mean each pthread consumes kernel resources, scheduler attention, etc. In contrast a Fiber is a comparitively light-weight construct. Each fiber has a distinct execution context, but they don't require the direct attention of the system scheduler. The run inside the context of the kernel level thread.One aspect of my confusing statement is meant to convey that I could not find this intra-thead multitasking concept already implemented in Mac OS X, nor could I find primitives like makecontext and swapcontext upon which it could be built easily.The other idea I meant to convey is that I also cannot find an API whose syntax directly supports the concept of cooperatively scheduled threads of execution. The Carbon Thread Manager presents such an API. The API itself reflects the model of cooperative threads of execution, yielding between them, etc.As you point out, you can emulate the cooperative model using the various threading APIs using threads and synchronization primitives. Doing so allows you to implement the semantic of that model. My point, however, is that if you want a nice syntax that expresses the cooperative model you're going to have to do the plumbing yourself, and add your own façade. The OS no longer provides one.Scott___ MacRuby-devel mailing list [email protected] http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Fibers and Enumerators
On Aug 13, 2010, at 03:34 PM, "Jordan K. Hubbard" wrote:Remember, nobody is defending the POSIX pthread APIs here, they're saying that GCD makes Fibers less relevant...GCD makes Fibers less relevant so long as you are willing to lock your code to the MacRuby platform. ;-) But I think we've all agreed on that point as well.It's also been said many times already, but I'll say it again: No one is saying Fiber support would be undesirable, if for Ruby 1.9 compatibility reasons alone. What they're saying is that someone needs to VOLUNTEER to do the work involved or the discussion is and will remain purely academic. Unfortunately you are going to have to have someone who has the requisite experience. Since the OS doesn't provide the primitives I thought it might, that person is going to have to understand the Intel 64-bit ABI, (evidently) DWARF exception handling, Mac OS X's implementation of pthreads, and that's before you can start figuring out how to plug it all into the MacRuby interpreter.As an "application framework level guy" I know that I certainly don't have the requisite skills. I might be able to build them up, but I imagine would probably take longer than anyone interested in MacRuby cares to wait.Scott___ MacRuby-devel mailing list [email protected] http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Fibers and Enumerators
On Aug 13, 2010, at 03:46 PM, easco wrote:Unfortunately you are going to have to have someone who has the requisite experience. Since the OS doesn't provide the primitives I thought it might, that person is going to have to understand the Intel 64-bit ABI, (evidently) DWARF exception handling, Mac OS X's implementation of pthreads, and that's before you can start figuring out how to plug it all into the MacRuby interpreter.As an "application framework level guy" I know that I certainly don't have the requisite skills. I might be able to build them up, but I imagine would probably take longer than anyone interested in MacRuby cares to wait.After thinking about it a bit... I guess you wouldn't really care about the Intel ABI so much as some aspect of the LLVM?Does LLVM already have some mechanism for supporting multiple "stacks" and switching between them? (rhetorical question... I imagine I can Google the answer myself).Scott___ MacRuby-devel mailing list [email protected] http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
