Re: [9fans] "Blocks" in C

[erik quanstrom]

On Fri Sep 18 11:52:23 EDT 2009, driv...@0xabadba.be wrote:
>  Is there some method of lock profiling on plan9?  For example when I do 
> work on freebsd and say remove a giant lock from the keyboard subsystem; I 
> run the lock profiler before and after the change to see how long the system 
> was sitting at the in kernel locks.  
>  I am doing my thesis (undergrad) on these kind of topics and am 
> interested in seeing how I may be of use.

use kprof.  or add your own instrumentation.

if you have very specific ideas about what needs
to be tracked, it's quite easy to implement.  that
would be an interesting project.  port/taslock.c
already tracks some lock statistics.  esp. lockstats.inglare
port/qlock has qlocks and rwlocks.

the tracking could use atomic increments.  as
it stands, i don't believe it is accurate because two
locks can be inglare at the same time.  thus inglar
would appear to be less than it should be.

- erik

Re: [9fans] "Blocks" in C

[drivers]

 Is there some method of lock profiling on plan9?  For example when I do 
work on freebsd and say remove a giant lock from the keyboard subsystem; I run 
the lock profiler before and after the change to see how long the system was 
sitting at the in kernel locks.  
 I am doing my thesis (undergrad) on these kind of topics and am interested 
in seeing how I may be of use.

=jt
--Original Message--
From: erik quanstrom
Sender: 9fans-boun...@9fans.net
To: 9fans@9fans.net
ReplyTo: Fans of the OS Plan 9 from Bell Labs
Subject: Re: [9fans] "Blocks" in C
Sent: Sep 18, 2009 07:41

> it isn't really a `big kernel lock' in the linux sense.

you're right, technically it is a very different problem.

the effects seem similar.  the lock is just in the block allocator
rather than the syscall interface.  if you're doing a lot of i/o
in a standard kernel, there's a lot of block allocation.

> with error checking. an alternative allocator that uses quickfit on top of
> a variant of the simple allocator in kernighan and ritchie is
> roughly 30 times faster on some malloc-thrashing tests (i'm just remembering 
> that
> from tests i did last year, so i might have the number wrong, but it's 
> substantial).

i see the same thing.  with the allocator in a specific packet-pushing
test, i see 185000 packets/s, with a block pool i see ~45 packets/s.
all the packets are the same size.

would you be willing to share your allocator?  this may help a number
of cases i'm currently fighting.  10gbe and 2×10gbe + quad processors
have a way of putting a sharp point on an otherwise dull problem.

i also wonder if an ilock is really the right protection
mechanism on machines with 8 or 16 logical processors
for malloc.

the reason i mention this is that it turns out that the lock
on block pools can be hot enough that splitting the lock
per controller can have a big positive effect.  i need to set
up a faster target, but i've seen significant jumps in bandwidth
for 2 82563 controllers when using 2 pools instead of one
even on a fairly bored machine.

sorry for the lack of real numbers.

- erik



=jt

Re: [9fans] "Blocks" in C

[erik quanstrom]

> it isn't really a `big kernel lock' in the linux sense.

you're right, technically it is a very different problem.

the effects seem similar.  the lock is just in the block allocator
rather than the syscall interface.  if you're doing a lot of i/o
in a standard kernel, there's a lot of block allocation.

> with error checking. an alternative allocator that uses quickfit on top of
> a variant of the simple allocator in kernighan and ritchie is
> roughly 30 times faster on some malloc-thrashing tests (i'm just remembering 
> that
> from tests i did last year, so i might have the number wrong, but it's 
> substantial).

i see the same thing.  with the allocator in a specific packet-pushing
test, i see 185000 packets/s, with a block pool i see ~45 packets/s.
all the packets are the same size.

would you be willing to share your allocator?  this may help a number
of cases i'm currently fighting.  10gbe and 2×10gbe + quad processors
have a way of putting a sharp point on an otherwise dull problem.

i also wonder if an ilock is really the right protection
mechanism on machines with 8 or 16 logical processors
for malloc.

the reason i mention this is that it turns out that the lock
on block pools can be hot enough that splitting the lock
per controller can have a big positive effect.  i need to set
up a faster target, but i've seen significant jumps in bandwidth
for 2 82563 controllers when using 2 pools instead of one
even on a fairly bored machine.

sorry for the lack of real numbers.

- erik

Re: [9fans] "Blocks" in C

[Charles Forsyth]

> i mentioned that the pool library can act as a big
> kernel lock a few weeks ago.  i don't know if anyone
> has thoughts on how to deal with this.

it isn't really a `big kernel lock' in the linux sense.
the big kernel lock was the device by which operating systems
written with only a uniprocessor in mind were made to `work' on a 
multiprocessor.
typically, the equivalent of qlock and qunlock of a
single global lock were placed at strategic points to
bracket (say) every system call that affected non-trivial state.
then the developers started splitting the lock region.
it's easier than thinking. processes would queue for use of
the kernel (but they'd go to sleep, not spin).

for the pool library it's a spin lock, which is intended to
protect small blocks of code. really the effects of collision
should not be as bad as they are.
although it's probably good especially with many cores to have some 
processor-local allocation
of cpu and memory resources without interlock, or make the main storage pool 
locks more refined,
or using wait-free cleverness,
and indeed there are schemes to do all those things, the big initial problem 
with
the pool library is that it's remarkably slow. in fact,
that's been my experience with every tree-based allocator
i've had to use, so it's not just that pool is replete
with error checking. an alternative allocator that uses quickfit on top of
a variant of the simple allocator in kernighan and ritchie is
roughly 30 times faster on some malloc-thrashing tests (i'm just remembering 
that
from tests i did last year, so i might have the number wrong, but it's 
substantial).
for a reasonable choice of block quantum it also has better fragmentation 
properties,
which is amusing since that's supposed to be one of the advantages of the 
tree-based allocators.

Re: [9fans] "Blocks" in C

[Daniel Lyons]

It would be easy to say that list should be divided, in practice
though, I'm not sure the folks who I would like to have a privilege
of addressing would voluntarily subscribe to the #3 type of list.



Being a curious sometime user I guess I fall in category 3. Which  
seems natural since the list is called "9fans" not "plan9-dev." Not  
sure our community is large enough to survive being partitioned off  
into little tiny segments.


—
Daniel Lyons

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-17 at 17:02 -0400, erik quanstrom wrote:
> > ... In case anyone is wondering what they could be doing instead of feeding
> > this massive thread more fatty foods.
> 
> there's lots of complaining on the list about the
> content of the list.
> 
> it's not like there aren't good meaty issues to discuss.
> what happened with either of the recently-reported
> fossil lockup problems, for instance?
> 
> i mentioned that the pool library can act as a big
> kernel lock a few weeks ago.  i don't know if anyone
> has thoughts on how to deal with this.
> 
> it seems to me we deserve the list we create.

I think you're right. Perhaps different folks are approaching
9fans with different assumptions, though. I see 3 camps present:
   1. The actual *developers* of the technology related to
   Plan 9. That would include you, Russ, Inferno folks and
   perhaps Nemo & co.

   2. The actual users of Plan9.

   3. The folks who do not use Plan9 in day-to-day life, but
   are fundamentally convinced in a WWP9D (what would Plan 9 do)
   principle.

And, of course, there are also trolls.

Trolls, aside, it looks like #1 and #2 might be slightly misaligned
with #3. At least my personal experience of getting frustrated on
9fans comes mostly from the cases where I didn't positioned my
question as the WWP9D one.

It would be easy to say that list should be divided, in practice
though, I'm not sure the folks who I would like to have a privilege
of addressing would voluntarily subscribe to the #3 type of list.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[Steve Simon]

> what happened with either of the recently-reported
> fossil lockup problems, for instance?

As I now have two servers at home (old and new) I have been trying
to provoke the old one into locking up so I can take a snap of its fossil.

sadly the old server has been irriatingly reliable and the only lockup
I have had so far was on the new machine (typical!).

If anyone has any thoughts/hints/guesses on how to provoke the fossil
lockup I would be very interested to hear. I have tried mirroring
sources which used to be a rich source of lockups but it seems to work now.

I am going to try sending this machine fake emails I have a feeling that
that can annoy fossil, and also speed up the rate of taking temporary
snaps which others have reported as being a source of problems.

-Steve

Re: [9fans] "Blocks" in C

[Uriel]

http://ninetimes.cat-v.org/news/2009/09/07/0-mplayer9/

On Thu, Sep 17, 2009 at 10:31 PM, Anant Narayanan  wrote:
> On Sep 17, 2009, at 10:26 PM, erik quanstrom wrote:
>>>
>>> Good luck trying to get Plan 9 to play video!
>>>
>>
>> minooka; lc /sys/src/9/pc/*tv*.c
>> devtv.c         vgatvp3020.c    vgatvp3026.c
>
> Sure, if you have a TV tuner. What I was referring to was Plan 9's ability
> (or lack thereof) to decode and play digital video codecs. Just one of those
> things that prevent someone from running only Plan 9 on their computers --
> you need one of the big 3 for web browser + video.
>
> --
> Anant
>
>
>

Re: [9fans] "Blocks" in C

[erik quanstrom]

> ... In case anyone is wondering what they could be doing instead of feeding
> this massive thread more fatty foods.

there's lots of complaining on the list about the
content of the list.

it's not like there aren't good meaty issues to discuss.
what happened with either of the recently-reported
fossil lockup problems, for instance?

i mentioned that the pool library can act as a big
kernel lock a few weeks ago.  i don't know if anyone
has thoughts on how to deal with this.

it seems to me we deserve the list we create.

- erik

Re: [9fans] "Blocks" in C

[Akshat Kumar]

> What I was referring to was Plan 9's ability
> (or lack thereof) to decode and play digital video codecs. Just one of those
> things that prevent someone from running only Plan 9 on their computers --
> you need one of the big 3 for web browser + video.

With Cinap Lenrek's work on linuxemu, one can use mplayer (with ffmpeg)
to play videos, and flash in browser works, so one can also watch videos
on Youtube.

Federico G. Benavento released a tool to play mpeg videos a while back.
It's not perfect and could use work.


... In case anyone is wondering what they could be doing instead of feeding
this massive thread more fatty foods.
ak

Re: [9fans] "Blocks" in C

[Anant Narayanan]

On Sep 17, 2009, at 10:26 PM, erik quanstrom wrote:

Good luck trying to get Plan 9 to play video!



minooka; lc /sys/src/9/pc/*tv*.c
devtv.c vgatvp3020.cvgatvp3026.c


Sure, if you have a TV tuner. What I was referring to was Plan 9's  
ability (or lack thereof) to decode and play digital video codecs.  
Just one of those things that prevent someone from running only Plan 9  
on their computers -- you need one of the big 3 for web browser + video.


--
Anant

Re: [9fans] "Blocks" in C

[erik quanstrom]

> Good luck trying to get Plan 9 to play video!
> 

minooka; lc /sys/src/9/pc/*tv*.c
devtv.c vgatvp3020.cvgatvp3026.c

- erik

Re: [9fans] "Blocks" in C

[Anant Narayanan]

Not meaning to add fuel to the fire, but;

On Sep 17, 2009, at 7:38 PM, Jack Norton wrote:
I hate iTunes with a passion. It is a huge monolithic godlike  
creature that tries to do everything for me (usually when I don't  
want it to).  It brings my 12" powerbook to a screeching halt (I get  
beach balled to death), and it doesn't natively support many audio/ 
video codecs/containers (and isn't that easily extended, which  
brings up quicktime   ).


Somebody wrapped ffmpeg in Quicktime, which lets it play almost any  
imaginable media format: http://perian.org/


Quicktime X isn't too bad compared to the old QT, atleast they're  
making progress. You can't say the same of iTunes though, I think it's  
been going downhill since version 7. However, I do hope that when they  
decide to rewrite it in 64-bit Cocoa ("iTunes X"), they'll make it  
half as insane as compared to now. It's bound to happen.


 Just about the simplest way to play audio on a computer, save for  
the methods in plan9 :)  (I'm _trying_ to get us back on topic...)


Good luck trying to get Plan 9 to play video!

--
Anant

Re: [9fans] "Blocks" in C

[Jack Norton]

David Leimbach wrote:



On Thu, Sep 17, 2009 at 1:43 AM, Charles Forsyth 
mailto:fors...@terzarima.net>> wrote:


we'd have been much better off if Apple had instead spent the
time and effort writing a decent iTunes, or opening their platform
interfaces enough that someone else could do it (and on Linux, not
just Mac or Windows).


What's your gripe on iTunes?  I've had a few issues with it, but it 
does seem to get the job done somehow.  Honestly just curious.

I'd like to jump in on this.
I hate iTunes with a passion. It is a huge monolithic godlike creature 
that tries to do everything for me (usually when I don't want it to).  
It brings my 12" powerbook to a screeching halt (I get beach balled to 
death), and it doesn't natively support many audio/video 
codecs/containers (and isn't that easily extended, which brings up 
quicktime   ). 
Then again, I grew up using winamp, and I absolutely love the old style 
winamp.   No bloody database, no crazy multi-tiered file browser, and no 
video player.  Just select a song(s) and play. 
Itunes is not a media player, it is a platform in and of itself.  It is 
the emacs of media players (in that it is all encompassing, there is a 
church/cult, etc...). 
Just about the simplest way to play audio on a computer, save for the 
methods in plan9 :)  (I'm _trying_ to get us back on topic...)
Ok I'm done. 


-Jack

Re: [9fans] "Blocks" in C

[Daniel Lyons]

On Sep 17, 2009, at 3:19 AM, Andrew Simmons wrote:

And no doubt we'd have been much better off if Apple had instead  
spent the

time and effort making a decent iPod.



Um... what is it you dislike about the iPod?

—
Daniel Lyons

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 17, 2009 at 1:43 AM, Charles Forsyth wrote:

> we'd have been much better off if Apple had instead spent the
> time and effort writing a decent iTunes, or opening their platform
> interfaces enough that someone else could do it (and on Linux, not just Mac
> or Windows).
>
>
What's your gripe on iTunes?  I've had a few issues with it, but it does
seem to get the job done somehow.  Honestly just curious.

Re: [9fans] "Blocks" in C

[LiteStar numnums]

Like shuffle db (i.e. no iTunes).

On Thu, Sep 17, 2009 at 5:19 AM, Andrew Simmons  wrote:

> >> we'd have been much better off if Apple had instead spent the
> >> time and effort writing a decent iTunes
>
> And no doubt we'd have been much better off if Apple had instead spent the
> time and effort making a decent iPod.
>
> The iTunes on my computer strikes me as at worst perfectly decent, in
> general outstandingly good. What do you feel a decent iTunes should
> look like?
>
>


-- 
And in the "Only Prolog programmers will find this funny" department:

Q: How many Prolog programmers does it take to change a lightbulb?

A: No.
 -- Ovid

   "By cosmic rule, as day yields night, so winter summer, war peace, plenty
famine. All things change. Air penetrates the lump of myrrh, until the
joining bodies die and rise again in smoke called incense."

   "Men do not know how that which is drawn in different directions
harmonises with itself. The harmonious structure of the world depends upon
opposite tension like that of the bow and the lyre."

   "This universe, which is the same for all, has not been made by any god
or man, but it always has been, is, and will be an ever-living fire,
kindling itself by regular measures and going out by regular measures"
-- Heraclitus

Re: [9fans] "Blocks" in C

[Daniel Lyons]

On Sep 17, 2009, at 2:43 AM, Charles Forsyth wrote:


opening their platform interfaces


Any in particular?

—
Daniel Lyons

Re: [9fans] "Blocks" in C

[Charles Forsyth]

we'd have been much better off if Apple had instead spent the
time and effort writing a decent iTunes, or opening their platform
interfaces enough that someone else could do it (and on Linux, not just Mac or 
Windows).

Re: [9fans] "Blocks" in C

[Lawrence E. Bakst]

For those that care:

http://www.friday.com/bbum/2009/08/29/basic-blocks/

http://www.friday.com/bbum/2009/08/29/blocks-tips-tricks/#more-1505


-- 
l...@iridescent.org

Re: [9fans] "Blocks" in C

[Lawrence E. Bakst]

Apple has an "interesting" process for releasing open source code. One of the 
guys that works on it wrote something up on it once, but I am sorry I don't 
have a pointer right now.

I would not assume that the it's the same "bits" that are used inside Apple. 
Certainly at the very least, many comments are removed from the source code. 
Nor would I assume that building the source code will give you the same binary. 
It might be interesting to see if you can recreate the same binary that Snow 
Leopard uses.

You might get the same binaries for something as simple as "cat", but I know 
for sure that this was not the case for some of the disk utilities, the last 
time I checked. For at least a few "sensitive" code bases I suspect that Apple 
probably maintains two versions, one for internal use and the other for open 
source release.

I am not making any value judgements here, just trying to state what the facts 
as I know them.

Having said all that, I suspect that Blocks, Clang, LLVM, and Autozone (Apple's 
GC) are all very close to what Apple uses in Snow Leopard and internally.

As a contrived example of the kind of code that could removed, suppose Apple's 
GC used a private, internal only API, to mark certain pages in the address 
space for paging behavior. That is the kind of code that would be removed 
before the open source was released, in order to prevent revealing the private 
API. Again this is a contrived example.

Best,

leb

At 2:28 PM -0700 9/11/09, David Leimbach wrote:
>On Fri, Sep 11, 2009 at 1:36 PM, Roman V Shaposhnik 
><r...@sun.com> wrote:
>
>On Fri, 2009-09-11 at 15:15 -0300, Iruata Souza wrote:
>> On Tue, Sep 8, 2009 at 1:40 PM, Bakul Shah 
>> <bakul+pl...@bitblocks.com> wrote:
>
> > > int x;
>> >
>> > void trash_x() { x = -42; }
>> >
>> > ... ^{ trash_x(); } ...
>> >
>> > My view: if you can't solve a problem cleanly and in a
>> > general way with a feature, it does not belong in a language
>> > (but may belong in a library).
>> >
>>
>> for those who still care
>> http://libdispatch.macosforge.org/
>
>I still do care very much (and in fact, I've been meaning
>to provide some of the answers on this mailing list, but
>apparently one can't upgrade to Snow Leopard over the
>net so I have to physically drive to the Mac store :-().
>
>Anyway, for a non Mac person, can somebody please clarify
>whether macosforge.org has the very same bits that go 
>into
>the Mac OS itself, or do I still have to get the real thing?
>
>
>This is the exported open source stuff from Apple.  So, yes it's the same.
>
>Of course they could have "forks" inside apple you don't get to see, but it's 
>the same source base.
>
>Dave
> 
>
>Thanks,
>Roman.


-- 
l...@iridescent.org

Re: [9fans] "Blocks" in C

[David Leimbach]

On Sat, Sep 12, 2009 at 4:08 AM, Anant Narayanan  wrote:

> On Sep 11, 2009, at 10:36 PM, Roman V Shaposhnik wrote:
>
>> I still do care very much (and in fact, I've been meaning
>> to provide some of the answers on this mailing list, but
>> apparently one can't upgrade to Snow Leopard over the
>> net so I have to physically drive to the Mac store :-().
>>
>> Anyway, for a non Mac person, can somebody please clarify
>> whether macosforge.org has the very same bits that go into
>> the Mac OS itself, or do I still have to get the real thing?
>>
>
> The source as put on macosforge has been "generalized" (apparently to make
> it easy to port to other platforms), but AFAIK the code that actually runs
> in Snow Leopard has a few "performance optimizations" that takes advantage
> of a similar low-level API that is supported by the new xnu kernel.
>


My understanding is it's the same source on macosforge, it can just be built
with or without the kernel hints.  (They're part of the kqueues stuff, and
that's being ported to FreeBSD as well).




>
> The xnu kernel itself is also open-source,
> http://www.opensource.apple.com/source/xnu/xnu-1456.1.26/ is the version
> with GCD primitives.
>
> --
> Anant
>
>
>

Re: [9fans] "Blocks" in C

[Anant Narayanan]

On Sep 11, 2009, at 10:36 PM, Roman V Shaposhnik wrote:

I still do care very much (and in fact, I've been meaning
to provide some of the answers on this mailing list, but
apparently one can't upgrade to Snow Leopard over the
net so I have to physically drive to the Mac store :-().

Anyway, for a non Mac person, can somebody please clarify
whether macosforge.org has the very same bits that go into
the Mac OS itself, or do I still have to get the real thing?


The source as put on macosforge has been "generalized" (apparently to  
make it easy to port to other platforms), but AFAIK the code that  
actually runs in Snow Leopard has a few "performance optimizations"  
that takes advantage of a similar low-level API that is supported by  
the new xnu kernel.


The xnu kernel itself is also open-source, http://www.opensource.apple.com/source/xnu/xnu-1456.1.26/ 
 is the version with GCD primitives.


--
Anant

Re: [9fans] "Blocks" in C

[David Leimbach]

On Fri, Sep 11, 2009 at 1:36 PM, Roman V Shaposhnik  wrote:

> On Fri, 2009-09-11 at 15:15 -0300, Iruata Souza wrote:
> > On Tue, Sep 8, 2009 at 1:40 PM, Bakul Shah 
> > >
> wrote:
> > > int x;
> > >
> > > void trash_x() { x = -42; }
> > >
> > > ... ^{ trash_x(); } ...
> > >
> > > My view: if you can't solve a problem cleanly and in a
> > > general way with a feature, it does not belong in a language
> > > (but may belong in a library).
> > >
> >
> > for those who still care
> > http://libdispatch.macosforge.org/
>
> I still do care very much (and in fact, I've been meaning
> to provide some of the answers on this mailing list, but
> apparently one can't upgrade to Snow Leopard over the
> net so I have to physically drive to the Mac store :-().
>
> Anyway, for a non Mac person, can somebody please clarify
> whether macosforge.org has the very same bits that go into
> the Mac OS itself, or do I still have to get the real thing?
>
>
This is the exported open source stuff from Apple.  So, yes it's the same.

Of course they could have "forks" inside apple you don't get to see, but
it's the same source base.

Dave


> Thanks,
> Roman.
>
>
>

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Fri, 2009-09-11 at 15:15 -0300, Iruata Souza wrote:
> On Tue, Sep 8, 2009 at 1:40 PM, Bakul Shah  wrote:
> > int x;
> >
> > void trash_x() { x = -42; }
> >
> > ... ^{ trash_x(); } ...
> >
> > My view: if you can't solve a problem cleanly and in a
> > general way with a feature, it does not belong in a language
> > (but may belong in a library).
> >
> 
> for those who still care
> http://libdispatch.macosforge.org/

I still do care very much (and in fact, I've been meaning
to provide some of the answers on this mailing list, but
apparently one can't upgrade to Snow Leopard over the
net so I have to physically drive to the Mac store :-().

Anyway, for a non Mac person, can somebody please clarify
whether macosforge.org has the very same bits that go into
the Mac OS itself, or do I still have to get the real thing?

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[David Leimbach]

On Fri, Sep 11, 2009 at 11:15 AM, Iruata Souza  wrote:

> On Tue, Sep 8, 2009 at 1:40 PM, Bakul Shah 
> >
> wrote:
> > On Tue, 08 Sep 2009 08:31:28 PDT David Leimbach 
>  wrote:
> >>
> >> Having wrestled with this stuff a little bit, and written "something".
>  I
> >> can immediately see how one can get away from needing to "select" in
> code so
> >> much, and fire off blocks to handle client server interactions etc.
>  It's
> >> kind of neat.
> >
> > alt(3) is a nicer way to avoid select().
> >
> > I still say CSP is the way to go. In plan9/limbo channels
> > work across coroutines in one process. Seems to me extending
> > channels to work across preemptive threads (running on
> > multiple cores) or across processes or machines is might lead
> > to a more elegant and no less performant model.  It seems
> > to be a more natural model when you have zillions of
> > processors on a chip (like TileraPro64, with zillion = 64).
> > They can't all go to shared external memory without paying a
> > substantial cost but neighbor to neighbor communication is
> > far faster (tilera claims 37Tbps onchip interconnect b/w and
> > 50Gbps of I/O bw).
> >
> > It is nice that a Apple C block treats all non local
> > variables (except __block ones) as read only variables.  But
> > every time I look at blocks I see new problems. What if a
> > block calls a function that modifies a global like in the
> > example below? If this works, what is the point of treating
> > globals as readonly? If this doesn't work, how do ensure
> > trash_x() causes a seg fault, particularly when it is defined
> > in another file?
> >
> > int x;
> >
> > void trash_x() { x = -42; }
> >
> > ... ^{ trash_x(); } ...
> >
> > My view: if you can't solve a problem cleanly and in a
> > general way with a feature, it does not belong in a language
> > (but may belong in a library).
> >
>
> for those who still care
> http://libdispatch.macosforge.org/
>
>
And the FreeBSD port is underway too.  It's going to have the kernel
scheduling hints too it seems.

Dave

Re: [9fans] "Blocks" in C

[Iruata Souza]

On Tue, Sep 8, 2009 at 1:40 PM, Bakul Shah  wrote:
> On Tue, 08 Sep 2009 08:31:28 PDT David Leimbach   wrote:
>>
>> Having wrestled with this stuff a little bit, and written "something".  I
>> can immediately see how one can get away from needing to "select" in code so
>> much, and fire off blocks to handle client server interactions etc.  It's
>> kind of neat.
>
> alt(3) is a nicer way to avoid select().
>
> I still say CSP is the way to go. In plan9/limbo channels
> work across coroutines in one process. Seems to me extending
> channels to work across preemptive threads (running on
> multiple cores) or across processes or machines is might lead
> to a more elegant and no less performant model.  It seems
> to be a more natural model when you have zillions of
> processors on a chip (like TileraPro64, with zillion = 64).
> They can't all go to shared external memory without paying a
> substantial cost but neighbor to neighbor communication is
> far faster (tilera claims 37Tbps onchip interconnect b/w and
> 50Gbps of I/O bw).
>
> It is nice that a Apple C block treats all non local
> variables (except __block ones) as read only variables.  But
> every time I look at blocks I see new problems. What if a
> block calls a function that modifies a global like in the
> example below? If this works, what is the point of treating
> globals as readonly? If this doesn't work, how do ensure
> trash_x() causes a seg fault, particularly when it is defined
> in another file?
>
> int x;
>
> void trash_x() { x = -42; }
>
> ... ^{ trash_x(); } ...
>
> My view: if you can't solve a problem cleanly and in a
> general way with a feature, it does not belong in a language
> (but may belong in a library).
>

for those who still care
http://libdispatch.macosforge.org/

Re: [9fans] "Blocks" in C

[Bakul Shah]

On Tue, 08 Sep 2009 08:31:28 PDT David Leimbach   wrote:
> 
> Having wrestled with this stuff a little bit, and written "something".  I
> can immediately see how one can get away from needing to "select" in code so
> much, and fire off blocks to handle client server interactions etc.  It's
> kind of neat.

alt(3) is a nicer way to avoid select().

I still say CSP is the way to go. In plan9/limbo channels
work across coroutines in one process. Seems to me extending
channels to work across preemptive threads (running on
multiple cores) or across processes or machines is might lead
to a more elegant and no less performant model.  It seems
to be a more natural model when you have zillions of
processors on a chip (like TileraPro64, with zillion = 64).
They can't all go to shared external memory without paying a
substantial cost but neighbor to neighbor communication is
far faster (tilera claims 37Tbps onchip interconnect b/w and
50Gbps of I/O bw).

It is nice that a Apple C block treats all non local
variables (except __block ones) as read only variables.  But
every time I look at blocks I see new problems. What if a
block calls a function that modifies a global like in the
example below? If this works, what is the point of treating
globals as readonly? If this doesn't work, how do ensure
trash_x() causes a seg fault, particularly when it is defined
in another file?

int x;

void trash_x() { x = -42; }

... ^{ trash_x(); } ...

My view: if you can't solve a problem cleanly and in a
general way with a feature, it does not belong in a language
(but may belong in a library).

Re: [9fans] "Blocks" in C

[Uriel]

On Tue, Sep 8, 2009 at 5:31 PM, David Leimbach wrote:
>
> [snip]
>
> I guess we'll see what happens.

We all know what will happen: more and more layers of crud will be added.

Just as Russ predicted:

From: r...@plan9.bell-labs.com (Russ Cox)
Subject: Re: [9fans] design clairvoyance & the 9 way
Date: Thu, 8 May 2003 04:05:31 GMT

> What does tomorrow's unix look like?

I'm confident that tomorrow's Unix will look like today's Unix, only cruftier.

Russ

Re: [9fans] "Blocks" in C

[David Leimbach]

On Mon, Sep 7, 2009 at 2:40 AM, Uriel  wrote:

> On Mon, Sep 7, 2009 at 11:05 AM, Greg Comeau wrote:
> > In article <25cf9336-c071-44a5-ab04-6bb042bc5...@kix.in>,
> > Anant Narayanan  wrote:
> >>I understand the argument that blocks don't "feel" C-like, but the
> >>argument that you can do everything with just using function pointers
> >>is BS.
> >
> > Even one step further, even if we all agree blocks are BS,
>
> As I was reading this thread I kept hearing inside my head boyd's
> voice screaming this at frequent intervals:
>
> "Blocks are bollocks!"
>
> uriel
>
>
>
In case anyone is interested, I've got my concurrent prime sieve working
just fine now (with help from Kevin Van Vechten at apple) using libdispatch,
simulating the "actor model" by hooking up blocks to pipe fds.

http://paste.lisp.org/display/86549#3

It turns out I don't need the semaphore and the counter either, because I
can dynamically add stuff to groups, and then wait on that stuff to leave
the group for synchronizing the program.  Again this advice came from my
buddy Kevin at apple.

Having wrestled with this stuff a little bit, and written "something".  I
can immediately see how one can get away from needing to "select" in code so
much, and fire off blocks to handle client server interactions etc.  It's
kind of neat.

My understanding is that Apple is going through the process to open up the
source to all of this stuff soon, and another friend of mine has already
done his own version of blocks that works for the iPhone and Leopard, and
another person is looking to make an API compatible with libdispatch readily
available.

I guess we'll see what happens.

Dave

Re: [9fans] "Blocks" in C

[Uriel]

On Mon, Sep 7, 2009 at 11:05 AM, Greg Comeau wrote:
> In article <25cf9336-c071-44a5-ab04-6bb042bc5...@kix.in>,
> Anant Narayanan  wrote:
>>I understand the argument that blocks don't "feel" C-like, but the
>>argument that you can do everything with just using function pointers
>>is BS.
>
> Even one step further, even if we all agree blocks are BS,

As I was reading this thread I kept hearing inside my head boyd's
voice screaming this at frequent intervals:

"Blocks are bollocks!"

uriel


> moving more generally, that something can already be done using
> feature X had BETTER be a consideration, but that consideration alone
> is often insufficient.
> --
> Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
> Comeau C/C++ ONLINE ==>     http://www.comeaucomputing.com/tryitout
> World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
> Comeau C/C++ with Dinkumware's Libraries... Have you tried it?
>
>

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article <8214db3a-3368-4b61-b0cd-bac5f2be7...@sun.com>,
Roman Shaposhnik  wrote:
>There's been a *lot* of speculation on this thread and very little fact.
>I'd encourage everybody to play with the feature before forming
>any kind of final judgement.

This is true, a good point, etc, however

>On Sep 3, 2009, at 8:52 PM, erik quanstrom wrote:
>>> Did you even read the article or any of the examples? There are  
>>> plenty
>>> of things that you can "do" with blocks that you can't with just
>>> function pointers. That's besides the fact that some of them are more
>>> elegantly expressed with blocks that look sort of ugly with function
>>> pointers.
>>
>> on the other hand, apple says this is illegal
>
>Where exactly does it say that?
>
>>dispatch_block_t p;
>>
>>  if(cond){
>>  p =^ { print("cond\n"); };
>>  }else{
>>  p =^ { print("cond\n"); };
>>  }
>>  p();
>>
>> since the first part is equivalent to
>>
>>  if(cond){
>>  struct Block _t = ...;
>>  p = &_t;
>>  }
>
>I'm pretty sure Apple's compiler is smart enough to allow the  
>construct that you've just mentioned. In fact, I'd be willing
>to bet on it.

 I'm fairly certain I was confused about this when I originally
read that document.  I think (I can't recall) the example being
in a different language than C so coughed it up to some different
scoping rules or something in that other language.

So, the clincher is, if it's the case, since it seems obvious that
the construct could work the way one would intuitively?? think it
would work, then, why doesn't it?
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article <25cf9336-c071-44a5-ab04-6bb042bc5...@kix.in>,
Anant Narayanan  wrote:
>I understand the argument that blocks don't "feel" C-like, but the  
>argument that you can do everything with just using function pointers  
>is BS.

Even one step further, even if we all agree blocks are BS,
moving more generally, that something can already be done using
feature X had BETTER be a consideration, but that consideration alone
is often insufficient.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[David Leimbach]

On Sun, Sep 6, 2009 at 4:03 PM, Uriel  wrote:

> On Fri, Sep 4, 2009 at 4:56 PM, David Leimbach wrote:
> >
> >
> > On Fri, Sep 4, 2009 at 7:20 AM, erik quanstrom 
> > wrote:
> >>
> >> > I could be wrong, but I feel like you're not really interested in
> >> > entertaining that this idea could be useful, but more interested in
> >> > shooting
> >> > it down [...]
> >>
> >> remember, if a guy says to the king, hey you're fly's undone,
> >> we send that guy to the stockades for a week.  meanwhile
> >> the king's fly remains undone.
> >>
> >> since the raison d'etre of blocks is ease of programming,
> >> i would think it would follow that it should be uniformly
> >> easier across the board.  if there are big exceptions to this
> >> (like extra locking), i would think the feature would earn
> >> a fail.
> >>
> >
> > I am totally agreeing with you so far on all points you've just made.
>  And I
> > think that's why Apple is seeking feedback.
>
> Here is some feedback for Apple: Fire your whole software and
> programming division, they are making the GNU and Gnome crack monkeys
> look sane, competent and responsible.
>
> uriel
>
>
> Why am I not surprised that this is your reaction?  At least you're
consistent :-)

Re: [9fans] "Blocks" in C

[David Leimbach]

On Sun, Sep 6, 2009 at 3:35 PM, Uriel  wrote:

> On Thu, Sep 3, 2009 at 9:50 PM, David Leimbach wrote:
> >
> >
> > On Thu, Sep 3, 2009 at 12:36 PM, erik quanstrom 
> > wrote:
> >>
> >> > > > Apple's using it all over the place in Snow Leopard, in all their
> >> > > > native
> >> > > > apps to write cleaner, less manual-lock code.  At least, that's
> the
> >> > > > claim
> >> > > > :-).
> >> > >
> >> > > could someone explain this to me?  i'm just missing how
> >> > > naming a block of code could change its locking properties.
> >> > >
> >> > >
> >> > The explanation is in the manual I linked to earlier in this
> discussion.
> >> >  If
> >> > you want to see examples there's two I can think of available for
> >> > download.
> >> >  One is called DispatchLife the other is DispatchFractal.
> >> >
> >> > I've looked at DispatchLife, and there's no explicit locking of state
> >> > for
> >> > every cell being concurrently update in Conway's game of life.
> >>
> >> i can't find DispatchLife after a few minutes of googling.
> >> i've read the manual, and it looks like csp to me.  clearly
> >> i am a reprobate outside the apple reality distortion field.
> >>
> >
> > Google doesn't have all the answers, I actually had to use Bing today,
> and
> > it worked... anyway here's the link to DispatchLife.
> > http://developer.apple.com/mac/library/samplecode/DispatchLife/
> >
> >>
> >> could you explain why this isn't csp and why this can't be done
> >> with regular c (that is why we need the concept of an
> >> unnamed function pointer) and the thread library?
> >
> > I'm actually planning to figure this stuff out a bit more and "blog"
> about
> > it, hopefully by Friday sometime (tomorrow).
> > I don't agree that any of this stuff is strictly needed.  One can plod
> along
> > with pthreads and do it wrong all day.  One doesn't *need* C either, I've
>
> What do pthreads have to do with anything? Ever heard of libthread,
> libtask and rfork? I guess not, Apple certainly hasn't, and they sure
> are doing a great job at making their developers keep track of as much
> Apple-induced crap as possible...
>

Guess what percentage of developers for Apple know anything at all about
libthread, and libtask.  I bet it's like me and maybe 4 or 5 other people.

Also rfork was in the Mac OS X man pages early on, because the pages were
essentially taken out of FreeBSD, which also had an rfork implementation.

Pthreads, being a part of POSIX, are a lot more well known.  I almost said
understood, but I actually doubt the number of people who know what pthreads
are know how to use them well.  I doubt I could write much with them even
today without looking over the man pages thoroughly as I've been spoiled by
other ways.

Since pthreads are mapped to Mac OS X's mach_threads, it seems to be a path
of least resistance to try to implement stuff that works with them.

And yeah, Apple does do a bit of "developer lock in" with their APIs.  The
only other place Cocoa stuff is even applicable is GNUStep, and even there
it's a subset, though my understanding is the Cocoa bindings of GNU Emacs
work on GNUStep equally well.

I guess my point is, if you invent even the best new way to program
something, or even a superior microprocessor or really anything at all, you
have to overcome the big gorilla's out there, and convince them that your
way is worth adopting.

Also, keep in mind they needed to come up with something that worked with
Objective-C, C++ and C equally well.  Since they don't develop all that
stuff in public, there's no telling what things they tried before settling
in on blocks.


> uriel
>
>
> > seen whole OSes for x86 written in assembly.
> > It all depends on how much crap you want to keep track of.
> > Dave
> >
> >>
> >> - erik
> >>
> >
> >
>
>

Re: [9fans] "Blocks" in C

[Uriel]

On Fri, Sep 4, 2009 at 4:56 PM, David Leimbach wrote:
>
>
> On Fri, Sep 4, 2009 at 7:20 AM, erik quanstrom 
> wrote:
>>
>> > I could be wrong, but I feel like you're not really interested in
>> > entertaining that this idea could be useful, but more interested in
>> > shooting
>> > it down [...]
>>
>> remember, if a guy says to the king, hey you're fly's undone,
>> we send that guy to the stockades for a week.  meanwhile
>> the king's fly remains undone.
>>
>> since the raison d'etre of blocks is ease of programming,
>> i would think it would follow that it should be uniformly
>> easier across the board.  if there are big exceptions to this
>> (like extra locking), i would think the feature would earn
>> a fail.
>>
>
> I am totally agreeing with you so far on all points you've just made.  And I
> think that's why Apple is seeking feedback.

Here is some feedback for Apple: Fire your whole software and
programming division, they are making the GNU and Gnome crack monkeys
look sane, competent and responsible.

uriel

> The advantage of the higher
> level languages that were designed with concurrency in the language, is that
> you don't feel like you're twiddling the bits manually so much to express an
> algorithm.
>
>>
>> i'm just noting that if blocks require locking as you mention,
>> then this is inferior to calling a function through a pointer.
>
> Indeed.
>
>>
>> unless you don't accept more locking is worse, it's hard to
>> argue this point.
>
> My entire point is this will often allow you to get away with far less
> locking, or at least, that's what the intention of all of this is.
>
>>
>> you can accuse me of hating, that won't change how blocks
>> work.
>
>
> I'm not saying you are hating on blocks, I said I'm beginning to feel as if
> perhaps you're trying to find holes in it, but you keep saying things that I
> don't think are true about them. For example, claiming that blocks require
> more locking is evidently false if you look at example code that's been
> provided.  DispatchLife, for example, includes contextual data pointing to
> neighboring cells, and because of the serial nature of the queues involved,
> it's impossible to have two threads updating the same shared data at once.
> The story is different if you take those same blocks, and have them
> scheduled to run on the global concurrent queue, in which case many of them
> could be running at once.  So if you code it up incorrectly, you sure could
> have to do some locking.  But if you take advantage of the (too many in my
> opinion) abstractions provided that help to guarantee you will not need
> locking, then you shouldn't need to do any explicit locking of shared state.
> And that is exactly why Apple bothered to do all of this.  If not, they're
> totally wasting their time, because, as you've said, to create something
> that would require more locking is a big lose.
>
>>
>> > Deep down inside, I want people to stop trying to code stuff like this
>> > in C
>> > and try the massively scaled parallelism/concurrency stuff in other
>> > languages better suited to the problem space.
>>
>> why would you use c then?
>
> Because if I wanted to write something for Mac OS X, and I needed it to work
> with Grand Central Dispatch, I've not been provided a lot of options at the
> moment to do anything else.
> Dave
>>
>> - erik
>>
>
>

Re: [9fans] "Blocks" in C

[Uriel]

On Thu, Sep 3, 2009 at 9:50 PM, David Leimbach wrote:
>
>
> On Thu, Sep 3, 2009 at 12:36 PM, erik quanstrom 
> wrote:
>>
>> > > > Apple's using it all over the place in Snow Leopard, in all their
>> > > > native
>> > > > apps to write cleaner, less manual-lock code.  At least, that's the
>> > > > claim
>> > > > :-).
>> > >
>> > > could someone explain this to me?  i'm just missing how
>> > > naming a block of code could change its locking properties.
>> > >
>> > >
>> > The explanation is in the manual I linked to earlier in this discussion.
>> >  If
>> > you want to see examples there's two I can think of available for
>> > download.
>> >  One is called DispatchLife the other is DispatchFractal.
>> >
>> > I've looked at DispatchLife, and there's no explicit locking of state
>> > for
>> > every cell being concurrently update in Conway's game of life.
>>
>> i can't find DispatchLife after a few minutes of googling.
>> i've read the manual, and it looks like csp to me.  clearly
>> i am a reprobate outside the apple reality distortion field.
>>
>
> Google doesn't have all the answers, I actually had to use Bing today, and
> it worked... anyway here's the link to DispatchLife.
> http://developer.apple.com/mac/library/samplecode/DispatchLife/
>
>>
>> could you explain why this isn't csp and why this can't be done
>> with regular c (that is why we need the concept of an
>> unnamed function pointer) and the thread library?
>
> I'm actually planning to figure this stuff out a bit more and "blog" about
> it, hopefully by Friday sometime (tomorrow).
> I don't agree that any of this stuff is strictly needed.  One can plod along
> with pthreads and do it wrong all day.  One doesn't *need* C either, I've

What do pthreads have to do with anything? Ever heard of libthread,
libtask and rfork? I guess not, Apple certainly hasn't, and they sure
are doing a great job at making their developers keep track of as much
Apple-induced crap as possible...

uriel


> seen whole OSes for x86 written in assembly.
> It all depends on how much crap you want to keep track of.
> Dave
>
>>
>> - erik
>>
>
>

Re: [9fans] "Blocks" in C

[Uriel]

On Thu, Sep 3, 2009 at 5:54 PM, erik quanstrom wrote:
>> Plan 9 has a lot to offer and a lot for others to learn from. Concurrency
>> framework that could scale up to 1K [virtual]cores in an SMP
>> configuration is not one of those features though.
>
> forgive the ignorance, but is there any such thing as a
> 1k-core smp machine?  and is apple doing such a thing?
>
> even commodity intel and amd mp offerings are numa.
> they're not very n, but they're still n.

That, and I still don't see how C 'blocks' solve the problem in question.

But I'm dumber than the Apple geniuses, and grateful for my stupidity.

uriel

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Fri, 2009-09-04 at 13:42 -0400, erik quanstrom wrote:
> as i believe was originally explained,
> i ripped that example *directly* from the apple grand central
> documentation on page 37 in the "Data Types" section:
> 
> http://developer.apple.com/mac/library/documentation/Performance/Reference/GCD_libdispatch_Ref/GCD_libdispatch_Ref.pdf
> 
> maybe you don't believe the documentation?

Sure I do. I also believed in Apple doing the right thing here :-(

Anyway, the only thing I can say in my defense here is that since
your example was mixed among pretty funny speculations about executable
code being copied to the stack and such I couldn't possible imagine
that it came from the docs *verbatim*.

Thanks,
Roman.

P.S. And yes, technically -- it is up to compiler to either have or
not have that limitation, since the blocks that you create within
a function scope are all *known* to it and can be rearranged at will.

Re: [9fans] "Blocks" in C

[erik quanstrom]

> as i believe was originally explained,
> i ripped that example *directly* from the apple grand central
> documentation on page 37 in the "Data Types" section:
> 
> http://developer.apple.com/mac/library/documentation/Performance/Reference/GCD_libdispatch_Ref/GCD_libdispatch_Ref.pdf
> 
> maybe you don't believe the documentation?

also http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1370.pdf
page 2 it says blocks use automatic storage.

- erik

Re: [9fans] "Blocks" in C

[erik quanstrom]

> Where exactly does it say that?
> 
> >dispatch_block_t p;
> >
> > if(cond){
> > p =^ { print("cond\n"); };
> > }else{
> > p =^ { print("cond\n"); };
> > }
> > p();
> >
> > since the first part is equivalent to
> >
> > if(cond){
> > struct Block _t = ...;
> > p = &_t;
> > }
> 
> I'm pretty sure Apple's compiler is smart enough to allow the  
> construct that
> you've just mentioned. In fact, I'd be willing to bet on it. Sadly  
> just like about
> anybody else on this thread I don't have an access to Snow Leopard Mac  
> OS X
> at the moment. So if there needs to be an experiment run -- that'll  
> have to
> wait till next week.

as i believe was originally explained,
i ripped that example *directly* from the apple grand central
documentation on page 37 in the "Data Types" section:

http://developer.apple.com/mac/library/documentation/Performance/Reference/GCD_libdispatch_Ref/GCD_libdispatch_Ref.pdf

maybe you don't believe the documentation?

- erik

Re: [9fans] "Blocks" in C

[Roman Shaposhnik]

On Sep 4, 2009, at 9:58 AM, Iruata Souza wrote:

On Fri, Sep 4, 2009 at 1:44 PM, Roman Shaposhnik wrote:
There's been a *lot* of speculation on this thread and very little  
fact.

(...)
Trust me, I've seen how it is generated.



so we should trust you and not the facts? is that what you are saying?
because i haven't seen any 'factual' code yet.


I don't understand the question. When I say "trust me" I was commenting
on one *particular* aspect of implementation that I *happen* to have  
seen.

I think it is fair to speak from experience where experience exists.

I don't have answers to most of the intelligent questions asked by  
Bakul,

for example and that's why I keep my mouth shut on those portions
of the thread up until I can lay my hands on Snow Leopard.

Finally, if you are interested in this topic go grab http://clang.llvm.org/ 
 and
play with it. It'll give you 90% of the answers. The rest is in  
Apple's implementation.

My understanding is that it is slightly different from the OS one.

Thanks,
Roman.

P.S. Or do you want *my* help with the code?

Re: [9fans] "Blocks" in C

[Iruata Souza]

On Fri, Sep 4, 2009 at 1:44 PM, Roman Shaposhnik wrote:
> There's been a *lot* of speculation on this thread and very little fact.
> (...)
> Trust me, I've seen how it is generated.
>

so we should trust you and not the facts? is that what you are saying?
because i haven't seen any 'factual' code yet.
iru

Re: [9fans] "Blocks" in C

[Roman Shaposhnik]

On Sep 4, 2009, at 2:15 AM, Greg Comeau wrote:

In article <1251993672.16936.4779.ca...@work.sfbay.sun.com>,
Roman V Shaposhnik  wrote:

On Thu, 2009-09-03 at 08:44 -0700, David Leimbach wrote:


The blocks aren't interesting at all by themselves, I totally agree
with that.  However what they do to let you write a function inline,
that can be pushed to another function, to be executed on a  
concurrent

FIFO, is where the real power comes out.


I'm not 100% sure why the heck they did it this way, which is  
totally
different from any other version of concurrent programming setup  
I've

seen, except maybe that Apple likes to "think different"?


It seems that quite a few concurrency frameworks worth the paper  
their

APIs are written on, are converging on this model. The ultimate goal
has to do with encapsulation of the computation into idempotent units
and describing the topology between those units. That separates the
executor from the semantics of computations nicely and has all  
sorts of
bonuses as far as bridging the gap between SMP and distributes  
systems

are concerned.

I think the semantics of what needs to be done is well understood.  
The
million dollar question is how to best express such a semantics in  
what

still looks like a programming language.

What Apple has done is one way of attacking the problem. Where I  
sit we
explore CPS for doing very similar sort of thing. One point is  
clear --

there no consensus yet.


I don't think I follow what you just said


I can explain if there's interest. Perhaps in a different thread.

Thanks,
Roman.

P.S. I've just spent past couple of months working on a prototype of
a parallel framework. I'm not sure I'll be allowed to continue that  
work,

so I might as well brag about it here.

Re: [9fans] "Blocks" in C

[Roman Shaposhnik]

On Sep 4, 2009, at 5:14 AM, erik quanstrom wrote:

But this has no more to do with parallelism than any other
feature of C. If you used __block vars in a block, you'd
still need to lock them when the block is called from
different threads.


that's a lot worse than a function pointer.  with
a function pointer your going to get unique space
on the stack for each invocation.


Erik, you're usually pretty pedantic and factual. This
post is neither.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[Roman Shaposhnik]

There's been a *lot* of speculation on this thread and very little fact.
I'd encourage everybody to play with the feature before forming
any kind of final judgement.

On Sep 3, 2009, at 8:52 PM, erik quanstrom wrote:
Did you even read the article or any of the examples? There are  
plenty

of things that you can "do" with blocks that you can't with just
function pointers. That's besides the fact that some of them are more
elegantly expressed with blocks that look sort of ugly with function
pointers.


on the other hand, apple says this is illegal


Where exactly does it say that?


   dispatch_block_t p;

if(cond){
p =^ { print("cond\n"); };
}else{
p =^ { print("cond\n"); };
}
p();

since the first part is equivalent to

if(cond){
struct Block _t = ...;
p = &_t;
}


I'm pretty sure Apple's compiler is smart enough to allow the  
construct that
you've just mentioned. In fact, I'd be willing to bet on it. Sadly  
just like about
anybody else on this thread I don't have an access to Snow Leopard Mac  
OS X
at the moment. So if there needs to be an experiment run -- that'll  
have to

wait till next week.


intuitive?  easy to read? pretty?


What part are you complaining about? The imaginary failure?


also, if this from david's example is allowed (i'll assume
that the original examples' print(X+y) for int X and y
was a bit of a typo --- i hope!)

block =^ (int x){ print("%d\n", x ); };

that sucker is on the stack.  by-by no-execute stack.
how does it get to the stack?  is it just copied from
the text segment or is it compiled at run time?



The structure is on the stack, the code itself is in .text as
anything else in your C program. Trust me, I've seen how
it is generated.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[Bakul Shah]

On Fri, 04 Sep 2009 08:04:40 PDT David Leimbach   wrote:
> On Fri, Sep 4, 2009 at 7:41 AM, Bakul Shah
> 
> > wrote:
> 
> > On Fri, 04 Sep 2009 00:47:18 PDT David Leimbach 
> >  wrote:
> > > On Fri, Sep 4, 2009 at 12:11 AM, Bakul Shah
> > > <
> > bakul%2bpl...@bitblocks.com >
> > > > wrote:
> > >
> > > > But this has no more to do with parallelism than any other
> > > > feature of C. If you used __block vars in a block, you'd
> > > > still need to lock them when the block is called from
> > > > different threads.
> > > >
> > > I just wrote a prime sieve with terrible shutdown synchronization you can
> > > look at here:
> > >
> > > http://paste.lisp.org/display/86549
> >
> > Not sure how your program invalidates what I said.  Blocks do
> > provide more syntactic sugar but that "benefit" is independent
> > of GCD (grand central dispatch) or what have you. Given that
> > __block vars are shared, I don't see how you can avoid locking
> > if blocks get used in parallel.
> >

To be precise, I meant to write "avoid locking if blocks get
executed in parallel and access a __block variable".

> You've said it yourself.  "if blocks get used in parallel".  If the blocks
> are scheduled to the same non-concurrent queue, there shouldn't be a
> problem, unless you've got blocks scheduled and running on multiple serial
> queues.   There are 3 concurrent queues, each with different priorities in
> GCD, and you can't create any more concurrent queues to the best of my
> knowledge, the rest are serial queues, and they schedule blocks in FIFO
> order.
>
> Given that you can arrange your code such that no two blocks sharing the
> same state can execute at the same time now, why would you lock it?

Consider this example:

int
main(int c, char**v)
{
int n = c > 1? atoi(v[1]) : 1000;
__block int x;
x = 0;
parallel_execute(n, ^{ for (int i = 0; i < n; i++) ++x; });
while (x != n*n)
sleep(1);
}

Where parallel_execute() spawns off n copies of the block and
tries to execute as many of them in parallel as possible.
Presumably this is implementable?  Will this prorgam ever
terminate (for any value of n upto 2^15-1)?  How would you
avoid sharing here except by turning parallel_execute() in
serial_execute()?

> I should note that for some reason my code falls apart in terms of actually
> working as I expected it after MAX is set to something over 700, so I'm
> probably *still* not doing something correctly, or I did something Apple
> didn't expect.

:-)

Re: [9fans] "Blocks" in C

[David Leimbach]

On Fri, Sep 4, 2009 at 7:41 AM, Bakul Shah

> wrote:

> On Fri, 04 Sep 2009 00:47:18 PDT David Leimbach 
>  wrote:
> > On Fri, Sep 4, 2009 at 12:11 AM, Bakul Shah
> > <
> bakul%2bpl...@bitblocks.com >
> > > wrote:
> >
> > > But this has no more to do with parallelism than any other
> > > feature of C. If you used __block vars in a block, you'd
> > > still need to lock them when the block is called from
> > > different threads.
> > >
> > I just wrote a prime sieve with terrible shutdown synchronization you can
> > look at here:
> >
> > http://paste.lisp.org/display/86549
>
> Not sure how your program invalidates what I said.  Blocks do
> provide more syntactic sugar but that "benefit" is independent
> of GCD (grand central dispatch) or what have you. Given that
> __block vars are shared, I don't see how you can avoid locking
> if blocks get used in parallel.
>

You've said it yourself.  "if blocks get used in parallel".  If the blocks
are scheduled to the same non-concurrent queue, there shouldn't be a
problem, unless you've got blocks scheduled and running on multiple serial
queues.   There are 3 concurrent queues, each with different priorities in
GCD, and you can't create any more concurrent queues to the best of my
knowledge, the rest are serial queues, and they schedule blocks in FIFO
order.

Given that you can arrange your code such that no two blocks sharing the
same state can execute at the same time now, why would you lock it?  What I
did was allocate context data for the actual read end of a pipe fd on the
heap, such that when an associated block was launched by the runtime (when
something was written to the write fd of a pipe) it would get it's context
pointer in it's block struct, which I could access by get_context.

I should note that for some reason my code falls apart in terms of actually
working as I expected it after MAX is set to something over 700, so I'm
probably *still* not doing something correctly, or I did something Apple
didn't expect.

Dave

Re: [9fans] "Blocks" in C

[David Leimbach]

On Fri, Sep 4, 2009 at 7:20 AM, erik quanstrom wrote:

> > I could be wrong, but I feel like you're not really interested in
> > entertaining that this idea could be useful, but more interested in
> shooting
> > it down [...]
>
> remember, if a guy says to the king, hey you're fly's undone,
> we send that guy to the stockades for a week.  meanwhile
> the king's fly remains undone.
>
> since the raison d'etre of blocks is ease of programming,
> i would think it would follow that it should be uniformly
> easier across the board.  if there are big exceptions to this
> (like extra locking), i would think the feature would earn
> a fail.
>
>
I am totally agreeing with you so far on all points you've just made.  And I
think that's why Apple is seeking feedback.  The advantage of the higher
level languages that were designed with concurrency in the language, is that
you don't feel like you're twiddling the bits manually so much to express an
algorithm.


> i'm just noting that if blocks require locking as you mention,
> then this is inferior to calling a function through a pointer.
>

Indeed.


>
> unless you don't accept more locking is worse, it's hard to
> argue this point.
>

My entire point is this will often allow you to get away with far less
locking, or at least, that's what the intention of all of this is.


>
> you can accuse me of hating, that won't change how blocks
> work.
>


I'm not saying you are hating on blocks, I said I'm beginning to feel as if
perhaps you're trying to find holes in it, but you keep saying things that I
don't think are true about them. For example, claiming that blocks require
more locking is evidently false if you look at example code that's been
provided.  DispatchLife, for example, includes contextual data pointing to
neighboring cells, and because of the serial nature of the queues involved,
it's impossible to have two threads updating the same shared data at once.

The story is different if you take those same blocks, and have them
scheduled to run on the global concurrent queue, in which case many of them
could be running at once.  So if you code it up incorrectly, you sure could
have to do some locking.  But if you take advantage of the (too many in my
opinion) abstractions provided that help to guarantee you will not need
locking, then you shouldn't need to do any explicit locking of shared state.

And that is exactly why Apple bothered to do all of this.  If not, they're
totally wasting their time, because, as you've said, to create something
that would require more locking is a big lose.



> > Deep down inside, I want people to stop trying to code stuff like this in
> C
> > and try the massively scaled parallelism/concurrency stuff in other
> > languages better suited to the problem space.
>
> why would you use c then?
>

Because if I wanted to write something for Mac OS X, and I needed it to work
with Grand Central Dispatch, I've not been provided a lot of options at the
moment to do anything else.

Dave

>
> - erik
>
>

Re: [9fans] "Blocks" in C

[Bakul Shah]

On Fri, 04 Sep 2009 00:47:18 PDT David Leimbach   wrote:
> On Fri, Sep 4, 2009 at 12:11 AM, Bakul Shah
> 
> > wrote:
> 
> > But this has no more to do with parallelism than any other
> > feature of C. If you used __block vars in a block, you'd
> > still need to lock them when the block is called from
> > different threads.
> >
> I just wrote a prime sieve with terrible shutdown synchronization you can
> look at here:
> 
> http://paste.lisp.org/display/86549

Not sure how your program invalidates what I said.  Blocks do
provide more syntactic sugar but that "benefit" is independent
of GCD (grand central dispatch) or what have you. Given that
__block vars are shared, I don't see how you can avoid locking
if blocks get used in parallel.

Re: [9fans] "Blocks" in C

[erik quanstrom]

> I could be wrong, but I feel like you're not really interested in
> entertaining that this idea could be useful, but more interested in shooting
> it down [...]

remember, if a guy says to the king, hey you're fly's undone,
we send that guy to the stockades for a week.  meanwhile
the king's fly remains undone.

since the raison d'etre of blocks is ease of programming,
i would think it would follow that it should be uniformly
easier across the board.  if there are big exceptions to this
(like extra locking), i would think the feature would earn
a fail.

i'm just noting that if blocks require locking as you mention,
then this is inferior to calling a function through a pointer.

unless you don't accept more locking is worse, it's hard to
argue this point.

you can accuse me of hating, that won't change how blocks
work.

> Deep down inside, I want people to stop trying to code stuff like this in C
> and try the massively scaled parallelism/concurrency stuff in other
> languages better suited to the problem space.

why would you use c then?

- erik

Re: [9fans] "Blocks" in C

[matt]

I could be wrong, but I feel like you're not really interested in 
entertaining that this idea could be useful, but more interested in 
shooting it down.  That's fine, people do that all the time.  People 
are *constantly* saying Plan 9 is a huge waste of time too.  And if 
you count the number of actual users, they're probably right.



using plan9 as a stick to beat one of us with hurts us all

Re: [9fans] "Blocks" in C

[David Leimbach]

On Fri, Sep 4, 2009 at 5:14 AM, erik quanstrom wrote:

> > But this has no more to do with parallelism than any other
> > feature of C. If you used __block vars in a block, you'd
> > still need to lock them when the block is called from
> > different threads.
>
> that's a lot worse than a function pointer.  with
> a function pointer your going to get unique space
> on the stack for each invocation.
>

Unless the variable is static right?  And not automatic?  So __block is the
block's static local data?

I could be wrong, but I feel like you're not really interested in
entertaining that this idea could be useful, but more interested in shooting
it down.  That's fine, people do that all the time.  People are *constantly*
saying Plan 9 is a huge waste of time too.  And if you count the number of
actual users, they're probably right.

I happen to like Plan 9, so I do some stuff with it.  I don't know if I like
libdispatch, GCD or these new C closures yet.  I am finding myself wanting
to try to like them, but I can't decide yet how bad or good they really are.

Deep down inside, I want people to stop trying to code stuff like this in C
and try the massively scaled parallelism/concurrency stuff in other
languages better suited to the problem space.


>
> the variable capture thing seems to me to just
> confuse the issue.  c doesn't otherwise work
> like that.
>

And now you've arrived at the conclusion apple probably arrived at for
changing the language the way they did.  C doesn't otherwise work like that
:-)

Note I've written a concurrent prime sieve with these facilities.

http://paste.lisp.org/display/86549

I should also note that once I crank max up a little over 700 it all seems
to fall apart.

This might be my bug, or apple's bug.  But I've been really good at finding
bugs in apple's new features for years now.  I totally froze up a system
playing with kernel queues when they added them at first too, and they
contacted me personally to retry my code when they thought they had it
fixed.

They may come off as pompous, but I know people in apple looking for people
to examine this stuff in a serious way, and give feedback.

Dave



>
> - erik
>
>

Re: [9fans] "Blocks" in C

[erik quanstrom]

> But this has no more to do with parallelism than any other
> feature of C. If you used __block vars in a block, you'd
> still need to lock them when the block is called from
> different threads.

that's a lot worse than a function pointer.  with
a function pointer your going to get unique space
on the stack for each invocation.

the variable capture thing seems to me to just
confuse the issue.  c doesn't otherwise work
like that.

- erik

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article <3e1162e60909030844r8760a8fu1b27d6e60965e...@mail.gmail.com>,
David Leimbach  wrote:
>The blocks aren't interesting at all by themselves, I totally agree with
>that.  However what they do to let you write a function inline, that can be
>pushed to another function, to be executed on a concurrent FIFO, is where
>the real power comes out.

This seems to be the first post that does not appear to be a kind of
knee jerk reaction.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article <1251993672.16936.4779.ca...@work.sfbay.sun.com>,
Roman V Shaposhnik  wrote:
>On Thu, 2009-09-03 at 08:44 -0700, David Leimbach wrote:
>
>> The blocks aren't interesting at all by themselves, I totally agree
>> with that.  However what they do to let you write a function inline,
>> that can be pushed to another function, to be executed on a concurrent
>> FIFO, is where the real power comes out.
>> 
>> 
>> I'm not 100% sure why the heck they did it this way, which is totally
>> different from any other version of concurrent programming setup I've
>> seen, except maybe that Apple likes to "think different"?
>
>It seems that quite a few concurrency frameworks worth the paper their
>APIs are written on, are converging on this model. The ultimate goal
>has to do with encapsulation of the computation into idempotent units
>and describing the topology between those units. That separates the
>executor from the semantics of computations nicely and has all sorts of
>bonuses as far as bridging the gap between SMP and distributes systems
>are concerned.
>
>I think the semantics of what needs to be done is well understood. The
>million dollar question is how to best express such a semantics in what
>still looks like a programming language.
>
>What Apple has done is one way of attacking the problem. Where I sit we
>explore CPS for doing very similar sort of thing. One point is clear --
>there no consensus yet.

I don't think I follow what you just said, but your conclusion
is probably exactly right.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article ,
erik quanstrom  wrote:
>> Apple's using it all over the place in Snow Leopard, in all their native
>> apps to write cleaner, less manual-lock code.  At least, that's the claim
>> :-).
>
>could someone explain this to me?  i'm just missing how
>naming a block of code could change its locking properties.

Unless I'm completely not recalling properly (so I could be
completely wrong) it's not the naming or unname per se,
but that because the stuff happens at runtime and I seem
to recall can also be tagged, then, then that combo allows
the underlying machinery can pick up on the tag, or not,
to do the locking.  Automatically.  If I'm not remembering
correctly, or, I'm thinking about something totally different,
I'm certain somebody will correct any misspeaks I've just made.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article <5d375e920909030832i17c62bc7mbb1afc55708e0...@mail.gmail.com>,
Uriel  wrote:
>So libthread must be a figment of 9fan's imagination...
>
>Of course, for Apple (or anyone else) to learn from Plan 9 would be
>impossible, so instead they had to add a new 'feature' to C.

Apple, and many other companies, sure can be pompous often,
bringing even "the market chooses" to be rhetoric.
That said, I really can't say how true the above is in this case.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[David Leimbach]

On Fri, Sep 4, 2009 at 12:11 AM, Bakul Shah

> wrote:

> On Thu, 03 Sep 2009 22:35:35 PDT David Leimbach 
>  wrote:
> >
> >
> > Actually, reading on a bit more they deal with the "variable capture"
> > talking about const copies.
> >
> > Automatic storage variables not marked with __block are imported as
> > const copies.
> >
> > The simplest example is that of importing a variable of type int.
> >
> >int x = 10;
> >void (^vv)(void) = ^{ printf("x is %d\n", x); }
> >x = 11;
> >vv();
> >
> > would be compiled
> >
> > struct __block_literal_2 {
> > void *isa;
> > int flags;
> > int reserved;
> > void (*invoke)(struct __block_literal_2 *);
> > struct __block_descriptor_2 *descriptor;
> > const int x;
> > };
> >
> > void __block_invoke_2(struct __block_literal_2 *_block) {
> > printf("x is %d\n", _block->x);
> > }
> >
> > static struct __block_descriptor_2 {
> > unsigned long int reserved;
> > unsigned long int Block_size;
> > } __block_descriptor_2 = { 0, sizeof(struct __block_literal_2) };
> >
> > and
> >
> >   struct __block_literal_2 __block_literal_2 = {
> >   &_NSConcreteStackBlock,
> >   (1<<29), ,
> >   __block_invoke_2,
> >   &__block_descriptor_2,
> > x
> >};
> >
> > In summary, scalars, structures, unions, and function pointers are
> > generally imported as const copies with no need for helper functions.
>
> Just read this after posting my last message.
>
> But this has no more to do with parallelism than any other
> feature of C. If you used __block vars in a block, you'd
> still need to lock them when the block is called from
> different threads.
>
>
I just wrote a prime sieve with terrible shutdown synchronization you can
look at here:

http://paste.lisp.org/display/86549

Dave

Re: [9fans] "Blocks" in C

[Bakul Shah]

On Thu, 03 Sep 2009 22:35:35 PDT David Leimbach   wrote:
> 
> 
> Actually, reading on a bit more they deal with the "variable capture"
> talking about const copies.
> 
> Automatic storage variables not marked with __block are imported as
> const copies.
> 
> The simplest example is that of importing a variable of type int.
> 
>int x = 10;
>void (^vv)(void) = ^{ printf("x is %d\n", x); }
>x = 11;
>vv();
> 
> would be compiled
> 
> struct __block_literal_2 {
> void *isa;
> int flags;
> int reserved;
> void (*invoke)(struct __block_literal_2 *);
> struct __block_descriptor_2 *descriptor;
> const int x;
> };
> 
> void __block_invoke_2(struct __block_literal_2 *_block) {
> printf("x is %d\n", _block->x);
> }
> 
> static struct __block_descriptor_2 {
> unsigned long int reserved;
> unsigned long int Block_size;
> } __block_descriptor_2 = { 0, sizeof(struct __block_literal_2) };
> 
> and
> 
>   struct __block_literal_2 __block_literal_2 = {
>   &_NSConcreteStackBlock,
>   (1<<29), ,
>   __block_invoke_2,
>   &__block_descriptor_2,
> x
>};
> 
> In summary, scalars, structures, unions, and function pointers are
> generally imported as const copies with no need for helper functions.

Just read this after posting my last message.

But this has no more to do with parallelism than any other
feature of C. If you used __block vars in a block, you'd
still need to lock them when the block is called from
different threads.

Re: [9fans] "Blocks" in C

[Bakul Shah]

On Fri, 04 Sep 2009 00:44:35 EDT erik quanstrom   wrote:
> > > that sucker is on the stack.  by-by no-execute stack.

I don't think so. See below.

> > > how does it get to the stack?  is it just copied from
> > > the text segment or is it compiled at run time?
> > >
> > 
> > I don't think I posted the whole code, so that's my bad.  The X was on the
> > stack to begin with as the first X was an automatic variable in a function.
> >  I'd be a little surprised to find an automatic variable in the text
> > segment, but perhaps that's just my not remembering things properly.
> >  (didn't mean that tongue in cheek, I don't think about that stuff much
> > these days, as I've spent the last year or so doing Erlang and Haskell.)
> 
> it is the block itself that apple claims is on the stacp
> (your grand centeral reference, p. 38).  and i wonder
> how it gets there.  is it just copied from the text segment?
> that seems kind of pointless.  why not just execute it
> from the text segment?  or is it modified (compiled?)
> at run time?

[Note: I am simply guessing and have no idea how they
 actually do this but this model seems workable]

Consider this example:

int foo(int a) {
__block int b = 0;
int (^g()) = ^{ return a + ++b; }
...
return g() + g();
}


My guess is the above will be translated to something like this:

struct anon1 {
int (*f)(struct anon1*);
const int   a;
int *const  bp;
};
int anon1_f(struct anon1* x) {
return x->a + ++(*x->bp);
}
int foo(int a) {
int *bp = malloc(sizeof *bp); // not quite. see the correction below
*bp = 0;
struct anon1 _anon = { &anon1_f, a, &b };
struct anon1* g = &_anon;
...
return g->f(&_anon) + g->f(&_anon);
}

As you can see, _anon will disappear when foo() is exited.
But if you were to Block_copy() _anon, it will be allocated
on the heap and a ptr to it returned. Now you do have
everything needed to call this anon function even after
returning from foo().  &_anon can also be passed to another
thread etc. with no problem.

Most likely __block variables are allocated on the heap and
ref counted.  Ref count is decremented on exit from the
lexical scope where a __block var is defined.  Block_copy()
increments refcount of every __block var referenced by the
block, Block_release() decrements it.

So this is basically a function closure. They seem to have
very carefully navigated around C's semantic sand bars.

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 10:31 PM, David Leimbach  wrote:

>
>
> On Thu, Sep 3, 2009 at 9:44 PM, erik quanstrom wrote:
>
>> > > that sucker is on the stack.  by-by no-execute stack.
>> > > how does it get to the stack?  is it just copied from
>> > > the text segment or is it compiled at run time?
>> > >
>> >
>> > I don't think I posted the whole code, so that's my bad.  The X was on
>> the
>> > stack to begin with as the first X was an automatic variable in a
>> function.
>> >  I'd be a little surprised to find an automatic variable in the text
>> > segment, but perhaps that's just my not remembering things properly.
>> >  (didn't mean that tongue in cheek, I don't think about that stuff much
>> > these days, as I've spent the last year or so doing Erlang and Haskell.)
>>
>> it is the block itself that apple claims is on the stacp
>> (your grand centeral reference, p. 38).  and i wonder
>> how it gets there.  is it just copied from the text segment?
>> that seems kind of pointless.  why not just execute it
>> from the text segment?  or is it modified (compiled?)
>> at run time?
>>
>
> Right, my understanding is that blocks live on the stack, however it
> appears that they get copied to a particular queue before being run, either
> explicitly or implicitly depending on how it gets submitted.  If you use
> dispatch_after, it gets copied and released automatically after a certain
> amount of time.
>
> The whole thing is very "eventy".
>
> Some of the documentation is not terribly explicit about how things get
> copied if they do at all.  Example is the dispatch_async call, which behaves
> based on the queue to which it is submitted, but whether or not a copy
> occurs is not mentioned.
>
> This document goes into more detail:
> http://clang.llvm.org/docs/BlockImplementation.txt
>
> "Block literals may occur within functions where the structure is created
> in stack local memory. They may also appear as initialization expressions
> for Block variables of global or static local variables.
>
> When a Block literal expression is evaluated the stack based structure is 
> initialized as follows:
>
> 1) static descriptor structure is declared and initialized as follows:
> 1a) the invoke function pointer is set to a function that takes the Block 
> structure as its first argument and the rest of the arguments (if any) to the 
> Block and executes the Block compound statement.
> 1b) the size field is set to the size of the following Block literal 
> structure.
> 1c) the copy_helper and dispose_helper function pointers are set to 
> respective helper functions if they are required by the Block literal
> 2) a stack (or global) Block literal data structure is created and 
> initialized as follows:
> 2a) the isa field is set to the address of the external 
> _NSConcreteStackBlock, which is a block of uninitialized memory supplied in 
> libSystem, or _NSConcreteGlobalBlock if this is a static or file level block 
> literal.
> 2) The flags field is set to zero unless there are variables imported into 
> the block that need helper functions for program level Block_copy() and 
> Block_release() operations, in which case the (1<<25) flags bit is set."
>
> I'm still left feeling somewhat not understanding how it all works :-)
>
>
>

Actually, reading on a bit more they deal with the "variable capture"
talking about const copies.

Automatic storage variables not marked with __block are imported as
const copies.

The simplest example is that of importing a variable of type int.

   int x = 10;
   void (^vv)(void) = ^{ printf("x is %d\n", x); }
   x = 11;
   vv();

would be compiled

struct __block_literal_2 {
void *isa;
int flags;
int reserved;
void (*invoke)(struct __block_literal_2 *);
struct __block_descriptor_2 *descriptor;
const int x;
};

void __block_invoke_2(struct __block_literal_2 *_block) {
printf("x is %d\n", _block->x);
}

static struct __block_descriptor_2 {
unsigned long int reserved;
unsigned long int Block_size;
} __block_descriptor_2 = { 0, sizeof(struct __block_literal_2) };

and

  struct __block_literal_2 __block_literal_2 = {
&_NSConcreteStackBlock,
(1<<29), ,
__block_invoke_2,
&__block_descriptor_2,
x
   };

In summary, scalars, structures, unions, and function pointers are
generally imported as const copies with no need for helper functions.




>
>> - erik
>>
>>
>

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 9:44 PM, erik quanstrom wrote:

> > > that sucker is on the stack.  by-by no-execute stack.
> > > how does it get to the stack?  is it just copied from
> > > the text segment or is it compiled at run time?
> > >
> >
> > I don't think I posted the whole code, so that's my bad.  The X was on
> the
> > stack to begin with as the first X was an automatic variable in a
> function.
> >  I'd be a little surprised to find an automatic variable in the text
> > segment, but perhaps that's just my not remembering things properly.
> >  (didn't mean that tongue in cheek, I don't think about that stuff much
> > these days, as I've spent the last year or so doing Erlang and Haskell.)
>
> it is the block itself that apple claims is on the stacp
> (your grand centeral reference, p. 38).  and i wonder
> how it gets there.  is it just copied from the text segment?
> that seems kind of pointless.  why not just execute it
> from the text segment?  or is it modified (compiled?)
> at run time?
>

Right, my understanding is that blocks live on the stack, however it appears
that they get copied to a particular queue before being run, either
explicitly or implicitly depending on how it gets submitted.  If you use
dispatch_after, it gets copied and released automatically after a certain
amount of time.

The whole thing is very "eventy".

Some of the documentation is not terribly explicit about how things get
copied if they do at all.  Example is the dispatch_async call, which behaves
based on the queue to which it is submitted, but whether or not a copy
occurs is not mentioned.

This document goes into more detail:
http://clang.llvm.org/docs/BlockImplementation.txt

"Block literals may occur within functions where the structure is created in
stack local memory. They may also appear as initialization expressions for
Block variables of global or static local variables.

When a Block literal expression is evaluated the stack based structure
is initialized as follows:

1) static descriptor structure is declared and initialized as follows:
1a) the invoke function pointer is set to a function that takes the
Block structure as its first argument and the rest of the arguments
(if any) to the Block and executes the Block compound statement.
1b) the size field is set to the size of the following Block literal structure.
1c) the copy_helper and dispose_helper function pointers are set to
respective helper functions if they are required by the Block literal
2) a stack (or global) Block literal data structure is created and
initialized as follows:
2a) the isa field is set to the address of the external
_NSConcreteStackBlock, which is a block of uninitialized memory
supplied in libSystem, or _NSConcreteGlobalBlock if this is a static
or file level block literal.
2) The flags field is set to zero unless there are variables imported
into the block that need helper functions for program level
Block_copy() and Block_release() operations, in which case the (1<<25)
flags bit is set."

I'm still left feeling somewhat not understanding how it all works :-)



>
> - erik
>
>

Re: [9fans] "Blocks" in C

[erik quanstrom]

> > that sucker is on the stack.  by-by no-execute stack.
> > how does it get to the stack?  is it just copied from
> > the text segment or is it compiled at run time?
> >
> 
> I don't think I posted the whole code, so that's my bad.  The X was on the
> stack to begin with as the first X was an automatic variable in a function.
>  I'd be a little surprised to find an automatic variable in the text
> segment, but perhaps that's just my not remembering things properly.
>  (didn't mean that tongue in cheek, I don't think about that stuff much
> these days, as I've spent the last year or so doing Erlang and Haskell.)

it is the block itself that apple claims is on the stacp
(your grand centeral reference, p. 38).  and i wonder
how it gets there.  is it just copied from the text segment?
that seems kind of pointless.  why not just execute it
from the text segment?  or is it modified (compiled?)
at run time?

- erik

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 8:52 PM, erik quanstrom wrote:

> > Did you even read the article or any of the examples? There are plenty
> > of things that you can "do" with blocks that you can't with just
> > function pointers. That's besides the fact that some of them are more
> > elegantly expressed with blocks that look sort of ugly with function
> > pointers.
>
> on the other hand, apple says this is illegal
>
>dispatch_block_t p;
>
>if(cond){
>p =^ { print("cond\n"); };
>}else{
>p =^ { print("cond\n"); };
>}
>p();
>
> since the first part is equivalent to
>
>if(cond){
>struct Block _t = ...;
>p = &_t;
>}
>
> intuitive?  easy to read? pretty?
>
> also, if this from david's example is allowed (i'll assume
> that the original examples' print(X+y) for int X and y
> was a bit of a typo --- i hope!)
>
>block =^ (int x){ print("%d\n", x ); };
>

Wasn't my example... came off an email I cut and pasted from another email
after googling around a bit for block stuff.  Also I think you mean printf
since we're being pedantic? :-)


>
> that sucker is on the stack.  by-by no-execute stack.
> how does it get to the stack?  is it just copied from
> the text segment or is it compiled at run time?
>

I don't think I posted the whole code, so that's my bad.  The X was on the
stack to begin with as the first X was an automatic variable in a function.
 I'd be a little surprised to find an automatic variable in the text
segment, but perhaps that's just my not remembering things properly.
 (didn't mean that tongue in cheek, I don't think about that stuff much
these days, as I've spent the last year or so doing Erlang and Haskell.)


>
> - erik
>
>

Re: [9fans] "Blocks" in C

[erik quanstrom]

> Did you even read the article or any of the examples? There are plenty  
> of things that you can "do" with blocks that you can't with just  
> function pointers. That's besides the fact that some of them are more  
> elegantly expressed with blocks that look sort of ugly with function  
> pointers.

on the other hand, apple says this is illegal

dispatch_block_t p;

if(cond){
p =^ { print("cond\n"); };
}else{
p =^ { print("cond\n"); };
}
p();

since the first part is equivalent to

if(cond){
struct Block _t = ...;
p = &_t;
}

intuitive?  easy to read? pretty?

also, if this from david's example is allowed (i'll assume
that the original examples' print(X+y) for int X and y
was a bit of a typo --- i hope!)

block =^ (int x){ print("%d\n", x ); };

that sucker is on the stack.  by-by no-execute stack.
how does it get to the stack?  is it just copied from
the text segment or is it compiled at run time?

- erik

Re: [9fans] "Blocks" in C

[Anant Narayanan]

On Sep 3, 2009, at 5:15 PM, Uriel wrote:

Exactly, I still fail to understand the point of this "feature",
function points have worked fine for ages, but then I never understood
any religion, and that is what Apple seems to be all about.


Did you even read the article or any of the examples? There are plenty  
of things that you can "do" with blocks that you can't with just  
function pointers. That's besides the fact that some of them are more  
elegantly expressed with blocks that look sort of ugly with function  
pointers.


I understand the argument that blocks don't "feel" C-like, but the  
argument that you can do everything with just using function pointers  
is BS.


--
Anant

Re: [9fans] "Blocks" in C

[erik quanstrom]

On Thu Sep  3 21:38:30 EDT 2009, driv...@0xabadba.be wrote:
> To ensure only one thread in the kernel at a time?

yes.  http://en.wikipedia.org/wiki/Giant_lock
it allows only one kernel thread to run at a time.

the pool lock allows as many threads to run as
one would like, but they can't allocate concurrently
since blocks and kernel buffers are allocated through
the pool library.  this effectively ties kernel procs doing
io together by their shoelaces.  it's a big reason that a
number of ethernet drivers have private frame pools.

cutting the lock (any lock) out of the mix when shuffling
very small blocks around the kernel was worth double
the blocks/s in some tests i did.  sorry to be vague.  i
don't have a good writeup.

- erik

Re: [9fans] "Blocks" in C

[drivers]

To ensure only one thread in the kernel at a time?

-jt-
--Original Message--
From: erik quanstrom
Sender: 9fans-boun...@9fans.net
To: 9fans@9fans.net
ReplyTo: Fans of the OS Plan 9 from Bell Labs
Subject: Re: [9fans] "Blocks" in C
Sent: Sep 3, 2009 21:32

> what does BLK stand for?

big kernel lock.

- erik



=jt

Re: [9fans] "Blocks" in C

[erik quanstrom]

> what does BLK stand for?

big kernel lock.

- erik

Re: [9fans] "Blocks" in C

[Russ Cox]

> i'll grant you this in implementation.  the pool library's lock
> in effect becomes plan 9's BLK.  since the pool library is used
> in the kernel and user space, a user space application gets hit
> twice.  i've been doing some full-tilt boogie testing with 2x10gbe
> and even with 2 cores, the BLK^wpool lock is devastating.

what does BLK stand for?

russ

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-03 at 12:44 -0400, erik quanstrom wrote:
> On Thu Sep  3 12:20:09 EDT 2009, r...@sun.com wrote:
> > On Thu, 2009-09-03 at 11:54 -0400, erik quanstrom wrote:
> > > > Plan 9 has a lot to offer and a lot for others to learn from. 
> > > > Concurrency
> > > > framework that could scale up to 1K [virtual]cores in an SMP
> > > > configuration is not one of those features though.
> > > 
> > > forgive the ignorance, but is there any such thing as a
> > > 1k-core smp machine? 
> > 
> [...]
> > True. But even for those platforms good SMP frameworks are quite
> > difficult to come by. And here I do mean computation, not how
> > to accommodate scalable IO.
> 
> i'll grant you this in implementation.  the pool library's lock
> in effect becomes plan 9's BLK.  since the pool library is used
> in the kernel and user space, a user space application gets hit
> twice.  i've been doing some full-tilt boogie testing with 2x10gbe
> and even with 2 cores, the BLK^wpool lock is devastating.

I'm not surprised here. I'm also not surprised that it gets used
twice -- that's precisely what Sun does with slab allocator.

> where do you see the theoretical limitation in plan 9?

I don't think there's anything limiting plan 9, except for 
people willing to invest their time and coding skills in
tuning these things and coming up with programming frameworks
that take us further than manually juggling processes/threads.

My view is a bit skewed towards computations, and there its
all about your kernel scheduler and memory usage patterns.
At least at the low level it is. So I guess as long as these
things are ok, you can then build abstractions a'la GCD on
top of them.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-03 at 17:35 -0400, erik quanstrom wrote:
> On Thu Sep  3 17:09:01 EDT 2009, r...@sun.com wrote:
> > Anything can be done using regular C and threads. The trick here
> > is to make everything *scalable* and *painless* enough so that
> > mere mortals can start benefiting from parallelism in their code.
> > 
> > The other trick here is to find a model that makes things *natural*, and
> > that means practically no explicit locking, less shared state, etc.
> > 
> > The search for the model is meaningless unless it is used for
> > solving *practical* challenges. In that respect, one of my
> > favorite article is how implementation of a chess engine 
> > influenced Cilk framework (which almost has the notion of a "block")
> >http://supertech.csail.mit.edu/papers/icca99.pdf
> > 
> > Read it, I don't think we can be on the same page (and escape the
> > armchair philosophy trap) unless we are talking about practical
> > applications of the framework.
> > 
> > Look at the chess example -- can the same be done with pure C? Sure!
> > Did Cilk make it less painful? Absolutely!
> 
> my question was, what's naming your function pointers
> or not got to do with locking?  i'm asking about the language
> construct, not the library er i mean "framework" and maybe runtime
> that goes with it.

It has nothing to do with locking. You are correct. However, the style
of programming that it allows lets you build lock-free computations
much more *elegantly*. 

I use the word elegantly on purpose here. Since the discussion is
already in the personal taste territory all I can do to back my
claim is to note that regardless of whether blocks are closures 
or not (they are not) they are true lambdas.

Are lambdas useful? I sure think so, but I don't think I can convince
you by any other means except for offering links to "LAMBDA the
ultimate..." set of articles:
http://library.readscheme.org/page1.html

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[Daniel Lyons]

On Sep 3, 2009, at 10:02 AM, erik quanstrom wrote:


in c, i don't see why such a bolt-on would be useful in
c, especially since your concurrent fifo would be limited
to one shared-memory node unless you're going to add a runtime
compiler.



It's primarily an aesthetic benefit. From 
http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/13

- (IBAction)analyzeDocument:(NSButton *)sender
{
  NSDictionary *stats = [myDoc analyze];
  [myModel setDict:stats];
  [myStatsView setNeedsDisplay:YES];
  [stats release];
}

becomes

- (IBAction)analyzeDocument:(NSButton *)sender
{
  dispatch_async(dispatch_get_global_queue(0, 0), ^{
  NSDictionary *stats = [myDoc analyze];
  dispatch_async(dispatch_get_main_queue(), ^{
  [myModel setDict:stats];
  [myStatsView setNeedsDisplay:YES];
  [stats release];
});
  });
}

Two more lines of code, and it will execute that block in another  
thread without having to do much thinking about it.


—
Daniel Lyons

Re: [9fans] "Blocks" in C

[Daniel Lyons]

On Sep 3, 2009, at 9:44 AM, David Leimbach wrote:

I'm not 100% sure why the heck they did it this way, which is  
totally different from any other version of concurrent programming  
setup I've seen, except maybe that Apple likes to "think different"?


This API looks a lot to me like doing event-driven programming with  
jQuery. It lets you structure the code about the same as you would  
blocking, single-threaded code. That's the main benefit. The new stuff  
isn't intended to help you do Erlang-style concurrency or run on  
clusters. It's intended to help regular developers spawn a couple  
threads when it might help a little. Ease of use to the developer is  
priority 1 here.


Apple's strategy is, I think, fairly transparent: they want to make  
programming easier, and they want to make programming all of their  
different platforms seem similar. It works both intellectually (if  
they are similar, it's less to learn) and emotionally (it feels like a  
good investment to learn one, because it might at least apply to the  
others.) When you don't have the dominant platform, it's wise to cater  
to the laziest developers, because they are the ones who make the  
platform seem large and inviting to the customers. The more fun they  
make programming, the more it helps their bottom line.


—
Daniel Lyons

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 2:45 PM, David Leimbach  wrote:

>
>
> On Thu, Sep 3, 2009 at 2:35 PM, erik quanstrom wrote:
>
>> On Thu Sep  3 17:09:01 EDT 2009, r...@sun.com wrote:
>> > Anything can be done using regular C and threads. The trick here
>> > is to make everything *scalable* and *painless* enough so that
>> > mere mortals can start benefiting from parallelism in their code.
>> >
>> > The other trick here is to find a model that makes things *natural*, and
>> > that means practically no explicit locking, less shared state, etc.
>> >
>> > The search for the model is meaningless unless it is used for
>> > solving *practical* challenges. In that respect, one of my
>> > favorite article is how implementation of a chess engine
>> > influenced Cilk framework (which almost has the notion of a "block")
>> >http://supertech.csail.mit.edu/papers/icca99.pdf
>> >
>> > Read it, I don't think we can be on the same page (and escape the
>> > armchair philosophy trap) unless we are talking about practical
>> > applications of the framework.
>> >
>> > Look at the chess example -- can the same be done with pure C? Sure!
>> > Did Cilk make it less painful? Absolutely!
>>
>> my question was, what's naming your function pointers
>> or not got to do with locking?  i'm asking about the language
>> construct, not the library er i mean "framework" and maybe runtime
>> that goes with it.
>>
>>
> Maybe if you see the block implementation you wouldn't think it was merely
> naming a function pointer?
>
> http://clang.llvm.org/docs/BlockImplementation.txt
>
>
also

{
 int X;
 call_a_block(^(int y) {print (X+y); });
}

The block has a snapshot of that stack variable "X".

It really does work a bit more like a closure than a function pointer.

Dave


> Dave
>
>
>> - erik
>>
>>
>

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 2:35 PM, erik quanstrom wrote:

> On Thu Sep  3 17:09:01 EDT 2009, r...@sun.com wrote:
> > Anything can be done using regular C and threads. The trick here
> > is to make everything *scalable* and *painless* enough so that
> > mere mortals can start benefiting from parallelism in their code.
> >
> > The other trick here is to find a model that makes things *natural*, and
> > that means practically no explicit locking, less shared state, etc.
> >
> > The search for the model is meaningless unless it is used for
> > solving *practical* challenges. In that respect, one of my
> > favorite article is how implementation of a chess engine
> > influenced Cilk framework (which almost has the notion of a "block")
> >http://supertech.csail.mit.edu/papers/icca99.pdf
> >
> > Read it, I don't think we can be on the same page (and escape the
> > armchair philosophy trap) unless we are talking about practical
> > applications of the framework.
> >
> > Look at the chess example -- can the same be done with pure C? Sure!
> > Did Cilk make it less painful? Absolutely!
>
> my question was, what's naming your function pointers
> or not got to do with locking?  i'm asking about the language
> construct, not the library er i mean "framework" and maybe runtime
> that goes with it.
>
>
Maybe if you see the block implementation you wouldn't think it was merely
naming a function pointer?

http://clang.llvm.org/docs/BlockImplementation.txt

Dave


> - erik
>
>

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 2:49 PM, David Leimbach  wrote:

>
>
> On Thu, Sep 3, 2009 at 2:45 PM, David Leimbach  wrote:
>
>>
>>
>> On Thu, Sep 3, 2009 at 2:35 PM, erik quanstrom wrote:
>>
>>> On Thu Sep  3 17:09:01 EDT 2009, r...@sun.com wrote:
>>> > Anything can be done using regular C and threads. The trick here
>>> > is to make everything *scalable* and *painless* enough so that
>>> > mere mortals can start benefiting from parallelism in their code.
>>> >
>>> > The other trick here is to find a model that makes things *natural*,
>>> and
>>> > that means practically no explicit locking, less shared state, etc.
>>> >
>>> > The search for the model is meaningless unless it is used for
>>> > solving *practical* challenges. In that respect, one of my
>>> > favorite article is how implementation of a chess engine
>>> > influenced Cilk framework (which almost has the notion of a "block")
>>> >http://supertech.csail.mit.edu/papers/icca99.pdf
>>> >
>>> > Read it, I don't think we can be on the same page (and escape the
>>> > armchair philosophy trap) unless we are talking about practical
>>> > applications of the framework.
>>> >
>>> > Look at the chess example -- can the same be done with pure C? Sure!
>>> > Did Cilk make it less painful? Absolutely!
>>>
>>> my question was, what's naming your function pointers
>>> or not got to do with locking?  i'm asking about the language
>>> construct, not the library er i mean "framework" and maybe runtime
>>> that goes with it.
>>>
>>>
>> Maybe if you see the block implementation you wouldn't think it was merely
>> naming a function pointer?
>>
>> http://clang.llvm.org/docs/BlockImplementation.txt
>>
>>
> also
>
> {
>  int X;
>  call_a_block(^(int y) {print (X+y); });
> }
>
> The block has a snapshot of that stack variable "X".
>
> It really does work a bit more like a closure than a function pointer.
>
> Dave
>

Also this doc is the spec: http://clang.llvm.org/docs/BlockLanguageSpec.txt

>
>
>> Dave
>>
>>
>>> - erik
>>>
>>>
>>
>

Re: [9fans] "Blocks" in C

[James Tomaschke]

Roman V Shaposhnik wrote:
> On Thu, 2009-09-03 at 11:54 -0400, erik quanstrom wrote:

>> even commodity intel and amd mp offerings are numa.
>> they're not very n, but they're still n.
> 
> True. But even for those platforms good SMP frameworks are quite
> difficult to come by. And here I do mean computation, not how
> to accommodate scalable IO.

We have enjoyed our 4x quad system that we mainly use for sparse matrix
circuit solvers, though the current algorithms used don't seem to scale
well.  Instead of trying to run a single simulation on as many cores as
possible, we do many different simulations at once.  This trend might
move us towards more quadratic programming or differential evolution
where you are wanting to search a n-dimensional parameter space for
optimization.

Re: [9fans] "Blocks" in C

[erik quanstrom]

On Thu Sep  3 17:09:01 EDT 2009, r...@sun.com wrote:
> Anything can be done using regular C and threads. The trick here
> is to make everything *scalable* and *painless* enough so that
> mere mortals can start benefiting from parallelism in their code.
> 
> The other trick here is to find a model that makes things *natural*, and
> that means practically no explicit locking, less shared state, etc.
> 
> The search for the model is meaningless unless it is used for
> solving *practical* challenges. In that respect, one of my
> favorite article is how implementation of a chess engine 
> influenced Cilk framework (which almost has the notion of a "block")
>http://supertech.csail.mit.edu/papers/icca99.pdf
> 
> Read it, I don't think we can be on the same page (and escape the
> armchair philosophy trap) unless we are talking about practical
> applications of the framework.
> 
> Look at the chess example -- can the same be done with pure C? Sure!
> Did Cilk make it less painful? Absolutely!

my question was, what's naming your function pointers
or not got to do with locking?  i'm asking about the language
construct, not the library er i mean "framework" and maybe runtime
that goes with it.

- erik

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-03 at 15:36 -0400, erik quanstrom wrote:
> > > > Apple's using it all over the place in Snow Leopard, in all their native
> > > > apps to write cleaner, less manual-lock code.  At least, that's the 
> > > > claim
> > > > :-).
> > >
> > > could someone explain this to me?  i'm just missing how
> > > naming a block of code could change its locking properties.
> > >
> > >
> > The explanation is in the manual I linked to earlier in this discussion.  If
> > you want to see examples there's two I can think of available for download.
> >  One is called DispatchLife the other is DispatchFractal.
> > 
> > I've looked at DispatchLife, and there's no explicit locking of state for
> > every cell being concurrently update in Conway's game of life.
> 
> i can't find DispatchLife after a few minutes of googling.
> i've read the manual, and it looks like csp to me.  clearly
> i am a reprobate outside the apple reality distortion field.
> 
> could you explain why this isn't csp 

Please define what is it that you refer to as csp. The reason
I'm asking is simple: csp-like things are needed but *not*
sufficient for implementing a nice abstraction for the 
parallel framework.

> and why this can't be done
> with regular c (that is why we need the concept of an
> unnamed function pointer) and the thread library?

Anything can be done using regular C and threads. The trick here
is to make everything *scalable* and *painless* enough so that
mere mortals can start benefiting from parallelism in their code.

The other trick here is to find a model that makes things *natural*, and
that means practically no explicit locking, less shared state, etc.

The search for the model is meaningless unless it is used for
solving *practical* challenges. In that respect, one of my
favorite article is how implementation of a chess engine 
influenced Cilk framework (which almost has the notion of a "block")
   http://supertech.csail.mit.edu/papers/icca99.pdf

Read it, I don't think we can be on the same page (and escape the
armchair philosophy trap) unless we are talking about practical
applications of the framework.

Look at the chess example -- can the same be done with pure C? Sure!
Did Cilk make it less painful? Absolutely!

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[Iruata Souza]

On Thu, Sep 3, 2009 at 4:50 PM, David Leimbach wrote:
>
>
> On Thu, Sep 3, 2009 at 12:36 PM, erik quanstrom 
> wrote:
>>
>> > > > Apple's using it all over the place in Snow Leopard, in all their
>> > > > native
>> > > > apps to write cleaner, less manual-lock code.  At least, that's the
>> > > > claim
>> > > > :-).
>> > >
>> > > could someone explain this to me?  i'm just missing how
>> > > naming a block of code could change its locking properties.
>> > >
>> > >
>> > The explanation is in the manual I linked to earlier in this discussion.
>> >  If
>> > you want to see examples there's two I can think of available for
>> > download.
>> >  One is called DispatchLife the other is DispatchFractal.
>> >
>> > I've looked at DispatchLife, and there's no explicit locking of state
>> > for
>> > every cell being concurrently update in Conway's game of life.
>>
>> i can't find DispatchLife after a few minutes of googling.
>> i've read the manual, and it looks like csp to me.  clearly
>> i am a reprobate outside the apple reality distortion field.
>>
>
> Google doesn't have all the answers, I actually had to use Bing today, and
> it worked... anyway here's the link to DispatchLife.
> http://developer.apple.com/mac/library/samplecode/DispatchLife/
>
>>
>> could you explain why this isn't csp and why this can't be done
>> with regular c (that is why we need the concept of an
>> unnamed function pointer) and the thread library?
>
> I'm actually planning to figure this stuff out a bit more and "blog" about
> it, hopefully by Friday sometime (tomorrow).
> I don't agree that any of this stuff is strictly needed.  One can plod along
> with pthreads and do it wrong all day.  One doesn't *need* C either, I've
> seen whole OSes for x86 written in assembly.
> It all depends on how much crap you want to keep track of.
> Dave
>

or how much crap you want to dive into without noticing.

iru

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 12:36 PM, erik quanstrom wrote:

> > > > Apple's using it all over the place in Snow Leopard, in all their
> native
> > > > apps to write cleaner, less manual-lock code.  At least, that's the
> claim
> > > > :-).
> > >
> > > could someone explain this to me?  i'm just missing how
> > > naming a block of code could change its locking properties.
> > >
> > >
> > The explanation is in the manual I linked to earlier in this discussion.
>  If
> > you want to see examples there's two I can think of available for
> download.
> >  One is called DispatchLife the other is DispatchFractal.
> >
> > I've looked at DispatchLife, and there's no explicit locking of state for
> > every cell being concurrently update in Conway's game of life.
>
> i can't find DispatchLife after a few minutes of googling.
> i've read the manual, and it looks like csp to me.  clearly
> i am a reprobate outside the apple reality distortion field.
>
>
Google doesn't have all the answers, I actually had to use Bing today, and
it worked... anyway here's the link to DispatchLife.

http://developer.apple.com/mac/library/samplecode/DispatchLife/


> could you explain why this isn't csp and why this can't be done
> with regular c (that is why we need the concept of an
> unnamed function pointer) and the thread library?
>

I'm actually planning to figure this stuff out a bit more and "blog" about
it, hopefully by Friday sometime (tomorrow).

I don't agree that any of this stuff is strictly needed.  One can plod along
with pthreads and do it wrong all day.  One doesn't *need* C either, I've
seen whole OSes for x86 written in assembly.

It all depends on how much crap you want to keep track of.

Dave


>
> - erik
>
>

Re: [9fans] "Blocks" in C

[erik quanstrom]

> > > Apple's using it all over the place in Snow Leopard, in all their native
> > > apps to write cleaner, less manual-lock code.  At least, that's the claim
> > > :-).
> >
> > could someone explain this to me?  i'm just missing how
> > naming a block of code could change its locking properties.
> >
> >
> The explanation is in the manual I linked to earlier in this discussion.  If
> you want to see examples there's two I can think of available for download.
>  One is called DispatchLife the other is DispatchFractal.
> 
> I've looked at DispatchLife, and there's no explicit locking of state for
> every cell being concurrently update in Conway's game of life.

i can't find DispatchLife after a few minutes of googling.
i've read the manual, and it looks like csp to me.  clearly
i am a reprobate outside the apple reality distortion field.

could you explain why this isn't csp and why this can't be done
with regular c (that is why we need the concept of an
unnamed function pointer) and the thread library?

- erik

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 11:58 AM, erik quanstrom wrote:

> > Apple's using it all over the place in Snow Leopard, in all their native
> > apps to write cleaner, less manual-lock code.  At least, that's the claim
> > :-).
>
> could someone explain this to me?  i'm just missing how
> naming a block of code could change its locking properties.
>
>
The explanation is in the manual I linked to earlier in this discussion.  If
you want to see examples there's two I can think of available for download.
 One is called DispatchLife the other is DispatchFractal.

I've looked at DispatchLife, and there's no explicit locking of state for
every cell being concurrently update in Conway's game of life.

Dave


> - erik
>
>

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 9:02 AM, erik quanstrom wrote:

> > The blocks aren't interesting at all by themselves, I totally agree with
> > that.  However what they do to let you write a function inline, that can
> be
> > pushed to another function, to be executed on a concurrent FIFO, is where
> > the real power comes out.
>
> this reminds me of paul and byron's shell, es which had
> anonymous blocks.  in fact, that's how the if statement
> worked.
>
> in c, i don't see why such a bolt-on would be useful in
> c, especially since your concurrent fifo would be limited
> to one shared-memory node unless you're going to add a runtime
> compiler.
>
>
Apple's using it all over the place in Snow Leopard, in all their native
apps to write cleaner, less manual-lock code.  At least, that's the claim
:-).

If by node you mean "single machine" then I suppose I agree, but this is not
a distributed computing solution in the world of multi-node programming.

Dave


> - erik
>
>

Re: [9fans] "Blocks" in C

[erik quanstrom]

> Apple's using it all over the place in Snow Leopard, in all their native
> apps to write cleaner, less manual-lock code.  At least, that's the claim
> :-).

could someone explain this to me?  i'm just missing how
naming a block of code could change its locking properties.

- erik

Re: [9fans] "Blocks" in C

[erik quanstrom]

On Thu Sep  3 12:20:09 EDT 2009, r...@sun.com wrote:
> On Thu, 2009-09-03 at 11:54 -0400, erik quanstrom wrote:
> > > Plan 9 has a lot to offer and a lot for others to learn from. Concurrency
> > > framework that could scale up to 1K [virtual]cores in an SMP
> > > configuration is not one of those features though.
> > 
> > forgive the ignorance, but is there any such thing as a
> > 1k-core smp machine? 
> 
[...]
> True. But even for those platforms good SMP frameworks are quite
> difficult to come by. And here I do mean computation, not how
> to accommodate scalable IO.

i'll grant you this in implementation.  the pool library's lock
in effect becomes plan 9's BLK.  since the pool library is used
in the kernel and user space, a user space application gets hit
twice.  i've been doing some full-tilt boogie testing with 2x10gbe
and even with 2 cores, the BLK^wpool lock is devastating.

where do you see the theoretical limitation in plan 9?

- erik

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-03 at 11:54 -0400, erik quanstrom wrote:
> > Plan 9 has a lot to offer and a lot for others to learn from. Concurrency
> > framework that could scale up to 1K [virtual]cores in an SMP
> > configuration is not one of those features though.
> 
> forgive the ignorance, but is there any such thing as a
> 1k-core smp machine? 

I put "virtual" there on purpose. Here's what I can publically
share: http://www.tgdaily.com/content/view/43776/135/ and there's
more to come.

On a public front it is still 512 not 1K, though:
   4 cpus per box * 16 physical cores per cpu * 8 virtual threads per core

>  and is apple doing such a thing?

Nobody knows what Apple is doing for sure. I know 3 things though:
   1. Apple badly wants a share of a server market
   2. Apple is still doing POWER, though not releasing it (just like
  they did x86 for 7 or so years without telling anyone)
   3. POWER is getting to 1K SMP as well.
   
> even commodity intel and amd mp offerings are numa.
> they're not very n, but they're still n.

True. But even for those platforms good SMP frameworks are quite
difficult to come by. And here I do mean computation, not how
to accommodate scalable IO.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[erik quanstrom]

> The blocks aren't interesting at all by themselves, I totally agree with
> that.  However what they do to let you write a function inline, that can be
> pushed to another function, to be executed on a concurrent FIFO, is where
> the real power comes out.

this reminds me of paul and byron's shell, es which had
anonymous blocks.  in fact, that's how the if statement
worked.

in c, i don't see why such a bolt-on would be useful in
c, especially since your concurrent fifo would be limited
to one shared-memory node unless you're going to add a runtime
compiler.

- erik

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-03 at 08:44 -0700, David Leimbach wrote:

> The blocks aren't interesting at all by themselves, I totally agree
> with that.  However what they do to let you write a function inline,
> that can be pushed to another function, to be executed on a concurrent
> FIFO, is where the real power comes out.
> 
> 
> I'm not 100% sure why the heck they did it this way, which is totally
> different from any other version of concurrent programming setup I've
> seen, except maybe that Apple likes to "think different"?

It seems that quite a few concurrency frameworks worth the paper their
APIs are written on, are converging on this model. The ultimate goal
has to do with encapsulation of the computation into idempotent units
and describing the topology between those units. That separates the
executor from the semantics of computations nicely and has all sorts of
bonuses as far as bridging the gap between SMP and distributes systems
are concerned.

I think the semantics of what needs to be done is well understood. The
million dollar question is how to best express such a semantics in what
still looks like a programming language.

What Apple has done is one way of attacking the problem. Where I sit we
explore CPS for doing very similar sort of thing. One point is clear --
there no consensus yet.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[erik quanstrom]

> Plan 9 has a lot to offer and a lot for others to learn from. Concurrency
> framework that could scale up to 1K [virtual]cores in an SMP
> configuration is not one of those features though.

forgive the ignorance, but is there any such thing as a
1k-core smp machine?  and is apple doing such a thing?

even commodity intel and amd mp offerings are numa.
they're not very n, but they're still n.

- erik

Re: [9fans] "Blocks" in C

[Roman V Shaposhnik]

On Thu, 2009-09-03 at 17:32 +0200, Uriel wrote:
> So libthread must be a figment of 9fan's imagination...
> 
> Of course, for Apple (or anyone else) to learn from Plan 9 would be
> impossible, so instead they had to add a new 'feature' to C.

Plan 9 has a lot to offer and a lot for others to learn from. Concurrency
framework that could scale up to 1K [virtual]cores in an SMP
configuration is not one of those features though.

Thanks,
Roman.

Re: [9fans] "Blocks" in C

[David Leimbach]

On Thu, Sep 3, 2009 at 8:15 AM, Uriel  wrote:

> On Wed, Sep 2, 2009 at 4:20 PM, Devon H. O'Dell
> wrote:
> > 2009/9/2 Uriel :
> >> On Wed, Sep 2, 2009 at 10:04 AM, Anant Narayanan wrote:
> >>> Mac OS 10.6 introduced a new C compiler frontend (clang), which added
> >>> support for "blocks" in C [1]. Blocks basically add closures and
> anonymous
> >>> functions to C (and it's derivatives). Full details with examples are
> in the
> >>> linked article. I think the feature is quite elegant and might be
> useful in
> >>> cases where you want map/reduce like functionality in C.
> >>
> >> Er., I might be more dumb than usual, but why on earth would you
> >> need/want this garbage to get map/reduce functionality in C?
> >>
> >> To me it seems the typical "lets come up with some cute 'feature' and
> >> then we will figure out how to hype ourselves all the way to hell".
> >
> > I don't see why you'd particularly need / want this in C, but the
> > argument here seems silly given that you've stressed your affinity to
> > other languages that implement closures / anonymous functions.
>
> My affinity is to language that display *conceptual integrity*.
>
>
> > In any case, implementing closures in C isn't too difficult, and if
> > you want to return a function, just return a pointer to it.
>
> Exactly, I still fail to understand the point of this "feature",
> function points have worked fine for ages, but then I never understood
> any religion, and that is what Apple seems to be all about.
>

The blocks aren't interesting at all by themselves, I totally agree with
that.  However what they do to let you write a function inline, that can be
pushed to another function, to be executed on a concurrent FIFO, is where
the real power comes out.

I'm not 100% sure why the heck they did it this way, which is totally
different from any other version of concurrent programming setup I've seen,
except maybe that Apple likes to "think different"?

I'm not sure I think this stuff is "insanely great" though.  I'm planning to
try using it before judging it further though.

Dave


>
> Peace
>
> uriel
>
>

Re: [9fans] "Blocks" in C

[Uriel]

So libthread must be a figment of 9fan's imagination...

Of course, for Apple (or anyone else) to learn from Plan 9 would be
impossible, so instead they had to add a new 'feature' to C.

uriel



On Wed, Sep 2, 2009 at 5:07 PM, David Leimbach wrote:
> Has anyone actually looked at the spec or is this just armchair philosophy?
> I've actually looked at these, and used em a little bit.  They're not at all
> as bad as I once thought they could be, and the reason they're there is to
> work with a concurrency framework onto which blocks can be scheduled to run
> on various queues, to do concurrent programming.
> If you're running Snow Leopard, they're being used all over the place for
> thread management.
> Here's my objective and actually informed review:
> Can something go horribly wrong with these blocks if you don't use them
> properly?  You bet!  Imagine many concurrently running blocks playing with
> shared global pointers... Luckily the serial queues are able to prevent that
> exact thing from happening. :-)
> Does it make the code easier to read?  So far my answer is no, not at all.
>
> Does it succeed in the goal of making it possible to do lockless programming
> of concurrent applications?  Yes, there's several rather interesting
> examples showing the concurrent computation of all the cells in Conway's
> Life for example.  I'd say this beats the snot out of coding with pthreads.
> However, if I want real concurrency programming, I would NEVER use C over
> Erlang or Haskell by choice (or Limbo if it's available).
> Blocks themselves are really not terribly useful, you need the libdispatch
> library to make the real value in them come out, which does all the queue
> management by talking to the subsystem of Snow Leopard called "Grand Central
> Dispatch".
> Dave
>
> On Wed, Sep 2, 2009 at 4:40 AM, Eris Discordia 
> wrote:
>>
>> Perl people love closures. It's one of their common programming
>> techniques. Closures in C? Way to screw its clarity and closeness to the
>> real (or virtual) machine. And in the end closure or no closure doesn't
>> change how the binary looks but allows programmers to pepper source with
>> brain-teasers (now, what does _that_ evaluate to?). Not good at all.
>>
>> --On Wednesday, September 02, 2009 10:04 +0200 Anant Narayanan
>>  wrote:
>>
>>> Mac OS 10.6 introduced a new C compiler frontend (clang), which added
>>> support for "blocks" in C [1]. Blocks basically add closures and
>>> anonymous functions to C (and it's derivatives). Full details with
>>> examples are in the linked article. I think the feature is quite elegant
>>> and might be useful in cases where you want map/reduce like functionality
>>> in C.
>>>
>>> How much effort would it be to support a feature similar to blocks in 8c
>>> (and family)? What are your thoughts on the idea in general?
>>>
>>> --
>>> Anant
>>>
>>> [1] http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/10
>>>
>>
>>
>>
>>
>>
>
>

Re: [9fans] "Blocks" in C

[Uriel]

On Wed, Sep 2, 2009 at 4:20 PM, Devon H. O'Dell wrote:
> 2009/9/2 Uriel :
>> On Wed, Sep 2, 2009 at 10:04 AM, Anant Narayanan wrote:
>>> Mac OS 10.6 introduced a new C compiler frontend (clang), which added
>>> support for "blocks" in C [1]. Blocks basically add closures and anonymous
>>> functions to C (and it's derivatives). Full details with examples are in the
>>> linked article. I think the feature is quite elegant and might be useful in
>>> cases where you want map/reduce like functionality in C.
>>
>> Er., I might be more dumb than usual, but why on earth would you
>> need/want this garbage to get map/reduce functionality in C?
>>
>> To me it seems the typical "lets come up with some cute 'feature' and
>> then we will figure out how to hype ourselves all the way to hell".
>
> I don't see why you'd particularly need / want this in C, but the
> argument here seems silly given that you've stressed your affinity to
> other languages that implement closures / anonymous functions.

My affinity is to language that display *conceptual integrity*.


> In any case, implementing closures in C isn't too difficult, and if
> you want to return a function, just return a pointer to it.

Exactly, I still fail to understand the point of this "feature",
function points have worked fine for ages, but then I never understood
any religion, and that is what Apple seems to be all about.

Peace

uriel

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article ,
Akshat Kumar  wrote:
>Greg Comeau wrote:
>[... stuff ...]
>
>These four posts seem to be in indirect
>conversation with Charles Forsyth's,
>"one thing that gets me is that i've had
>people fulminate about the few minor
>changes in Plan 9's C compilers,
>because `they are not standard'"
>
>Perhaps he can further elaborate.

Not sure what you mean, but "for the record" definitely never had
any such conversation with Charles that I know of -- plus, it's
probably not my position, so should not be part of any conversation
I don't know of either :).  (FWIW, in these four posts, I did not
present my closure position, just trying to understand some of the
responses).

I'll probably google this string to see what you're referring to,
but I'll *guess* that Charles is probably referring to some people
feeling that Plan 9 C extensions are permission to open the floodgates.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?

Re: [9fans] "Blocks" in C

[Charles Forsyth]

>Perhaps he [me?] can further elaborate.

i certainly did not have comeau in mind.

Re: [9fans] "Blocks" in C

[Akshat Kumar]

Greg Comeau wrote:
[... stuff ...]

These four posts seem to be in indirect
conversation with Charles Forsyth's,
"one thing that gets me is that i've had
people fulminate about the few minor
changes in Plan 9's C compilers,
because `they are not standard'"

Perhaps he can further elaborate.


ak

Re: [9fans] "Blocks" in C

[Greg Comeau]

In article <5d375e920909020532p1c3bd46l75d89db4f2243...@mail.gmail.com>,
Uriel  wrote:
>On Wed, Sep 2, 2009 at 10:04 AM, Anant Narayanan wrote:
>> Mac OS 10.6 introduced a new C compiler frontend (clang), which added
>> support for "blocks" in C [1]. Blocks basically add closures and anonymous
>> functions to C (and it's derivatives). Full details with examples are in the
>> linked article. I think the feature is quite elegant and might be useful in
>> cases where you want map/reduce like functionality in C.
>
>Er., I might be more dumb than usual, but why on earth would you
>need/want this garbage to get map/reduce functionality in C?
>
>To me it seems the typical "lets come up with some cute 'feature' and
>then we will figure out how to hype ourselves all the way to hell".

Ok, I'll plan devil's advocate: This cute stuff has been around
for year and years and year already, just not in C.  Also,
cute stuff like designated initializers, compound literals
(which actually at least to me look like this closure-like stuff), etc.
in C I believe comes from Plan 9 C.  So, it seems to me that if
one wants compound literals then one probably would go for
something like this block stuff.   Certainly they do different
things, but, still, just a thought.
-- 
Greg Comeau / 4.3.10.1 with C++0xisms now in beta!
Comeau C/C++ ONLINE ==> http://www.comeaucomputing.com/tryitout
World Class Compilers:  Breathtaking C++, Amazing C99, Fabulous C90.
Comeau C/C++ with Dinkumware's Libraries... Have you tried it?