Re: What concurrency models to document?

[Leonardo Borges]

Hey,

A bit late to the party but something I'd love to see in the book is
an approachable summary/description/use cases of the main concurrency
models at play today: event-based and thread-based concurrency. I see
this section helping people compare Clojure with something like
Node.js or Ruby with EventMachine (Clojure has aleph[1] for that
purpose).

There is a lot of controversy around the topic and I believe even a
brief discussion on the subject will be beneficial.

This paper has a lot a good info:
http://static.usenix.org/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/
(it is biased towards a threaded model though).

Would this be way out of scope for your book? - Even not exactly what
you were after when you asked about concurrency models?

[1] https://github.com/ztellman/aleph/

Cheers,
Leonardo Borges
www.leonardoborges.com


On Fri, Aug 3, 2012 at 1:11 PM, cej38  wrote:
> I think that you have to talk about concurrency!  It is on everyone's mind.
> I would like to see the discussion go further than what I have seen in most
> other Clojure books.  If you are REALLY interested in concurrency, you are
> probably interested in looking at using more than one node in a cluster.
> Two areas that are always interested in concurrency are big data and high
> performance computing.  I come from a background where the only idea of how
> to do concurrency is through the use of MPI.  I would like to learn how to
> get nodes on a cluster to talk to each other within a Clojure enviroment.
>
> Further, a paragraph or two about what use cases each of the node
> interconnect models would work best for would be absolutely awesome.  As an
> example, I know that all of the following exist but I don't know when to use
> what (I know clojure-hadoop is NOT what I want for my use cases).
>
> clojure-hadoop
> swamiji
> cacalog
> zookeeper-clj
> storm
> Avout
> lein-condor
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[cej38]

I think that you have to talk about concurrency!  It is on everyone's mind. 
 I would like to see the discussion go further than what I have seen in 
most other Clojure books.  If you are REALLY interested in concurrency, you 
are probably interested in looking at using more than one node in a 
cluster.  Two areas that are always interested in concurrency are big data 
and high performance computing.  I come from a background where the only 
idea of how to do concurrency is through the use of MPI.  I would like to 
learn how to get nodes on a cluster to talk to each other within a Clojure 
enviroment.  

Further, a paragraph or two about what use cases each of the node 
interconnect models would work best for would be absolutely awesome.  As an 
example, I know that all of the following exist but I don't know when to 
use what (I know clojure-hadoop is NOT what I want for my use cases).


   - clojure-hadoop 
   - swamiji 
   - cacalog 
   - zookeeper-clj 
   - storm 
   - Avout
   - lein-condor

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Brian Hurt]

Adding my two cents:

I'd say immutablity, transactions/refs, futures, and actors.  I'd also talk
a lot about what I call the "four horsemen of the parallel apocalypse"-
race conditions, deadlocks, live locks, and priority inversions.  These are
all bugs that, while they *can* show up in senquential code, show up a heck
of a lot more often, and are bigger problems, in parallel code.  They are
also all "heisenbugs", in that they are all timing dependent, so
"inconsequential" changes in other parts of the code can change the
expression of, or even hide, the bug.  And they are not amenable for
classic testing or debugging strategies to handle.  Any mutable value needs
to come with an iron clad guarantee that either a) that value is only ever
accessed from a single thread, or b) that the code is properly synchronized
when it accesses that variable.  Any thing else will simply lead to tears
and recriminations.

Thus the importance of immutability as a default for most objects-
immutable objects don't need to be synchronized on.  Transactions and refs
come with guarantees of synchronization correctness, as do actors.  So to
do atoms, but in a less rigorous, and more error prone, way.  Futures and
actors are two ways to express parallelism.  For the record, I agree with
SPJ- I don't think there is one right way to express parallelism.

Brian

On Wed, Aug 1, 2012 at 8:29 PM, Brian Marick  wrote:

> I'm writing a book on /Functional Programming for the Object-Oriented
> Programmer/.  It uses Clojure as the teaching
> language.
>
> Since one of the things that attracts programmers to FP is the hope that
> ordinary mortals can write multicore programs, I want to have a chapter on
> concurrency. Of the various types of concurrency Clojure offers, which do
> you think would be most useful to explain? My inclination is:
> auto-concurrency due to immutability, futures (I love futures), and atoms.
> What do you think would be the right set?
>
> -
> Brian Marick, Artisanal Labrador
> Contract programming in Ruby and Clojure
> Occasional consulting on Agile
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Marshall T. Vandegrift]

Brian Marick  writes:

> Which raises the question: *is* concurrency actually a strong selling
> point for functional languages?

It definitely was/is for me.  I'd avoided threads for most of my
career-to-date because in imperative languages any thread could mutate
any value to be anything at any point.  This is difficult to reason
about when the code is written correctly, and then in C you have the
added fun that an error could lead to that malevolent thread writing to
absolutely any address in memory.  Complete undebuggable madness!

So I'd generally reached for processes in order to achieve concurrency.
Using processes doesn't get rid of deadlocks etc, and introduces
additional copying overhead if e.g. communicating over pipes, but does
shrink the problem one needs to think about down to just the set of
mechanisms explicitly used to communicate between the processes.

IMHO, FP in general and Clojure in particular does two big things
regarding concurrency:

  (1) By using immutable values, provides the same "conceptual
  shrinking" as using processes does in imperative languages, only
  without (well, with reduced) copying overhead.  The concurrency
  problem is limited to just the interactions over the relatively
  small number of explicit concurrency points.

  (2) The ability to build higher-order concurrency abstractions which
  completely encapsulate recurring patterns.  When using pure
  functions on immutable data, e.g. `pmap` and `reducers/fold`
  really do give you auto-concurrency for certain patterns of
  concurrent operations.

Just my 2 cents,

-Marshall

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Timothy Baldridge]

>
> > You have to put quite a bit of thought in to get things right.
>
> Which raises the question: *is* concurrency actually a strong selling
> point for functional languages?
>
>
Yes, but I would question the concept of "free". I would sell the
concurrency features thusly:

1) Assert that functions should be pure
2) Assert that all data should be immutable

Now, using that we can show how concurrency is easier (not free), to
implement using existing OOP concepts. For instance:

Var = [ThreadLocal] in C# (kindof)
Atom = Interlocked.CompareExchange  in .NET
Agent = Queue with a processing thread
Refs = um...yeah

All of these concurrency "features" don't mean a thing if your data itself
is mutable, or your functions impure. So it's not so much that thread
safety is simple in FP, it's more that it's almost impossible in OOP.

Timothy

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Ben Mabey]

On 8/2/12 10:04 AM, Brian Marick wrote:

On Aug 2, 2012, at 8:50 AM, Meikel Brandmeyer (kotarak) wrote:


You have to put quite a bit of thought in to get things right.

Which raises the question: *is* concurrency actually a strong selling point for 
functional languages?


Sane defaults are a selling point for functional languages.  _ Java 
Concurrency in Practice_ repeatedly talks about how immutability in many 
cases will make your concurrency problems simpler and more 
approachable.  As Meikel says, things will still require thought, but it 
will require less thought than if you are dealing with mutable state 
flying around everywhere. :)  As an aside, I think "unpure" languages 
like Clojure are really the best of both worlds.  You start out with a 
sane default but it allows you to deal with state in a uniform way when 
you need to and allows you to drop down to mutable state if performance 
requires it.


BTW, I love the idea for the book and I think it will help a lot of 
programmers understand the FP paradigm shift from mutable OO.


-Ben

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Jay Fields]

I don't find it to be one of the reasons I use Clojure... Nor do I
find it significantly easier than other options I've used in the past
(e.g. Java + Jetlang)

I feel like concurrency is the half-truth we tell to encourage
borderline adopters or spark interest. I don't believe it's a
motivating factor for adoption.

Of course, I could name 10+ other motivating reasons, but that's
probably a topic for another time...

Sent from my iPhone

On Aug 2, 2012, at 12:05 PM, Brian Marick  wrote:

>
> On Aug 2, 2012, at 8:50 AM, Meikel Brandmeyer (kotarak) wrote:
>
>> You have to put quite a bit of thought in to get things right.
>
> Which raises the question: *is* concurrency actually a strong selling point 
> for functional languages?
>
> In last year's state of clojure survey [*], concurrency ranked 6 out of 13 
> for the "biggest wins" question, but I don't know what conclusions to draw 
> from that.
>
> I added this chapter to the outline partly as a way to force myself to learn 
> more about functional concurrency. Perhaps I need to find another excuse.
>
>
> (*) 
> http://cemerick.com/2011/07/11/results-of-the-2011-state-of-clojure-survey/
>
> -
> Brian Marick, Artisanal Labrador
> Contract programming in Ruby and Clojure
> Occasional consulting on Agile
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Brian Marick]

On Aug 2, 2012, at 8:50 AM, Meikel Brandmeyer (kotarak) wrote:

> You have to put quite a bit of thought in to get things right. 

Which raises the question: *is* concurrency actually a strong selling point for 
functional languages? 

In last year's state of clojure survey [*], concurrency ranked 6 out of 13 for 
the "biggest wins" question, but I don't know what conclusions to draw from 
that.

I added this chapter to the outline partly as a way to force myself to learn 
more about functional concurrency. Perhaps I need to find another excuse.


(*) http://cemerick.com/2011/07/11/results-of-the-2011-state-of-clojure-survey/

-
Brian Marick, Artisanal Labrador
Contract programming in Ruby and Clojure
Occasional consulting on Agile


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Meikel Brandmeyer (kotarak)]

Hi,

I agree with the previous posters. Concurrency in Clojure is neither free 
nor automatic. You have to put quite a bit of thought in to get things 
right. I'm always reminded of the classic memoize 
discussion. 
Things like futures or promises take away boilerplate, but they don't 
protect you from eg. dead locks.

Kind regards
Meikel

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Jay Fields]

I find auto-concurrency in clojure to be misleadingly thrown around as
free. I think a good description of what is free would be helpful, and
examples of concurrency patterns that are still necessary above
clojure data structures.

A common example from my work is a screen that will need to display a
snapshot of data when opened and insert new data at the top on new
events. It's not acceptable to publish the same data twice or never,
it must be an immediately consistent view at all times. I've found
ways to do this with both agents (when some latency is acceptable) and
single threading the storing and publishing logic. I've also solved
this problem by always sending full snapshots of the data and removing
the data that's already known to be on the screen.

None of these solutions felt elegant.

Sent from my iPhone

On Aug 2, 2012, at 2:53 AM, Christian Sperandio
 wrote:

> Hi,
>
> I don't like the term auto-concurrency  because auto-concurrency
> doesn't exist. The immutability make easier the concurrency
> programming  because you don't have to manage the shared data but, you
> always have the synchronising management and write specific code for
> concurrency (using pmap, futures, promises and so on).
>
> About the types of concurrency, I think you should talk about ref and
> agent. Refs for their transactions management and agents because for
> people coming from OOP, They remember them things like actor pattern.
>
> Good luck for your book.
>
>
> Christian
>
> Le 2 août 2012 à 02:29, Brian Marick  a écrit :
>
>> I'm writing a book on /Functional Programming for the Object-Oriented 
>> Programmer/.  It uses Clojure as the teaching 
>> language.
>>
>> Since one of the things that attracts programmers to FP is the hope that 
>> ordinary mortals can write multicore programs, I want to have a chapter on 
>> concurrency. Of the various types of concurrency Clojure offers, which do 
>> you think would be most useful to explain? My inclination is: 
>> auto-concurrency due to immutability, futures (I love futures), and atoms. 
>> What do you think would be the right set?
>>
>> -
>> Brian Marick, Artisanal Labrador
>> Contract programming in Ruby and Clojure
>> Occasional consulting on Agile
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with your 
>> first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: What concurrency models to document?

[Christian Sperandio]

Hi,

I don't like the term auto-concurrency  because auto-concurrency
doesn't exist. The immutability make easier the concurrency
programming  because you don't have to manage the shared data but, you
always have the synchronising management and write specific code for
concurrency (using pmap, futures, promises and so on).

About the types of concurrency, I think you should talk about ref and
agent. Refs for their transactions management and agents because for
people coming from OOP, They remember them things like actor pattern.

Good luck for your book.


Christian

Le 2 août 2012 à 02:29, Brian Marick  a écrit :

> I'm writing a book on /Functional Programming for the Object-Oriented 
> Programmer/.  It uses Clojure as the teaching 
> language.
>
> Since one of the things that attracts programmers to FP is the hope that 
> ordinary mortals can write multicore programs, I want to have a chapter on 
> concurrency. Of the various types of concurrency Clojure offers, which do you 
> think would be most useful to explain? My inclination is: auto-concurrency 
> due to immutability, futures (I love futures), and atoms. What do you think 
> would be the right set?
>
> -
> Brian Marick, Artisanal Labrador
> Contract programming in Ruby and Clojure
> Occasional consulting on Agile
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

What concurrency models to document?

[Brian Marick]

I'm writing a book on /Functional Programming for the Object-Oriented 
Programmer/.  It uses Clojure as the teaching 
language.

Since one of the things that attracts programmers to FP is the hope that 
ordinary mortals can write multicore programs, I want to have a chapter on 
concurrency. Of the various types of concurrency Clojure offers, which do you 
think would be most useful to explain? My inclination is: auto-concurrency due 
to immutability, futures (I love futures), and atoms. What do you think would 
be the right set?

-
Brian Marick, Artisanal Labrador
Contract programming in Ruby and Clojure
Occasional consulting on Agile


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en