[Chicken-users] Parallel procedures in CHICKEN
Hi! Is there a way to implement map, for-each and other procedures in a parallel way so (use srfi-1) (map (lambda (x) (+ x 1)) (iota 100) would automatically split the list into smaller lists according to the number of CPU cores and then gather the results back? Thanks in advance. Cheers, Arthur ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Parallel procedures in CHICKEN
On Sat, 24 Dec 2016 02:11:37 -0200 Arthur Maciel wrote: > Is there a way to implement map, for-each and other procedures in a > parallel way so > > (use srfi-1) > (map (lambda (x) (+ x 1)) (iota 100) > > would automatically split the list into smaller lists according to the > number of CPU cores and then gather the results back? I guess you could spawn a process pool and send these processes a thunk that calculates their part and send back the result. You could use s11n egg for that, I believe. I’m not sure it would be faster than the regular functions though. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Parallel procedures in CHICKEN
Hi Kooda! Em 24 de dez de 2016 07:00, "Kooda" escreveu: On Sat, 24 Dec 2016 02:11:37 -0200 Arthur Maciel wrote: > Is there a way to implement map, for-each and other procedures in a > parallel way so > > (use srfi-1) > (map (lambda (x) (+ x 1)) (iota 100) > > would automatically split the list into smaller lists according to the > number of CPU cores and then gather the results back? I guess you could spawn a process pool and send these processes a thunk that calculates their part and send back the result. You could use s11n egg for that, I believe. I’m not sure it would be faster than the regular functions though. Do you recommend any specific way to create the pool and especially to communicate between the processes? About the speed, I'll test and report the results. Thanks! Arthur ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Parallel procedures in CHICKEN
Hi Arthur, You might find this bit of exploratory code useful: http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54. It is an example of spawning off dozens of sub-processes and using nanomsg to communicate the data back. We needed to find changed files in gigs of data where the originating process would not be able to directly see the files (the program will be setuid). With this proof of concept code on a 32 processor machine we saw a task that would take hours drop to minutes. No surprise there but it was nice that it worked. It might be handy if this idea was abstracted into an egg with special versions of map, for-each etc. but I don't have time to attempt that now. Nanomsg is pretty neat but it has quirks. To work reliably all open and closing of ports had to be protected with mutexes which seems very odd. Note: the code was only a proof of concept trial and it probably won't work for you out of the box. It is nice that with nanomsg you simply change the connection URL to switch from in process communication to inter-process to across hosts. We saw no difference between IPC and tcp but we were only handing back and forth tiny amounts of data so that is to be expected. Matt -=- On Wed, Dec 28, 2016 at 8:46 AM, Arthur Maciel wrote: > Hi Kooda! > > Em 24 de dez de 2016 07:00, "Kooda" escreveu: > > On Sat, 24 Dec 2016 02:11:37 -0200 > Arthur Maciel wrote: > > Is there a way to implement map, for-each and other procedures in a > > parallel way so > > > > (use srfi-1) > > (map (lambda (x) (+ x 1)) (iota 100) > > > > would automatically split the list into smaller lists according to the > > number of CPU cores and then gather the results back? > > I guess you could spawn a process pool and send these processes a thunk > that calculates their part and send back the result. You could use s11n > egg for that, I believe. > > I’m not sure it would be faster than the regular functions though. > > > Do you recommend any specific way to create the pool and especially to > communicate between the processes? > > About the speed, I'll test and report the results. > > Thanks! > Arthur > > ___ > Chicken-users mailing list > Chicken-users@nongnu.org > https://lists.nongnu.org/mailman/listinfo/chicken-users > > ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Parallel procedures in CHICKEN
Hi Matt! Thank you very much for the code! If I have any success I'll report here. Have a Happy New Year! Cheers, Arthur Em 28 de dez de 2016 15:29, "Matt Welland" escreveu: > Hi Arthur, > > You might find this bit of exploratory code useful: > http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54. > It is an example of spawning off dozens of sub-processes and using nanomsg > to communicate the data back. We needed to find changed files in gigs of > data where the originating process would not be able to directly see the > files (the program will be setuid). With this proof of concept code on a 32 > processor machine we saw a task that would take hours drop to minutes. No > surprise there but it was nice that it worked. It might be handy if this > idea was abstracted into an egg with special versions of map, for-each etc. > but I don't have time to attempt that now. > > Nanomsg is pretty neat but it has quirks. To work reliably all open and > closing of ports had to be protected with mutexes which seems very odd. > Note: the code was only a proof of concept trial and it probably won't work > for you out of the box. It is nice that with nanomsg you simply change the > connection URL to switch from in process communication to inter-process to > across hosts. We saw no difference between IPC and tcp but we were only > handing back and forth tiny amounts of data so that is to be expected. > > Matt > -=- > > On Wed, Dec 28, 2016 at 8:46 AM, Arthur Maciel > wrote: > >> Hi Kooda! >> >> Em 24 de dez de 2016 07:00, "Kooda" escreveu: >> >> On Sat, 24 Dec 2016 02:11:37 -0200 >> Arthur Maciel wrote: >> > Is there a way to implement map, for-each and other procedures in a >> > parallel way so >> > >> > (use srfi-1) >> > (map (lambda (x) (+ x 1)) (iota 100) >> > >> > would automatically split the list into smaller lists according to the >> > number of CPU cores and then gather the results back? >> >> I guess you could spawn a process pool and send these processes a thunk >> that calculates their part and send back the result. You could use s11n >> egg for that, I believe. >> >> I’m not sure it would be faster than the regular functions though. >> >> >> Do you recommend any specific way to create the pool and especially to >> communicate between the processes? >> >> About the speed, I'll test and report the results. >> >> Thanks! >> Arthur >> >> ___ >> Chicken-users mailing list >> Chicken-users@nongnu.org >> https://lists.nongnu.org/mailman/listinfo/chicken-users >> >> > ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Parallel procedures in CHICKEN
You could build something with the posix-shm egg, and the posix unit. If you need locks, there’s the posix-semaphore egg. -Dan From: Chicken-users [mailto:chicken-users-bounces+dan=ironoxide...@nongnu.org] On Behalf Of Arthur Maciel Sent: December 28, 2016 7:47 AM To: Kooda Cc: chicken-users Subject: Re: [Chicken-users] Parallel procedures in CHICKEN Hi Kooda! Em 24 de dez de 2016 07:00, "Kooda" mailto:ko...@upyum.com> > escreveu: On Sat, 24 Dec 2016 02:11:37 -0200 Arthur Maciel mailto:arthurmac...@gmail.com> > wrote: > Is there a way to implement map, for-each and other procedures in a > parallel way so > > (use srfi-1) > (map (lambda (x) (+ x 1)) (iota 100) > > would automatically split the list into smaller lists according to the > number of CPU cores and then gather the results back? I guess you could spawn a process pool and send these processes a thunk that calculates their part and send back the result. You could use s11n egg for that, I believe. I’m not sure it would be faster than the regular functions though. Do you recommend any specific way to create the pool and especially to communicate between the processes? About the speed, I'll test and report the results. Thanks! Arthur ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Parallel procedures in CHICKEN
Thanks for pointing it, Dan! Do you know if it is supported by CHICKEN on Windows too? Cheers, Arthur Em 30 de dez de 2016 03:43, "Dan Leslie" escreveu: > You could build something with the posix-shm egg, and the posix unit. If > you need locks, there’s the posix-semaphore egg. > > > > -Dan > > > > *From:* Chicken-users [mailto:chicken-users-bounces+dan= > ironoxide...@nongnu.org] *On Behalf Of *Arthur Maciel > *Sent:* December 28, 2016 7:47 AM > *To:* Kooda > *Cc:* chicken-users > *Subject:* Re: [Chicken-users] Parallel procedures in CHICKEN > > > > Hi Kooda! > > > > Em 24 de dez de 2016 07:00, "Kooda" escreveu: > > On Sat, 24 Dec 2016 02:11:37 -0200 > Arthur Maciel wrote: > > Is there a way to implement map, for-each and other procedures in a > > parallel way so > > > > (use srfi-1) > > (map (lambda (x) (+ x 1)) (iota 100) > > > > would automatically split the list into smaller lists according to the > > number of CPU cores and then gather the results back? > > I guess you could spawn a process pool and send these processes a thunk > that calculates their part and send back the result. You could use s11n > egg for that, I believe. > > I’m not sure it would be faster than the regular functions though. > > > > Do you recommend any specific way to create the pool and especially to > communicate between the processes? > > > > About the speed, I'll test and report the results. > > > > Thanks! > > Arthur > ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users