Re: [julia-users] Re: eachline() work with pmap() is slow

2016-10-16 Thread lovebufan
codes link is the same link in my first reply. On Monday, October 17, 2016 at 12:47:15 AM UTC+8, Jeremy McNees wrote: > > Care to share the code you used? > > On Sun, Oct 16, 2016 at 9:24 AM wrote: > >>

[julia-users] Re: eachline() work with pmap() is slow

2016-10-16 Thread Páll Haraldsson
On Sunday, October 16, 2016 at 5:57:49 PM UTC, Páll Haraldsson wrote: > > > > On Saturday, October 15, 2016 at 2:54:53 AM UTC, love...@gmail.com wrote: >> >> @Páll, do you mean that pmap will first do a ``collect`` operation, >> > > yes and no.. > Should have said here, just no. I meant, yes,

[julia-users] Re: eachline() work with pmap() is slow

2016-10-16 Thread Páll Haraldsson
On Saturday, October 15, 2016 at 2:54:53 AM UTC, love...@gmail.com wrote: > > I have change the code to parallel on files rather than lines. codes are > available here > if > anyone have interests. > However, the speed is not

Re: [julia-users] Re: eachline() work with pmap() is slow

2016-10-16 Thread Jeremy McNees
Care to share the code you used? On Sun, Oct 16, 2016 at 9:24 AM wrote: > Problem solved by hand writing an implementation of ``pmap`` & codes > updated :) > >

[julia-users] Re: eachline() work with pmap() is slow

2016-10-16 Thread lovebufan
Problem solved by hand writing an implementation of ``pmap`` & codes updated :)

[julia-users] Re: eachline() work with pmap() is slow

2016-10-14 Thread lovebufan
I have change the code to parallel on files rather than lines. codes are available here if anyone have interests. However, the speed is not satisfactory still (total processing speed approx. 10M/s, ideally it should be 100M/s,

[julia-users] Re: eachline() work with pmap() is slow

2016-10-14 Thread Páll Haraldsson
On Friday, October 14, 2016 at 3:45:36 AM UTC, love...@gmail.com wrote: > > I want to process each line of a large text file (100G) in parallel using > the following code > > pmap(process_fun, eachline(the_file)) > > however, it seems that pmap is slow. following is a dummy experiment: > >

[julia-users] Re: eachline() work with pmap() is slow

2016-10-14 Thread Jeremy McNees
I need to run something similar due to a large number of text files that I have. They are too large to load into memory at one-time, let alone multiple files at the same time. I find that pmap() works very well here. First, you should wrap your for loop in a function. In general you should

[julia-users] Re: eachline() work with pmap() is slow

2016-10-14 Thread Jeremy McNees
I need to run something similar due to a large number of text files that I have. They are too large to load into memory at one-time, let alone multiple files at the same time. I find that pmap() works very well here. First, you should wrap your for loop in a function. In general you should