Re: [julia-users] Re: eachline() work with pmap() is slow

[lovebufan]

codes link 
 is the 
same link in my first reply. 

On Monday, October 17, 2016 at 12:47:15 AM UTC+8, Jeremy McNees wrote:
>
> Care to share the code you used?
>
> On Sun, Oct 16, 2016 at 9:24 AM > wrote:
>
>> Problem solved by hand writing an implementation of ``pmap`` & codes 
>> updated :)
>>
>>
At some point, I thought ``asyncmap(remote(process), iter)`` would solve 
the problem without handwriting a pmap (ref discussion here 
). However, when I test 
the code, it seems reading a lot of file contents without processing. That 
was confusing...

[julia-users] Re: eachline() work with pmap() is slow

[lovebufan]

Problem solved by hand writing an implementation of ``pmap`` & codes 
updated :)

[julia-users] Re: eachline() work with pmap() is slow

[lovebufan]

I have change the code to parallel on files rather than lines. codes are 
available here 
 if 
anyone have interests.
However, the speed is not satisfactory still (total processing speed 
approx. 10M/s, ideally it should be 100M/s, the network speed). 
CPU not full, IO not full, and I cannot find the bottleneck...

@Jeremy, thanks for the reply. The bottleneck is IO. You need days just to 
stream all files at full speed. Thus waiting to load the whole file will 
waste a lot of time. Ideally it will be that when I streamed the data one 
pass, the processing is also done without extra time.
@Páll, do you mean that pmap will first do a ``collect`` operation, then 
processing? So even you give pmap an iterator, it will not benefit from it? 
That will be sad. 

[julia-users] eachline() work with pmap() is slow

[lovebufan]

I want to process each line of a large text file (100G) in parallel using 
the following code

pmap(process_fun, eachline(the_file))

however, it seems that pmap is slow. following is a dummy experiment:

julia> writedlm("tmp.txt",rand(10,100)) # produce a large file
julia> @time for l in eachline("tmp.txt")
  split(l)
  end
  5.678517 seconds (11.00 M allocations: 732.637 MB, 40.67% gc time)

julia> addprocs() # 32 core

julia> @time map(split, eachline("tmp.txt"));
  4.834571 seconds (11.00 M allocations: 734.638 MB, 32.84% gc time)

julia> @time pmap(split, eachline("tmp.txt"));
112.275411 seconds (227.06 M allocations: 10.024 GB, 50.72% gc time)

the goal is to process those files (300+) as fast as possible. and maybe 
there are better ways to call pmap?

[julia-users] Re: How to convert a jpeg base64 encoded string to Image

[lovebufan]

no, it does not... just because "image/jpeg" will produce an error.

On Thursday, October 13, 2016 at 3:55:22 PM UTC+8, Andreas Lobinger wrote:
>
> Why does stringmime("image/png" produce jpeg?
>
> On Thursday, October 13, 2016 at 7:44:42 AM UTC+2, love...@gmail.com 
> wrote:
>>
>> I have some jpeg images saved as base64 encoded strings (such strings can 
>> be produced by ```stringmime("image/png", convert(Image, rand(5,5)))``` 
>> using Images.jl). but I tried a whole day and cannot convert them back 
>> (without disk I/O)...
>>
>

[julia-users] Re: How to convert a jpeg base64 encoded string to Image

[lovebufan]

Well, by trial & error, I found the correct function is ``readblob``.

On Thursday, October 13, 2016 at 1:44:42 PM UTC+8, love...@gmail.com wrote:
>
> I have some jpeg images saved as base64 encoded strings (such strings can 
> be produced by ```stringmime("image/png", convert(Image, rand(5,5)))``` 
> using Images.jl). but I tried a whole day and cannot convert them back 
> (without disk I/O)...
>

[julia-users] How to convert a jpeg base64 encoded string to Image

[lovebufan]

I have some jpeg images saved as base64 encoded strings (such strings can 
be produced by ```stringmime("image/png", convert(Image, rand(5,5)))``` 
using Images.jl). but I tried a whole day and cannot convert them back 
(without disk I/O)...

Re: [julia-users] Re: How to install 0.4.5 on Ubuntu?

[lovebufan]

Agree. 

On Saturday, July 16, 2016 at 8:40:18 AM UTC+8, Uwe Fechner wrote:
>
> Well, you can always pin a version. See: 
> https://help.ubuntu.com/community/PinningHowto
>
> For me, using Ubuntu packages is the easiest way to go, in particular if I 
> have to explain students how
> to install Julia. 
>
> Uwe
>
> On Friday, July 15, 2016 at 10:16:59 PM UTC+2, Tony Kelman wrote:
>>
>> The ppa is barely maintained. nightlies have not been working very well 
>> at all, and I wouldn't expect to see 0.5 packaged very rapidly in 
>> juliareleases. Not everyone uses ubuntu, and doing that packaging is effort 
>> that is better spent elsewhere. The counterpoint of automatic updates is 
>> that you should be prepared for your julia 0.4 install to get upgraded to 
>> 0.5 as soon as Elliot does get around to packaging it, which will likely 
>> break much of your code.
>
>

[julia-users] Re: What should Julia do on "Ubuntu compatibility mode in Windows 10"?

[lovebufan]

Just run Ubuntu version of Julia on Ubuntu mode on windows 10. 
Everything seems good.

On Sunday, April 3, 2016 at 3:27:04 AM UTC+8, Páll Haraldsson wrote:
>
> I guess just run as a regular Linux ELF binary..
>
> Since the news, and Windows Julia works, we can just ignore and run Julia 
> in non-Ubuntu mode..
>
> -- 
> Palli.
>
>

[julia-users] Re: What features are interesting in a VS Code plug-in?

[lovebufan]

any updates?

On Thursday, October 8, 2015 at 2:04:51 PM UTC+8, Tomas Lycken wrote:
>
> I've just been invited to an early look at a plug-in SDK for Visual Studio 
> Code, and I want to start experimenting with it in a plug-in for Julia 
> development. 
>
> What features are interesting in such a plug-in? 
>
> I have no idea what the SDK will support, so your favorite feature might 
> not be possible to implement (at least not at the moment) but I'd still 
> love to learn what features you all would find the most useful. 
>
> Hit me! :) 
>
> // T 
>
>

[julia-users] How to check whether a macro is defined

[lovebufan]

to suppress some annoying method overwritten warnings, I found a useful 
@nowarn macro.
However, I need to check whether this macro is defined and include the 
definition if not.

Question, how to check whether a macro is defined?

If one thread can ask multiple questions, I'm also interested in
how to suppress warnings? especially those relates to redefinition.

Thanks!