Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

I run the code on 0.3.0. It did not improve things (in fact, there was a 
3-5% deterioration)



On Tuesday, June 17, 2014 1:57:47 PM UTC-4, David Anthoff wrote:
>
> I submitted three pull requests to the original repo that get rid of three 
> different array allocations in loops and that make things a fair bit faster 
> altogether:
>
>  
>
> https://github.com/jesusfv/Comparison-Programming-Languages-Economics/pulls
>
>  
>
> I think it would also make sense to run these benchmarks on julia 0.3.0 
> instead of 0.2.1, given that there have been a fair number of performance 
> imrpovements.
>
>  
>
> *From:* julia...@googlegroups.com  [mailto:
> julia...@googlegroups.com ] *On Behalf Of *Florian Oswald
> *Sent:* Tuesday, June 17, 2014 10:50 AM
> *To:* julia...@googlegroups.com 
> *Subject:* Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < 
> Julia < Java < Matlab < the rest
>
>  
>
> thanks peter. I made that devectorizing change after dalua suggested so. 
> It made a massive difference!
>
> On Tuesday, 17 June 2014, Peter Simon > 
> wrote:
>
> You're right.  Replacing the NumericExtensions function calls with a small 
> loop
>
>  
>
> maxDifference  = 0.0
> for k = 1:length(mValueFunction)
> maxDifference = max(maxDifference, abs(mValueFunction[k]- 
> mValueFunctionNew[k]))
> end
>
>
> makes no significant difference in execution time or memory allocation and 
> eliminates the dependency.
>
>  
>
> --Peter
>
>
>
> On Tuesday, June 17, 2014 10:05:03 AM UTC-7, Andreas Noack Jensen wrote:
>
> ...but the Numba version doesn't use tricks like that. 
>
>  
>
> The uniform metric can also be calculated with a small loop. I think that 
> requiring dependencies is against the purpose of the exercise.
>
>  
>
> 2014-06-17 18:56 GMT+02:00 Peter Simon :
>
> As pointed out by Dahua, there is a lot of unnecessary memory allocation. 
>  This can be reduced significantly by replacing the lines
>
>  
>
> maxDifference  = maximum(abs(mValueFunctionNew-mValueFunction))
> mValueFunction= mValueFunctionNew
> mValueFunctionNew = zeros(nGridCapital,nGridProductivity)
>
>  
>
>  
>
> with
>
>  
>
> maxDifference  = maximum(abs!(subtract!(mValueFunction, 
> mValueFunctionNew)))
> (mValueFunction, mValueFunctionNew) = (mValueFunctionNew, 
> mValueFunction)
> fill!(mValueFunctionNew, 0.0)
>
>  
>
> abs! and subtract! require adding the line
>
>  
>
> using NumericExtensions
>
>  
>
> prior to the function line.  I think the OP used Julia 0.2; I don't 
> believe that NumericExtensions will work with that old version.  When I 
> combine these changes with adding 
>
>  
>
> @inbounds begin
> ...
> end
>
>  
>
> block around the "while" loop, I get about 25% reduction in execution 
> time, and reduction of memory allocation from roughly 700 MByte to 180 MByte
>
>  
>
> --Peter
>
>
>
> On Tuesday, June 17, 2014 9:32:34 AM UTC-7, John Myles White wrote:
>
> Sounds like we need to rerun these benchmarks after the new GC branch gets 
> updated.
>
>  
>
>  -- John
>
>  
>
> On Jun 17, 2014, at 9:31 AM, Stefan Karpinski  
> wrote:
>
>  
>
> That definitely smells like a GC issue. Python doesn't have this 
> particular problem since it uses reference counting.
>
>  
>
> On Tue, Jun 17, 2014 at 12:21 PM, Cristóvão Duarte Sousa  
> wrote:
>
> I've just done measurements of algorithm inner loop times in my machine by 
> changing the code has shown in this commit 
> 
> .
>
>  
>
> I've found out something... see for yourself:
>
>  
>
> using Winston
> numba_times = readdlm("numba_times.dat")[10:end];
> plot(numba_times)
>
>
> 
>
> julia_times = readdlm("julia_times.dat")[10:end];
> plot(julia_times)
>
>  
>
>
> 
>
> println((median(numba_times), mean(numba_times), var(numba_times)))
>
> (0.0028225183486938477,0.0028575707378805993,2.4830103817464292e-8)
>
>  
>
> println((median(julia_times), mean(julia_times), var(julia_times)))
>
> (0.00282404404,0.0034863882123824454,1.7058255003790299e-6)
>
>  
>
> So, while inner loop times have more or less the same median on both Julia 
> and Numba tests, the mean and variance are higher in Julia.
>
>  
>
> Can that be due to the garbage collector being kicking in?
>
>
>
> On Monday, June 16, 2014 4:52:07 PM UTC+1, Florian Oswald wrote:
>
> Dear all,
>
>  
>
> I thought you might find this paper interesting: 
> http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf
>
>  
>
> It takes a standard model from macro economics and computes it's solution 
> with an identical algorithm in several languages. Julia is roughly 2.6 
> times slower than the best C++ executable. 

Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

Thanks! I'll learn those tools. In any case, paper updated online, github 
page with new commit. This is really great. Nice example of aggregation of 
information. Economists love that :)

On Tuesday, June 17, 2014 9:11:08 AM UTC-4, Stefan Karpinski wrote:
>
> Not your fault at all. We need to make this kind of thing easier to 
> discover. Eg with
>
> https://github.com/astrieanna/TypeCheck.jl
>
> On Jun 17, 2014, at 8:35 AM, Jesus Villaverde  > wrote:
>
> Ah Sorry, over 20 years of coding in Matlab :(
>
> Yes, you are right, once I change that line, the type definition is 
> irrelevant. We should change the paper and the code ASAP
>
> On Tuesday, June 17, 2014 12:03:29 AM UTC-4, Peter Simon wrote:
>>
>> By a process of elimination, I determined that the only variable whose 
>> declaration affected the run time was vGridCapital.  The variable is 
>> declared to be of type Array{Float64,1}, but is initialized as
>>
>>
>> vGridCapital = 0.5*capitalSteadyState:0.1:1.5*capitalSteadyState
>>
>> which, unlike in Matlab, produces a Range object, rather than an array. 
>>  If the line above is modified to
>>
>> vGridCapital = [0.5*capitalSteadyState:0.1:1.5*capitalSteadyState]
>>
>> then the type instability is eliminated, and all type declarations can be 
>> removed with no effect on execution time.
>>
>> --Peter
>>
>>
>> On Monday, June 16, 2014 2:59:31 PM UTC-7, Jesus Villaverde wrote:
>>>
>>> Also, defining
>>>
>>> mylog(x::Float64) = ccall((:log, "libm"), Float64, (Float64,), x)
>>>
>>> made quite a bit of difference for me, from 1.92 to around 1.55. If I also 
>>> add @inbounds, I go down to 1.45, making Julia only twice as sslow as C++. 
>>> Numba still beats Julia, which kind of bothers me a bit
>>>
>>>
>>> Thanks for the suggestions.
>>>
>>>
>>> On Monday, June 16, 2014 4:56:34 PM UTC-4, Jesus Villaverde wrote:
>>>>
>>>> Hi
>>>>
>>>> 1) Yes, we pre-compiled the function.
>>>>
>>>> 2) As I mentioned before, we tried the code with and without type 
>>>> declaration, it makes a difference.
>>>>
>>>> 3) The variable names turns out to be quite useful because this code 
>>>> will be eventually nested into a much larger project where it is 
>>>> convenient 
>>>> to have very explicit names.
>>>>
>>>> Thanks 
>>>>
>>>> On Monday, June 16, 2014 12:13:44 PM UTC-4, Dahua Lin wrote:
>>>>>
>>>>> First, I agree with John that you don't have to declare the types in 
>>>>> general, like in a compiled language. It seems that Julia would be able 
>>>>> to 
>>>>> infer the types of most variables in your codes.
>>>>>
>>>>> There are several ways that your code's efficiency may be improved:
>>>>>
>>>>> (1) You can use @inbounds to waive bound checking in several places, 
>>>>> such as line 94 and 95 (in RBC_Julia.jl)
>>>>> (2) Line 114 and 116 involves reallocating new arrays, which is 
>>>>> probably unnecessary. Also note that Base.maxabs can compute the maximum 
>>>>> of 
>>>>> absolute value more efficiently than maximum(abs( ... ))
>>>>>
>>>>> In terms of measurement, did you pre-compile the function before 
>>>>> measuring the runtime?
>>>>>
>>>>> A side note about code style. It seems that it uses a lot of Java-ish 
>>>>> descriptive names with camel case. Julia practice tends to encourage more 
>>>>> concise naming.
>>>>>
>>>>> Dahua
>>>>>
>>>>>
>>>>>
>>>>> On Monday, June 16, 2014 10:55:50 AM UTC-5, John Myles White wrote:
>>>>>>
>>>>>> Maybe it would be good to verify the claim made at 
>>>>>> https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_Julia.jl#L9
>>>>>>  
>>>>>>
>>>>>> I would think that specifying all those types wouldn’t matter much if 
>>>>>> the code doesn’t have type-stability problems. 
>>>>>>
>>>>>>  — John 
>>>>>>
>>>>>> On Jun 16, 2014, at 8:52 AM, Florian Oswald  
>>>>>> wrote: 
>>>>>>
>>>>>> > Dear all, 
>>>>>> > 
>>>>>> > I thought you might find this paper interesting: 
>>>>>> http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf 
>>>>>> > 
>>>>>> > It takes a standard model from macro economics and computes it's 
>>>>>> solution with an identical algorithm in several languages. Julia is 
>>>>>> roughly 
>>>>>> 2.6 times slower than the best C++ executable. I was bit puzzled by the 
>>>>>> result, since in the benchmarks on http://julialang.org/, the 
>>>>>> slowest test is 1.66 times C. I realize that those benchmarks can't 
>>>>>> cover 
>>>>>> all possible situations. That said, I couldn't really find anything 
>>>>>> unusual 
>>>>>> in the Julia code, did some profiling and removed type inference, but 
>>>>>> still 
>>>>>> that's as fast as I got it. That's not to say that I'm disappointed, I 
>>>>>> still think this is great. Did I miss something obvious here or is there 
>>>>>> something specific to this algorithm? 
>>>>>> > 
>>>>>> > The codes are on github at 
>>>>>> > 
>>>>>> > 
>>>>>> https://github.com/jesusfv/Comparison-Programming-Languages-Economics 
>>>>>> > 
>>>>>> > 
>>>>>>
>>>>>>

Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

I think so! Matlab is just too slow for many things and a bit old in some 
dimensions. I often use C++ but for a lot of stuff, it is just to 
cumbersome.

On Tuesday, June 17, 2014 8:50:02 AM UTC-4, Bruno Rodrigues wrote:
>
> Hi Pr. Villaverde, just wanted to say that it was your paper that made me 
> try Julia. I must say that I am very happy with the switch! Will you 
> continue using Julia for your research?
>

Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

Ah Sorry, over 20 years of coding in Matlab :(

Yes, you are right, once I change that line, the type definition is 
irrelevant. We should change the paper and the code ASAP

On Tuesday, June 17, 2014 12:03:29 AM UTC-4, Peter Simon wrote:
>
> By a process of elimination, I determined that the only variable whose 
> declaration affected the run time was vGridCapital.  The variable is 
> declared to be of type Array{Float64,1}, but is initialized as
>
>
> vGridCapital = 0.5*capitalSteadyState:0.1:1.5*capitalSteadyState
>
> which, unlike in Matlab, produces a Range object, rather than an array. 
>  If the line above is modified to
>
> vGridCapital = [0.5*capitalSteadyState:0.1:1.5*capitalSteadyState]
>
> then the type instability is eliminated, and all type declarations can be 
> removed with no effect on execution time.
>
> --Peter
>
>
> On Monday, June 16, 2014 2:59:31 PM UTC-7, Jesus Villaverde wrote:
>>
>> Also, defining
>>
>> mylog(x::Float64) = ccall((:log, "libm"), Float64, (Float64,), x)
>>
>> made quite a bit of difference for me, from 1.92 to around 1.55. If I also 
>> add @inbounds, I go down to 1.45, making Julia only twice as sslow as C++. 
>> Numba still beats Julia, which kind of bothers me a bit
>>
>>
>> Thanks for the suggestions.
>>
>>
>> On Monday, June 16, 2014 4:56:34 PM UTC-4, Jesus Villaverde wrote:
>>>
>>> Hi
>>>
>>> 1) Yes, we pre-compiled the function.
>>>
>>> 2) As I mentioned before, we tried the code with and without type 
>>> declaration, it makes a difference.
>>>
>>> 3) The variable names turns out to be quite useful because this code 
>>> will be eventually nested into a much larger project where it is convenient 
>>> to have very explicit names.
>>>
>>> Thanks 
>>>
>>> On Monday, June 16, 2014 12:13:44 PM UTC-4, Dahua Lin wrote:
>>>>
>>>> First, I agree with John that you don't have to declare the types in 
>>>> general, like in a compiled language. It seems that Julia would be able to 
>>>> infer the types of most variables in your codes.
>>>>
>>>> There are several ways that your code's efficiency may be improved:
>>>>
>>>> (1) You can use @inbounds to waive bound checking in several places, 
>>>> such as line 94 and 95 (in RBC_Julia.jl)
>>>> (2) Line 114 and 116 involves reallocating new arrays, which is 
>>>> probably unnecessary. Also note that Base.maxabs can compute the maximum 
>>>> of 
>>>> absolute value more efficiently than maximum(abs( ... ))
>>>>
>>>> In terms of measurement, did you pre-compile the function before 
>>>> measuring the runtime?
>>>>
>>>> A side note about code style. It seems that it uses a lot of Java-ish 
>>>> descriptive names with camel case. Julia practice tends to encourage more 
>>>> concise naming.
>>>>
>>>> Dahua
>>>>
>>>>
>>>>
>>>> On Monday, June 16, 2014 10:55:50 AM UTC-5, John Myles White wrote:
>>>>>
>>>>> Maybe it would be good to verify the claim made at 
>>>>> https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_Julia.jl#L9
>>>>>  
>>>>>
>>>>> I would think that specifying all those types wouldn’t matter much if 
>>>>> the code doesn’t have type-stability problems. 
>>>>>
>>>>>  — John 
>>>>>
>>>>> On Jun 16, 2014, at 8:52 AM, Florian Oswald  
>>>>> wrote: 
>>>>>
>>>>> > Dear all, 
>>>>> > 
>>>>> > I thought you might find this paper interesting: 
>>>>> http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf 
>>>>> > 
>>>>> > It takes a standard model from macro economics and computes it's 
>>>>> solution with an identical algorithm in several languages. Julia is 
>>>>> roughly 
>>>>> 2.6 times slower than the best C++ executable. I was bit puzzled by the 
>>>>> result, since in the benchmarks on http://julialang.org/, the slowest 
>>>>> test is 1.66 times C. I realize that those benchmarks can't cover all 
>>>>> possible situations. That said, I couldn't really find anything unusual 
>>>>> in 
>>>>> the Julia code, did some profiling and removed type inference, but still 
>>>>> that's as fast as I got it. That's not to say that I'm disappointed, I 
>>>>> still think this is great. Did I miss something obvious here or is there 
>>>>> something specific to this algorithm? 
>>>>> > 
>>>>> > The codes are on github at 
>>>>> > 
>>>>> > 
>>>>> https://github.com/jesusfv/Comparison-Programming-Languages-Economics 
>>>>> > 
>>>>> > 
>>>>>
>>>>>

Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

Also, defining

mylog(x::Float64) = ccall((:log, "libm"), Float64, (Float64,), x)

made quite a bit of difference for me, from 1.92 to around 1.55. If I also add 
@inbounds, I go down to 1.45, making Julia only twice as sslow as C++. Numba 
still beats Julia, which kind of bothers me a bit


Thanks for the suggestions.


On Monday, June 16, 2014 4:56:34 PM UTC-4, Jesus Villaverde wrote:
>
> Hi
>
> 1) Yes, we pre-compiled the function.
>
> 2) As I mentioned before, we tried the code with and without type 
> declaration, it makes a difference.
>
> 3) The variable names turns out to be quite useful because this code will 
> be eventually nested into a much larger project where it is convenient to 
> have very explicit names.
>
> Thanks 
>
> On Monday, June 16, 2014 12:13:44 PM UTC-4, Dahua Lin wrote:
>>
>> First, I agree with John that you don't have to declare the types in 
>> general, like in a compiled language. It seems that Julia would be able to 
>> infer the types of most variables in your codes.
>>
>> There are several ways that your code's efficiency may be improved:
>>
>> (1) You can use @inbounds to waive bound checking in several places, such 
>> as line 94 and 95 (in RBC_Julia.jl)
>> (2) Line 114 and 116 involves reallocating new arrays, which is probably 
>> unnecessary. Also note that Base.maxabs can compute the maximum of absolute 
>> value more efficiently than maximum(abs( ... ))
>>
>> In terms of measurement, did you pre-compile the function before 
>> measuring the runtime?
>>
>> A side note about code style. It seems that it uses a lot of Java-ish 
>> descriptive names with camel case. Julia practice tends to encourage more 
>> concise naming.
>>
>> Dahua
>>
>>
>>
>> On Monday, June 16, 2014 10:55:50 AM UTC-5, John Myles White wrote:
>>>
>>> Maybe it would be good to verify the claim made at 
>>> https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_Julia.jl#L9
>>>  
>>>
>>> I would think that specifying all those types wouldn’t matter much if 
>>> the code doesn’t have type-stability problems. 
>>>
>>>  — John 
>>>
>>> On Jun 16, 2014, at 8:52 AM, Florian Oswald  
>>> wrote: 
>>>
>>> > Dear all, 
>>> > 
>>> > I thought you might find this paper interesting: 
>>> http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf 
>>> > 
>>> > It takes a standard model from macro economics and computes it's 
>>> solution with an identical algorithm in several languages. Julia is roughly 
>>> 2.6 times slower than the best C++ executable. I was bit puzzled by the 
>>> result, since in the benchmarks on http://julialang.org/, the slowest 
>>> test is 1.66 times C. I realize that those benchmarks can't cover all 
>>> possible situations. That said, I couldn't really find anything unusual in 
>>> the Julia code, did some profiling and removed type inference, but still 
>>> that's as fast as I got it. That's not to say that I'm disappointed, I 
>>> still think this is great. Did I miss something obvious here or is there 
>>> something specific to this algorithm? 
>>> > 
>>> > The codes are on github at 
>>> > 
>>> > https://github.com/jesusfv/Comparison-Programming-Languages-Economics 
>>> > 
>>> > 
>>>
>>>

Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

Hi

1) Yes, we pre-compiled the function.

2) As I mentioned before, we tried the code with and without type 
declaration, it makes a difference.

3) The variable names turns out to be quite useful because this code will 
be eventually nested into a much larger project where it is convenient to 
have very explicit names.

Thanks 

On Monday, June 16, 2014 12:13:44 PM UTC-4, Dahua Lin wrote:
>
> First, I agree with John that you don't have to declare the types in 
> general, like in a compiled language. It seems that Julia would be able to 
> infer the types of most variables in your codes.
>
> There are several ways that your code's efficiency may be improved:
>
> (1) You can use @inbounds to waive bound checking in several places, such 
> as line 94 and 95 (in RBC_Julia.jl)
> (2) Line 114 and 116 involves reallocating new arrays, which is probably 
> unnecessary. Also note that Base.maxabs can compute the maximum of absolute 
> value more efficiently than maximum(abs( ... ))
>
> In terms of measurement, did you pre-compile the function before measuring 
> the runtime?
>
> A side note about code style. It seems that it uses a lot of Java-ish 
> descriptive names with camel case. Julia practice tends to encourage more 
> concise naming.
>
> Dahua
>
>
>
> On Monday, June 16, 2014 10:55:50 AM UTC-5, John Myles White wrote:
>>
>> Maybe it would be good to verify the claim made at 
>> https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_Julia.jl#L9
>>  
>>
>> I would think that specifying all those types wouldn’t matter much if the 
>> code doesn’t have type-stability problems. 
>>
>>  — John 
>>
>> On Jun 16, 2014, at 8:52 AM, Florian Oswald  
>> wrote: 
>>
>> > Dear all, 
>> > 
>> > I thought you might find this paper interesting: 
>> http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf 
>> > 
>> > It takes a standard model from macro economics and computes it's 
>> solution with an identical algorithm in several languages. Julia is roughly 
>> 2.6 times slower than the best C++ executable. I was bit puzzled by the 
>> result, since in the benchmarks on http://julialang.org/, the slowest 
>> test is 1.66 times C. I realize that those benchmarks can't cover all 
>> possible situations. That said, I couldn't really find anything unusual in 
>> the Julia code, did some profiling and removed type inference, but still 
>> that's as fast as I got it. That's not to say that I'm disappointed, I 
>> still think this is great. Did I miss something obvious here or is there 
>> something specific to this algorithm? 
>> > 
>> > The codes are on github at 
>> > 
>> > https://github.com/jesusfv/Comparison-Programming-Languages-Economics 
>> > 
>> > 
>>
>>

Re: [julia-users] Benchmarking study: C++ < Fortran < Numba < Julia < Java < Matlab < the rest

[Jesus Villaverde]

Hi

I am one of the authors of the paper :)

Our first version of the code did not declare types. It was thanks to 
Florian's suggestion that we started doing it. We discovered, to our 
surprise, that it reduced execution time by around 25%. I may be mistaken 
but I do not think there are type-stability problems. We have a version of 
the code that is nearly identical in C++ and we did not have any of those 
type problems.

On Monday, June 16, 2014 11:55:50 AM UTC-4, John Myles White wrote:
>
> Maybe it would be good to verify the claim made at 
> https://github.com/jesusfv/Comparison-Programming-Languages-Economics/blob/master/RBC_Julia.jl#L9
>  
>
> I would think that specifying all those types wouldn’t matter much if the 
> code doesn’t have type-stability problems. 
>
>  — John 
>
> On Jun 16, 2014, at 8:52 AM, Florian Oswald  > wrote: 
>
> > Dear all, 
> > 
> > I thought you might find this paper interesting: 
> http://economics.sas.upenn.edu/~jesusfv/comparison_languages.pdf 
> > 
> > It takes a standard model from macro economics and computes it's 
> solution with an identical algorithm in several languages. Julia is roughly 
> 2.6 times slower than the best C++ executable. I was bit puzzled by the 
> result, since in the benchmarks on http://julialang.org/, the slowest 
> test is 1.66 times C. I realize that those benchmarks can't cover all 
> possible situations. That said, I couldn't really find anything unusual in 
> the Julia code, did some profiling and removed type inference, but still 
> that's as fast as I got it. That's not to say that I'm disappointed, I 
> still think this is great. Did I miss something obvious here or is there 
> something specific to this algorithm? 
> > 
> > The codes are on github at 
> > 
> > https://github.com/jesusfv/Comparison-Programming-Languages-Economics 
> > 
> > 
>
>