On 4 Sep 2010, at 20:42, Joe Vornehm Jr. wrote:
> I think corresponding with the authors is the best approach. It is
> too late to change the article since it's already in press. The
> authors may choose to publish a comment clarifying the results. If
> Martin is still dissatisfied with their response, perhaps he might
> submit a comment to the journal (author instructions here).
>
> I feel there are some additional issues in this paper, even beyond
> the possible benchmark errors. There are major or minor
> inaccuracies in almost every section that mentions Octave. I agree
> that they should have been using Octave 3.2.x, which was available
> well before the January submission date listed on the paper. I
> don't know how they didn't find the documentation website, and
> anyway, I don't know why "help" and "lookfor" don't qualify as
> online documentation and indexing, or why they are considered "not
> user-friendly." They neglected Java support and the packages
> available via Octave-Forge. I do wish they would make their
> methodology (especially their code) public so it can be inspected.
> And I hope they compared the numerical results of the different
> products to make sure their code was correct.
>
> But to put my concerns in the proper context, I think it's an
> excellent idea for a paper, and I'm glad they pursued it. It's just
> the kind of reference that is needed for scientists and others
> choosing a computational package. I hope they are willing to work
> through the issues being raised here.
>
> Joe V.
In case anyone wants to check the results shown in that paper I
prepared a script performing (what is my interpretetion of)
the benchmarks listed in the tables, this could be helpful for keeping
the discussion with the authors on technical grounds rather than
starting a flamewar.
I also reported the results I obtained on my 2.5 years old core2-duo
macbook running OS X 10.5.8 and Octave.app 3.2.3.
My guess is that those guys somehow got a version of Octave linked to
a poor implementation of BLAS/LAPACK
HTH,
c.
P.S. tests from the statistics section are missing as I went out of
time, sorry
----------8<-----------
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Miscellaneous operations freemat mathnium
octave R Scilab
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Loop test 10,000 × 10,000 601.606 798.788 1526.000
261.077 271.713
tic, for ii=1:1e4, for jj=1:1e4, end, end, toc()
%Elapsed time is 14.674 seconds.
% 2000 × 2000 random matrix^1000 1.573 3.886
0.592 0.745 29.398
a = rand(2000);
tic, a^1e3; toc()
%Elapsed time is 23.488 seconds.
tic, a.^1e3; toc()
%Elapsed time is 0.42365 seconds.
% Sorting of 5,000,000 random values 4.545 94.692
1.581 1.449 2.300
a = rand (5e6, 1);
tic (), sort (a); toc ()
%Elapsed time is 1.2075 seconds.
% FFT over 220 random values 0.405 23.912
0.137 0.763 0.991
a = rand (220, 1);
tic (), fft (a); toc ()
%Elapsed time is 0.00012302 seconds.
% Calculation of 2,000,000 Fibonacci numbers 1.798 81.205
2.514 0.430 3.047
nf = 2e6;
%with for loop
tic; fib = ones (nf, 1); for ii=3:nf; fib(ii) = fib(ii-1)+fib(ii-2);
end; toc()
%Elapsed time is 39.529878 seconds.
%with filter
tic (); x = [1; zeros(nf-1, 1)]; a = [1 -1 -1]; b = 1; fibfil =
filter(b, a, x); toc ()
%Elapsed time is 0.8 seconds.
isequal (fib, fibfil)
%ans = 1
% Factorial of a big integer (10 digits) 0.002 0.003
0.007 0.008 0.003
a = floor (rand (1)*1e10)
%a = 8.2000e+09
tic, factorial (a); toc
%Elapsed time is 0.000299 seconds.
% Plot 2-D on 200,000 points 0.563 1.072
0.128 7.988 19.292
%with gnuplot
a = rand (2e5, 1);
tic (); plot (a); toc ()
%Elapsed time is 0.6472 seconds. (but the window takes much longer to
show up)
close all
backend ('fltk')
tic (); plot (a); toc ()
%Elapsed time is 0.04016 seconds. (and the window comes up very quickly)
% Plot 3-D on 200,000 points 1.105 3.691
0.091 0.216 1.789
close all
backend ('gnuplot')
tic (); plot3 (a); toc ()
%Elapsed time is 0.4186 seconds. (but the window takes forever to show
up)
close all
backend ('fltk')
tic (); plot3 (a); toc ()
%Elapsed time is 0.04582 seconds. (and the window comes up very quickly)
% Average performance for this group 37.71% 14.57%
68.49% 57.62% 31.01%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Matrix operations
freemat mathnium octave R Scilab
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Matrix multiplication among two 2000 × 2000 random matrices
8.187 171.389 18.664 0.070 4.626
a = rand (2e3, 2e3);
b = rand (2e3, 2e3);
tic (); a*b; toc ()
%Elapsed time is 1.3 seconds.
tic (); a.*b; toc ()
%Elapsed time is 0.06476 seconds.
% Transpose of a 2000 × 2000 random matrix
0.311 2.494 0.110 1.853 0.127
tic (); a'; toc ()
%Elapsed time is 0.06137 seconds.
tic (); a.'; toc ()
%Elapsed time is 0.066 seconds.
% Creation of a 2000 × 2000 Hilbert matrix
0.042 - 0.351 0.519 0.229
tic (); a = hilb (2e3); toc ()
%Elapsed time is 0.3474 seconds.
% Hessenberg form of a 2000 × 2000 random matrix
- 501.603 1274.100 - 29.412
a = rand (2e3, 2e3);
tic (); hess (a); toc ()
%Elapsed time is 11.886 seconds.
% Rank of a 2000 × 2000 random matrix
32.150 308.015 27.225 15.597 29.273
tic (); rank (a); toc ()
%Elapsed time is 13.795 seconds.
% Trace of a 2000 × 2000 random matrix
60.195 0.679 0.028 0.038 0.005
tic (); trace (a); toc ()
%Elapsed time is 0.003068 seconds.
% Condition number of a 2000 × 2000 random matrix 491.47
3939.406 20.735 16.853 29.257
tic (); cond (a); toc ()
%Elapsed time is 13.394 seconds.
% Kronecker product of two 200 × 200 random matrices
- 20.367 0.210 0.337 0.102
a = rand (200);
b = rand (200);
tic; kron (a, b); toc ()
%error: memory exhausted or requested size too large for range of
Octave's index type -- trying to return to prompt
% Average performance for the tests of this group
31.35% 2.42% 39.75% 50.90% 64.68%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Basic algebra freemat
mathnium octave R Scilab
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Determinant of a 2000 × 2000 random matrix 3.945
33.214 6.007 5.796 3.249
a = rand (2000);
tic; det (a); toc ()
%Elapsed time is 0.739024 seconds.
% Inverse of a 2000 × 2000 random matrix 533.880
78.364 18.991 24.409 9.489
tic; inv (a); toc ()
%Elapsed time is 2.23666 seconds.
% Eigenvalues of a 2000 × 2000 random matrix 44.679
3645.349 58.462 59.596 46.147
tic; eig (a); toc ()
%Elapsed time is 47.4915 seconds.
% Eigenvectors over a 2000 × 2000 random matrix 104.053
3787.40 125.200 126.105 540.456
tic; [v, l] = eig (a); toc ()
%Elapsed time is 120.715 seconds.
% 2000 × 2000 dot product matrix 8.665
181.192 18.763 11.995 4.751
a = rand (2e3, 1); b = rand (1, 2e3);
tic; a * b; toc () %% Is that what is meant by dot product matrix??
%Elapsed time is 0.0591681 seconds.
% Norm of a 2000 × 2000 random matrix 30.000
6.623 27.196 0.180 29.240
a = rand (2e3);
tic; norm (a); toc ()
%Elapsed time is 13.2002 seconds.
% Linear system solve of 1500 equations 1.903
7.151 2.764 2.677 76.925
a = rand (1.5e3); b = rand (1.5e3, 1);
tic; a \b; toc ()
%Elapsed time is 0.40785 seconds.
% Average performance for the tests of this group 62.80%
8.26% 51.20% 66.16% 59.88%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Advanced algebra freemat
mathnium octave R Scilab
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Cholesky decomposition of a 2000 × 2000 random matrix –
- 51.931 2.843 2.920 1.716
a = rand (2e3);
a = a*a';
tic; chol (a); toc ()
%Elapsed time is 0.388352 seconds.
% Lu decomposition of a 1500 × 1500 random matrix
1.667 19.799 6.709 0.003 1.687
a = rand (1500);
tic; lu (a); toc ()
%Elapsed time is 0.689038 seconds.
% Qr decomposition of a 1200 × 1200 random matrix
3.055 54.817 24.710 3.234 2.970
a = rand (1200);
tic; qr (a); toc ()
%Elapsed time is 0.819952 seconds.
% Singular value decomposition of a 2000 × 2000 random matrix 59.016
3871.344 205.740 86.328 29.167
a = rand (2000);
tic; svd (a); toc ()
%Elapsed time is 13.3108 seconds.
% Schur decomposition of a 1500 × 1500 random matrix
- 325.062 32.601 51.172 30.227
a = rand (1500);
tic; schur (a); toc ()
%Elapsed time is 29.2586 seconds.
% Reduced Row Echelon Form of a 2000 × 2000 random matrix
311.442 21.890 269.770 – - 144.047
a = rand (2000);
tic; rref (a); toc ()
%Elapsed time is 196.287 seconds.
% Average performance for the tests of this group 38.47% 19.80%
31.24% 68.69% 69.23%
% Statistics
% PCA over a 3000 × 300 random matrix – – 1.5395 1.7020
11.5281
% Gaussian error function on a 2000 × 2000 random matrix 0.110
1.003 0.571 11.202 0.161
% Linear regression over a 2000 × 2000 random matrix 3.717
172.287 6.025 17.848 3.412
% Arithmetic mean over a 2 × 106 random vector 0.034 4.686
0.153 0.057 0.001
% Geometric mean over a 2 × 106 random vector – – 1.284
4.937 0.002
% Harmonic mean over a 2 × 106 random vector – – 0.599 0.224
0.002
% Standard deviation over a 2 × 106 random vector 0.128 1.156
0.383 0.151 1.178
% Variance of a 1200 × 1200 random matrix 0.041 – 0.448
3.145 2.945
% Skewness of 5 × 106 random values 0.325 106.453 1.211
0.674 2.297
% Kurtosis of 5 × 106 random values 0.329 153.826 1.205
0.771 2.250
% Range values of a 2000 × 2000 random array 0.123 0.594 0.249
0.102 0.087
% Average performance for the tests of this group 83.19% 5.61%
28.07% 34.19% 56.59%
------------------------------------------------------------------------------
This SF.net Dev2Dev email is sponsored by:
Show off your parallel programming skills.
Enter the Intel(R) Threading Challenge 2010.
http://p.sf.net/sfu/intel-thread-sfd
_______________________________________________
Octave-dev mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/octave-dev
