Hi All,
I have some performance updates for the jar tool and for the Zip/Jar writing components, including some code to allow parallel writing of Jar and ZIP files (in java.util)
This work is not finished as yet but I am looking to see if anyone has any
views as to the shape this should move in
Currently it is a testbed for comparing different techniques, but largely based
on the Jar utility
The changes allow the work to be spread across multiple CPUs and optimise the
some of the code and I/O paths
This comparative figures do not include the effect of the nio changes that I
proposed in earlier emails
Command line changes
0--9 - I have added support for specifying different compression levels (the
existing jar command just allows default compression or '0' for no compression
D This allows the files to all be written with the date of now, lather than the
file date (the conversion of the date to zip format is a CPU hog, and not
needed in some use-cases)
Z0-5 - these are the different mechanisms to allow different parallel execution
models - I would not expect this to be a production qualifier
The test environment is a 4 core Intel core2 pc running windows vista 64, the test case is jaring up the content of rt.jar to a jar file.
Each test is repeated 6 times and the last 5 are averaged to produce the answers. Each test is run in a fresh VM
The performance figures are below as a CSV. The last column is the duration of
the task in ms.
In summary the existing jar utility takes (for uncompressed, compressed) 8.4 , 9.4 seconds to complete and this can be reduced to 1.6, 2.3 seconds
The different parallel algorithms are
0 - none all in one thread as before
1 - file scanning in one core, 10 threads loading and buffering files, zip
writing in a single thread using the existing ZipOuputStream
2. - file scanning in one core, 10 threads loading and buffering files, zip
writing mostly mutithreaded (e.g. parallel compression, single write to the
output stream)
3 - as 2 but writes to a file rather than a stream
4. as 2 but uses channels to be to write with direct buffers
5 as 4 but using heap buffers
3-5 have the zip capability in the code to seek and update headers that are
incomplete, but this is not much tested
C:\Program Files\Java\jdk1.6.0_24\bin\java.exe, C:\Program
Files\Java\jdk1.6.0_24\lib\tools.jar, -cf0, java 1.6 rt -cf0, 8482
C:\Program Files\Java\jdk1.6.0_24\bin\java.exe, C:\Program
Files\Java\jdk1.6.0_24\lib\tools.jar, -cf, java 1.6 rt -cf, 9318
C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Program
Files\Java\jdk1.7.0\lib\tools.jar, -cf0, java 1.7 rt -cf0, 8497
C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Program
Files\Java\jdk1.7.0\lib\tools.jar, -cf, java 1.7 rt -cf, 9518
C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Test\Archive\baseline.jar,
-cf0, orig 1.7 rt -cf0, 8448
C:\Program Files\Java\jdk1.7.0\bin\java.exe, C:\Test\Archive\baseline.jar, -cf,
orig 1.7 rt -cf, 9484
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0, project 1.7 rt
-cf0, 3133
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0D, project 1.7 rt
-cf0D, 2824
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0Z0, project 1.7 rt
-cf0 parallel 0, 3026
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ0, project 1.7 rt
-cf0D parallel 0, 2961
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ1, project 1.7 rt
-cf0D parallel 1, 2022
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ2, project 1.7 rt
-cf0D parallel 2, 1757
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ3, project 1.7 rt
-cf0D parallel 3, 1632
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ4, project 1.7 rt
-cf0D parallel 4, 1994
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf0DZ5, project 1.7 rt
-cf0D parallel 5, 1978
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1, project 1.7 rt
-cf1, 5237
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1D, project 1.7 rt
-cf1D, 5073
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1Z0, project 1.7 rt
-cf1 parallel 0, 5367
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ0, project 1.7 rt
-cf1D parallel 0, 5002
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ1, project 1.7 rt
-cf1D parallel 1, 5125
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ2, project 1.7 rt
-cf1D parallel 2, 2257
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ3, project 1.7 rt
-cf1D parallel 3, 2145
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ4, project 1.7 rt
-cf1D parallel 4, 2505
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf1DZ5, project 1.7 rt
-cf1D parallel 5, 2549
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf2, project 1.7 rt
-cf2, 5371
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf3, project 1.7 rt
-cf3, 5409
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf4, project 1.7 rt
-cf4, 5778
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf5, project 1.7 rt
-cf5, 5906
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6, project 1.7 rt
-cf6, 6082
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf7, project 1.7 rt
-cf7, 6070
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf8, project 1.7 rt
-cf8, 6251
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9, project 1.7 rt
-cf9, 6191
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6D, project 1.7 rt
-cf6D, 5843
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6Z0, project 1.7 rt
-cf6 parallel 0, 6095
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ0, project 1.7 rt
-cf6D parallel 0, 5907
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ1, project 1.7 rt
-cf6D parallel 1, 5957
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ2, project 1.7 rt
-cf6D parallel 2, 2388
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ3, project 1.7 rt
-cf6D parallel 3, 2351
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ4, project 1.7 rt
-cf6D parallel 4, 2694
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf6DZ5, project 1.7 rt
-cf6D parallel 5, 2830
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9D, project 1.7 rt
-cf9D, 6134
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9Z0, project 1.7 rt
-cf9 parallel 0, 6258
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ0, project 1.7 rt
-cf9D parallel 0, 6066
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ1, project 1.7 rt
-cf9D parallel 1, 6203
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ2, project 1.7 rt
-cf9D parallel 2, 2490
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ3, project 1.7 rt
-cf9D parallel 3, 2361
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ4, project 1.7 rt
-cf9D parallel 4, 2788
C:\Program Files\Java\jdk1.7.0\bin\java.exe,
C:\NetBeansProjects\JavaProject1\dist\javaproject1.jar, -cf9DZ5, project 1.7 rt
-cf9D parallel 5, 2847
regards
Mike
