hg: jdk8/tl/jdk: 4243978: (ref) Race condition in Reference.enqueue(); ...
Changeset: 5f2838744544 Author:ysr Date: 2011-10-31 17:38 -0700 URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/5f2838744544 4243978: (ref) Race condition in Reference.enqueue() 4268317: (ref) Reference.isEnqueued() can return true when instance not enqueued Summary: The reference handler now declares, and assumes, that the discovered field, rather than the next field, is (to be) used to link the entries in the pending list, thus allowing a reference object to be safely enqueued even while it is in the pending state. Also added slightly modified regression tests from the two bug reports. Reviewed-by: mchung, alanb, jcoomes ! src/share/classes/java/lang/ref/Reference.java ! src/share/javavm/export/jvm.h ! src/share/native/common/jdk_util.c + test/java/lang/ref/ReferenceEnqueue.java + test/java/lang/ref/ReferenceEnqueuePending.java
Re: performance updates to jar and zip
On 10/31/2011 5:11 PM, Mike Skells wrote: Hi Sherman, (1) Are the number here with writing the LOC in the background thread, in not I don't understand the comment The writeLOC in the base code version is very slow. each int written is written as 4 seperate bytes, each of which takes out locks, checks to flush etc The number below has the loc written in pooled thread. YES It is faster than writing the loc in main thread. [@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cfT4 rt.jar rtjar Warmup:rtjar...done mtCreate: threadNum=4... Jar TotalTime:2754
Re: performance updates to jar and zip
Hi Sherman, (1) Are the number here with writing the LOC in the background thread, in not I don't understand the comment The writeLOC in the base code version is very slow. each int written is written as 4 seperate bytes, each of which takes out locks, checks to flush etc In the version that I submitted I rewrote this to write to a temporary buffer Running a profiler on you code to look for the CPU/Elapsed time I see (all figure are for the CPU/Elapsed time as a proportion of the main thread CPU/elapsed time total, and include time in called methods method writeLOC 16.7%/16% of which ZipCoder.getBytes 9.9%/8.9% To put this is proportion the BufferedOutputStream.write is 2.5%, 4.8% (2) Iti is true that this is not a performance gain by code improvement, but it is a performnce gain by specification. The same arguement applies to allowing a Zip Compression of 1 rather than the default. As for the spec, all I have seen is that it should provide the date is available. I am not that fussed if we include this in another section and I that all specification changes need to be considered seperately, as long as it is not lost I ran you test and I did not see more than 15% difference in the scanning with jav2DosTime includes/excluded. Minimal time in profiler reported as well Running the same time on the same files via the sample improved app that you supplied I see 16% of the time spent in javaToDosTime Could this be a micro-benchmark issue? (aqnd/or a profiling issue) In the benchmarks that I have for the full jar app I see usually 5-10% total time reduction in the tests that are CPU bound (3) It will not give much advantage if we have to process the files in a strict order, and really is part of the app structure not the improvements, ie the mechaism of how the pipeline work (4) It is ather hard to quantify the benifit of the approach. Certainly all of the code that I had did not lock the main thread until the IO buffers are full, so generally the thread didnt block until the data was ready to be pushed to the output. I am not sure that there is much extra complexity introduced by using a j.u.c.CompletionQueue, and the main thread polls that (5) if w want to limit memory use then a Semaphore would better avoid lots of JC cycles, which could adversely affect other part sof the system. I used a memory pool, so that the memory was not cycled and didnt grow, which seems a more efficient solution, but slightly more complex (6) I hd not expected to have a large impact of the u.u.zip package, but I think a Parallel ZipWriter is useful. I included 2 basic implementations, one that wrote bases in the user managing threads, and one that defferred to a Executor. The user APi for that is similar to ZipOutputStream, but has a simper API, in that you write an entry with and InputStream, and the implementation manages the parallelism. I would think that a capability for a parallel ZIP encoding stream would be a benifit to the community in general The majority of the code that I presented in the j.u.Zip and j.u.jar packages are for investigation (e.g. nio vs RandomAccessFile), and were only left in because I could not verify that the RandomAccessFile based IO was faster than nio channel based IO, and to verify If there was a better approach (other) one other improvement in Zip is related to the handling of stored mode. In the standard delivery the file is read twice, once to calculate the CRC and again to write the data Regards Mike > >From: Xueming Shen >To: Mike Skells >Cc: "core-libs-dev@openjdk.java.net" >Sent: Monday, 31 October 2011, 21:23 >Subject: Re: performance updates to jar and zip > > > >Hi Mike, > >(1) While it's not a "significant benefit" :-) obviously it helps the through put to move the "loc writing" >to the background work (together with the compression). I notice 10%+ improvement on one of my >4-core machine (vm OS installation, so the IO is supposed to be slow), when increasing the threads >from 3-4. > >@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cf rt.jar rtjar >Warmup:rtjar...done >Jar TotalTime:7211 > >[@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cfT3 rt.jar rtjar >Warmup:rtjar...done >mtCreate: threadNum=3... >Jar TotalTime:3266 > >[@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cfT4 rt.jar rtjar >Warmup:rtjar...done >mtCreate: threadNum=4... >Jar TotalTime:2754 > >(2) It's definitely fine with me to have a separate discussion regarding whether or not jar should >have a -D like option for those know that they will never need the lastModified info in the jar/zip >file they create. But I don't think we should count/include the time saved from using -D into the >"performance improvement/gain" here, you trade off the functionality for speed here, especially >this info is something specified by default in loc/cen tables. > >I also tried that
hg: jdk8/tl/jdk: 7053252: New regression test does not compile on windows-amd64
Changeset: b60e88ef5d8d Author:wetmore Date: 2011-10-31 16:23 -0700 URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/b60e88ef5d8d 7053252: New regression test does not compile on windows-amd64 Reviewed-by: valeriep ! test/ProblemList.txt ! test/sun/security/pkcs11/Provider/Absolute.java
Re: performance updates to jar and zip
Hi Mike,
(1) While it's not a "significant benefit" :-) obviously it helps the
through put to move the "loc writing"
to the background work (together with the compression). I notice 10%+
improvement on one of my
4-core machine (vm OS installation, so the IO is supposed to be slow),
when increasing the threads
from 3-4.
@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cf rt.jar rtjar
Warmup:rtjar...done
Jar TotalTime:7211
[@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cfT3 rt.jar
rtjar
Warmup:rtjar...done
mtCreate: threadNum=3...
Jar TotalTime:3266
[@flicker-vm2:/tmp/sherman/test]../mtjar-linux/bin/java Jar cfT4 rt.jar
rtjar
Warmup:rtjar...done
mtCreate: threadNum=4...
Jar TotalTime:2754
(2) It's definitely fine with me to have a separate discussion regarding
whether or not jar should
have a -D like option for those know that they will never need the
lastModified info in the jar/zip
file they create. But I don't think we should count/include the time
saved from using -D into the
"performance improvement/gain" here, you trade off the functionality
for speed here, especially
this info is something specified by default in loc/cen tables.
I also tried that javaToDosTime calculation in FIter2.java
http://cr.openjdk.java.net/~sherman/mtjar2/FIter2.java
I did not see any significant performance impact by doing javaToDosTime
calculation, if I did not
mis-understand what you meant.
(3) I actually tried ArrayBlockingQueue, but it does not appear to help
the performance in my setup,
actually it slowdown the process a little, so I took it off the table. I
might give it a try later.
(4) Not separating the file loading and compression is by purpose, this
way it helps to preserve the
"order" of the files/dir scanned. The cost is as you suggested the main
thread might get blocked on
the queue, if the first in line has not finished the load/compression
yet. But I did not see an easy way
to preserve the sequential order by using a separate "completed" queue
(doable, but makes thing
complicated). While preserving the "order" is not a hard request, zip
spec never specifies the order
/structure of entries included, I don't any reason to break the existing
behavior if the change does
not bring in something significant.
(5) There is potential memory issue with the current code, in worse case
if the writing thread can
not catch up with all compressing threads and you have an "unlimited"
files to zip in. It can be
addressed by either monitoring the memory usage or simply wrap the
allocation with the try
block, if we exhaust the memory, just pass the file directly without
submit it into the job queue.
But this is something we can consider later, the purpose of my code is
to have some measure
to see how far we can go, mostly because we don't have your code work on
jdk8 yet.
(6) I would not expect that we are going to add bunch of new public
classes/apis just for this
particular performance tuning, if those classes/apis don't bring in too
much value for general
jar/zip operation, for example, in my experimental code, I've added
ZOS.writeNextEntry
for the convenience of the experimenting/testing, but if we finally go
this direction, I would
assume we will end up having a "customized" ZipOutputStream in
sun.tool.jar for this purpose
instead of exposing that "writeNextEntry" API, as it probably serves
nobody. Yes, that means
those "public" classes/APIs in your proposal will have to have a very
good story to back them
to be "public".
I'm looking for a workable JDK8 patch to test/work on:-) we need some
data first, and then
decide what will be in and what will be left behind.
-Sherman
http://cr.openjdk.java.net/~sherman/mtjar2/webrev/
On 10/27/2011 4:07 PM, Mike Skells wrote:
Hi Sherman,
I think that you will get significant benefit from generating the data
structures in the background threads.
I think that is you profile the usageyou will see that the generation
of the header information is the dominant feature.
That is why I parallelised the writing process.
There are several bottlenecks such as the encoding of the name name
and (although you dismiss it) the calculation of the dos time format
is a CPU hog (the -D qualifier). I hink that it is about 10% of the
overall CPU load
This is by the way pretty much in line with the extraction feature
below added in java 6, so I cant see that there is a great reason
against it,
after all why spend time storing information that (in most use cases)
is not read (either because the jar utility does not by default
maintain it, and most jar files are
probably not expanded anyway
/**
* If true, maintain compatibility with JDK releases prior to 6.0 by
* timestamping extracted files with the time at which they are
extracted.
* Default is to use the time given in the archive.
*/
private static final boolean useExtractionTime =
Boolean.getBoolean("sun.tools.jar.useEx
hg: jdk8/tl/jdk: 7105780: Add SSLSocket client/SSLEngine server to templates directory
Changeset: 8681362a2f04 Author:wetmore Date: 2011-10-31 11:54 -0700 URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/8681362a2f04 7105780: Add SSLSocket client/SSLEngine server to templates directory Reviewed-by: xuelei + test/sun/security/ssl/templates/SSLSocketSSLEngineTemplate.java
Collections.checkedQueue() offer method should not call add.
Darryl, CheckedQueue.offer should call 'this.queue.offer' instead of 'this.add'. If you pass a Queue with bounded capacity (ArrayBlockingQueue) the CQ.offer method should return false when the queue is full but will instead throw an IllegalStateException. The current version also is performing the type check twice. Jason Changeset: c5c91589b126 Author:mduigou Date: 2011-10-19 14:17 -0700 URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/c5c91589b126 5029031: Add Collections.checkedQueue() Reviewed-by: mduigou Contributed-by: darryl.mocek at oracle.com ! src/share/classes/java/util/Collections.java + test/java/util/Collections/CheckedQueue.java
