Great, thanks a lot for these data! Xavier
On 6/14/07, Buck, Robert <[EMAIL PROTECTED]> wrote:
What the performance test did is: 1. increase by powers of 2 a file size variable 2. for each file size, create/open a new file 3. write specified number of bytes to file 4. close file Tests were performed on two systems: 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k 146GB disks in RAID 1 configuration; 16 GB RAM; this is production grade hardware. Both sytems produced similar profiles (shape), however, on the production hardware with Linux (ext2, I believe) the numbers were substantially greater (makes sense when you consider the fast disks I have; at 3GB/sec rates, these SAS drives scream). In both cases buffers were configured for both Old IO and New IO. With NIO direct buffers were used. In both cases, I determined the ideal buffer sizes based upon experimentation to be around 32kb - 64 kb. Going to buffer sizes less than, or greater than, this range reduced overall throughput. -Bob > -----Original Message----- > From: Xavier Hanin [mailto:[EMAIL PROTECTED] > Sent: Thursday, June 14, 2007 8:38 AM > To: [email protected] > Subject: Re: ivy:retrieve performance > > On 6/14/07, Buck, Robert <[EMAIL PROTECTED]> wrote: > > > > Folks, > > > > We benchmarked a number of JDK IO API's for an internal project. To > > neutralize any questions regarding NIO vs Old IO, please > take a look > > at the attached diagram. These rates will be largely > identical on both > > Linux and Windows. Blue line NIO, red line old io. > > > I've uploaded the image here: > http://www.flickr.com/photos/[EMAIL PROTECTED]/548163404/ > > Could you give a little bit more details about how you did > the tests: jvm used, size of buffer used for old IO, ... > > According to your tests it seems that NIO should be preferred > in any case, it wasn't what some users in the javalobby > thread seemed to say. So I wonder what makes the difference. > > Xavier > > If you do not get the attached JPEG file, let me know. I can send it > > directly to you if you so request. > > > > Cheers, > > > > Bob > > > > > -----Original Message----- > > > From: Xavier Hanin [mailto:[EMAIL PROTECTED] > > > Sent: Thursday, June 14, 2007 6:22 AM > > > To: [email protected] > > > Subject: Re: ivy:retrieve performance > > > > > > On 6/14/07, Gilles Scokart <[EMAIL PROTECTED]> wrote: > > > > > > > > > 2. The buffer in FileUtils.java is too small. It's set at > > > 8192. It > > > > > seems to > > > > > > be much better for me to set it much larger. This is due to > > > > > > the fact > > > > > that > > > > > > it > > > > > > needs to read and write simultaneously. The bigger the > > > buffer is, > > > > > > the smaller number of time, HD header has to move. For > > > me, 65536 > > > > > > seems to perform much better but I haven't tried > other numbers. > > > > > > > > > > > > > > > I'd like to get more feedback on this. One use case is > > > not the other. > > > > This > > > > > size has been borrowed from Ant copy mechanism. Maybe > > > what we could > > > > > do > > > > is > > > > > make this configurable, so that one could adapt to its > > > needs. Or try > > > > > to guess a good size depending on the size (when it's > possible > > > > > to get an > > > > idea > > > > > of the size before copying). > > > > > > > > > > Xavier > > > > > > > > > > > > > > > > > Couldn't we use the nio for that? (See > > > > http://www.javalobby.org/java/forums/t17036.html) > > > > > > > > > According to comments 10 and 11 NIO have bad performance > for large > > > files on linux, and input stream with byte buffer is > pretty close to > > > NIO for small files (see comment 13 conclusion). So I'm not sure > > > switching to NIO would indeed help a lot. According to > the tests in > > > the thread you pointed using a 64kB buffer seems to be a > good choice > > > (which confirms testn tests), at least for large files. OTOH the > > > last conclusion (comment 17) is different. > > > So I don't really know what to think about that. We > should make some > > > tests on several platforms and jvms to draw conclusion > ourself, but > > > it takes time. > > > > > > > > > Xavier > > > > > > Gilles > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Xavier Hanin - Independent Java Consultant Manage your > dependencies > > > with Ivy! > > > http://incubator.apache.org/ivy/ > > > > > > > > > -- > Xavier Hanin - Independent Java Consultant Manage your > dependencies with Ivy! > http://incubator.apache.org/ivy/ >
-- Xavier Hanin - Independent Java Consultant Manage your dependencies with Ivy! http://incubator.apache.org/ivy/
