I have tested that code and read a bit about sparse files.
Tests first: I had on average 65 s on Windows 10, it is down now to 50
to 55 seconds. The huge problem is that we cannot use @BeforeClass to
create the file once because JUnit pre-annotations does not support hat.
To the sparse stuff: according the some Stackoverflow answers,
RandomAccessFile does create a sparse file automatically on Linux,
Solaris, etc. where as on Windows you need to set a bit in native code
for NTFS. The only options for sparse files is either 'fsutil' or NIO2.
I'd stick to fsutil the way it is right now.
Do you see any room for improvement without huge effort? Otherwise I'd
go on and commit your improvement.
I still do not undertand why it runs so slow on your Windows box.
Michael
Am 2016-12-29 um 19:10 schrieb Guillaume Boué:
I have a Toshiba SSHD (MQ02ABD100H). I think the issue is that the Java
code should create a sparse file to have things faster. Using setLength
on a random access file probably does it depending on the OS and type of
drive, but it isn't creating one in my situation.
When run on Ubuntu, creating the test file is done practically instantly
whereas it takes 60 seconds on my Windows machine. Downloading takes
around 30 seconds in Ubuntu, whereas it takes 150 seconds in Windows.
I changed the method creating the test file (makeHugeFile) to:
if ( Os.isFamily( Os.FAMILY_WINDOWS ) )
{
Process p = new ProcessBuilder( "fsutil", "file",
"createnew", hugeFile.getAbsolutePath(),
String.valueOf(
HUGE_FILE_SIZE ) ).start();
p.waitFor();
}
else
{
RandomAccessFile ra = new RandomAccessFile(
hugeFile.getPath(), "rw" );
ra.setLength( HUGE_FILE_SIZE + 1 );
ra.seek( HUGE_FILE_SIZE );
ra.write( 1 );
ra.close();
}
This matches with the timings I get on Ubuntu (both for the creation of
the file, and the downloading via Wagon). I ran the build several times
with this change without any timeout issues. Can you test this change on
your side?
Side-note: I added "ra.close();" in the code above, because the random
access file wasn't closed otherwise, and then the file wasn't getting
deleted properly.
Guillaume
Le 29/12/2016 à 15:50, Michael Osipov a écrit :
Am 2016-12-29 um 12:24 schrieb Guillaume Boué:
I ran them at least 10 times, and there was the timeout issue each time.
Yes, the timeout is Surefire waiting for the forked VM. I applied the
patch (I had done similar tries also) but I still have the issue on
Windows 10.
You'll find the logs attached. It seems that the download is really
advancing (inside target, I can see the file huge<numbers>txt slowly
growing), but is just taking a long time. Additionally, the test class
HugeFileDownloadTest is composed of 2 tests, downloading the same file 2
times with different parameters: from the logs, it seems one of them is
succeeding (taking 140 seconds of the 400 in total), and then it times
out performing the second.
Also, part of the issue is probably with the time it takes to create the
4 Go test file that is later downloaded through Wagon (maybe it should
be done by calling the fsutil command when the test runs on Windows).
I'll try to do more timings when running the tests with Ubuntu and see
where the difference is.
Thanks for the analysis. I think the cheapest improvement we can do is
to create the file only once (setUp(), tearDown(). It is created for
each test again. I do not think that this is really necessary.
What type of disks do you have? This test passes on my machine within
70 seconds on a Samsung SSD.
I will apply the patch to the branch because those fixes are necessary
anyway.
Please keep me updated.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---
L'absence de virus dans ce courrier électronique a été vérifiée par le
logiciel antivirus Avast.
https://www.avast.com/antivirus
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
