RE: ZeroCopyPut mystery
I know this is an old thread, but I wanted to relate the final scheme I came up
with for optimal uploading of DVD files on windows,
in case anyone else has to deal with this issue:
1) issue Powershell command from Java to recursively list all files in a root
directory, and copy each file to dev null, as follows:
String cmd = "powershell.exe \"Get-Childitem -Recurse " +
rootDirectory +
" | where {!$_.PsIsContainer} | %{ Get-Content
$_.FullName > $null; echo $_.FullName;} \" ";
2) Notice the echo command at the end; now that the file is in the windows file
cache, I can read this echoed name in java and issue an asynchronous zero copy
put on the file via
httpasyncclient (this is now very fast, since the file is in the cache)
So, this method pipelines the reads and puts, so that put time can be ignored,
and time to upload is just the time to read the files from DVD.
Cheers,
Aaron
-Original Message-
From: Boxer, Aaron
Sent: Friday, January 17, 2014 3:26 PM
To: HttpClient User Discussion
Subject: RE: ZeroCopyPut mystery
On Fri, 2014-01-17 at 16:28 +, Boxer, Aaron wrote:
> I made a simple test: I wrote a few lines of Java using File channel and
> transferTo(...) method to read files from my DVD and write to the hard drive.
> With this simple test, the transferTo(...) method gets about 4 MB /S
> transfer rate to disk.
>
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...)
> method, and ran my application, which copies files from DVD
> to a socket. And the transferTo(...) from file to socket was getting about
> 0.7 MB /s . This is a 6 X degradation in performance.
> And, if you recall, if the files are already in the OS file cache, I get
> about 100 MB/S, so the socket is not slowing me down.
>
I wanted to share the solution I came up with that is pretty optimal, in my
opinion:
1) sequentially read files from DVD, from FileChannel into Direct ByteBuffer -
once read, these files are now in the Windows OS file cache
2) after every 50 files read (or certain amount of MB), I send these files out
over the asynch client( good transfer rate; asynchronous execution)
I do this until all files are read.
The total cost to me is just the cost of reading the DVD, which I can't really
avoid. So, this is pretty optimal.
This e-mail may contain confidential and/or privileged information for the sole
use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was
originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete
all copies.
Opinions, conclusions or other information contained in this e-mail may not be
that of the organization.
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
This e-mail may contain confidential and/or privileged information for the sole
use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was
originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete
all copies.
Opinions, conclusions or other information contained in this e-mail may not be
that of the organization.
RE: ZeroCopyPut mystery
On Fri, 2014-01-17 at 16:28 +, Boxer, Aaron wrote: > I made a simple test: I wrote a few lines of Java using File channel and > transferTo(...) method to read files from my DVD and write to the hard drive. > With this simple test, the transferTo(...) method gets about 4 MB /S > transfer rate to disk. > > Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) > method, and ran my application, which copies files from DVD > to a socket. And the transferTo(...) from file to socket was getting about > 0.7 MB /s . This is a 6 X degradation in performance. > And, if you recall, if the files are already in the OS file cache, I get > about 100 MB/S, so the socket is not slowing me down. > I wanted to share the solution I came up with that is pretty optimal, in my opinion: 1) sequentially read files from DVD, from FileChannel into Direct ByteBuffer - once read, these files are now in the Windows OS file cache 2) after every 50 files read (or certain amount of MB), I send these files out over the asynch client( good transfer rate; asynchronous execution) I do this until all files are read. The total cost to me is just the cost of reading the DVD, which I can't really avoid. So, this is pretty optimal. This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this e-mail may not be that of the organization. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: ZeroCopyPut mystery
On Fri, 2014-01-17 at 16:28 +, Boxer, Aaron wrote: > I made a simple test: I wrote a few lines of Java using File channel and > transferTo(...) method to read files from my DVD and write to the hard drive. > With this simple test, the transferTo(...) method gets about 4 MB /S > transfer rate to disk. > > Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) > method, and ran my application, which copies files from DVD > to a socket. And the transferTo(...) from file to socket was getting about > 0.7 MB /s . This is a 6 X degradation in performance. > And, if you recall, if the files are already in the OS file cache, I get > about 100 MB/S, so the socket is not slowing me down. > > I have done a little more testing: transferTo from DVD to local socket gets > 3-4 MB/S transfer rate. > I tried using a thread pool, and performance degrades (not surprisingly) as > thread count rises from 1. > > Since DVD seeks are very slow, multiple threads trying to read at the same > time will kill performance. > > But, I am not sure this is the issue here, because I have set > > connManager.setMaxTotal(1); > connManager.setDefaultMaxPerRoute(1); > > > So, there should only be one thread doing the transfer (is this true?) > Actually HttpAsyncClient uses multiple i/o dispatch threads, one per CPU core. I suppose even with a cap of 1 total connection multiple threads might end up trying to access the same physical device. You can force HttpAsyncClient to use only one i/o dispatch thread by using a custom IOReactorConfig Thanks, Oleg. Here is my reactor config: IOReactorConfig ioReactorConfig = IOReactorConfig.custom() .setIoThreadCount(1) .setConnectTimeout(3) .setSoTimeout(3) .setTcpNoDelay(true) .build(); And the problem is still there with a thread count of 1. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this e-mail may not be that of the organization. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: ZeroCopyPut mystery
On Fri, 2014-01-17 at 16:28 +, Boxer, Aaron wrote: > I made a simple test: I wrote a few lines of Java using File channel and > transferTo(...) method to read files from my DVD and write to the hard drive. > With this simple test, the transferTo(...) method gets about 4 MB /S > transfer rate to disk. > > Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) > method, and ran my application, which copies files from DVD > to a socket. And the transferTo(...) from file to socket was getting about > 0.7 MB /s . This is a 6 X degradation in performance. > And, if you recall, if the files are already in the OS file cache, I get > about 100 MB/S, so the socket is not slowing me down. > > I have done a little more testing: transferTo from DVD to local socket gets > 3-4 MB/S transfer rate. > I tried using a thread pool, and performance degrades (not surprisingly) as > thread count rises from 1. > > Since DVD seeks are very slow, multiple threads trying to read at the same > time will kill performance. > > But, I am not sure this is the issue here, because I have set > > connManager.setMaxTotal(1); > connManager.setDefaultMaxPerRoute(1); > > > So, there should only be one thread doing the transfer (is this true?) > Actually HttpAsyncClient uses multiple i/o dispatch threads, one per CPU core. I suppose even with a cap of 1 total connection multiple threads might end up trying to access the same physical device. You can force HttpAsyncClient to use only one i/o dispatch thread by using a custom IOReactorConfig Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: ZeroCopyPut mystery
I made a simple test: I wrote a few lines of Java using File channel and
transferTo(...) method to read files from my DVD and write to the hard drive.
With this simple test, the transferTo(...) method gets about 4 MB /S transfer
rate to disk.
Then, I put some timing code into the LengthDelimitedEncoder.transfer(...)
method, and ran my application, which copies files from DVD
to a socket. And the transferTo(...) from file to socket was getting about 0.7
MB /s . This is a 6 X degradation in performance.
And, if you recall, if the files are already in the OS file cache, I get about
100 MB/S, so the socket is not slowing me down.
I have done a little more testing: transferTo from DVD to local socket gets
3-4 MB/S transfer rate.
I tried using a thread pool, and performance degrades (not surprisingly) as
thread count rises from 1.
Since DVD seeks are very slow, multiple threads trying to read at the same time
will kill performance.
But, I am not sure this is the issue here, because I have set
connManager.setMaxTotal(1);
connManager.setDefaultMaxPerRoute(1);
So, there should only be one thread doing the transfer (is this true?)
Anyways, I guess I will switch back to the synchronous client for my
application.
Aaron
This e-mail may contain confidential and/or privileged information for the sole
use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was
originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete
all copies.
Opinions, conclusions or other information contained in this e-mail may not be
that of the organization.
B CB [
X ܚX KK[XZ[
Y[
]\ \ ][ X ܚX P˘\X K ܙ B ܈Y][ۘ[ [X[ K[XZ[
Y[
]\ \ Z[ ˘\X K ܙ B B
This e-mail may contain confidential and/or privileged information for the sole
use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was
originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete
all copies.
Opinions, conclusions or other information contained in this e-mail may not be
that of the organization.
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: ZeroCopyPut mystery
> If it is only slower for DVD copies, that seems more like an issue with DVD > caching. > > Are you sure that the synch and asynch tests both started from exactly the > same environment? > > What do you mean by environment? Both tests read the same DVD, after > flushing the Windows OS file cache. I meant the phsyical environment. There are other potential hardware caches. For example the DVD drive. > But, the clients are, of course, set up differently. Would you like me to > post how I am configuring the clients? > What about the DVD cache? Did you reset the drive? Did you repeat the tests multiple times? Sometimes with async first, other times with sync first? Yes, this is completely repeatable. Each run, I clear the OS file cache, and each time, asynch is about the same slow rate compared to synch. > > > >> Perhaps it has to do with buffer size, thread count, . ? >> >> -Original Message- >> From: Oleg Kalnichevski [mailto:ol...@apache.org] >> Sent: Thursday, January 16, 2014 11:15 AM >> To: HttpClient User Discussion >> Subject: Re: ZeroCopyPut mystery >> >> On Thu, 2014-01-16 at 04:33 +, Boxer, Aaron wrote: >>> Hello, >>> >>> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a >>> network transfer rate of about 250 MBPS for first time read of files; 850 >>> MBPS if the files are already in the Windows OS file cache. Synchronous put >>> gives me about 30 MBPS. >>> >>> But, when I am dealing with files on DVD, I get a network transfer rate of >>> 5 MBPS for first time read of files, and 850 MBPS if files are in the OS >>> file cache. >>> Synchronous put gives me about 30 MBPS. >>> >>> So, in the case of first read of DVD files, synchronous client is superior. >>> Synchronous should be faster, because it can read and send at the same >>> time. >>> >>> Does this make sense? It seems a little strange to me. >>> >> >> Aaron, >> >> Zero copy data transfer is very platform specific. Java merely provides a >> common APIs, which however can have radically different implementations on >> different platforms and therefore can have different performance >> characteristics. >> >> I guess these numbers have more to do with Windows than with HttpAsyncClient. >> >> Oleg >> >> >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> >> This e-mail may contain confidential and/or privileged information for the >> sole use of the intended recipient. >> Any review or distribution by anyone other than the person for whom it was >> originally intended is strictly prohibited. >> If you have received this e-mail in error, please contact the sender and >> delete all copies. >> Opinions, conclusions or other information contained in this e-mail may not >> be that of the organization. > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > This e-mail may contain confidential and/or privileged information for the > sole use of the intended recipient. > Any review or distribution by anyone other than the person for whom it was > originally intended is strictly prohibited. > If you have received this e-mail in error, please contact the sender and > delete all copies. > Opinions, conclusions or other information contained in this e-mail may not > be that of the organization. > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this e-mail may not be that of the organization. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: ZeroCopyPut mystery
"Boxer, Aaron" wrote: >Thanks, Oleg. I am not trying to criticize the project in any way; just >wondering if anyone has insight into why, on Windows, Java NIO is so >much slower than stream IO. > >Perhaps it has to do with buffer size, thread count, . ? > You are welcome to criticize. I am merely trying to say this might be not the best list for this question. Thanks, Oleg. I do think there is something in the library, or how I am configuring the library, that is slowing me down: I made a simple test: I wrote a few lines of Java using File channel and transferTo(...) method to read files from my DVD and write to the hard drive. With this simple test, the transferTo(...) method gets about 4 MB /S transfer rate to disk. Then, I put some timing code into the LengthDelimitedEncoder.transfer(...) method, and ran my application, which copies files from DVD to a socket. And the transferTo(...) from file to socket was getting about 0.7 MB /s . This is a 6 X degradation in performance. And, if you recall, if the files are already in the OS file cache, I get about 100 MB/S, so the socket is not slowing me down. So, it is in the library somewhere. Any advice on how to debug this would be very much appreciated. Cheers, Aaron This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
RE: ZeroCopyPut mystery
"Boxer, Aaron" wrote: >Thanks, Oleg. I am not trying to criticize the project in any way; just >wondering if anyone has insight into why, on Windows, >Java NIO is so much slower than stream IO. > >Perhaps it has to do with buffer size, thread count, . ? > You are welcome to criticize. I am merely trying to say this might be not the best list for this question. Oleg -- Sent from my Android device with K-9 Mail. Please excuse my brevity. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: ZeroCopyPut mystery
On 16 January 2014 16:54, Boxer, Aaron wrote: > On 16 January 2014 16:35, Boxer, Aaron wrote: >> Thanks, Oleg. I am not trying to criticize the project in any way; >> just wondering if anyone has insight into why, on Windows, Java NIO is so >> much slower than stream IO. > > If it is only slower for DVD copies, that seems more like an issue with DVD > caching. > > Are you sure that the synch and asynch tests both started from exactly the > same environment? > > What do you mean by environment? Both tests read the same DVD, after > flushing the Windows OS file cache. I meant the phsyical environment. There are other potential hardware caches. For example the DVD drive. > But, the clients are, of course, set up differently. Would you like me to > post how I am configuring the clients? > What about the DVD cache? Did you reset the drive? Did you repeat the tests multiple times? Sometimes with async first, other times with sync first? > > > >> Perhaps it has to do with buffer size, thread count, . ? >> >> -Original Message- >> From: Oleg Kalnichevski [mailto:ol...@apache.org] >> Sent: Thursday, January 16, 2014 11:15 AM >> To: HttpClient User Discussion >> Subject: Re: ZeroCopyPut mystery >> >> On Thu, 2014-01-16 at 04:33 +, Boxer, Aaron wrote: >>> Hello, >>> >>> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a >>> network transfer rate of about 250 MBPS for first time read of files; 850 >>> MBPS if the files are already in the Windows OS file cache. Synchronous put >>> gives me about 30 MBPS. >>> >>> But, when I am dealing with files on DVD, I get a network transfer rate of >>> 5 MBPS for first time read of files, and 850 MBPS if files are in the OS >>> file cache. >>> Synchronous put gives me about 30 MBPS. >>> >>> So, in the case of first read of DVD files, synchronous client is superior. >>> Synchronous should be faster, because it can read and send at the same >>> time. >>> >>> Does this make sense? It seems a little strange to me. >>> >> >> Aaron, >> >> Zero copy data transfer is very platform specific. Java merely provides a >> common APIs, which however can have radically different implementations on >> different platforms and therefore can have different performance >> characteristics. >> >> I guess these numbers have more to do with Windows than with HttpAsyncClient. >> >> Oleg >> >> >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> >> >> This e-mail may contain confidential and/or privileged information for the >> sole use of the intended recipient. >> Any review or distribution by anyone other than the person for whom it was >> originally intended is strictly prohibited. >> If you have received this e-mail in error, please contact the sender and >> delete all copies. >> Opinions, conclusions or other information contained in this e-mail may not >> be that of the organization. > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > This e-mail may contain confidential and/or privileged information for the > sole use of the intended recipient. > Any review or distribution by anyone other than the person for whom it was > originally intended is strictly prohibited. > If you have received this e-mail in error, please contact the sender and > delete all copies. > Opinions, conclusions or other information contained in this e-mail may not > be that of the organization. > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: ZeroCopyPut mystery
On 16 January 2014 16:35, Boxer, Aaron wrote: > Thanks, Oleg. I am not trying to criticize the project in any way; > just wondering if anyone has insight into why, on Windows, Java NIO is so > much slower than stream IO. If it is only slower for DVD copies, that seems more like an issue with DVD caching. Are you sure that the synch and asynch tests both started from exactly the same environment? What do you mean by environment? Both tests read the same DVD, after flushing the Windows OS file cache. But, the clients are, of course, set up differently. Would you like me to post how I am configuring the clients? > Perhaps it has to do with buffer size, thread count, . ? > > -Original Message- > From: Oleg Kalnichevski [mailto:ol...@apache.org] > Sent: Thursday, January 16, 2014 11:15 AM > To: HttpClient User Discussion > Subject: Re: ZeroCopyPut mystery > > On Thu, 2014-01-16 at 04:33 +, Boxer, Aaron wrote: >> Hello, >> >> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a >> network transfer rate of about 250 MBPS for first time read of files; 850 >> MBPS if the files are already in the Windows OS file cache. Synchronous put >> gives me about 30 MBPS. >> >> But, when I am dealing with files on DVD, I get a network transfer rate of 5 >> MBPS for first time read of files, and 850 MBPS if files are in the OS file >> cache. >> Synchronous put gives me about 30 MBPS. >> >> So, in the case of first read of DVD files, synchronous client is superior. >> Synchronous should be faster, because it can read and send at the same time. >> >> Does this make sense? It seems a little strange to me. >> > > Aaron, > > Zero copy data transfer is very platform specific. Java merely provides a > common APIs, which however can have radically different implementations on > different platforms and therefore can have different performance > characteristics. > > I guess these numbers have more to do with Windows than with HttpAsyncClient. > > Oleg > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > This e-mail may contain confidential and/or privileged information for the > sole use of the intended recipient. > Any review or distribution by anyone other than the person for whom it was > originally intended is strictly prohibited. > If you have received this e-mail in error, please contact the sender and > delete all copies. > Opinions, conclusions or other information contained in this e-mail may not > be that of the organization. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this e-mail may not be that of the organization. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: ZeroCopyPut mystery
On 16 January 2014 16:35, Boxer, Aaron wrote: > Thanks, Oleg. I am not trying to criticize the project in any way; just > wondering if anyone has insight into why, on Windows, > Java NIO is so much slower than stream IO. If it is only slower for DVD copies, that seems more like an issue with DVD caching. Are you sure that the synch and asynch tests both started from exactly the same environment? > Perhaps it has to do with buffer size, thread count, . ? > > -Original Message- > From: Oleg Kalnichevski [mailto:ol...@apache.org] > Sent: Thursday, January 16, 2014 11:15 AM > To: HttpClient User Discussion > Subject: Re: ZeroCopyPut mystery > > On Thu, 2014-01-16 at 04:33 +, Boxer, Aaron wrote: >> Hello, >> >> When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a >> network transfer rate of about 250 MBPS for first time read of files; 850 >> MBPS if the files are already in the Windows OS file cache. Synchronous put >> gives me about 30 MBPS. >> >> But, when I am dealing with files on DVD, I get a network transfer rate of 5 >> MBPS for first time read of files, and 850 MBPS if files are in the OS file >> cache. >> Synchronous put gives me about 30 MBPS. >> >> So, in the case of first read of DVD files, synchronous client is superior. >> Synchronous should be faster, because it can read and send at the same time. >> >> Does this make sense? It seems a little strange to me. >> > > Aaron, > > Zero copy data transfer is very platform specific. Java merely provides a > common APIs, which however can have radically different implementations on > different platforms and therefore can have different performance > characteristics. > > I guess these numbers have more to do with Windows than with HttpAsyncClient. > > Oleg > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > This e-mail may contain confidential and/or privileged information for the > sole use of the intended recipient. > Any review or distribution by anyone other than the person for whom it was > originally intended is strictly prohibited. > If you have received this e-mail in error, please contact the sender and > delete all copies. > Opinions, conclusions or other information contained in this e-mail may not > be that of the organization. - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
RE: ZeroCopyPut mystery
Thanks, Oleg. I am not trying to criticize the project in any way; just wondering if anyone has insight into why, on Windows, Java NIO is so much slower than stream IO. Perhaps it has to do with buffer size, thread count, . ? -Original Message- From: Oleg Kalnichevski [mailto:ol...@apache.org] Sent: Thursday, January 16, 2014 11:15 AM To: HttpClient User Discussion Subject: Re: ZeroCopyPut mystery On Thu, 2014-01-16 at 04:33 +, Boxer, Aaron wrote: > Hello, > > When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a > network transfer rate of about 250 MBPS for first time read of files; 850 > MBPS if the files are already in the Windows OS file cache. Synchronous put > gives me about 30 MBPS. > > But, when I am dealing with files on DVD, I get a network transfer rate of 5 > MBPS for first time read of files, and 850 MBPS if files are in the OS file > cache. > Synchronous put gives me about 30 MBPS. > > So, in the case of first read of DVD files, synchronous client is superior. > Synchronous should be faster, because it can read and send at the same time. > > Does this make sense? It seems a little strange to me. > Aaron, Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics. I guess these numbers have more to do with Windows than with HttpAsyncClient. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org This e-mail may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this e-mail may not be that of the organization.
Re: ZeroCopyPut mystery
On Thu, 2014-01-16 at 04:33 +, Boxer, Aaron wrote: > Hello, > > When I use ZeroCopyPut with hard drive (spinning disk ) disk files, I get a > network transfer rate of about 250 MBPS for first time read of files; 850 > MBPS if the files are already in the Windows OS file cache. Synchronous put > gives me about 30 MBPS. > > But, when I am dealing with files on DVD, I get a network transfer rate of 5 > MBPS for first time read of files, and 850 MBPS if files are in the OS file > cache. > Synchronous put gives me about 30 MBPS. > > So, in the case of first read of DVD files, synchronous client is superior. > Synchronous should be faster, because it can read and send at the same time. > > Does this make sense? It seems a little strange to me. > Aaron, Zero copy data transfer is very platform specific. Java merely provides a common APIs, which however can have radically different implementations on different platforms and therefore can have different performance characteristics. I guess these numbers have more to do with Windows than with HttpAsyncClient. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
