Re: [Resteasy-users] Slow PUT request payload processing in client framework
Apologies, this DefferredOutputSTream stuff was a pull request I didn't
review thoroughly enough. I generally buffer these types of things.
The reason why this case exists is because an incompatibility with
JAX-RS and Apache Http Client. Apache HC does not let you modify
headers within HttpEntity calls. JAX-RS MessageBodyWriters require
header modification. So, I have to buffer the marshalling to memory or
disk.
We can probably put in a special case for FileInputStream as you note.
On 5/23/2013 1:15 PM, Gendler, Samuel wrote:
> There are also many utility classes available for copying between input
> and output streams or readers and writers which will utilize
> best-practices for maximizing throughput. Those should be used rather
> than a single byte copy in a tight loop, which is NEVER efficient unless
> the data is only arriving a byte at a time, and that case is handled by
> the fact that any read into a buffer will read UP TO buffer size bytes.
> See ByteStreams.copy() from google’s Guava or IOUtils in commons-io for
> well regarded utility classes.
>
> *From:*Ruusu Reino [mailto:reino.ru...@vtt.fi]
> *Sent:* Thursday, May 23, 2013 9:21 AM
> *To:* resteasy-users@lists.sourceforge.net
> *Subject:* Re: [Resteasy-users] Slow PUT request payload processing in
> client framework
>
> Hello,
>
> I’ve now looked into this a bit deeper. The culprit is a loop that
> copies data a single byte at a time from the InputStream into a
> DeferredFileOutputStream. This loop is in the method
> InputStreamProvider.writeTo().
>
> If the message body is delivered as a File instead of an InputStream,
> the process is much faster, as the data transfer occurs in packages of
> 2048 bytes in a loop at ProviderHelper.writeTo(), called by
> FileProvider.writeTo().
>
> Still, as the data is copied verbatim from the source into a temporary
> file, even when the input is itself a File object, the whole operation
> seems to be quite a bit of waste of valuable resources.
>
> Is there a way to entirely avoid the creation of this quite unnecessary
> temporary file?
>
> Alternatively, there seems to be a property for controlling the
> threshold for the creation of this temporary file, which defaults to
> only 1 MB, in the ApacheHttpClient4Executor class. What is the simplest
> way to tweak this parameter from client code?
>
> -- *
> *Reino Ruusu
> Research Scientist
> VTT Technical Research Centre of Finland / Systems Engineering
>
> *From:*Ruusu Reino [mailto:reino.ru...@vtt.fi]
> *Sent:* 23. toukokuuta 2013 15:49
> *To:* resteasy-users@lists.sourceforge.net
> <mailto:resteasy-users@lists.sourceforge.net>
> *Subject:* [Resteasy-users] Slow PUT request payload processing in
> client framework
>
> Hi,
>
> I’m a new person on this list, and I have encountered an issue with the
> RESTEasy Client Framework.
>
> While trying to transfer a 7.5 MB file over a simple PUT request using
> the client framework, the client freezes for minutes while very slowly
> building a temporary file, with 100% CPU utilization. RESTEasy version
> is 2.3.5_final.
>
> The behaviour is reproduced on a code as simple the one below. The
> temporary file is built before even looking up the address of the
> server. I don’t quite understand why a temporary file is necessary in
> the first place.
>
> Can others reproduce this problem or is it unique to my circumstances?
>
> Code:
>
> *import*java.io.FileInputStream;
>
> *import*java.io.FileNotFoundException;
>
> *import*java.io.InputStream;
>
> *import*javax.ws.rs.Consumes;
>
> *import*javax.ws.rs.PUT;
>
> *import*javax.ws.rs.Path;
>
> *import*org.jboss.resteasy.client.ProxyFactory;
>
> *import*org.junit.Test;
>
> *public**class*DummyTest {
>
> @Path("/dummy")
>
> *interface*DummyService {
>
> @PUT
>
> @Consumes("*/*")
>
> String process( InputStream data );
>
>}
>
> @Test
>
> *public**void*test() *throws*FileNotFoundException {
>
> DummyService service = ProxyFactory./create/(
> DummyService.*class*, "http://nonexistent.domain/rest/";);
>
> service.process( *new*FileInputStream( " file>") );
>
>}
>
> }
>
> -- *
> *Reino Ruusu
> Research Scientist
> VTT Technical Research Centre of Finland / Systems Engineering
>
> **
> E-mail sent through the Internet is not secure. Western Asset therefore
> recommends that you do not send any confidential or sensitive
> information to us via electronic mail, including social security
> numbers, account numbers, or personal identificati
Re: [Resteasy-users] Slow PUT request payload processing in client framework
There are also many utility classes available for copying between input and
output streams or readers and writers which will utilize best-practices for
maximizing throughput. Those should be used rather than a single byte copy in
a tight loop, which is NEVER efficient unless the data is only arriving a byte
at a time, and that case is handled by the fact that any read into a buffer
will read UP TO buffer size bytes. See ByteStreams.copy() from google's Guava
or IOUtils in commons-io for well regarded utility classes.
From: Ruusu Reino [mailto:reino.ru...@vtt.fi]
Sent: Thursday, May 23, 2013 9:21 AM
To: resteasy-users@lists.sourceforge.net
Subject: Re: [Resteasy-users] Slow PUT request payload processing in client
framework
Hello,
I've now looked into this a bit deeper. The culprit is a loop that copies data
a single byte at a time from the InputStream into a DeferredFileOutputStream.
This loop is in the method InputStreamProvider.writeTo().
If the message body is delivered as a File instead of an InputStream, the
process is much faster, as the data transfer occurs in packages of 2048 bytes
in a loop at ProviderHelper.writeTo(), called by FileProvider.writeTo().
Still, as the data is copied verbatim from the source into a temporary file,
even when the input is itself a File object, the whole operation seems to be
quite a bit of waste of valuable resources.
Is there a way to entirely avoid the creation of this quite unnecessary
temporary file?
Alternatively, there seems to be a property for controlling the threshold for
the creation of this temporary file, which defaults to only 1 MB, in the
ApacheHttpClient4Executor class. What is the simplest way to tweak this
parameter from client code?
--
Reino Ruusu
Research Scientist
VTT Technical Research Centre of Finland / Systems Engineering
From: Ruusu Reino [mailto:reino.ru...@vtt.fi]
Sent: 23. toukokuuta 2013 15:49
To:
resteasy-users@lists.sourceforge.net<mailto:resteasy-users@lists.sourceforge.net>
Subject: [Resteasy-users] Slow PUT request payload processing in client
framework
Hi,
I'm a new person on this list, and I have encountered an issue with the
RESTEasy Client Framework.
While trying to transfer a 7.5 MB file over a simple PUT request using the
client framework, the client freezes for minutes while very slowly building a
temporary file, with 100% CPU utilization. RESTEasy version is 2.3.5_final.
The behaviour is reproduced on a code as simple the one below. The temporary
file is built before even looking up the address of the server. I don't quite
understand why a temporary file is necessary in the first place.
Can others reproduce this problem or is it unique to my circumstances?
Code:
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import javax.ws.rs.Consumes;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import org.jboss.resteasy.client.ProxyFactory;
import org.junit.Test;
public class DummyTest {
@Path("/dummy")
interface DummyService {
@PUT
@Consumes("*/*")
String process( InputStream data );
}
@Test
public void test() throws FileNotFoundException {
DummyService service = ProxyFactory.create( DummyService.class,
"http://nonexistent.domain/rest/"; );
service.process( new FileInputStream( "" ) );
}
}
--
Reino Ruusu
Research Scientist
VTT Technical Research Centre of Finland / Systems Engineering
**
E-mail sent through the Internet is not secure. Western Asset
therefore recommends that you do not send any confidential or
sensitive information to us via electronic mail, including social
security numbers, account numbers, or personal identification
numbers. Delivery, and or timely delivery of Internet mail is not
guaranteed. Western Asset therefore recommends that you do not send
time sensitive or action-oriented messages to us via electronic
mail.
**
--
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may___
Resteasy-users mailing list
Resteasy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/resteasy-users
Re: [Resteasy-users] Slow PUT request payload processing in client framework
Hello,
I've now looked into this a bit deeper. The culprit is a loop that copies data
a single byte at a time from the InputStream into a DeferredFileOutputStream.
This loop is in the method InputStreamProvider.writeTo().
If the message body is delivered as a File instead of an InputStream, the
process is much faster, as the data transfer occurs in packages of 2048 bytes
in a loop at ProviderHelper.writeTo(), called by FileProvider.writeTo().
Still, as the data is copied verbatim from the source into a temporary file,
even when the input is itself a File object, the whole operation seems to be
quite a bit of waste of valuable resources.
Is there a way to entirely avoid the creation of this quite unnecessary
temporary file?
Alternatively, there seems to be a property for controlling the threshold for
the creation of this temporary file, which defaults to only 1 MB, in the
ApacheHttpClient4Executor class. What is the simplest way to tweak this
parameter from client code?
--
Reino Ruusu
Research Scientist
VTT Technical Research Centre of Finland / Systems Engineering
From: Ruusu Reino [mailto:reino.ru...@vtt.fi]
Sent: 23. toukokuuta 2013 15:49
To: resteasy-users@lists.sourceforge.net
Subject: [Resteasy-users] Slow PUT request payload processing in client
framework
Hi,
I'm a new person on this list, and I have encountered an issue with the
RESTEasy Client Framework.
While trying to transfer a 7.5 MB file over a simple PUT request using the
client framework, the client freezes for minutes while very slowly building a
temporary file, with 100% CPU utilization. RESTEasy version is 2.3.5_final.
The behaviour is reproduced on a code as simple the one below. The temporary
file is built before even looking up the address of the server. I don't quite
understand why a temporary file is necessary in the first place.
Can others reproduce this problem or is it unique to my circumstances?
Code:
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import javax.ws.rs.Consumes;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import org.jboss.resteasy.client.ProxyFactory;
import org.junit.Test;
public class DummyTest {
@Path("/dummy")
interface DummyService {
@PUT
@Consumes("*/*")
String process( InputStream data );
}
@Test
public void test() throws FileNotFoundException {
DummyService service = ProxyFactory.create( DummyService.class,
"http://nonexistent.domain/rest/"; );
service.process( new FileInputStream( "" ) );
}
}
--
Reino Ruusu
Research Scientist
VTT Technical Research Centre of Finland / Systems Engineering
--
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may___
Resteasy-users mailing list
Resteasy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/resteasy-users
[Resteasy-users] Slow PUT request payload processing in client framework
Hi,
I'm a new person on this list, and I have encountered an issue with the
RESTEasy Client Framework.
While trying to transfer a 7.5 MB file over a simple PUT request using the
client framework, the client freezes for minutes while very slowly building a
temporary file, with 100% CPU utilization. RESTEasy version is 2.3.5_final.
The behaviour is reproduced on a code as simple the one below. The temporary
file is built before even looking up the address of the server. I don't quite
understand why a temporary file is necessary in the first place.
Can others reproduce this problem or is it unique to my circumstances?
Code:
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.InputStream;
import javax.ws.rs.Consumes;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import org.jboss.resteasy.client.ProxyFactory;
import org.junit.Test;
public class DummyTest {
@Path("/dummy")
interface DummyService {
@PUT
@Consumes("*/*")
String process( InputStream data );
}
@Test
public void test() throws FileNotFoundException {
DummyService service = ProxyFactory.create( DummyService.class,
"http://nonexistent.domain/rest/"; );
service.process( new FileInputStream( "" ) );
}
}
--
Reino Ruusu
Research Scientist
VTT Technical Research Centre of Finland / Systems Engineering
--
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may___
Resteasy-users mailing list
Resteasy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/resteasy-users
