Hi Oleg,
*if (src.remaining() < 1024 && buffer().remaining() > src.remaining()) {*
Problem with this above condition is that even though buffer has capacity
to store complete contents of src, this condition will not satisfy if src
has 1024 bytes or more data (may well be in limit of actual size provided
during creation of buffer) to be written to buffer
Simple way to reproduce the problem is (Same has been demonstrated with
sample program in previous mail)
- Create SharedOutputBuffer with 4096 bytes of capacity
- Now try to write 1024 bytes of data to above created buffer
second step still will cause in a hang as first if condition is not
satisfied due to *src.remaining() < 1024* even though there is enough
capacity to store data and it proceeds to second condition which will cause
the program to wait on a condition for flush (even though this is the first
write we are performing)
*if (buffer().position() > 0 || dataStreamChannel == null) {*
* waitFlush();*
*}*
Regards,
Sreenivas
On Mon, Mar 6, 2023 at 9:12 PM Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2023-03-06 at 21:03 +0530, sreenivas somavarapu wrote:
> > Hi Oleg,
> >
> > My sample program (it's a kind of unit test for SharedOutputBuffer to
> > demonstrate that it is not able to write single byte if input byte
> > array
> > size is greater than 1023) doesn't have flush as my intention is to
> > show
> > the issue with SharedOutputBuffer alone and it is not a complete
> > program
> > which sends the data to the actual server and get response back. Main
> > issue
> > lies while writing data greater than 1023 bytes at first go itself
> > to SharedOutputBuffer from the byte array.
> >
> > I think the below conditions in SharedOutputBuffer is causing this
> > behavior
> >
> > while (src.hasRemaining()) {
> > // always buffer small chunks
> > *if (src.remaining() < 1024 && buffer().remaining() >
> > src.remaining()) {*
> > buffer().put(src);
> > } else {
> > * if (buffer().position() > 0 || dataStreamChannel
> > ==
> > null) {*
> > waitFlush();
> > }
> >
>
> What behavior? This code blocks if it is unable to store input data
> until the internal buffer gets flushed and space in it frees up, which
> is what is it supposed to do.
>
> If you think SharedOutputBuffer is wrong, please just use your own
> custom buffer implementation.
>
> Oleg
>
>
> > Regards,
> > Sreenivas
> >
> > On Mon, Mar 6, 2023 at 8:52 PM Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> >
> > > On Mon, 2023-03-06 at 20:20 +0530, sreenivas somavarapu wrote:
> > > > Hi Oleg,
> > > >
> > > > It looks like some issue with SharedOutputBuffer implementation
> > > > and
> > > > it is
> > > > also not expandable as we have to allocate complete size during
> > > > initialization itself. If I change below two lines from my
> > > > program it
> > > > proceeds further and completes the request (Basically
> > > > SharedOutputBuffer is
> > > > not able to write 1024 or above bytes of data at single go (If we
> > > > declare
> > > > temp block size as 1023 also it works)). It looks like there is
> > > > an
> > > > explicit 1024 length check in SharedOutputBuffer which is causing
> > > > this
> > > > behavior.
> > > >
> > > > SharedOutputBuffer buffer = new
> > > > SharedOutputBuffer(req.contentLength() +
> > > > 1024);
> > >
> > > What you are doing is _completely_ wrong. SharedOutputBuffer is not
> > > meant to be expandable.
> > >
> > > You cannot just write into the buffer and expect it to work by
> > > magic.
> > > You also must flush the buffer.
> > >
> > > Oleg
> > >
> > > >
> > > > byte[] tmp = new byte[512];
> > > >
> > > > Earlier those 2 lines were as below
> > > >
> > > > SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> > > >
> > > > byte[] tmp = new byte[4 * 1024];
> > > >
> > > > Below is my sample program on SharedOutputBuffer which
> > > > demonstrated
> > > > this
> > > > behavior
> > > >
> > > > final StringBuilder sbuffer = new StringBuilder("123");
> > > > System.out.println("O/P: " + sbuffer.toString());
> > > > for (long i = 0; i < (64 * 1000); ++i) {
> > > > sbuffer.append('A');
> > > > }
> > > >
> > > > String postData = sbuffer.toString();
> > > > System.out.println("O/P1: " + postData);
> > > >
> > > > byte[] inputData =
> > > > postData.getBytes(StandardCharsets.UTF_8);
> > > > ByteArrayInputStream inputStream = new
> > > > ByteArrayInputStream(inputData);
> > > > SharedOutputBuffer outputStream = new
> > > > SharedOutputBuffer(100
> > > > *
> > > > 1024); *<-- If we keep this as 4 * 1024 also it hangs.*
> > > > *byte[] tmp = new byte[1024]; <-- If we change this to
> > > > 1023
> > > > then
> > > > it works*
> > > > int l = -1;
> > > > while ((l = inputStream.read(tmp)) != -1) {
> > > > System.out.println("Writing Data");
> > > > System.out.println(l);
> > > > outputStream.write(tmp, 0, l);
> > > > System.out.println("Wrote Data");
> > > > }
> > > >
> > > > System.out.println("Done Writing Data");
> > > > outputStream.writeCompleted();
> > > > System.out.println(outputStream.toString());
> > > >
> > > > Regards,
> > > > Sreenivas
> > > >
> > > > On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski
> > > > <ol...@apache.org>
> > > > wrote:
> > > >
> > > > > On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> > > > > > Hi Team,
> > > > > >
> > > > > > We are using AbstractClassicEntityProducer with a
> > > > > > customization
> > > > > > of
> > > > > > storing
> > > > > > / computing contentLength as well which works fine if we have
> > > > > > small
> > > > > > POST
> > > > > > data, but when POST data is big (for example 64 KB), it seems
> > > > > > OutPutstream
> > > > > > (internally using ContentOutputStream a wrapper over
> > > > > > SharedOutputBuffer of
> > > > > > initial size 1024) is not able to write data (We are using
> > > > > > temporary
> > > > > > buffer
> > > > > > block of 4kb but still it is not able to write 4KB of data).
> > > > > > Is
> > > > > > there
> > > > > > any
> > > > > > limitation or some configuration problem?
> > > > > >
> > > > > > *Code being used*
> > > > > > SharedOutputBuffer buffer = new
> > > > > > SharedOutputBuffer(1024);
> > > > > >
> > > > > > produceData(contentType, new
> > > > > > ContentOutputStream(buffer));
> > > > > >
> > > > > > protected void produceData(ContentType contentType,
> > > > > > OutputStream
> > > > > > outputStream) throws IOException {
> > > > > > byte[] tmp = new byte[4 * 1024];
> > > > > > int l = -1;
> > > > > > while ((l = inputStream.read(tmp)) != -1) {
> > > > > > * outputStream.write(tmp, 0, l); <---- This is
> > > > > > where
> > > > > > request
> > > > > > hangs / waits*
> > > > >
> > > > > This is likely due to a bug in your code that prevents the
> > > > > shared
> > > > > buffer from being flushed.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> > > > > ----
> > > > > --
> > > > > To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscr...@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-h...@hc.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-h...@hc.apache.org
> > >
> > >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
>
--
Cheers,
S. Sreenivas
