Correct. So you have to make sure that you have enough memory to handle the fetchSize * concurrent requests.
On Tue, Oct 6, 2015 at 10:34 AM Sumit Nigam <sumit_o...@yahoo.com> wrote: > Thanks Samarth and Jesse. > > So, in effect setting the batch size (say, stmt.setFetchSize()) ensures > that only that much data is copied over the wire en-mass? And > 'behind-the-scenes', Phoenix driver would fetch next batch as each fetch > size is exhausted? > > Thanks, > Sumit > > ------------------------------ > *From:* Samarth Jain <samarth.j...@gmail.com> > *To:* "user@phoenix.apache.org" <user@phoenix.apache.org> > *Cc:* Sumit Nigam <sumit_o...@yahoo.com> > *Sent:* Tuesday, October 6, 2015 9:20 PM > *Subject:* Re: ResultSet size > > To add to what Jesse said, you can override the default scanner fetch size > programmatically via Phoenix by calling statement.setFetchSize(int). > On Tuesday, October 6, 2015, Jesse Yates <jesse.k.ya...@gmail.com> wrote: > > > So HBase (and by extension, Phoenix) does not do true "streaming" of rows > - rows are copied into memory from the HFiles and then eventually copied > en-mass onto the wire. On the client they are pulled off in chunks and > paged through by the client scanner. You can control the batch size (amount > of rows in each 'page') via the usual HBase client configurations > > On Tue, Oct 6, 2015 at 8:38 AM Sumit Nigam <sumit_o...@yahoo.com> wrote: > > Hi, > > Does Phoenix buffer the result set internally? I mean when I fire a huge > skip scan IN clause, then the data being returned may be too huge to > contain in memory. So, ideally I'd like to stream data through the > resultset.next() method. So, my question is does Phoenix really stream > results? > > And if so, is there a way to control how much is loads in one time in > client side before its next() fetches next batch of data from region > servers to client? > > Best regards, > Sumit > > > >
