Correct. So you have to make sure that you have enough memory to handle the fetchSize * concurrent requests.
On Tue, Oct 6, 2015 at 10:34 AM Sumit Nigam <[email protected]> wrote: > Thanks Samarth and Jesse. > > So, in effect setting the batch size (say, stmt.setFetchSize()) ensures > that only that much data is copied over the wire en-mass? And > 'behind-the-scenes', Phoenix driver would fetch next batch as each fetch > size is exhausted? > > Thanks, > Sumit > > ------------------------------ > *From:* Samarth Jain <[email protected]> > *To:* "[email protected]" <[email protected]> > *Cc:* Sumit Nigam <[email protected]> > *Sent:* Tuesday, October 6, 2015 9:20 PM > *Subject:* Re: ResultSet size > > To add to what Jesse said, you can override the default scanner fetch size > programmatically via Phoenix by calling statement.setFetchSize(int). > On Tuesday, October 6, 2015, Jesse Yates <[email protected]> wrote: > > > So HBase (and by extension, Phoenix) does not do true "streaming" of rows > - rows are copied into memory from the HFiles and then eventually copied > en-mass onto the wire. On the client they are pulled off in chunks and > paged through by the client scanner. You can control the batch size (amount > of rows in each 'page') via the usual HBase client configurations > > On Tue, Oct 6, 2015 at 8:38 AM Sumit Nigam <[email protected]> wrote: > > Hi, > > Does Phoenix buffer the result set internally? I mean when I fire a huge > skip scan IN clause, then the data being returned may be too huge to > contain in memory. So, ideally I'd like to stream data through the > resultset.next() method. So, my question is does Phoenix really stream > results? > > And if so, is there a way to control how much is loads in one time in > client side before its next() fetches next batch of data from region > servers to client? > > Best regards, > Sumit > > > >
