Yeah, so you can see that code creates a String array containing the whole
result set. Usually a very bad idea for 400K-row result sets. You want to
process results incrementally, probably via paging using row-value
constructors and LIMIT.
On Mon, May 18, 2015 at 12:00 PM, Isart Montane <isart.mont...@gmail.com>
wrote:
> Thanks James.
>
> That code is from the node driver, I will try to get some advice from it's
> developer.
>
> Thanks,
>
>
> On Mon, May 18, 2015 at 6:34 PM, James Taylor <jamestay...@apache.org>
> wrote:
>
>> Hi Isart,
>> That code isn't Phoenix code. This sounds like a Node JS issue. Vaclav
>> has done a lot with Node JS, so he may be able to give you some tips.
>> Thanks,
>> James
>>
>> On Mon, May 18, 2015 at 9:06 AM, Isart Montane <isart.mont...@gmail.com>
>> wrote:
>> > Hi Eli,
>> >
>> > thanks a lot for your comments. I think you are right. I found the
>> client
>> > code that's causing the issue. Do you have an example I can use to
>> patch it?
>> > is that the recommended way to access phoenix? I've seen on the web that
>> > there's also a query server available, is it worth a try?
>> >
>> >
>> > public String[] query(String sql)
>> > {
>> > List<String> lsResults = new ArrayList();
>> > Connection conn = null;
>> > try
>> > {
>> > conn = this.dataSource.getConnection();
>> > ResultSet rs = conn.createStatement().executeQuery(sql);
>> > ResultSetMetaData data = rs.getMetaData();
>> > int numberOfColumns = data.getColumnCount();
>> > List<String> lsRows = new ArrayList();
>> > for (int i = 1; i <= numberOfColumns; i++) {
>> > lsRows.add(data.getColumnName(i));
>> > }
>> > lsResults.add(join("\t", lsRows));
>> > lsRows.clear();
>> > while (rs.next())
>> > {
>> > for (int i = 1; i <= numberOfColumns; i++) {
>> > lsRows.add(rs.getString(i));
>> > }
>> > lsResults.add(join("\t", lsRows));
>> > lsRows.clear();
>> > }
>> > rs.close();
>> > conn.close();
>> > }
>> > catch (Exception e)
>> > {
>> > e.printStackTrace();
>> > return null;
>> > }
>> > return (String[])lsResults.toArray(new String[lsResults.size()]);
>> > }
>> >
>> > On Mon, May 18, 2015 at 5:43 PM, Eli Levine <elilev...@gmail.com>
>> wrote:
>> >>
>> >> I don't have info on what your app does with results from Phoenix. If
>> the
>> >> app is constructing some sort of object representations from Phoenix
>> results
>> >> and holding on to them, I would look at what the memory footprint of
>> that
>> >> is. I know this isn't very helpful but at this point I would try to dig
>> >> deeper into your app and the NodeJS driver rather than Phoenix, since
>> you
>> >> mentioned the same queries run fine in sqlline.
>> >>
>> >> On Mon, May 18, 2015 at 7:30 AM, Isart Montane <
>> isart.mont...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Eli,
>> >>>
>> >>> thanks a lot for your answer. That might be a workaround but I was
>> hoping
>> >>> to get a more generic answer I can apply to the driver/phoenix since
>> that
>> >>> will require me lots of changes to the code.
>> >>>
>> >>> Any clue on why it works with sqline but not trough the node driver?
>> >>>
>> >>> On Mon, May 18, 2015 at 4:20 PM, Eli Levine <elilev...@gmail.com>
>> wrote:
>> >>>>
>> >>>> Have you looked at paging [1] using Phoenix's row-value constructors
>> >>>> together with the LIMIT clause? That might be what you are looking
>> for.
>> >>>>
>> >>>> [1] http://phoenix.apache.org/paged.html
>> >>>>
>> >>>> Eli
>> >>>>
>> >>>>
>> >>>> On Mon, May 18, 2015 at 6:46 AM, Isart Montane <
>> isart.mont...@gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> the company I work for is performing some tests on Phoenix with
>> NodeJS.
>> >>>>> For simple queries I didn't had any problem, but as soon as I start
>> to use
>> >>>>> our app I'm getting "process out of memory" errors on the client
>> when I runs
>> >>>>> queries that return a big number of rows (i.e. 400k) . I think the
>> problem
>> >>>>> is that the client tries to buffer all the results in RAM and that
>> kills it.
>> >>>>> The same query runs fine when I run it with sqline.
>> >>>>>
>> >>>>> So, is there a way to tell the client to stream the results (or
>> batch
>> >>>>> them) instead of buffering them all? is raising the client memory
>> the only
>> >>>>> solution?
>> >>>>>
>> >>>>> I'm using phoenix-4.3.1 and
>> https://github.com/gaodazhu/phoenix-client
>> >>>>> as the NodeJS driver
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Isart Montane
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
>
