YannickR wrote:
>
>
> YannickR wrote:
>>
>>
>> YannickR wrote:
>>>
>>>
>>> YannickR wrote:
>>>>
>>>>
>>>> Yannick Richard wrote:
>>>>>
>>>>>
>>>>> Thomas Vandahl-2 wrote:
>>>>>>
>>>>>> Yannick Richard wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I am currently working on a Torque project that will handle database
>>>>>>> synchronization.
>>>>>>>
>>>>>>> The problem we have is an Out of Memory exception while selecting a
>>>>>>> big
>>>>>>> bunch of data from the database.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here is the command we are using :
>>>>>>>
>>>>>>> List ObjectsFromDB = ObjectPeer.doSelect(criteria, connection);
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I saw the LargeSelect class you worked on but cannot find any Java
>>>>>>> example that could help me go forward.
>>>>>>>
>>>>>>> Could you help me point to an example or help me understand how to
>>>>>>> integrate LargeSelect ?
>>>>>>
>>>>>> Just a few hints, I don't have a complete example at hand:
>>>>>>
>>>>>> LargeSelect ls = new LargeSelect(criteria, pageSize,
>>>>>> memoryPageLimit,
>>>>>> ObjectPeer.class.getName());
>>>>>>
>>>>>> where the pageSize defines how many records to get with one call and
>>>>>> the
>>>>>> memoryPageLimit defines how many of these pages to "read ahead".
>>>>>>
>>>>>> With this object you can now loop through the pages and LargeSelect
>>>>>> will
>>>>>> load the necessary data as needed, (pageSize * memoryPageLimit)
>>>>>> records
>>>>>> at a time. Like:
>>>>>>
>>>>>> while (ls.getNextResultsAvailable())
>>>>>> {
>>>>>> List ObjectsFromDB = ls.getNextResults();
>>>>>> // do what is necessary
>>>>>> }
>>>>>>
>>>>>> See the JavaDoc at
>>>>>> http://db.apache.org/torque/releases/torque-3.3/runtime/apidocs/index.html
>>>>>> for more information.
>>>>>>
>>>>>> Bye, Thomas.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> With your explanations I succeeded in running LargeSelect. It is now
>>>>> working while reading 10 000 records with a PageSize of 1000 and a
>>>>> MemoryPageLimit of 20. I don't know if it is normal, but when I use a
>>>>> MemoryPageLimit of 5 (5x1000 records), it is reading 1 - 5000 of 10000
>>>>> again and again and never get out of the loop...
>>>>>
>>>>> Next, I tried to use PageSize of 5000/MemoryPageLimit of 50 in order
>>>>> to read less than 250 000 records, I had following exception :
>>>>> Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap
>>>>> space
>>>>>
>>>>> I then modified the Eclipse shortcut arguments to better manage Heap
>>>>> memory, etc :
>>>>> -vmargs -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -Xms128M -Xmx1024M
>>>>>
>>>>> Unfortunately, it jammed on first call of .getNextResults() for a
>>>>> night...
>>>>> When I debug it I can see that the Thread is sleeping in
>>>>> getResults(start, size) for all night on following command: while
>>>>> (((start + size - 1) > currentlyFilledTo) && !queryCompleted)
>>>>>
>>>>> Is this a memory problem or I did something wrong ?
>>>>>
>>>>> Regards,
>>>>> Yannick Richard
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Never mind, I found the solution :jumping:
>>> Readind Torque documentation, I tought MSSQL was not supported with
>>> Limit and Offset features used by LargeSelect so I installed TORQUE-84
>>> patch from https://issues.apache.org/jira/browse/TORQUE-84 and this was
>>> causing getNextResultsAvailable() to not function correctly...
>>>
>>> I was also misunderstanding the use of PageSize and MemoryPageLimit.
>>> Total batch size in memory is not only the number of records in a page
>>> but PageSize * MemoryPageLimit...
>>>
>>> So without that patch, that is not completed anyway, and some time spent
>>> on RTFM... Everything is fine now ! Thanks for your help !
>>>
>>>
>>
>
> Sorry for confusion...
> I tough I succeeded to run LargeSelect with SQL server 2005, but it
> fails... continuing to read even after there are no more records...
> I did install TORQUE-84 patch in order to support Limit/Offset.
>
> My last post on the forum, talking about removing the Patch, is wrong...
> It only works when your specify PageSize * MemoryPageLimit that will cover
> all your records... If not, it will end reading before the end of the
> records.
>
> Is the patch working or not ? The status on
> https://issues.apache.org/jira/browse/TORQUE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> seems to be unresolved...
>
> Could someone help me on that one ?
> Can I still use LargeSelect with MSSQL 2005 ?
>
> Regards,
> Yannick Richard
>
--
View this message in context:
http://www.nabble.com/LargeSelect-example---tf4605414.html#a13430608
Sent from the Apache DB - Torque Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
