One thing to keep in mind is that with database queries and OFBiz there is no
inherent nature of how it handles large data sets. Each bit of code and each
query needs to be designed with a certain expected data set size, and typically
the more data you want to allow for in the data store the more effort and
testing it requires.
In other words, a query and bit of code that doesn't handle a certain large
table means absolutely nothing about how other queries and other code will
handle similarly large tables.
-David
On Aug 17, 2010, at 9:33 AM, Ruth Hoffman wrote:
> Hi Micha:
>
> Could I ask how big are your data sets? The reason I ask is that several
> years ago, prior to 9.04 I had an out-of-memory problem that was never
> resolved. The database tables that I was querying had between 4 - 12 million
> records. My database (and OFBiz usage of the same) went away before I
> couldn't do any further investigation into the problem.
>
> So, I'm still curious about OFBiz support for really large data sets.
>
> Regards,
> Ruth
>
> Michał Cukierman wrote:
>> Thanks for the quick answer.
>>
>> I have allocated 2GB of memory, so it's not a case. The problem is that
>> I work on huge data sets. Version is 10.4 but 9.04 is has the same code.
>> I have ommited those informations 'couse as you see - it's not relevant
>> to the topic.
>>
>> I don't understand your last assumption. Maybe it will someting in my
>> understanding?
>>
>> For now I think the best would be to improve the code I have proposed as
>> it fixes:
>> 1) My issues
>> 2) Coding way of selecting everything to get a couple of categories.
>>
>> So, thats why I asked about:
>>
>>
>>> 1) How to implement Right/Left joins on DynamicViewEntity
>> (I was able to find out how to creat inner join only)
>>
>>>> 2) How to implement my whereString
>> (Haven't found in a code such a use case)
>>
>>
>> Regards,
>> Michał
>>
>>
>>
>> Dnia 2010-08-17, wto o godzinie 07:54 -0700, BJ Freeman pisze:
>>
>>> as far as performance how much memory have you allocated to ofbiz?
>>> what version of ofbiz are you using?
>>> There is a top Category that should be checked for first.
>>> have not been in that part of the code for a while.
>>>
>>> Michał Cukierman sent the following on 8/17/2010 7:41 AM:
>>>
>>>> Hi,
>>>>
>>>> I have a problem with performence (and Out of Memmory Issue) with
>>>> catalog management module. There is a section invoked after
>>>> Choose Top Category
>>>>
>>>> Collection<GenericValue> allCategories =
>>>> delegator.findList("ProductCategory", null, null, null, null, false);
>>>> for (GenericValue curCat: allCategories) {
>>>> Collection<GenericValue> parentCats =
>>>> curCat.getRelatedCache("CurrentProductCategoryRollup");
>>>>
>>>> if (parentCats.isEmpty())
>>>> results.add(curCat);
>>>> }
>>>>
>>>> As we see. To get categories without parents we need to fetch for all
>>>> categories and for each of them, find a parent.
>>>>
>>>> I do not know Ofbiz api very well for now, so I have implemented:
>>>> EntityWhereString where =
>>>> EntityCondition.makeConditionWhere("PRODUCT_CATEGORY.PRODUCT_CATEGORY_ID
>>>> not in" +
>>>> " (select PRODUCT_CATEGORY_ROLLUP.PRODUCT_CATEGORY_ID from
>>>> PRODUCT_CATEGORY_ROLLUP)");
>>>> Set<String> fieldsToSelect = FastSet.newInstance();
>>>> fieldsToSelect.add("productCategoryId");
>>>> results.addAll(delegator.findList("ProductCategory",
>>>> where,
>>>> fieldsToSelect, null, null, false));
>>>>
>>>>
>>>> I simply do invoke a SQL query in the database:
>>>>
>>>> select PRODUCT_CATEGORY.PRODUCT_CATEGORY_ID from PRODUCT_CATEGORY where
>>>> PRODUCT_CATEGORY.PRODUCT_CATEGORY_ID not in
>>>> (select PRODUCT_CATEGORY_ROLLUP.PRODUCT_CATEGORY_ID from
>>>> PRODUCT_CATEGORY_ROLLUP);
>>>>
>>>> Is there a way ( for sure is ) to implement it in a better manner? For
>>>> now it works on my system, but after improving the query it could be
>>>> passed to a production.
>>>>
>>>> So I got two questions in fact:
>>>>
>>>> 1) How to implement Right/Left joins on DynamicViewEntity
>>>> 2) How to implement my whereString
>>>>
>>>>
>>>> Thank you in advance,
>>>>
>>>>
>>>>
>>
>>
>>
