subject:"Re\: \[rules\-users\] Needs advice to help Drools on a big problem $and avoid GC overhead$"

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

2014-03-21 Thread Raphael Jolivet

Guys,

Splitting the problem into smaller ones and retracting facts totally solved
the memory problem and also speed up the process.
I would like to thank you for your responsiveness and expertise.

I am truly amazed by the activity of this community, and Drools is an
awesome piece of software !

Thanks,
Raphael

2014-03-20 21:57 GMT+01:00 Raphael Jolivet raphael.joli...@gmail.com:

Hmm, sure it makes sense. The operations and resources are independent. I
think I can safely split the problem into smaller ones. I will try this
approach.

Thanks for the tip
Le 20 mars 2014 20:51, Wolfgang Laun wolfgang.l...@gmail.com a écrit :

I may fail to understand the fundamentals of this problem, but it seems
to me that it doesn't have the dynamic requirements that would
warrant a full blow up of all Resources and Operations at the same time.

We have the 300 rules, and I'm assuming that the pattern is the one
shown in OP's mail: a Resource subset times an Operation subset. So,
if you insert just one Resource (or Operation) fact together with
all Operation (or Resource) facts and fire all rules, you'll get all
Forbids
for the one Resource with all Operations (or vice versa). Then, you
retract the singleton and insert the next one, fire all rules, and so on.

-W

On 20/03/2014, Raphael Jolivet raphael.joli...@gmail.com wrote:
Thanks for you quick answer.

Not yet,

For that I will need to switch to fact insertion instead of Java Set.
Do you think to will make a difference in the way the RETE Tree will be
built ?

2014-03-20 20:27 GMT+01:00 Mark Proctor mproc...@codehaus.org:

Did you try using the engine in equality mode? It will only ever allow
one
equal object to be inserted, others are just ignored.

Mark
On 20 Mar 2014, at 18:54, Raphael raphael.joli...@gmail.com wrote:

Hi,

In our project, we use Drools to compute forbidden allocations as
input
for
an allocation problem (for which we do not use Drools BTW).

The constraint of the problem (forbidden operation for given
resources)
are
expressed as drools rules.

We have about
-200 resources
-700 operations

Those are the Drools facts.
We have about 200 Drools rules describing the forbidden allocations

The problem here is that many rules produce duplicates forbids.
For instance :
Rule 1 may forbid Op1 on Res1 Res3
Rule 2 may forbid Op1 on Res2 res3

Currently the RHS of rules just insert forbids into a pure Java Set
(no
fact
insertion).
The Set takes care of avoiding duplicates, but still, we feel we
could
help
Drools by giving him a hint that
all rules are actually producing pairs of Resource, Allocation and
that he
should not try to produce them more than once.

Currently the structure of each rule looks like that :

when
$resource : Resource(someconditions)
$operation : Operation(someconditions)
then
globalSet.add(Pairresrouce, operation)
end

With the size of the problem, we often end up with outOfMem/ GC
limit.
When analyzing memory dump, we see about
300.000 drools left tuple object.

I am not sure if we could help drools to reduce the Rete tree here.
I mean, the allocation matrix is already : 200*700 = 140.000 cells.
Drools only uses 2 nodes for each possible forbids, which seems
already
well
optimized.

We had several idea to improve that though, but we are not enough
expert
in
the internal of the RETE algorithm to decide if that's a good way to
go
:

1) Insert forbids as facts in the RHS and check they are not already
there
in the LHS :
when
$resource : Resource(someconditions)
$operation : Operation(someconditions)
not Forbid(res=$resource, op=$operation)
then
insert(Forbid($resource, $operation))
end

2) Merge all rules in one big rule with a single RHS
Is it possible to have something like :
when
// Rule 1
($resource : Resource(someconditions)
$operation : Operation(someconditions))
or
// Rule 2
($resource : Resource(someconditions)
$operation : Operation(someconditions))
// Etc ...
then
globalSet.add(Pairresrouce, operation)
end

3) Define a common RHS somewhere and tell Drools all rules use this
one.
Is there a possibility in Drools to define reusable RHS ? Would it
help
?

4) Any other idea ?

Thanks in advance for your help.
Brgds,

Raphael Jolivet

--
View this message in context:

http://drools.46999.n3.nabble.com/Needs-advice-to-help-Drools-on-a-big-problem-and-avoid-GC-overhead-tp4028838.html
Sent from the Drools: User forum mailing list archive at Nabble.com.
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

2014-03-20 Thread Mark Proctor

Did you try using the engine in equality mode? It will only ever allow one 
“equal” object to be inserted, others are just ignored.

Mark
On 20 Mar 2014, at 18:54, Raphael raphael.joli...@gmail.com wrote:

 Hi,
 
 In our project, we use Drools to compute forbidden allocations as input for
 an allocation problem  (for which we do not use Drools BTW).
 
 The constraint of the problem (forbidden operation for given resources) are
 expressed as drools rules.
 
 We have about
 -200 resources
 -700 operations
 
 Those are the Drools facts.
 We have about 200 Drools rules describing the forbidden allocations
 
 The problem here is that many rules produce duplicates forbids.
 For instance :
 Rule 1 may forbid Op1 on Res1  Res3
 Rule 2 may forbid Op1 on Res2  res3
 
 Currently the RHS of rules just insert forbids into a pure Java Set (no fact
 insertion).
 The Set takes care of avoiding duplicates, but still, we feel we could help
 Drools by giving him a hint that
 all rules are actually producing pairs of Resource, Allocation and that he
 should not try to produce them more than once.
 
 Currently the structure of each rule looks like that :
 
 when
 $resource : Resource(someconditions)
 $operation : Operation(someconditions)
 then
  globalSet.add(Pairresrouce, operation) 
 end
 
 With the size of the problem, we often end up with outOfMem/ GC limit.
 When analyzing memory dump, we see about 
 300.000 drools left tuple object.
 
 I am not sure if we could help drools to reduce the Rete tree here.
 I mean, the allocation matrix is already : 200*700 = 140.000 cells.
 Drools only uses 2 nodes for each possible forbids, which seems already well
 optimized.
 
 We had several idea to improve that though, but we are not enough expert in
 the internal of the RETE algorithm to decide if that's a good way to go :
 
 1) Insert forbids as facts in the RHS and check they are not already there
 in the LHS :
 when
 $resource : Resource(someconditions)
 $operation : Operation(someconditions)
 not Forbid(res=$resource, op=$operation)
 then
  insert(Forbid($resource, $operation))
 end
 
 2) Merge all rules in one big rule with a single RHS
 Is it possible to have something like :
 when
 // Rule 1 
 ($resource : Resource(someconditions)
 $operation : Operation(someconditions))
  or
 // Rule 2 
 ($resource : Resource(someconditions)
 $operation : Operation(someconditions))
 // Etc ...
 then
  globalSet.add(Pairresrouce, operation) 
 end
 
 3) Define a common RHS somewhere and tell Drools all rules use this one.
 Is there a possibility in Drools to define reusable RHS ? Would it help ?
 
 4) Any other idea ?
 
 
 Thanks in advance for your help.
 Brgds,
 
 Raphael Jolivet
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 --
 View this message in context: 
 http://drools.46999.n3.nabble.com/Needs-advice-to-help-Drools-on-a-big-problem-and-avoid-GC-overhead-tp4028838.html
 Sent from the Drools: User forum mailing list archive at Nabble.com.
 ___
 rules-users mailing list
 rules-users@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/rules-users


___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

2014-03-20 Thread Raphael Jolivet

Thanks for you quick answer.

Not yet,

For that I will need to switch to fact insertion instead of Java Set.
Do you think to will make a difference in the way the RETE Tree will be
built ?




2014-03-20 20:27 GMT+01:00 Mark Proctor mproc...@codehaus.org:

 Did you try using the engine in equality mode? It will only ever allow one
 equal object to be inserted, others are just ignored.

 Mark
 On 20 Mar 2014, at 18:54, Raphael raphael.joli...@gmail.com wrote:

  Hi,
 
  In our project, we use Drools to compute forbidden allocations as input
 for
  an allocation problem  (for which we do not use Drools BTW).
 
  The constraint of the problem (forbidden operation for given resources)
 are
  expressed as drools rules.
 
  We have about
  -200 resources
  -700 operations
 
  Those are the Drools facts.
  We have about 200 Drools rules describing the forbidden allocations
 
  The problem here is that many rules produce duplicates forbids.
  For instance :
  Rule 1 may forbid Op1 on Res1  Res3
  Rule 2 may forbid Op1 on Res2  res3
 
  Currently the RHS of rules just insert forbids into a pure Java Set (no
 fact
  insertion).
  The Set takes care of avoiding duplicates, but still, we feel we could
 help
  Drools by giving him a hint that
  all rules are actually producing pairs of Resource, Allocation and
 that he
  should not try to produce them more than once.
 
  Currently the structure of each rule looks like that :
 
  when
  $resource : Resource(someconditions)
  $operation : Operation(someconditions)
  then
   globalSet.add(Pairresrouce, operation)
  end
 
  With the size of the problem, we often end up with outOfMem/ GC limit.
  When analyzing memory dump, we see about
  300.000 drools left tuple object.
 
  I am not sure if we could help drools to reduce the Rete tree here.
  I mean, the allocation matrix is already : 200*700 = 140.000 cells.
  Drools only uses 2 nodes for each possible forbids, which seems already
 well
  optimized.
 
  We had several idea to improve that though, but we are not enough expert
 in
  the internal of the RETE algorithm to decide if that's a good way to go :
 
  1) Insert forbids as facts in the RHS and check they are not already
 there
  in the LHS :
  when
  $resource : Resource(someconditions)
  $operation : Operation(someconditions)
  not Forbid(res=$resource, op=$operation)
  then
   insert(Forbid($resource, $operation))
  end
 
  2) Merge all rules in one big rule with a single RHS
  Is it possible to have something like :
  when
  // Rule 1
  ($resource : Resource(someconditions)
  $operation : Operation(someconditions))
   or
  // Rule 2
  ($resource : Resource(someconditions)
  $operation : Operation(someconditions))
  // Etc ...
  then
   globalSet.add(Pairresrouce, operation)
  end
 
  3) Define a common RHS somewhere and tell Drools all rules use this one.
  Is there a possibility in Drools to define reusable RHS ? Would it help ?
 
  4) Any other idea ?
 
 
  Thanks in advance for your help.
  Brgds,
 
  Raphael Jolivet
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  --
  View this message in context:
 http://drools.46999.n3.nabble.com/Needs-advice-to-help-Drools-on-a-big-problem-and-avoid-GC-overhead-tp4028838.html
  Sent from the Drools: User forum mailing list archive at Nabble.com.
  ___
  rules-users mailing list
  rules-users@lists.jboss.org
  https://lists.jboss.org/mailman/listinfo/rules-users


 ___
 rules-users mailing list
 rules-users@lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/rules-users

___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

2014-03-20 Thread Wolfgang Laun

We have the 300 rules, and I'm assuming that the pattern is the one
shown in OP's mail: a Resource subset times an Operation subset. So,
if you insert just one Resource (or Operation) fact together with
all Operation (or Resource) facts and fire all rules, you'll get all Forbids
for the one Resource with all Operations (or vice versa). Then, you
retract the singleton and insert the next one, fire all rules, and so on.

-W

On 20/03/2014, Raphael Jolivet raphael.joli...@gmail.com wrote:
Thanks for you quick answer.

Not yet,

For that I will need to switch to fact insertion instead of Java Set.
Do you think to will make a difference in the way the RETE Tree will be
built ?

2014-03-20 20:27 GMT+01:00 Mark Proctor mproc...@codehaus.org:

Did you try using the engine in equality mode? It will only ever allow
one
equal object to be inserted, others are just ignored.

Mark
On 20 Mar 2014, at 18:54, Raphael raphael.joli...@gmail.com wrote:

Hi,

In our project, we use Drools to compute forbidden allocations as input
for
an allocation problem (for which we do not use Drools BTW).

The constraint of the problem (forbidden operation for given resources)
are
expressed as drools rules.

We have about
-200 resources
-700 operations

Those are the Drools facts.
We have about 200 Drools rules describing the forbidden allocations

The problem here is that many rules produce duplicates forbids.
For instance :
Rule 1 may forbid Op1 on Res1 Res3
Rule 2 may forbid Op1 on Res2 res3

Currently the RHS of rules just insert forbids into a pure Java Set (no
fact
insertion).
The Set takes care of avoiding duplicates, but still, we feel we could
help
Drools by giving him a hint that
all rules are actually producing pairs of Resource, Allocation and
that he
should not try to produce them more than once.

Currently the structure of each rule looks like that :

when
$resource : Resource(someconditions)
$operation : Operation(someconditions)
then
globalSet.add(Pairresrouce, operation)
end

With the size of the problem, we often end up with outOfMem/ GC limit.
When analyzing memory dump, we see about
300.000 drools left tuple object.

We had several idea to improve that though, but we are not enough
expert
in
the internal of the RETE algorithm to decide if that's a good way to go
:

3) Define a common RHS somewhere and tell Drools all rules use this
one.
Is there a possibility in Drools to define reusable RHS ? Would it help
?

4) Any other idea ?

Thanks in advance for your help.
Brgds,

Raphael Jolivet

--
View this message in context:
http://drools.46999.n3.nabble.com/Needs-advice-to-help-Drools-on-a-big-problem-and-avoid-GC-overhead-tp4028838.html
Sent from the Drools: User forum mailing list archive at Nabble.com.
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

2014-03-20 Thread Mark Proctor

On 20 Mar 2014, at 19:34, Raphael Jolivet raphael.joli...@gmail.com wrote:

Thanks for you quick answer.

Not yet,

For that I will need to switch to fact insertion instead of Java Set.
Do you think to will make a difference in the way the RETE Tree will be built
?
No it’s just a “set” pre filter for insertions.

The only way to change how the network is built, is to change the rules
themselves. You can improve node sharing, if the patterns are in the same order
and have the same constraints.

You might find some improvements by moving to 6.0, where we have a new “lazy”
algorithm. Which will only produce the LeftTuples on demand, for when a rule
fires.
http://blog.athico.com/2013/11/rip-rete-time-to-get-phreaky.html

Mark

2014-03-20 20:27 GMT+01:00 Mark Proctor mproc...@codehaus.org:
Did you try using the engine in equality mode? It will only ever allow one
“equal” object to be inserted, others are just ignored.

Mark
On 20 Mar 2014, at 18:54, Raphael raphael.joli...@gmail.com wrote:

Hi,

In our project, we use Drools to compute forbidden allocations as input for
an allocation problem (for which we do not use Drools BTW).

The constraint of the problem (forbidden operation for given resources) are
expressed as drools rules.

We have about
-200 resources
-700 operations

Those are the Drools facts.
We have about 200 Drools rules describing the forbidden allocations

The problem here is that many rules produce duplicates forbids.
For instance :
Rule 1 may forbid Op1 on Res1 Res3
Rule 2 may forbid Op1 on Res2 res3

Currently the RHS of rules just insert forbids into a pure Java Set (no fact
insertion).
The Set takes care of avoiding duplicates, but still, we feel we could help
Drools by giving him a hint that
all rules are actually producing pairs of Resource, Allocation and that he
should not try to produce them more than once.

Currently the structure of each rule looks like that :

when
$resource : Resource(someconditions)
$operation : Operation(someconditions)
then
globalSet.add(Pairresrouce, operation)
end

With the size of the problem, we often end up with outOfMem/ GC limit.
When analyzing memory dump, we see about
300.000 drools left tuple object.

We had several idea to improve that though, but we are not enough expert in
the internal of the RETE algorithm to decide if that's a good way to go :

1) Insert forbids as facts in the RHS and check they are not already there
in the LHS :
when
$resource : Resource(someconditions)
$operation : Operation(someconditions)
not Forbid(res=$resource, op=$operation)
then
insert(Forbid($resource, $operation))
end

3) Define a common RHS somewhere and tell Drools all rules use this one.
Is there a possibility in Drools to define reusable RHS ? Would it help ?

4) Any other idea ?

Thanks in advance for your help.
Brgds,

Raphael Jolivet

___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

2014-03-20 Thread Raphael Jolivet

Hmm, sure it makes sense. The operations and resources are independent. I
think I can safely split the problem into smaller ones. I will try this
approach.

Thanks for the tip
Le 20 mars 2014 20:51, Wolfgang Laun wolfgang.l...@gmail.com a écrit :

-W

On 20/03/2014, Raphael Jolivet raphael.joli...@gmail.com wrote:
Thanks for you quick answer.

Not yet,

For that I will need to switch to fact insertion instead of Java Set.
Do you think to will make a difference in the way the RETE Tree will be
built ?

2014-03-20 20:27 GMT+01:00 Mark Proctor mproc...@codehaus.org:

Did you try using the engine in equality mode? It will only ever allow
one
equal object to be inserted, others are just ignored.

Mark
On 20 Mar 2014, at 18:54, Raphael raphael.joli...@gmail.com wrote:

Hi,

In our project, we use Drools to compute forbidden allocations as
input
for
an allocation problem (for which we do not use Drools BTW).

The constraint of the problem (forbidden operation for given
resources)
are
expressed as drools rules.

We have about
-200 resources
-700 operations

Those are the Drools facts.
We have about 200 Drools rules describing the forbidden allocations

The problem here is that many rules produce duplicates forbids.
For instance :
Rule 1 may forbid Op1 on Res1 Res3
Rule 2 may forbid Op1 on Res2 res3

Currently the RHS of rules just insert forbids into a pure Java Set
(no
fact
insertion).
The Set takes care of avoiding duplicates, but still, we feel we could
help
Drools by giving him a hint that
all rules are actually producing pairs of Resource, Allocation and
that he
should not try to produce them more than once.

Currently the structure of each rule looks like that :

when
$resource : Resource(someconditions)
$operation : Operation(someconditions)
then
globalSet.add(Pairresrouce, operation)
end

With the size of the problem, we often end up with outOfMem/ GC limit.
When analyzing memory dump, we see about
300.000 drools left tuple object.

We had several idea to improve that though, but we are not enough
expert
in
the internal of the RETE algorithm to decide if that's a good way to
go
:

3) Define a common RHS somewhere and tell Drools all rules use this
one.
Is there a possibility in Drools to define reusable RHS ? Would it
help
?

4) Any other idea ?

Thanks in advance for your help.
Brgds,

Raphael Jolivet

--
View this message in context:

___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

___
rules-users mailing list

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

Re: [rules-users] Needs advice to help Drools on a big problem (and avoid GC overhead)

6 matches

Site Navigation

Mail list logo

Footer information