Re: [rules-users] Speed up inserting of rules into knowledge base

[Drooliver]

I have a very similar issue - so I'm interested in any further performance
tips or findings:

We use Drools with an interface where users can update / edit rules. Those
rules are then stored (and versioned) in a database. Afterwards the rules
are fetched again from database and added one by one in the following way:

 
for (Rule rule ...) {
knowledgeBuilder.add(ResourceFactory.newByteArrayResource(rule.getRuleContent().getBytes()),
ResourceType.DRL);

 if (kbuilder.hasErrors()) { throw Error... }
}

 kbase = KnowledgeBaseFactory.newKnowledgeBase();
 kbase.addKnowledgePackages(knowledgeBuilder.getKnowledgePackages());



We have several hundred rules and a rule checking throughput of also several
hundred rules/second (on 2 nodes). With the number of rules increasing this
KnowledgeBase update takes longer and longer (compiling all those rules) and
during this time no rule can be checked. So the system stands still from the
user point of view.

There seems to exist no possibility to refresh a rule selectively - is this
correct? If yes, then how is the best way to handle such a situation? The
first idea that comes to mind is using two KnowledgeBases in parallel...

--
View this message in context: 
http://drools-java-rules-engine.46999.n3.nabble.com/Speed-up-inserting-of-rules-into-knowledge-base-tp2211394p2779221.html
Sent from the Drools - User mailing list archive at Nabble.com.
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Speed up inserting of rules into knowledge base

[Piotr Jedrychowski]

Hello.

Some time ago I wrote an e-mail with question how to parse and load a 
large group of rules in an efficient way. I've made a lot of tests and I 
have couple conclusions. Is there a possibility that someone could 
confirm (if I'm right) or deny (if I'm wrong) my conclusions, and 
explain why Drools behave in such a way.


1) Parse rules in groups.
Parsing of rules should be done in large groups. Send to 
knowledgeBuilder.add() method e.g. 100 rules at one time, instead of 
parsing one by one rule.


2) Parsing takes a lot of memory.
During parsing of group about 2 rules my test application takes 
about 500MB memory.


3) Use only one package.
Loading rules into one package is extremely faster than loading every 
rule into separated package.


4) Call KnowledgeBase.add() method only once.
Prepared set of rules should be added by one call of KnowledgeBase.add() 
method instead of adding each rule by separated method call.


5) Method Session.fireAllRules() works extremely fast.
Despite fact that I load thousands of rules into KnowledgeBase, 
inserting a new fact and call Session.fireAllRules() works very 
efficient (it takes below 1ms).


Regards,
Piotr
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Speed up inserting of rules into knowledge base

[Corneil du Plessis]

you may reconsider the structure of the rules.

Push instead of pull. This means compare the file time or content to the 
database and only recreate rules that have modified. This way you can 
update the serialized packages where appropriate.


Without knowing what your rules looks like I can offer the following:
Consider using decision tables where the rules are similar but based on 
data.
Consider having rules that to retrieve the data instead of rules 
generated from the data.

Consider that some of the data may be facts and not rules.

If you can generate rules from data then you can write rules that use 
the data as facts.


On 07/01/2011 16:16, Piotr Jedrychowski wrote:

Part:
knowledgeBuilder.add(resource, ResourceType.DRL);
is the most expensive part of my source code - rest of instructions 
are nothing when you compare them to above line.


I cannot serialize the compiled rule packages because rules are 
generated from data read from database and this data isn't static. 
Data is generated before loading rules from files that user can change 
- so I have to do this:

1) read files (which could be changed by user since last JBoss startup)
2) process files and generate rows for database
3) get rows from database and generate rules for Drools

Rules aren't stored directly into database - I wasn't accurate in my 
previous e-mail. - sorry.




On 2011-01-07 14:00, Corneil du Plessis wrote:

You should only repeat

knowledgeBuilder.add(resource, ResourceType.DRL);

for each rule.

The next thing you can do is to serialize the compiled rule packages.

You should also consider using Guvnor to manage your rules.

On 07/01/2011 14:05, Piotr Jedrychowski wrote:

Hello.

I'm loading a big amount of rules during starting of JBoss (2 
rules). All rules (in string format) are available before JBoss 
startup - they are stored into database. Rules are loaded one by one 
and it takes about 90 minutes. I want to speed up this process. Is 
there something like "bulk load" for inserting rules into knowledge 
base or another smart way to fast loading a big amount of rules?


I'm using:
1) Drools 5.1
2) JBoss 4.2.3

My source code looks like this:

String rule = ...
Resource resource = ResourceFactory.newReaderResource(new 
StringReader(rule));
KnowledgeBuilder knowledgeBuilder = 
KnowledgeBuilderFactory.newKnowledgeBuilder();

knowledgeBuilder.add(resource, ResourceType.DRL);
knowledgeBase.addKnowledgePackages(knowledgeBuilder.getKnowledgePackages());

Regards,
Piotr


___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users



___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users



___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Speed up inserting of rules into knowledge base

[Swindells, Thomas]

It depends on your use case but do you actually need to generate rules each 
time? Or could you have a static set of rules and insert extra data in your 
working memory to simulate each of the rules?

If your scenario was you have different pricing rules for different customers 
and your data was something like the following:
Customer, Discount
A, 0.9
B, 0.89
C. 0.95

You could either generate 3 rules:

Rule "Customer A pricing"
When
P: Purchase(customer == "A", finalPrice == null)
Then
Modify(p) {
p.setFinalPrice(p.getPrice()*0.9);
}
End

Rule "Customer B pricing"
When
P: Purchase(customer == "B", finalPrice == null)
Then
Modify(p) {
p.setFinalPrice(p.getPrice()*0.89);
}
End

Rule "Customer C pricing"
When
P: Purchase(customer == "C", finalPrice == null)
Then
Modify(p) {
p.setFinalPrice(p.getPrice()*0.95);
}
End

Or you could insert a CustomerDetails(customer, discount) for each customer and 
have a single rule
Rule "Customer pricing"
When
C : CustomerDetails()
P : Purchase(customer == c.name, finalPrice == null)
Then
Modify(p) {
p.setFinalPrice(p.getPrice()*c.getDiscount());
}

Don't know if it will help or not - depends on your scenario.

Thomas

From: rules-users-boun...@lists.jboss.org 
[mailto:rules-users-boun...@lists.jboss.org] On Behalf Of Piotr Jedrychowski
Sent: 07 January 2011 14:16
To: Rules Users List
Subject: Re: [rules-users] Speed up inserting of rules into knowledge base

Part:
knowledgeBuilder.add(resource, ResourceType.DRL);
is the most expensive part of my source code - rest of instructions are nothing 
when you compare them to above line.

I cannot serialize the compiled rule packages because rules are generated from 
data read from database and this data isn't static. Data is generated before 
loading rules from files that user can change - so I have to do this:
1) read files (which could be changed by user since last JBoss startup)
2) process files and generate rows for database
3) get rows from database and generate rules for Drools

Rules aren't stored directly into database - I wasn't accurate in my previous 
e-mail. - sorry.



On 2011-01-07 14:00, Corneil du Plessis wrote:
You should only repeat

knowledgeBuilder.add(resource, ResourceType.DRL);

for each rule.

The next thing you can do is to serialize the compiled rule packages.

You should also consider using Guvnor to manage your rules.

On 07/01/2011 14:05, Piotr Jedrychowski wrote:
Hello.

I'm loading a big amount of rules during starting of JBoss (2 rules). All 
rules (in string format) are available before JBoss startup - they are stored 
into database. Rules are loaded one by one and it takes about 90 minutes. I 
want to speed up this process. Is there something like "bulk load" for 
inserting rules into knowledge base or another smart way to fast loading a big 
amount of rules?

I'm using:
1) Drools 5.1
2) JBoss 4.2.3

My source code looks like this:

String rule = ...
Resource resource = ResourceFactory.newReaderResource(new StringReader(rule));
KnowledgeBuilder knowledgeBuilder = 
KnowledgeBuilderFactory.newKnowledgeBuilder();
knowledgeBuilder.add(resource, ResourceType.DRL);
knowledgeBase.addKnowledgePackages(knowledgeBuilder.getKnowledgePackages());

Regards,
Piotr




___

rules-users mailing list

rules-users@lists.jboss.org<mailto:rules-users@lists.jboss.org>

https://lists.jboss.org/mailman/listinfo/rules-users





___

rules-users mailing list

rules-users@lists.jboss.org<mailto:rules-users@lists.jboss.org>

https://lists.jboss.org/mailman/listinfo/rules-users



**
This message is confidential and intended only for the addressee. If you have 
received this message in error, please immediately notify the 
postmas...@nds.com and delete it from your system as well as any copies. The 
content of e-mails as well as traffic data may be monitored by NDS for 
employment and security purposes. To protect the environment please do not 
print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, 
United Kingdom. A company registered in England and Wales. Registered no. 
3080780. VAT no. GB 603 8808 40-00
**
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Speed up inserting of rules into knowledge base

[Piotr Jedrychowski]

Part:
knowledgeBuilder.add(resource, ResourceType.DRL);
is the most expensive part of my source code - rest of instructions are 
nothing when you compare them to above line.


I cannot serialize the compiled rule packages because rules are 
generated from data read from database and this data isn't static. Data 
is generated before loading rules from files that user can change - so I 
have to do this:

1) read files (which could be changed by user since last JBoss startup)
2) process files and generate rows for database
3) get rows from database and generate rules for Drools

Rules aren't stored directly into database - I wasn't accurate in my 
previous e-mail. - sorry.




On 2011-01-07 14:00, Corneil du Plessis wrote:

You should only repeat

knowledgeBuilder.add(resource, ResourceType.DRL);

for each rule.

The next thing you can do is to serialize the compiled rule packages.

You should also consider using Guvnor to manage your rules.

On 07/01/2011 14:05, Piotr Jedrychowski wrote:

Hello.

I'm loading a big amount of rules during starting of JBoss (2 
rules). All rules (in string format) are available before JBoss 
startup - they are stored into database. Rules are loaded one by one 
and it takes about 90 minutes. I want to speed up this process. Is 
there something like "bulk load" for inserting rules into knowledge 
base or another smart way to fast loading a big amount of rules?


I'm using:
1) Drools 5.1
2) JBoss 4.2.3

My source code looks like this:

String rule = ...
Resource resource = ResourceFactory.newReaderResource(new 
StringReader(rule));
KnowledgeBuilder knowledgeBuilder = 
KnowledgeBuilderFactory.newKnowledgeBuilder();

knowledgeBuilder.add(resource, ResourceType.DRL);
knowledgeBase.addKnowledgePackages(knowledgeBuilder.getKnowledgePackages());

Regards,
Piotr


___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users



___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

Re: [rules-users] Speed up inserting of rules into knowledge base

[Corneil du Plessis]

You should only repeat

knowledgeBuilder.add(resource, ResourceType.DRL);

for each rule.

The next thing you can do is to serialize the compiled rule packages.

You should also consider using Guvnor to manage your rules.

On 07/01/2011 14:05, Piotr Jedrychowski wrote:

Hello.

I'm loading a big amount of rules during starting of JBoss (2 
rules). All rules (in string format) are available before JBoss 
startup - they are stored into database. Rules are loaded one by one 
and it takes about 90 minutes. I want to speed up this process. Is 
there something like "bulk load" for inserting rules into knowledge 
base or another smart way to fast loading a big amount of rules?


I'm using:
1) Drools 5.1
2) JBoss 4.2.3

My source code looks like this:

String rule = ...
Resource resource = ResourceFactory.newReaderResource(new 
StringReader(rule));
KnowledgeBuilder knowledgeBuilder = 
KnowledgeBuilderFactory.newKnowledgeBuilder();

knowledgeBuilder.add(resource, ResourceType.DRL);
knowledgeBase.addKnowledgePackages(knowledgeBuilder.getKnowledgePackages());

Regards,
Piotr


___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users

[rules-users] Speed up inserting of rules into knowledge base

[Piotr Jedrychowski]

Hello.

I'm loading a big amount of rules during starting of JBoss (2 
rules). All rules (in string format) are available before JBoss startup 
- they are stored into database. Rules are loaded one by one and it 
takes about 90 minutes. I want to speed up this process. Is there 
something like "bulk load" for inserting rules into knowledge base or 
another smart way to fast loading a big amount of rules?


I'm using:
1) Drools 5.1
2) JBoss 4.2.3

My source code looks like this:

String rule = ...
Resource resource = ResourceFactory.newReaderResource(new 
StringReader(rule));
KnowledgeBuilder knowledgeBuilder = 
KnowledgeBuilderFactory.newKnowledgeBuilder();

knowledgeBuilder.add(resource, ResourceType.DRL);
knowledgeBase.addKnowledgePackages(knowledgeBuilder.getKnowledgePackages());

Regards,
Piotr
___
rules-users mailing list
rules-users@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/rules-users