Hi,
not really an optimization, but based on your current solution, you
could rewrite the root lookup operation as an Ignite computation.
If you model the computation as a continuation you can avoid recursion.
This way you can avoid deadlocks and stack overflows.
In the continuation, you still need to retain the path to the root to
check for circular relationships I assume, otherwise it may run forever.
However, the same would be true for recursion (you would probably get a
stack overflow in a circular scenario).
You could cache the information about whether some id is a root or not
in a node local cache. I guess you could also cache sub results of the
lookups to make faster jumps in the tree. This of course implies you may
see out of date values for some time.
Please have a look at
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/computegrid/ComputeFibonacciContinuationExample.java
. It demonstrates continuation and node local cache usage.
S pozdravom,
Kamil Mišúth
On 2019-01-29 11:13, shishal wrote:
Hi,
I am storing parent child relationship in Ignite cache. Its a runtime
infinite stream of data(From Kafka) and partitioned across many node.
Data is relevant for 30 days only ,so expiry is set to 30 days.
Right now my Input record structure look like following:
/{
id: (String: UUID)
parentId : (String: UUID)
}/
In Ignite I store these records with Id as key and Object as value with
both
Id and parentId indexed.
Following are assumption for input record:
- Same input records can come multiple times
- All record are considered as root if Its parentId , do not appear as
Id in
another records.
- Records can come any order. ie. For Example leaves Node can come
before
root node.
My Use case is, When I searched for an ID, I need to get all parent in
upward direction.( ie. parent node, parent to parent node...till root
node).
I am currently making recursive call to get it.
My question is, Hoe can I optimize it. Considering write speed should
not be
compromise as its has to keep up with incoming fast data. Right now its
10K-20K/Sec but should have capability to scale further.
I also though about Affinity Collocation, But not sure how to insure
all
records of a tree goes to same Ignite Node.
Note: My my previous solution was based on Neo4j but It need something
which
can scale horizontally as neo4j expect whole database to be on single
machine.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
