Customized affinity function

2018-04-10 Thread Prasad Bhalerao 
Hi,

I have following case.

I have around 1 subscriptions. Each subscription has data varying from
1 to 10 million.
Currently the affinity key is set on subscriptionId.

e.g:

Subscription1: 2 million rows
Subscription2: 10 million rows
Subscription3: 50 million rows
Subscription5: 30 million rows
Subscription6: 2 rows

Now I want make sure that all big subscription should be evenly distributed
across all nodes.

I meant Subscription2,Subscription3,Subscription4,Subscription5 should not
land on same node based on their affinity id.

Is there any way with which I can distribute all big subscriptions across
all node evenly or in round robin fashion?


Thanks,
Prasad

Re: Customized affinity function

2018-04-10 Thread Mikael 

Hi!

The |@AffinityKeyMapped annotation can be used on a field or a method, 
so you can have a method returning anything you want (based on size), 
would that not work ?|


|Mikael
|


Den 2018-04-11 kl. 07:59, skrev Prasad Bhalerao:

Hi,

I have following case.

I have around 1 subscriptions. Each subscription has data varying 
from 1 to 10 million.

Currently the affinity key is set on subscriptionId.

e.g:

Subscription1: 2 million rows
Subscription2: 10 million rows
Subscription3: 50 million rows
Subscription5: 30 million rows
Subscription6: 2 rows

Now I want make sure that all big subscription should be evenly 
distributed across all nodes.


I meant Subscription2,Subscription3,Subscription4,Subscription5 should 
not land on same node based on their affinity id.


Is there any way with which I can distribute all big subscriptions 
across all node evenly or in round robin fashion?



Thanks,
Prasad

Re: Customized affinity function

2018-04-11 Thread aealexsandrov 
Hi Prasad,

Affinity function does not directly map keys to nodes, it maps keys to
partitions. So it takes care about entities balance between nodes. To read
more please take a look here:

https://apacheignite.readme.io/docs/affinity-collocation#affinity-function

In case if for every subscriber you are going to create it's own cache then
you can try to setup the node filters for every subscriber:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html#setNodeFilter-org.apache.ignite.lang.IgnitePredicate-

If you have the same cache for all subscribers then try to setup your own
AffinityKey (as descibed above):

IgniteCache cache =
ignite.getOrCreateCache(cfg);

for (int i = 0 ; i < 1_000_000; i++) {
cache.put(new EntityKey(2*i, 1L), "Value1" + i); //sub id =
1
cache.put(new EntityKey(2*i + 1, 2L), "Value2" + i); //sub
id =2
}

where:

public static class EntityKey {
public EntityKey(long id, long subscriberId) {
this.id = id;
this.subscriberId = subscriberId;
}

private long id;

// Subscriber ID which will be used for affinity.
@AffinityKeyMapped
private long subscriberId;

public long getSubscriberId() {
return subscriberId;
}

public long getId() {
return id;
}
}

Thank you,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Customized affinity function

2018-04-11 Thread vkulichenko 
Prasad,

Are you collocating data by subscription? Do you have any actual issue with
data distribution? With 1 subscriptions, I don't see why would you have
one with default settings.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Customized affinity function

2018-04-19 Thread Prasad Bhalerao 
Hi Val,

Out 1 subscriptions, only 10-20 subscriptions are very big. I just want
to make sure that these big subscriptions do not land up in partitions
which are located on same node.





On Thu, Apr 12, 2018 at 7:44 AM, vkulichenko 
wrote:

> Prasad,
>
> Are you collocating data by subscription? Do you have any actual issue with
> data distribution? With 1 subscriptions, I don't see why would you have
> one with default settings.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Customized affinity function

2018-04-19 Thread vkulichenko 
Statistically it's very unlikely that all big subscriptions (or many of them)
will end up together. So I would check if there is an issue first and go
from there.

Technically you can implement your own AffinityFunction to customize
distribution. But that's not a trivial task as you will have to consider all
the cases of nodes going out of topology, scaling out, etc.

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/