Hi! Looks like you can't achieve it in Ignite 3 the same way as Ignite 2 (at least currently). Current implementation of affinity in Ignite 3 based on ignite-3/RendezvousAffinityFunction.java at main · apache/ignite-3 (github.com) <https://github.com/apache/ignite-3/blob/main/modules/affinity/src/main/java/org/apache/ignite/internal/affinity/RendezvousAffinityFunction.java> as said previously. But unfortunately you cannot customize table affinity function, as in Ignite 2 cache options. So, it looks like the only thing that you can specify is `COLOCATED BY` - colocation key. The key can be composite. Primary key must include colocation key. Was `affinity_key` in Ignite 2.x. when you create your table.
On Thu, Jan 26, 2023 at 3:08 PM Colin Cassidy via user < [email protected]> wrote: > Thanks for the response, but I'm afraid that a simple affinity key doesn't > help in this case. To clarify - I don't just want to ensure that data > records in Ignite are stored in the same node - I want to ensure that the > events from Kafka arrive on the right node too. > > Kafka uses a partitoning and assignment scheme very similar to the one > Ignite uses. It also supports event affinity. But the actual paritition & > node assignment will never match up between Kafka and Ignite. > > That means that an event e1 might land on a kafka consumer running on > Ignite node n1 - but Ignite may chose to store the data on node n2. For a > high throughput of events, this kills performance and scale. > > What I want to do is override the assignment to ensure that the event > landing on node n1 is also stored on node n1. Meaning that the data access > will normally be local and therefore fast. > > Second prize would be to ensure that at least the partitions are aligned. > So that the Kafka and Ignite partition number for a given key is the same. > So if k1 & k2 are mapped to partition 1 and k3, k4 are mapped to partition > 2 - that will be true both for Kafka and Ignite. Then at least the node > assignment can be handled on the consumer and doesn't require a change to > the producer. > > In any case, I think all of this could be achieved in Ignite 2 - but the > question is - can it be achieved also in Ignite 3? > > Thanks, > Colin. > > On Thursday, 26 January 2023 at 11:57:06 GMT, Stephen Darlington < > [email protected]> wrote: > > > You don’t need to override the RendezvousAffinityFunction to do what > you’re asking. Instead, if you define an affinity key to your table, it > will guarantee that related records will be kept together. However, make > sure your groups are not too coarse or you’ll get poor distribution across > your cluster. > > On 26 Jan 2023, at 11:37, Colin Cassidy via user <[email protected]> > wrote: > > I would like to override the Ignite AffinityFunction to delegate to > Kafka's partitioning algorithm. Possibly also the node assignment - but > definitely the partitioning > > This will allow me to ensure that events are processed local to where the > data is stored. > > I think this is possible in Ignite 2.x but there appear to be significant > changes in Ignite 3.x. Could I confirm - if I follow this path, will it be > possible to override the cache partitioning still in 3.x? > > Thanks in advance, > Colin. > > >
