I've created a ticket for this: https://issues.apache.org/jira/browse/IGNITE-13671
-Val On Wed, Nov 4, 2020 at 12:53 AM Valentin Kulichenko < [email protected]> wrote: > Thanks, Raymond. So the reason why you couldn't use the @AffinityKeyMapped > annotation or the CacheKeyConfiguration is that collocation is based on > *two fields*, not just one field. Is my understanding correct? > > If that's the case, I believe it can be easily improved by providing ways > to specify multiple fields for collocation. The fact that this scenario > requires a custom affinity mapper is a usability issue in my view. > > -Val > > On Tue, Nov 3, 2020 at 10:26 AM Raymond Wilson <[email protected]> > wrote: > >> If I have a primary key vector like this <x><y><t>, where <x> & <y> can >> have varying values for <t>, and I want all keys with the same <x> and <y> >> to reside in the same partition for processing colocation requirements >> then >> I can't use the standard Ignite mapping to do this, I need to use a custom >> mapper than uses just the <x> & <y> components. >> >> >> On Wed, Nov 4, 2020 at 1:33 AM Valentin Kulichenko < >> [email protected]> wrote: >> >> > Raymond, >> > >> > Thanks for the details. So it sounds like you have a custom affinity >> > mapper, not affinity function. This makes things simpler, but I'm still >> > failing to understand why standard mechanisms for collocation [1] didn't >> > work for you. Could you please clarify? >> > >> > [1] >> > >> https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation >> > >> > -Val >> > >> > On Tue, Nov 3, 2020 at 12:25 AM Raymond Wilson < >> [email protected] >> > > >> > wrote: >> > >> > > In terms of Key -> Partition -> Node mapping, we provide customer >> > affinity >> > > mappers for the Key -> Partition stage an allow Ignite to map >> partitions >> > to >> > > nodes. >> > > >> > > Our keys are structs with multiple fields forming a composite primary >> > key, >> > > parts of which are spatially identifying and parts contain other >> > > characteristics. We want to ensure all related spatial elements >> resolve >> > to >> > > the same partition and so will be placed on the same node for query >> > > colocation purposes. >> > > >> > > >> > > >> > > On Tue, Nov 3, 2020 at 9:20 PM Valentin Kulichenko < >> > > [email protected]> wrote: >> > > >> > > > Moti, Raymond, >> > > > >> > > > Could you please describe your use cases in more detail? What are >> the >> > > types >> > > > used as cache keys? What is the custom logic that you use for >> affinity >> > > > mapping? What was the exact reason to customize versus using >> built-in >> > > > collocation mechanisms? >> > > > >> > > > Ultimately, I'm sure that custom affinity functions will go away >> cause >> > > they >> > > > introduce multiple potential issues. However, I would definitely >> like >> > to >> > > > make sure that your use cases are still supported in Ignite 3.0 via >> > some >> > > > other means. >> > > > >> > > > -Val >> > > > >> > > > On Mon, Nov 2, 2020 at 12:21 PM Alexei Scherbakov < >> > > > [email protected]> wrote: >> > > > >> > > > > Thanks for the clarification. >> > > > > >> > > > > There was no intention to remove the customizable key to partition >> > > > mapping. >> > > > > >> > > > > Difficulties arise when mapping partitions to nodes, so it's >> > desirable >> > > to >> > > > > have internally tested implementation with a way to customize it's >> > > > behavior >> > > > > without additional coding on the user side. >> > > > > >> > > > > пн, 2 нояб. 2020 г. в 23:01, Raymond Wilson < >> > > [email protected] >> > > > >: >> > > > > >> > > > > > Just to be clear, the affinity functions we are using convert >> keys >> > to >> > > > > > partitions, we do not map partitions to nodes and leave that to >> > > Ignite. >> > > > > > >> > > > > > On Tue, Nov 3, 2020 at 8:48 AM Alexei Scherbakov < >> > > > > > [email protected]> wrote: >> > > > > > >> > > > > > > Hello. >> > > > > > > >> > > > > > > Custom affinity functions can cause weird bugs and data loss >> if >> > > > > > implemented >> > > > > > > wrongly. >> > > > > > > There is an intention to keep a backup filter based on user >> > > > attributes >> > > > > > > (with additional validation logic to ensure correctness) for >> > > > > controllable >> > > > > > > data placement. >> > > > > > > >> > > > > > > Can you describe more precisely why you had to implement >> custom >> > > > > affinity >> > > > > > > functions and not resort to default rendezvous affinity + >> backup >> > > > > filter ? >> > > > > > > >> > > > > > > >> > > > > > > пн, 2 нояб. 2020 г. в 21:45, Raymond Wilson < >> > > > > [email protected] >> > > > > > >: >> > > > > > > >> > > > > > > > We also use custom affinity functions (vis the C# client). >> > > > > > > > >> > > > > > > > The wish list mentions use of a particular annotation >> > > > > > > > (@CentralizedAffinityFunction): >> > > > > > > > Is the wish to remove just this annotation, or the ability >> to >> > > > define >> > > > > > > custom >> > > > > > > > affinity functions at all? >> > > > > > > > >> > > > > > > > In our case we use affinity functions to ensure particular >> > > > > distribution >> > > > > > > of >> > > > > > > > spatial data across a processing cluster to ensure even load >> > > > > > management. >> > > > > > > > >> > > > > > > > On Tue, Nov 3, 2020 at 5:31 AM Moti Nisenson < >> > > > > [email protected]> >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > I saw at >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist >> > > > > > > > > that custom affinity functions are on the potential >> wishlist >> > > for >> > > > > > > removal. >> > > > > > > > > The way we're using it's very critical that we be able to >> > > control >> > > > > the >> > > > > > > > > placement of data quite precisely - as part of that we >> > specify >> > > > > > > explicitly >> > > > > > > > > the partition we want in the key, and then our affinity >> > > function >> > > > > uses >> > > > > > > > that >> > > > > > > > > (else delegating to default rendezvous). We don't need all >> > the >> > > > > > > > > abilities there, although I think that often others do. >> > > > > > > > > >> > > > > > > > > This seems to me to be a case that the benefit of removing >> > this >> > > > is >> > > > > > > > minimal >> > > > > > > > > and could cause quite a lot of disruption to users. >> > > > > > > > > >> > > > > > > > > Thanks! >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > <http://www.trimble.com/> >> > > > > > > > Raymond Wilson >> > > > > > > > Solution Architect, Civil Construction Software Systems >> (CCSS) >> > > > > > > > 11 Birmingham Drive | Christchurch, New Zealand >> > > > > > > > +64-21-2013317 Mobile >> > > > > > > > [email protected] >> > > > > > > > >> > > > > > > > < >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > >> > > > > > > Best regards, >> > > > > > > Alexei Scherbakov >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > <http://www.trimble.com/> >> > > > > > Raymond Wilson >> > > > > > Solution Architect, Civil Construction Software Systems (CCSS) >> > > > > > 11 Birmingham Drive | Christchurch, New Zealand >> > > > > > +64-21-2013317 Mobile >> > > > > > [email protected] >> > > > > > >> > > > > > < >> > > > > > >> > > > > >> > > > >> > > >> > >> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > Best regards, >> > > > > Alexei Scherbakov >> > > > > >> > > > >> > > >> > > >> > > -- >> > > <http://www.trimble.com/> >> > > Raymond Wilson >> > > Solution Architect, Civil Construction Software Systems (CCSS) >> > > 11 Birmingham Drive | Christchurch, New Zealand >> > > +64-21-2013317 Mobile >> > > [email protected] >> > > >> > > < >> > > >> > >> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch >> > > > >> > > >> > >> >> >> -- >> <http://www.trimble.com/> >> Raymond Wilson >> Solution Architect, Civil Construction Software Systems (CCSS) >> 11 Birmingham Drive | Christchurch, New Zealand >> +64-21-2013317 Mobile >> [email protected] >> >> < >> https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch >> > >> >
