A word of caution: you’re generally better replicating your data across clusters than stretching a single cluster across data centres. If the latency is very low it should work, but it could degrade your throughput and you need to be careful about split-brain and other networking issues.
Regards, Stephen > On 5 Aug 2021, at 15:24, Courtney Robinson <courtney.robin...@hypi.io> wrote: > > Hi Alex, > Thanks for the reply. I'm glad I asked before the team went any further. > So we can achieve this with the built in affinity function and the backup > filter. The real complexity is going to be in migrating our existing caches. > > So to clarify the steps involved here are > because Ignite registers all env. vars as node attributes we can set e.g. > NODE_DC=<EU_WEST|EU_EAST|CAN0> as an environment var in each k8s cluster > Then set the backup filter's constructor-arg.value to be NODE_DC. This will > tell Ignite that two backups cannot be placed on any two nodes with the same > NODE_DC value - correct? > When we call create table, we must set template=myTemplateName > Before creating any tables, myTemplateName must be created and must include > the backup filter with NODE_DC > Have I got that right? > > If so, it seems simple enough. Now the real challenge is where you said the > cache has to be re-created. > > I can't see how we do this without major down time, we have functionality in > place that allows customers to effectively do a "copy from table A to B and > then delete A" but it will be impossible to get all of them to do this any > time soon. > > Has anyone else had to do something similar, how is the community generally > doing migrations like this? > > Side note: The only thing that comes to mind is that we will need to build a > virtual catalog that we maintain so that there isn't a one to one mapping > between customer tables and the actual Ignite table name. > So if a table is currently called A and we add a virtual catalog then we keep > a mapping that says when the user wants to call "A" it should really go to > table "A_v2" or something. This comes with its own challenge and a massive > testing overhead. > > Regards, > Courtney Robinson > Founder and CEO, Hypi > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io/> > > <https://hypi.io/>https://hypi.io <https://hypi.io/> > > On Thu, Aug 5, 2021 at 11:43 AM Alex Plehanov <plehanov.a...@gmail.com > <mailto:plehanov.a...@gmail.com>> wrote: > Hello, > > You can create your own cache templates with the affinity function you > require (currently you use a predefined "partitioned" template, which only > sets cache mode to "PARTITIONED"). See [1] for more information about cache > templates. > > > Is this the right approach > > How do we handle existing data, changing the affinity function will cause > > Ignite to not be able to find existing data right? > You can't change cache configuration after cache creation. In your example > these changes will be just ignored. The only way to change cache > configuration - is to create the new cache and migrate data. > > > How would you recommend implementing the affinity function to be aware of > > the data centre? > It's better to use the standard affinity function with a backup filter for > such cases. There is one shipped with Ignite (see [2]). > > [1]: > https://ignite.apache.org/docs/latest/configuring-caches/configuration-overview#cache-templates > > <https://ignite.apache.org/docs/latest/configuring-caches/configuration-overview#cache-templates> > [2]: > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/ClusterNodeAttributeAffinityBackupFilter.html > > <https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/ClusterNodeAttributeAffinityBackupFilter.html> > чт, 5 авг. 2021 г. в 09:40, Courtney Robinson <courtney.robin...@hypi.io > <mailto:courtney.robin...@hypi.io>>: > Hi all, > Our growth with Ignite continues and as we enter the next phase, we need to > support multi-cluster deployments for our platform. > We deploy Ignite and the rest of our stack in Kubernetes and we're in the > early stages of designing what a multi-region deployment should look like. > We are 90% SQL based when using Ignite, the other 10% includes Ignite > messaging, Queues and compute. > > In our case we have thousands of tables > CREATE TABLE IF NOT EXISTS Person ( > id int, > city_id int, > name varchar, > company_id varchar, > PRIMARY KEY (id, city_id) > ) WITH "template=..."; > In our case, most tables use a template that looks like this: > > partitioned,backups=2,data_region=hypi,cache_group=hypi,write_synchronization_mode=primary_sync,affinity_key=instance_id,atomicity=ATOMIC,cache_name=Person,key_type=PersonKey,value_type=PersonValue > > I'm aware of affinity co-location > (https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation > <https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation>) > and in the past when we used the key value APIs more than SQL we also used > custom affinity a function to control placement. > > What I don't know is how to best do this with SQL defined caches. > We will have at least 3 Kubernetes clusters, each in a different data centre, > let's say EU_WEST, EU_EAST, CAN0 > > Previously we provided environment variables that our custom affinity > function would use and we're thinking of providing the data centre name this > way. > > We have 2 backups in all cases + the primary and so we want the primary in > one DC and each backup to be in a different DC. > > There is no syntax in the SQL template that we could find to enables > specifying a custom affinity function. > Our instance_id column currently used has no common prefix or anything to > associate with a DC. > > We're thinking of getting the cache for each table and then setting the > affinity function to replace the default RendevousAffinityFunction the way we > did before we switched to SQL. > Something like this: > repo.ctx.ignite.cache("Person").getConfiguration(org.apache.ignite.configuration.CacheConfiguration) > .setAffinity(new org.apache.ignite.cache.affinity.AffinityFunction() { > ... > }) > > There are a few things unclear about this: > Is this the right approach? > How do we handle existing data, changing the affinity function will cause > Ignite to not be able to find existing data right? > How would you recommend implementing the affinity function to be aware of the > data centre? > Are there any other caveats we need to be thinking about? > There is a lot of existing data, we want to try to avoid a full copy/move to > new tables if possible, that will prove to be very difficult in production. > > Regards, > Courtney Robinson > Founder and CEO, Hypi > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io/> > > <https://hypi.io/>https://hypi.io <https://hypi.io/>
