Re: Ignite .NET + CDC replication using Kafka

[Nikolay Izhikov]

CDC not related to Ignite.NET .

It captures changes that happens on storage (WAL) layer of Ignite.
So, all you need is to follow the examples from documentation and configure CDC 
replication.

> 12 июня 2024 г., в 14:02, Pavel Tupitsyn  написал(а):
> 
> Can you  share some  working  examples  to  capture  the changes  happening  
> to  cache realtime using  .NET?
> 

Re: Ignite .NET + CDC replication using Kafka

[Pavel Tupitsyn]

> Is  this feature  available in  Ignite  .NET ? Can  you  share  the
sample example?

Yes, Continuous Query is available in Ignite.NET, sample code:
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs

RE: Ignite .NET + CDC replication using Kafka

[satyajit.mandal.barclays.com via user]

Thanks Pavel.

Is  this feature  available in  Ignite  .NET ? Can  you  share  the sample 
example?

https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Affinity.Rendezvous.ClusterNodeAttributeAffinityBackupFilter.html

Regards
Satyajit


From: Pavel Tupitsyn 
Sent: Wednesday, June 12, 2024 4:39 PM
To: Mandal, Satyajit: IT (PUN) 
Subject: Re: Ignite .NET + CDC replication using Kafka


CAUTION: This email originated from outside our organisation - 
ptupit...@apache.org<mailto:ptupit...@apache.org> Do not click on links, open 
attachments, or respond unless you recognize the sender and can validate the 
content is safe.
Some vendors also provide cluster replication with native .NET APIs based on 
Ignite:

https://www.gridgain.com/docs/latest/administrators-guide/data-center-replication/configuring-replication<https://clicktime.symantec.com/15t5z5LCoSPzxZnd8HV1f?h=idSuwKpOy7WVXyTrMyN6I_jp2pG9Xj2Vk1v_yWPSoTM=&u=https://www.gridgain.com/docs/latest/administrators-guide/data-center-replication/configuring-replication>

On Wed, Jun 12, 2024 at 2:02 PM Pavel Tupitsyn 
mailto:ptupit...@apache.org>> wrote:
Hi Satyajit,

CDC extension is not available directly in Ignite.NET, you can use a Compute 
job [1] or a Service [2] implemented in Java to access this functionality.

Alternatively, Continuous Query [3] provides a way to capture all changes in a 
specific cache. Example: [4]

[1] 
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html#Apache_Ignite_Core_Compute_ICompute_ExecuteJavaTask__1_System_String_System_Object_<https://clicktime.symantec.com/15t5pQwdtD2p8g8n3AgiR?h=0OmzUAZhJ4oIYVK3YnnrK4Utc8FX4mfLzbzlFY_zyVw=&u=https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html%23Apache_Ignite_Core_Compute_ICompute_ExecuteJavaTask__1_System_String_System_Object_>
[2] 
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Services.IServices.html<https://clicktime.symantec.com/15t5uF8vLpiQYcxhaj5s3?h=k1Ix2oO44D6JEfJZZylfedVJMd23dk5v3xOQ1LFGRJM=&u=https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Services.IServices.html>
[3] 
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Query.Continuous.ContinuousQuery-2.html<https://clicktime.symantec.com/15t5jakMRbMDijJrVcHZo?h=rnbT0tB2FA_fBxze82QMiPgYEpJAV9mFZFsm0ZGDCIY=&u=https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Query.Continuous.ContinuousQuery-2.html>
[4] 
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs<https://clicktime.symantec.com/15t5ZvMnWMz2tqf1QVVGZ?h=u43Fem4zddoJccj5n3cleHCQMjH1ZEAC0uTZQihPSl0=&u=https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs>

On Wed, Jun 12, 2024 at 9:52 AM 
mailto:satyajit.man...@barclays.com>> wrote:
Hi  Pavel,

How  can  we achieve  replication  from  Ignite .NET  cluster  to  another  
Ignite .NET cluster  as  backup using  Kafka  as  middleware? This  is  needed  
for resiliency  test  for Ignite cluster  where we  shutdown  one cluster  and  
move  all  application  service  to  backup  cluster  as  part  of  resiliency  
testing.

Basically  need  to  know  which  Ignite features supports to  capture changes  
in  caches  so  that we can write  programs  to  capture  those changes and  
stream  those  to  Kafka  using  .NET libraries.

Found  this  doc  but  not sure how  Ignite .NET fits  into  this ( CDC 
replication  using  Kafka)
https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions<https://clicktime.symantec.com/15t5ekZ4xyfdJnUvx3tRB?h=uLyfAFvoWOjEF9JvWuSOSIGaUYLH7IKGUW28dcFAg2M=&u=https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions>

Can you  share some  working  examples  to  capture  the changes  happening  to 
 cache realtime using  .NET?

Thanks
Satyajit



Barclays Execution Services Limited registered in England. Registered No. 
1767980. Registered office: 1 Churchill Place, London, E14 5HP

Barclays Execution Services Limited provides support and administrative 
services across Barclays group. Barclays Execution Services Limited is an 
appointed representative of Barclays Bank UK plc and Barclays Bank plc. 
Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential 
Regulation Authority and regulated by the Financial Conduct Authority and the 
Prudential Regulation Authority.

This email and any attachments are confidential and intended solely for the 
addressee and may also be privileged or exempt from disclosure under applicable 
law. If you are not the addressee, or have received this email in error, please 
notify the sender and immediately delete it and any attachments from your 
sys

Re: Ignite .NET + CDC replication using Kafka

[Pavel Tupitsyn]

Hi Satyajit,

CDC extension is not available directly in Ignite.NET, you can use a
Compute job [1] or a Service [2] implemented in Java to access this
functionality.

Alternatively, Continuous Query [3] provides a way to capture all changes
in a specific cache. Example: [4]

[1]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html#Apache_Ignite_Core_Compute_ICompute_ExecuteJavaTask__1_System_String_System_Object_
[2]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Services.IServices.html
[3]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Query.Continuous.ContinuousQuery-2.html
[4]
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs

On Wed, Jun 12, 2024 at 9:52 AM  wrote:

> Hi  Pavel,
>
>
>
> How  can  we achieve  replication  from  Ignite .NET  cluster  to
> another  Ignite .NET cluster  as  backup using  Kafka  as  middleware?
> This  is  needed  for resiliency  test  for Ignite cluster  where we
> shutdown  one cluster  and  move  all  application  service  to  backup
> cluster  as  part  of  resiliency  testing.
>
>
>
> Basically  need  to  know  which  Ignite features supports to  capture
> changes  in  caches  so  that we can write  programs  to  capture  those
> changes and  stream  those  to  Kafka  using  .NET libraries.
>
>
>
> Found  this  doc  but  not sure how  Ignite .NET fits  into  this ( CDC
> replication  using  Kafka)
>
>
> https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions
>
>
>
> Can you  share some  working  examples  to  capture  the changes
> happening  to  cache realtime using  .NET?
>
>
>
> Thanks
>
> Satyajit
>
>
>
>
>
> Barclays Execution Services Limited registered in England. Registered No.
> 1767980. Registered office: 1 Churchill Place, London, E14 5HP
>
> Barclays Execution Services Limited provides support and administrative
> services across Barclays group. Barclays Execution Services Limited is an
> appointed representative of Barclays Bank UK plc and Barclays Bank plc.
> Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential
> Regulation Authority and regulated by the Financial Conduct Authority and
> the Prudential Regulation Authority.
>
> This email and any attachments are confidential and intended solely for
> the addressee and may also be privileged or exempt from disclosure under
> applicable law. If you are not the addressee, or have received this email
> in error, please notify the sender and immediately delete it and any
> attachments from your system. Do not copy, use, disclose or otherwise act
> on any part of this email or its attachments.
>
> Internet communications are not guaranteed to be secure or virus-free. The
> Barclays group does not accept responsibility for any loss arising from
> unauthorised access to, or interference with, any internet communications
> by any third party, or from the transmission of any viruses. Replies to
> this email may be monitored by the Barclays group for operational or
> business reasons.
>
> Any opinion or other information in this email or its attachments that
> does not relate to the business of the Barclays group is personal to the
> sender and is not given or endorsed by the Barclays group.
>
> Unless specifically indicated, this e-mail is not an offer to buy or sell
> or a solicitation to buy or sell any securities, investment products or
> other financial product or service, an official confirmation of any
> transaction, or an official statement of Barclays.
>

Ignite .NET + CDC replication using Kafka

[satyajit.mandal.barclays.com via user]

Hi  Pavel,

How  can  we achieve  replication  from  Ignite .NET  cluster  to  another  
Ignite .NET cluster  as  backup using  Kafka  as  middleware? This  is  needed  
for resiliency  test  for Ignite cluster  where we  shutdown  one cluster  and  
move  all  application  service  to  backup  cluster  as  part  of  resiliency  
testing.

Basically  need  to  know  which  Ignite features supports to  capture changes  
in  caches  so  that we can write  programs  to  capture  those changes and  
stream  those  to  Kafka  using  .NET libraries.

Found  this  doc  but  not sure how  Ignite .NET fits  into  this ( CDC 
replication  using  Kafka)
https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions

Can you  share some  working  examples  to  capture  the changes  happening  to 
 cache realtime using  .NET?

Thanks
Satyajit



Barclays Execution Services Limited registered in England. Registered No. 
1767980. Registered office: 1 Churchill Place, London, E14 5HP

Barclays Execution Services Limited provides support and administrative 
services across Barclays group. Barclays Execution Services Limited is an 
appointed representative of Barclays Bank UK plc and Barclays Bank plc. 
Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential 
Regulation Authority and regulated by the Financial Conduct Authority and the 
Prudential Regulation Authority. 

This email and any attachments are confidential and intended solely for the 
addressee and may also be privileged or exempt from disclosure under applicable 
law. If you are not the addressee, or have received this email in error, please 
notify the sender and immediately delete it and any attachments from your 
system. Do not copy, use, disclose or otherwise act on any part of this email 
or its attachments.

Internet communications are not guaranteed to be secure or virus-free. The 
Barclays group does not accept responsibility for any loss arising from 
unauthorised access to, or interference with, any internet communications by 
any third party, or from the transmission of any viruses. Replies to this email 
may be monitored by the Barclays group for operational or business reasons.

Any opinion or other information in this email or its attachments that does not 
relate to the business of the Barclays group is personal to the sender and is 
not given or endorsed by the Barclays group.

Unless specifically indicated, this e-mail is not an offer to buy or sell or a 
solicitation to buy or sell any securities, investment products or other 
financial product or service, an official confirmation of any transaction, or 
an official statement of Barclays.