Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[adumalagan]

Unfortunately, db work is out of the scope of my responsibility, and I was
just told to work around the issue
Hence me asking about Colocation and MapReduce




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

> Colocation by Partition, but that API is not available for C#
Partition-based APIs were added in Ignite 2.9 [1]

> Is it possible to speed up loading cache by using Binary Object +
Colocating by Key, or Binary Object + MapReduce API?
Probably yes. But have you sorted out the DB connection bottleneck?
Let's make sure we are fixing the right problem.

[1]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html#Apache_Ignite_Core_Compute_ICompute_AffinityCall__1_System_Collections_Generic_IEnumerable_System_String__System_Int32_Apache_Ignite_Core_Compute_IComputeFunc___0__

On Thu, Dec 3, 2020 at 7:09 PM adumalagan  wrote:

> Thank you for the clarification!
>
> For this use case where data is being pulled from an underlying RDBMS and
> instantiated as an Associate object, is it possible to work the Associate
> object as a Binary Object similar to what you've done here?
>
> Also, I see than in Vladamir's loading example, he uses the Colocation by
> Partition, but that API is not available for C#. Is it possible to speed up
> loading cache by using Binary Object + Colocating by Key, or Binary Object
> + MapReduce API?
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[adumalagan]

Thank you for the clarification!
For this use case where data is being pulled from an underlying RDBMS and
instantiated as an Associate object, is it possible to work the Associate
object as a Binary Object similar to what you've done here? 
Also, I see than in Vladamir's loading example, he uses the Colocation by
Partition, but that API is not available for C#. Is it possible to speed up
loading cache by using Binary Object + Colocating by Key, or Binary Object +
MapReduce API? 
 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

> are there C# equivalencies to the Java interfaces BinaryObject and
Affinity

Yes, please see IIgnite.GetAffinity [1] and GetBinary [2].

[1]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.IIgnite.html#Apache_Ignite_Core_IIgnite_GetAffinity_System_String_
[2]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.IIgnite.html#Apache_Ignite_Core_IIgnite_GetBinary

On Thu, Dec 3, 2020 at 8:55 AM  wrote:

> Hi,
>
> I do not use SQL because my interest is to get max performance, I use
> key-value API instead. The same reason is why I use Java - it is native for
> Ignite. I think there should be wrappers forBinaryObject and Affinity in c#.
>
> Vladimir
>
> 4:39, 3 декабря 2020 г., adumalagan :
>
> I see, thanks for the clarification!
>
> I also have another question - are there C# equivalencies to the Java
> interfaces BinaryObject and Affinity?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
> --
> Отправлено из мобильного приложения Яндекс.Почты
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[vtchernyi]

Hi,I do not use SQL because my interest is to get max performance, I use key-value API instead. The same reason is why I use Java - it is native for Ignite. I think there should be wrappers forBinaryObject and Affinity in c#.Vladimir 4:39, 3 декабря 2020 г., adumalagan :I see, thanks for the clarification! I also have another question - are there C# equivalencies to the Javainterfaces BinaryObject and Affinity?-- Sent from: http://apache-ignite-users.70518.x6.nabble.com/-- Отправлено из мобильного приложения Яндекс.Почты

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[adumalagan]

I see, thanks for the clarification! 

I also have another question - are there C# equivalencies to the Java
interfaces BinaryObject and Affinity?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Denis Magda]

Yes, once the data is loaded into Ignite you can query it with SQL.

-
Denis


On Wed, Dec 2, 2020 at 9:41 AM adumalagan  wrote:

> Hi Vlad, I've started to read through your link and I just wanted to know
> that, if I were to implement your tutorial, could I perform SQL queries on
> the cache after loading?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[adumalagan]

Hi Vlad, I've started to read through your link and I just wanted to know
that, if I were to implement your tutorial, could I perform SQL queries on
the cache after loading?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[vtchernyi]

Hi,may be my recent tutorial [1] will shed some light to the question. The story is about my experience in loading big tables to Ignite. Hope it will help despite the Java language insideRegards,Vladimir[1] https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api21:06, 24 ноября 2020 г., Pavel Tupitsyn :Run your application in the same data center as the database, so that network costs are minimized.On Tue, Nov 24, 2020 at 8:54 PM ABDumalagan  wrote:I see - do you have any suggestions of how I can work around the bottleneck
and speed up data loading into cache? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

-- Отправлено из мобильного приложения Яндекс.Почты

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

Run your application in the same data center as the database, so that
network costs are minimized.

On Tue, Nov 24, 2020 at 8:54 PM ABDumalagan 
wrote:

> I see - do you have any suggestions of how I can work around the bottleneck
> and speed up data loading into cache?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

I see - do you have any suggestions of how I can work around the bottleneck
and speed up data loading into cache? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

The code looks ok to me.
Since the query is so simple, most likely your connection to the DB is the
bottleneck.

On Tue, Nov 24, 2020 at 8:30 PM ABDumalagan 
wrote:

> So I am currently running this
> 
> code to just pull all the rows from the underlying database, and the
> diagnostic session is hitting 20 mins and counting -
>
> Is this definitively a database issue or is it in the way I've written the
> code?
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

So I am currently running  this
  
code to just pull all the rows from the underlying database, and the
diagnostic session is hitting 20 mins and counting - 
Is this definitively a database issue or is it in the way I've written the
code?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

> how would I check to see if the underlying database is a bottleneck
Check how much time it takes to read 500 000 rows from there, without
loading into Ignite.
Or run a profiler.

On Tue, Nov 24, 2020 at 6:25 PM ABDumalagan 
wrote:

> I'll try a local scenario and stream with fake data right now, but how
> would I check to see if the underlying database is a bottleneck?
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

I'll try a local scenario and stream with fake data right now, but how would
I check to see if the underlying database is a bottleneck? 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

1) Yes
2) On my local machine with 2 nodes I can stream 500 000 entries in *2*
seconds (I tried to recreate the Associate class with some fake data).
Are you sure that Oracle is not a bottleneck? Can you try a local scenario
with fake data - how much time does it take?

On Tue, Nov 24, 2020 at 5:46 PM ABDumalagan 
wrote:

> I see - I'm actually going to scrap the LoadCache approach and implement
> your suggestion of DataStream - I just have a couple of questions regarding
> that:
>
> 1) Is it possible to use both DataStream and CacheStoreFactory together?
>
> 2) Right now, my Program.cs
>  of
> DataStream is naïve and loads 1000 entries in ~7 seconds, 1 entries in
> ~55 seconds, and 10 entries in 9 mins and 12 seconds
>
> Is this a normal time for loading bulk entries?
>
> Is there another way I can implement or configure DataStream to load bulk
> entries faster? I think at this rate, loading > 500,000 entries will be
> pushing around an hour.
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

I see - I'm actually going to scrap the LoadCache approach and implement your
suggestion of DataStream - I just have a couple of questions regarding that:

1) Is it possible to use both DataStream and CacheStoreFactory together?

2) Right now, my  Program.cs
   of
DataStream is naïve and loads 1000 entries in ~7 seconds, 1 entries in
~55 seconds, and 10 entries in 9 mins and 12 seconds 
Is this a normal time for loading bulk entries?
Is there another way I can implement or configure DataStream to load bulk
entries faster? I think at this rate, loading > 500,000 entries will be
pushing around an hour. 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

NullPointerException usually indicates a bug. Please provide full exception
stack trace.

On Tue, Nov 24, 2020 at 4:03 PM ABDumalagan 
wrote:

> Also, I tried changing the code from cache.LocalLoadCache(null) to
> cache.LoadCache(null) and I received a NullPointerException.
>
> Am I supposed to pass a non-null parameter into LoadCache?
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

DataStreamer is the best approach for loading large amounts of data into
Ignite

On Tue, Nov 24, 2020 at 3:50 PM ABDumalagan 
wrote:

> I see, that makes sense - I actually have a question regarding that:
>
> If I wanted to store a whole table from an underlying database as
> key-value pairs in cache, but the database had over 500,000 entries, how
> would you suggest going about that?
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

Ok, here is the problem, per documentation [1]:
"In case of partitioned caches, keys that are not mapped to this node,
either as primary or backups, will be automatically discarded by the cache."

Since you have two nodes in the cluster, but call localLoadCache only on
one node, part of the cache entries that belong to the other nodes are
discarded.
You can do one of the following:
1. Call LoadCache instead of LocalLoadCache, so that cache store is invoked
on every node. This will perform the same Oracle query multiple times,
which may be suboptimal.
2. Use DataStreamer [2] instead of LoadCache to load all the data from a
single node

[1] https://apacheignite-net.readme.io/docs/data-loading#icacheloadcache
[2] https://apacheignite-net.readme.io/docs/data-streamers

On Tue, Nov 24, 2020 at 1:25 AM ABDumalagan 
wrote:

> 1. I currently just have 2 Ignite nodes--first one remotely to start the
> cluster and the second one (this one) started programmatically with C#.
>
> 2. Adding Thread.Sleep(5000) doesn't change the result, unfortunately.
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

1. I currently just have 2 Ignite nodes--first one remotely to start the
cluster and the second one (this one) started programmatically with C#.
2. Adding Thread.Sleep(5000) doesn't change the result, unfortunately.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

1. How many Ignite nodes do you have?
2. What if you add Thread.Sleep(5000) before the last Console.WriteLine?
Does the resulting number change?

On Mon, Nov 23, 2020 at 6:01 PM ABDumalagan 
wrote:

> 1. Your program worked for me!
>
> 2. I added something to my LoadCache(Action, params
> object[] args) method in OracleStore.cs. I added the following 3 lines
> after the while loop:
>
> reader.Dispose();
>
> cmd.Dispose();
>
> con.Dispose();
>
> Console returned a non-zero cache size of 5136, however, the queries I
> wanted to do and the queries counted by the method is 10,000 - I was
> wondering what happened to the other ~5000 queries and why they aren't in
> cache?
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

1. Your program worked for me!

2. I added something to my LoadCache(Action, params object[]
args) method in OracleStore.cs. I added the following 3 lines after the
while loop: 
reader.Dispose();
cmd.Dispose();
con.Dispose();
Console returned a non-zero cache size of 5136, however, the queries I
wanted to do and the queries counted by the method is 10,000 - I was
wondering what happened to the other ~5000 queries and why they aren't in
cache? 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

Your code seems to be correct. It works for me in a simplified form:
https://gist.github.com/ptupitsyn/a64c899b32b73ab55cb706cd4a09e6e9

1. Can you try the program above - does it work for you?
2. Can you confirm that the Oracle query returns a non-empty result set?


On Mon, Nov 23, 2020 at 3:00 AM ABDumalagan 
wrote:

> I see - am I dealing with 1.) in my case?
>
> When I hover over the method LoadCache(Action, params
> object[] args) in Visual Studio, it says that *"...This method is called
> whenever
> Apache.Ignite.Core.Cache.ICache.LocalLoadCache(Apache.Ignite.Core.Cache.ICacheEntryFilter,
> params object[]) method is invoked which is usually to preload the cache
> from persistent storage"*.
>
> From the .NET docs
> ,
> it says that for LocalLoadCache(ICacheEntryFilter, Object[]), the
> loaded values will then be given to the optionally passed in predicate,
> and, if the predicate returns true, will be stored in cache. If predicate
> is null, then all loaded values will be stored in cache.
>
> In my case, I call cache.LocalLoadCache(null) in Program.cs
> ,
> where it then calls on LoadCache(Action act, params
> object[] args) in OracleStore.cs
> 
> and the Associate IDs are being printed to console.
>
> However, when I try to print the cache size in console, it returns 0 -
> does this mean that the values are not being stored in cache?
>
> Do I need to do something else to make sure the values are loaded into
> cache? I thought that LocalLoadCache would just put all the values queried
> from the underlying storage into cache.
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

I see - am I dealing with 1.) in my case?
When I hover over the method LoadCache(Action, params
object[] args)  in Visual Studio, it says that *"...This method is called
whenever
Apache.Ignite.Core.Cache.ICache.LocalLoadCache(Apache.Ignite.Core.Cache.ICacheEntryFilter,
params object[]) method is invoked which is usually to preload the cache
from persistent storage"*.
 
>From the  .NET docs

 
, it says that for LocalLoadCache(ICacheEntryFilter, Object[]), the
loaded values will then be given to the optionally passed in predicate, and,
if the predicate returns true, will be stored in cache. If predicate is
null, then all loaded values will be stored in cache. 
In my case, I call cache.LocalLoadCache(null) in  Program.cs
  ,
where it then calls on LoadCache(Action act, params object[]
args) in  OracleStore.cs
  
and the Associate IDs are being printed to console. 
However, when I try to print the cache size in console, it returns 0 - does
this mean that the values are not being stored in cache?
Do I need to do something else to make sure the values are loaded into
cache? I thought that LocalLoadCache would just put all the values queried
from the underlying storage into cache.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

Looks like we mixed up two different delegates:

1. ICache.LoadCache method takes an *optional *filter delegate in case you
want to skip some of the entries from cache store. You can pass null there
when filtering is not required.
2. ICacheStore.LoadCache method gives you (an implementor of the interface)
a delegate that you *should *call for every entry to be loaded

On Sat, Nov 21, 2020 at 11:05 PM ABDumalagan 
wrote:

> Got *(1)* to work - thank you!
>
> However, for *(2)*, I tried to implement a test Action delegate that
> writes the value of the entry in the console and I got an error stating:
>
> Argument 1: cannot convert from 'System.Action'
> to 'Apache.Ignite.Core.Cache.ICacheEntryFilter'
> This error occurred at line cache.Load(action, null)
>
> Am I going about this implementation the wrong way? If so, what would I
> need to do to correct this?
>
>
> Updated files:
>
> Runs program; test Action delegate:Program.cs
> 
>
> Inherits CacheStoreAdapter: OracleStore.cs
> 
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

Got *(1)* to work - thank you!
However, for *(2)*, I tried to implement a test Action delegate that writes
the value of the entry in the console and I got an error stating: 
Argument 1: cannot convert from 'System.Action'
to 'Apache.Ignite.Core.Cache.ICacheEntryFilter'
This error occurred at line cache.Load(action, null)
Am I going about this implementation the wrong way? If so, what would I need
to do to correct this? 


Updated files: 
Runs program; test Action delegate: Program.cs
  
Inherits CacheStoreAdapter:  OracleStore.cs
   




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: C# CacheStoreAdapter - Customizing Load, LoadCache methods

[Pavel Tupitsyn]

Hello,

*> (1)* Regarding the Load method, I was under the impression that calling
Get(key) would inherently call the Load(key) method from my OracleStore.cs

It would if you set CacheConfiguration.ReadThrough = true. Similarly,
enable WriteThrough if you want Ignite to call OracleStore.Write on
cache.Put.


> *(2)* With the LoadCache method, I essentially would be doing the same
thing as the Load method, but this time, I would be querying 10,000 entries
from the database and putting it into cache
>  am unsure about what to do for the Action parameter

You are supposed to call this action for every entry to be loaded into
cache, e.g.:

while(reader.Read())
{
/*
 * 1) Instantiate Associate object
 * 2) Put Associate object in cache
 */
  var associate = new Associate(reader);
  act(associate.Id, associate);
}


On Fri, Nov 20, 2020 at 8:23 PM ABDumalagan 
wrote:

> Hello,
>
> I have some confusion about implementing my own Load and LoadCache methods
> -
>
> I am trying to load data from an underlying Oracle database into cache,
> and the entries are key-value pairs where the key is the Associate ID and
> the value is the entire Associate object.
>
> *(1)* Regarding the Load method, I was under the impression that calling
> Get(key) would inherently call the Load(key) method from my OracleStore.cs
> file and load the key and corresponding object from the database into
> cache.
>
> Under the Load method in OracleStore.cs, I connect to the underlying
> Oracle database and execute a query which grabs the data that correlates to
> the Associate ID.
>
> However, after running my Program.cs file, I received an error saying 
> *"System.Collections.Generic.KeyNotFoundException:
> 'The given key was not present in the cache: 18755"* -- was I under the
> wrong impression that the Load(key) would just put the data into cache?
>
> I know that the Load method returns an object, and in my case, it returns
> an Associate object. However, do I need to do implement something else in
> order to put the object returned from the Load method into cache?
>
> *(2)* With the LoadCache method, I essentially would be doing the same
> thing as the Load method, but this time, I would be querying 10,000 entries
> from the database and putting it into cache. I also hold the same
> assumption that calling the LoadCache method will just load the 10,000
> entries in cache.
>
> Also, I am unsure about what to do for the Action parameter in
> LoadCache(Action, Object[])--would I need to define a method to
> pass as a parameter, and would that method be responsible for putting the
> entries in cache? If so, how would I go about it?
>
> Files attached below:
>
> Inherits CacheStoreAdapter: OracleStore.cs
> 
>
> Inherits IFactory: OracleStoreFactory.cs
> 
>
> Establishes Oracle database connection: DBConnection.cs
> 
>
> Defines Associate object: Associate.cs
> 
>
> Runs program (creates and starts node): Program.cs
> 
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>
>

C# CacheStoreAdapter - Customizing Load, LoadCache methods

[ABDumalagan]

Hello, 
I have some confusion about implementing my own Load and LoadCache methods - 
I am trying to load data from an underlying Oracle database into cache, and
the entries are key-value pairs where the key is the Associate ID and the
value is the entire Associate object. 
*(1)* Regarding the Load method, I was under the impression that calling
Get(key) would inherently call the Load(key) method from my OracleStore.cs
file and load the key and corresponding object from the database into cache. 
Under the Load method in OracleStore.cs, I connect to the underlying Oracle
database and execute a query which grabs the data that correlates to the
Associate ID.
However, after running my Program.cs file, I received an error saying
*"System.Collections.Generic.KeyNotFoundException: 'The given key was not
present in the cache: 18755"* -- was I under the wrong impression that the
Load(key) would just put the data into cache? 
I know that the Load method returns an object, and in my case, it returns an
Associate object. However, do I need to do implement something else in order
to put the object returned from the Load method into cache?
*(2)* With the LoadCache method, I essentially would be doing the same thing
as the Load method, but this time, I would be querying 10,000 entries from
the database and putting it into cache. I also hold the same assumption that
calling the LoadCache method will just load the 10,000 entries in cache.
Also, I am unsure about what to do for the Action parameter in
LoadCache(Action, Object[])--would I need to define a method to pass
as a parameter, and would that method be responsible for putting the entries
in cache? If so, how would I go about it? 
 
Files attached below:
Inherits CacheStoreAdapter:  OracleStore.cs
  
Inherits IFactory:  OracleStoreFactory.cs

  
Establishes Oracle database connection:  DBConnection.cs
  
Defines Associate object:  Associate.cs
  
Runs program (creates and starts node):  Program.cs
  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/