Re: Expected serialization performance of Ignite .NET
> 12 processor Intel i7-8850 laptop which is running 100% so you must have some good kit I got those numbers on desktop i7-9700k. Laptops are still no match for desktops (unfortunately). > Does Java version have any bearing on performance? Java version can certainly affect performance, but not in this case, where .NET part does most of the work On Wed, Dec 4, 2019 at 2:53 AM camer314 wrote: > Ok thanks. > > I have managed to get the times down to 13 seconds write and 7.5 seconds > read, basically double your times, on my 12 processor Intel i7-8850 laptop > which is running 100% so you must have some good kit! > > Does Java version have any bearing on performance? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Expected serialization performance of Ignite .NET
Ok thanks. I have managed to get the times down to 13 seconds write and 7.5 seconds read, basically double your times, on my 12 processor Intel i7-8850 laptop which is running 100% so you must have some good kit! Does Java version have any bearing on performance? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Expected serialization performance of Ignite .NET
I've tried your code and got similar numbers (11s to write, 3s to read). Profiler shows that serialization takes 90% of the streaming time (40% is your WriteBinaryInternal method, the rest is Ignite internals) Actual size of serialized object is 161 bytes, and DataRegionMetris.TotalAllocatedSize is ~2.3Gb, which is how much memory Ignite actually uses to store all that data Your code is good, and timings seem good to me too (serializing 2 gigabytes in 10 seconds is not bad in my books). I don't see anything to speed up here, tweaking JVM does not seem to do much. You could split source data in some batches and run multiple streamers in parallel, that brings the time down to 6s on my machine. > What is the most efficient way to iterate all items on a local cache Either ScanQuery without filter, or simply foreach (var entry in cache) - ICache implements IEnumerable On Tue, Dec 3, 2019 at 9:12 AM camer314 wrote: > It seems a combination of a better spec machine and a parallel for loop has > improved performance, although it still takes 8 seconds to run through all > the cache items. > > Here is some basic test code...would appreciate any tips on how to improve > access in this type of usage pattern: > > > https://wtwdeeplearning.blob.core.windows.net/temp/ignitetest.zip?st=2019-12-03T05%3A47%3A38Z=2019-12-12T05%3A47%3A00Z=rl=2018-03-28=b=t%2FXw4bpAFRo7aKdpIbwLfTFOB4Sv%2FeetSi%2FvVSRjg8w%3D > > On my VM it takes 25 seconds to populate and 8 seconds to retrieve. > > What is the most efficient way to iterate all items on a local cache? In a > real situation I would not know the keys i have in the cache and the only > way I could get decent throughput on the read was the parallel loop which > implies i know the keys already. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Expected serialization performance of Ignite .NET
It seems a combination of a better spec machine and a parallel for loop has improved performance, although it still takes 8 seconds to run through all the cache items. Here is some basic test code...would appreciate any tips on how to improve access in this type of usage pattern: https://wtwdeeplearning.blob.core.windows.net/temp/ignitetest.zip?st=2019-12-03T05%3A47%3A38Z=2019-12-12T05%3A47%3A00Z=rl=2018-03-28=b=t%2FXw4bpAFRo7aKdpIbwLfTFOB4Sv%2FeetSi%2FvVSRjg8w%3D On my VM it takes 25 seconds to populate and 8 seconds to retrieve. What is the most efficient way to iterate all items on a local cache? In a real situation I would not know the keys i have in the cache and the only way I could get decent throughput on the read was the parallel loop which implies i know the keys already. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Expected serialization performance of Ignite .NET
Can you please attach a working project to reproduce those numbers? It is hard to say without the code: a class with 21 properties can vary in size a lot. There are many other things at play - JVM options, RAM size, benchmark method, etc. On Mon, Dec 2, 2019 at 2:08 PM camer314 wrote: > I have a 21 property C# class (mix of int and string) and am using > IBinarizable interface as suggested in the documentation. > > My cache is configured such that each cache entry is a collection of these > objects, lets say each cache item is a List. > > I have 10 million instance of this class. For simplicity lets say each > cache > entry holds 10 of these objects, amounting to 1 million cache entries. > > Each object is roughly about 120 bytes long, so 10 million = ~1.2 gigabytes > of data stored. > > I am using a LOCAL cache and a simple foreach loop over the cache takes in > the region of 25 seconds. This seems like an eternity. I understand there > is > a lot of serialization happening, probably a lot of garbage collecting as > well, but it still seems like a large amount of time to effectively move > memory from one location to another. > > Does that time seem exorbitant to you given the above specs or is it > expected? > > What is the optimal way to lay out cache items locally for cache read > iteration (that is, compute needs to iterate the entire cache)? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
