I've tried your code and got similar numbers (11s to write, 3s to read). Profiler shows that serialization takes 90% of the streaming time (40% is your WriteBinaryInternal method, the rest is Ignite internals) Actual size of serialized object is 161 bytes, and DataRegionMetris.TotalAllocatedSize is ~2.3Gb, which is how much memory Ignite actually uses to store all that data
Your code is good, and timings seem good to me too (serializing 2 gigabytes in 10 seconds is not bad in my books). I don't see anything to speed up here, tweaking JVM does not seem to do much. You could split source data in some batches and run multiple streamers in parallel, that brings the time down to 6s on my machine. > What is the most efficient way to iterate all items on a local cache Either ScanQuery without filter, or simply foreach (var entry in cache) - ICache implements IEnumerable On Tue, Dec 3, 2019 at 9:12 AM camer314 <cameron.mur...@towerswatson.com> wrote: > It seems a combination of a better spec machine and a parallel for loop has > improved performance, although it still takes 8 seconds to run through all > the cache items. > > Here is some basic test code...would appreciate any tips on how to improve > access in this type of usage pattern: > > > https://wtwdeeplearning.blob.core.windows.net/temp/ignitetest.zip?st=2019-12-03T05%3A47%3A38Z&se=2019-12-12T05%3A47%3A00Z&sp=rl&sv=2018-03-28&sr=b&sig=t%2FXw4bpAFRo7aKdpIbwLfTFOB4Sv%2FeetSi%2FvVSRjg8w%3D > > On my VM it takes 25 seconds to populate and 8 seconds to retrieve. > > What is the most efficient way to iterate all items on a local cache? In a > real situation I would not know the keys i have in the cache and the only > way I could get decent throughput on the read was the parallel loop which > implies i know the keys already. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
