Re: Getting all data from cache via scan query is taking lot of time
Fix is complete and will be included in Ignite 2.8. Pre-release build is available right now: https://www.nuget.org/packages/Apache.Ignite/2.8.0-alpha20200122 On Mon, Jan 20, 2020 at 2:04 PM Pavel Tupitsyn wrote: > Ticket filed and fix is on the way: > https://issues.apache.org/jira/browse/IGNITE-12555 > > On Tue, Jan 14, 2020 at 6:44 PM Pavel Tupitsyn > wrote: > >> I've tried that code with a single local Ignite server, started with >> default configuration, the result is: >>Time for PutAll: 187 ms for rows: 7500 >>Time for fetching all data: 128 ms for rows: 7500 >> >> However, when I removed JSON serialization and stored the objects as is >> (IgniteCache): >>Time for PutAll: 1924 ms for rows: 7500 >>Time for fetching all data: 20288 ms for rows: 7500 >> >> There is a performance issue with DateTime value serialization. I'm >> investigating and will get back to you with details soon. >> >> Meanwhile, there is a *workaround*: force Timestamp serialization format >> and convert all values to UTC (which is a good idea anyway): >> new IgniteClientConfiguration >> { >> ... >> BinaryConfiguration = new BinaryConfiguration >> { >> Serializer = new BinaryReflectiveSerializer >> { >> ForceTimestamp = true >> } >> } >> }; >> >> Then call ToUniversalTime() on all DateTime values. The result is: >> Time for PutAll: 761 ms for rows: 7500 >> Time for fetching all data: 645 ms for rows: 7500 >> >> For reference, JSON serialization of those 7500 rows (CreateMapFromList) >> takes ~600ms on my machine. >> >> On Tue, Jan 14, 2020 at 4:09 PM Tunas >> wrote: >> >>> I am using Ignite 2.7.6 dll for Ignite and newtonSoft json for >>> converting C# >>> objects to json. >>> Ignite configuration is same as mentioned allow. (Server and client are >>> running on different machine) >>> >>> Sorry code class is little bit lengthy. >>> >>> One thing observed before publishing this code: addition of properties of >>> type ObservableCollection is taking more time. >>> >>> class Program >>> { >>> public static IIgniteClient IgniteThinClient; >>> private static IgniteClientConfiguration >>> GetIgniteClientConfiguration() >>> { >>> return new IgniteClientConfiguration >>> { >>> Endpoints = new[] { "MyWorkStation:10800" }, >>> SocketTimeout = TimeSpan.FromSeconds(60) >>> }; >>> } >>> static void Main(string[] args) >>> { >>> int rows = 7500; >>> var lst = GetData(rows); >>> >>> Ignition.ClientMode = true; >>> IgniteThinClient = >>> Ignition.StartClient(GetIgniteClientConfiguration()); >>> var cache = IgniteThinClient.GetOrCreateCache>> string>("TestSlowness"); >>> >>> var map1 = CreateMapFromList(lst); >>> Stopwatch sw = new Stopwatch(); >>> sw.Start(); >>> cache.PutAll(map1); >>> sw.Stop(); >>> Console.WriteLine($"Time for PutAll: >>> {sw.ElapsedMilliseconds} ms >>> for rows: {rows}"); >>> sw.Reset(); >>> >>> >>> ICacheClient cacheClient = >>> IgniteThinClient.GetOrCreateCache("TestSlowness"); >>> if(cacheClient!=null) >>> { >>> sw.Start(); >>> *var cacheEntries = cacheClient.Query(new >>> Apache.Ignite.Core.Cache.Query.ScanQuery(null)).GetAll();* >>> sw.Stop(); >>> Console.WriteLine($"Time for fetching all data: >>> {sw.ElapsedMilliseconds} ms for rows: {rows}"); >>> } >>> } >>> >>> //Actually below one is generic method(for simplicity changed to >>> non-generic) >>> private static ConcurrentDictionary >>> CreateMapFromList(IEnumerable collection) >>> { >>> var map = new ConcurrentDictionary(); >>> if (collection != null && collection.Any()) >>> { >>> foreach (EmpDataSet item in collection) >>> { >>> var val = SerializedEntity(item); >>> if (!string.IsNullOrWhiteSpace(val)) >>> { >>> map.TryAdd(item.DtId, val); >>> } >>> } >>> } >>> return map; >>> } >>> >>> >>> private static List GetData(int rows = 7000) >>> { >>> Random rand = new Random(); >>> >>> var lst = new List(); >>> for (int i=1; i<=rows;i++) >>> { >>> var obj1 = new EmpDataSet(); >>> obj1.DtId = i; >>> obj1.EmpRef = rand.Next(rows + rows); >>> obj1.DtName = RandomString(rand.Next(15, 20)); >>> obj1.DraftDtId = r
Re: Getting all data from cache via scan query is taking lot of time
Ticket filed and fix is on the way: https://issues.apache.org/jira/browse/IGNITE-12555 On Tue, Jan 14, 2020 at 6:44 PM Pavel Tupitsyn wrote: > I've tried that code with a single local Ignite server, started with > default configuration, the result is: >Time for PutAll: 187 ms for rows: 7500 >Time for fetching all data: 128 ms for rows: 7500 > > However, when I removed JSON serialization and stored the objects as is > (IgniteCache): >Time for PutAll: 1924 ms for rows: 7500 >Time for fetching all data: 20288 ms for rows: 7500 > > There is a performance issue with DateTime value serialization. I'm > investigating and will get back to you with details soon. > > Meanwhile, there is a *workaround*: force Timestamp serialization format > and convert all values to UTC (which is a good idea anyway): > new IgniteClientConfiguration > { > ... > BinaryConfiguration = new BinaryConfiguration > { > Serializer = new BinaryReflectiveSerializer > { > ForceTimestamp = true > } > } > }; > > Then call ToUniversalTime() on all DateTime values. The result is: > Time for PutAll: 761 ms for rows: 7500 > Time for fetching all data: 645 ms for rows: 7500 > > For reference, JSON serialization of those 7500 rows (CreateMapFromList) > takes ~600ms on my machine. > > On Tue, Jan 14, 2020 at 4:09 PM Tunas > wrote: > >> I am using Ignite 2.7.6 dll for Ignite and newtonSoft json for converting >> C# >> objects to json. >> Ignite configuration is same as mentioned allow. (Server and client are >> running on different machine) >> >> Sorry code class is little bit lengthy. >> >> One thing observed before publishing this code: addition of properties of >> type ObservableCollection is taking more time. >> >> class Program >> { >> public static IIgniteClient IgniteThinClient; >> private static IgniteClientConfiguration >> GetIgniteClientConfiguration() >> { >> return new IgniteClientConfiguration >> { >> Endpoints = new[] { "MyWorkStation:10800" }, >> SocketTimeout = TimeSpan.FromSeconds(60) >> }; >> } >> static void Main(string[] args) >> { >> int rows = 7500; >> var lst = GetData(rows); >> >> Ignition.ClientMode = true; >> IgniteThinClient = >> Ignition.StartClient(GetIgniteClientConfiguration()); >> var cache = IgniteThinClient.GetOrCreateCache> string>("TestSlowness"); >> >> var map1 = CreateMapFromList(lst); >> Stopwatch sw = new Stopwatch(); >> sw.Start(); >> cache.PutAll(map1); >> sw.Stop(); >> Console.WriteLine($"Time for PutAll: {sw.ElapsedMilliseconds} >> ms >> for rows: {rows}"); >> sw.Reset(); >> >> >> ICacheClient cacheClient = >> IgniteThinClient.GetOrCreateCache("TestSlowness"); >> if(cacheClient!=null) >> { >> sw.Start(); >> *var cacheEntries = cacheClient.Query(new >> Apache.Ignite.Core.Cache.Query.ScanQuery(null)).GetAll();* >> sw.Stop(); >> Console.WriteLine($"Time for fetching all data: >> {sw.ElapsedMilliseconds} ms for rows: {rows}"); >> } >> } >> >> //Actually below one is generic method(for simplicity changed to >> non-generic) >> private static ConcurrentDictionary >> CreateMapFromList(IEnumerable collection) >> { >> var map = new ConcurrentDictionary(); >> if (collection != null && collection.Any()) >> { >> foreach (EmpDataSet item in collection) >> { >> var val = SerializedEntity(item); >> if (!string.IsNullOrWhiteSpace(val)) >> { >> map.TryAdd(item.DtId, val); >> } >> } >> } >> return map; >> } >> >> >> private static List GetData(int rows = 7000) >> { >> Random rand = new Random(); >> >> var lst = new List(); >> for (int i=1; i<=rows;i++) >> { >> var obj1 = new EmpDataSet(); >> obj1.DtId = i; >> obj1.EmpRef = rand.Next(rows + rows); >> obj1.DtName = RandomString(rand.Next(15, 20)); >> obj1.DraftDtId = rand.Next(rows + rows); >> obj1.ApprovedDtId = rand.Next(rows + rows); >> obj1.Description = RandomString(rand.Next(20, 60)); >> obj1.Code1 = RandomString(rand.Next(1, 10), >> Convert.ToBoolean(rand.Next(0, 1))); >> obj1.Code2 = RandomString(rand.Next(1, 10), >> Convert.ToBoolean(rand.Next(0
Re: Getting all data from cache via scan query is taking lot of time
I've tried that code with a single local Ignite server, started with default configuration, the result is: Time for PutAll: 187 ms for rows: 7500 Time for fetching all data: 128 ms for rows: 7500 However, when I removed JSON serialization and stored the objects as is (IgniteCache): Time for PutAll: 1924 ms for rows: 7500 Time for fetching all data: 20288 ms for rows: 7500 There is a performance issue with DateTime value serialization. I'm investigating and will get back to you with details soon. Meanwhile, there is a *workaround*: force Timestamp serialization format and convert all values to UTC (which is a good idea anyway): new IgniteClientConfiguration { ... BinaryConfiguration = new BinaryConfiguration { Serializer = new BinaryReflectiveSerializer { ForceTimestamp = true } } }; Then call ToUniversalTime() on all DateTime values. The result is: Time for PutAll: 761 ms for rows: 7500 Time for fetching all data: 645 ms for rows: 7500 For reference, JSON serialization of those 7500 rows (CreateMapFromList) takes ~600ms on my machine. On Tue, Jan 14, 2020 at 4:09 PM Tunas wrote: > I am using Ignite 2.7.6 dll for Ignite and newtonSoft json for converting > C# > objects to json. > Ignite configuration is same as mentioned allow. (Server and client are > running on different machine) > > Sorry code class is little bit lengthy. > > One thing observed before publishing this code: addition of properties of > type ObservableCollection is taking more time. > > class Program > { > public static IIgniteClient IgniteThinClient; > private static IgniteClientConfiguration > GetIgniteClientConfiguration() > { > return new IgniteClientConfiguration > { > Endpoints = new[] { "MyWorkStation:10800" }, > SocketTimeout = TimeSpan.FromSeconds(60) > }; > } > static void Main(string[] args) > { > int rows = 7500; > var lst = GetData(rows); > > Ignition.ClientMode = true; > IgniteThinClient = > Ignition.StartClient(GetIgniteClientConfiguration()); > var cache = IgniteThinClient.GetOrCreateCache string>("TestSlowness"); > > var map1 = CreateMapFromList(lst); > Stopwatch sw = new Stopwatch(); > sw.Start(); > cache.PutAll(map1); > sw.Stop(); > Console.WriteLine($"Time for PutAll: {sw.ElapsedMilliseconds} > ms > for rows: {rows}"); > sw.Reset(); > > > ICacheClient cacheClient = > IgniteThinClient.GetOrCreateCache("TestSlowness"); > if(cacheClient!=null) > { > sw.Start(); > *var cacheEntries = cacheClient.Query(new > Apache.Ignite.Core.Cache.Query.ScanQuery(null)).GetAll();* > sw.Stop(); > Console.WriteLine($"Time for fetching all data: > {sw.ElapsedMilliseconds} ms for rows: {rows}"); > } > } > > //Actually below one is generic method(for simplicity changed to > non-generic) > private static ConcurrentDictionary > CreateMapFromList(IEnumerable collection) > { > var map = new ConcurrentDictionary(); > if (collection != null && collection.Any()) > { > foreach (EmpDataSet item in collection) > { > var val = SerializedEntity(item); > if (!string.IsNullOrWhiteSpace(val)) > { > map.TryAdd(item.DtId, val); > } > } > } > return map; > } > > > private static List GetData(int rows = 7000) > { > Random rand = new Random(); > > var lst = new List(); > for (int i=1; i<=rows;i++) > { > var obj1 = new EmpDataSet(); > obj1.DtId = i; > obj1.EmpRef = rand.Next(rows + rows); > obj1.DtName = RandomString(rand.Next(15, 20)); > obj1.DraftDtId = rand.Next(rows + rows); > obj1.ApprovedDtId = rand.Next(rows + rows); > obj1.Description = RandomString(rand.Next(20, 60)); > obj1.Code1 = RandomString(rand.Next(1, 10), > Convert.ToBoolean(rand.Next(0, 1))); > obj1.Code2 = RandomString(rand.Next(1, 10), > Convert.ToBoolean(rand.Next(0, 1))); > obj1.Label = RandomString(rand.Next(1, 10), > Convert.ToBoolean(rand.Next(0, 1))); > obj1.Etype = (EType)(rand.Next(1, 8)); > obj1.Estatus = (EStatus)(rand.Next(1, 4)); > obj1.Notes = RandomString(30, > Convert.ToBoolean(rand.Next(0,
Re: Getting all data from cache via scan query is taking lot of time
I am using Ignite 2.7.6 dll for Ignite and newtonSoft json for converting C# objects to json. Ignite configuration is same as mentioned allow. (Server and client are running on different machine) Sorry code class is little bit lengthy. One thing observed before publishing this code: addition of properties of type ObservableCollection is taking more time. class Program { public static IIgniteClient IgniteThinClient; private static IgniteClientConfiguration GetIgniteClientConfiguration() { return new IgniteClientConfiguration { Endpoints = new[] { "MyWorkStation:10800" }, SocketTimeout = TimeSpan.FromSeconds(60) }; } static void Main(string[] args) { int rows = 7500; var lst = GetData(rows); Ignition.ClientMode = true; IgniteThinClient = Ignition.StartClient(GetIgniteClientConfiguration()); var cache = IgniteThinClient.GetOrCreateCache("TestSlowness"); var map1 = CreateMapFromList(lst); Stopwatch sw = new Stopwatch(); sw.Start(); cache.PutAll(map1); sw.Stop(); Console.WriteLine($"Time for PutAll: {sw.ElapsedMilliseconds} ms for rows: {rows}"); sw.Reset(); ICacheClient cacheClient = IgniteThinClient.GetOrCreateCache("TestSlowness"); if(cacheClient!=null) { sw.Start(); *var cacheEntries = cacheClient.Query(new Apache.Ignite.Core.Cache.Query.ScanQuery(null)).GetAll();* sw.Stop(); Console.WriteLine($"Time for fetching all data: {sw.ElapsedMilliseconds} ms for rows: {rows}"); } } //Actually below one is generic method(for simplicity changed to non-generic) private static ConcurrentDictionary CreateMapFromList(IEnumerable collection) { var map = new ConcurrentDictionary(); if (collection != null && collection.Any()) { foreach (EmpDataSet item in collection) { var val = SerializedEntity(item); if (!string.IsNullOrWhiteSpace(val)) { map.TryAdd(item.DtId, val); } } } return map; } private static List GetData(int rows = 7000) { Random rand = new Random(); var lst = new List(); for (int i=1; i<=rows;i++) { var obj1 = new EmpDataSet(); obj1.DtId = i; obj1.EmpRef = rand.Next(rows + rows); obj1.DtName = RandomString(rand.Next(15, 20)); obj1.DraftDtId = rand.Next(rows + rows); obj1.ApprovedDtId = rand.Next(rows + rows); obj1.Description = RandomString(rand.Next(20, 60)); obj1.Code1 = RandomString(rand.Next(1, 10), Convert.ToBoolean(rand.Next(0, 1))); obj1.Code2 = RandomString(rand.Next(1, 10), Convert.ToBoolean(rand.Next(0, 1))); obj1.Label = RandomString(rand.Next(1, 10), Convert.ToBoolean(rand.Next(0, 1))); obj1.Etype = (EType)(rand.Next(1, 8)); obj1.Estatus = (EStatus)(rand.Next(1, 4)); obj1.Notes = RandomString(30, Convert.ToBoolean(rand.Next(0, 1))); obj1.Flag1 = Convert.ToBoolean(rand.Next(0, 1)); obj1.Flag2 = Convert.ToBoolean(rand.Next(0, 1)); obj1.Flag3 = Convert.ToBoolean(rand.Next(0, 1)); obj1.EmpAreaMap = ConstructEmpAreaMap(rand); obj1.EcType = (EcType)(rand.Next(1, 2)); obj1.Ref1 = rand.Next(50, 100); obj1.Ref1Name = RandomString(10, Convert.ToBoolean(rand.Next(0, 1))); obj1.DgId = rand.Next(24, 48); obj1.EmpRegs = GetEmpRegs(rand); obj1.DRegs = null; obj1.Fmap = new FMap { Id = rand.Next(1, rows + rows), Flts = ConstructFlts(rand, rand.Next(1, 4)) }; obj1.group1 = ConstructAgroup(rand); obj1.group2 = ConstructAgroup(rand); obj1.Str1 = RandomString(rand.Next(1, 50), Convert.ToBoolean(rand.Next(0, 1))); obj1.Str2 = RandomString(rand.Next(1, 50), Convert.ToBoolean(rand.Next(0, 1))); obj1.Str3 = RandomString(rand.Next(1, 50), Convert.ToBoolean(rand.Next(0, 1))); obj1.Str4 = RandomString(rand.Next(1, 50), Convert.ToBoolean(rand.Next(0, 1))); obj1.Str5 = RandomString(rand.Next(1, 50), Convert.ToBoolean(rand.Next(0, 1))); lst.Add(obj1); } return lst; } private static Agroup ConstructAgroup(Random rand) { int rows = rand.Next(1, 2); var obj1
Getting all data from cache via scan query is taking lot of time
Configuration of Ignite server backups = 1 persistenceEnabled = true atomicityMode = true I have two nodes on one server and only one ignite server instance is running. I am using C# thin client to fetch data from collection. I am storing custom user defined object collection in my cache. 1. custom object has around 30 fields 2. collection contains around 7000 custom objects. 3. I am storing object by converting it to JSON Now when i fetch whole collection from cache using "ScanQuery" it is taking more time then traditional database (SQL/Oracle/Sybase) I create cache for each custom object collection like below ICacheClient cacheClient = _igniteStore.IgniteThinClient.GetOrCreateCache(typeof(T).FullName); and when i want to fetch all data from collection i do it like below var cacheEntries = cacheClient.Query(new Apache.Ignite.Core.Cache.Query.*ScanQuery(null)).GetAll()*; I have tried multiple times but for my custom object which has 30 fields and collection has around 7k of this custom objects *It takes around 2 milliseconds out of which 18500 milliseconds taken by "ScanQuery" statement above and 1500 milliseconds by newton json converter. If i fetch same from my traditional RDBMS (SQL) then it takes only 5500 milliseconds.* Can some one please let me know what i am doing wrong? Anything to do with configuration of ignite server or any other way to fetch all data from cache. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Getting all data from cache via scan query is taking lot of time
Configuration of Ignite server backups = 1 persistenceEnabled = true atomicityMode = true I have two nodes on one server and only one ignite server instance is running. I am using C# thin client to fetch data from collection. I am storing custom user defined object collection in my cache. 1. custom object has around 30 fields 2. collection contains around 7000 custom objects. 3. I am storing object by converting it to JSON Now when i fetch whole collection from cache using "ScanQuery" it is taking more time then traditional database (SQL/Oracle/Sybase) I create cache for each custom object collection like below ICacheClient cacheClient = _igniteStore.IgniteThinClient.GetOrCreateCache(typeof(T).FullName); and when i want to fetch all data from collection i do it like below var cacheEntries = cacheClient.Query(new Apache.Ignite.Core.Cache.Query.*ScanQuery(null)).GetAll()*; I have tried multiple times but for my custom object which has 30 fields and collection has around 7k of this custom objects *It takes around 2 milliseconds out of which 18500 milliseconds taken by "ScanQuery" statement above and 1500 milliseconds by newton json converter. If i fetch same from my traditional RDBMS (SQL) then it takes only 5500 milliseconds.* Can some one please let me know what i am doing wrong? Anything to do with configuration of ignite server or any other way to fetch all data from cache. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Getting all data from cache via scan query is taking lot of time
20 seconds for a collection of 7k objects is way too slow. Can you please attach full reproducer so I can run it and dig deeper? On Mon, Jan 13, 2020 at 2:47 PM Tunas wrote: > Configuration of Ignite server > > backups = 1 > persistenceEnabled = true > atomicityMode = true > > I have two nodes on one server and only one ignite server instance is > running. > > I am using C# thin client to fetch data from collection. > > I am storing custom user defined object collection in my cache. > 1. custom object has around 30 fields > 2. collection contains around 7000 custom objects. > 3. I am storing object by converting it to JSON > > Now when i fetch whole collection from cache using "ScanQuery" it is taking > more time then traditional database (SQL/Oracle/Sybase) > > I create cache for each custom object collection like below > ICacheClient cacheClient = > _igniteStore.IgniteThinClient.GetOrCreateCache string>(typeof(T).FullName); > > > and when i want to fetch all data from collection i do it like below > var cacheEntries = cacheClient.Query(new > Apache.Ignite.Core.Cache.Query.*ScanQuery(null)).GetAll()*; > > > I have tried multiple times but for my custom object which has 30 fields > and > collection has around 7k of this custom objects > > *It takes around 2 milliseconds out of which 18500 milliseconds taken > by > "ScanQuery" statement above and 1500 milliseconds by newton json converter. > > If i fetch same from my traditional RDBMS (SQL) then it takes only 5500 > milliseconds.* > > > Can some one please let me know what i am doing wrong? > Anything to do with configuration of ignite server or any other way to > fetch > all data from cache. > > > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Getting all data from cache via scan query is taking lot of time
Configuration of Ignite server backups = 1 persistenceEnabled = true atomicityMode = true I have two nodes on one server and only one ignite server instance is running. I am using C# thin client to fetch data from collection. I am storing custom user defined object collection in my cache. 1. custom object has around 30 fields 2. collection contains around 7000 custom objects. 3. I am storing object by converting it to JSON Now when i fetch whole collection from cache using "ScanQuery" it is taking more time then traditional database (SQL/Oracle/Sybase) I create cache for each custom object collection like below ICacheClient cacheClient = _igniteStore.IgniteThinClient.GetOrCreateCache(typeof(T).FullName); and when i want to fetch all data from collection i do it like below var cacheEntries = cacheClient.Query(new Apache.Ignite.Core.Cache.Query.*ScanQuery(null)).GetAll()*; I have tried multiple times but for my custom object which has 30 fields and collection has around 7k of this custom objects *It takes around 2 milliseconds out of which 18500 milliseconds taken by "ScanQuery" statement above and 1500 milliseconds by newton json converter. If i fetch same from my traditional RDBMS (SQL) then it takes only 5500 milliseconds.* Can some one please let me know what i am doing wrong? Anything to do with configuration of ignite server or any other way to fetch all data from cache. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/