Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-04-23 Thread Andrey Mashenkov
Hi Naveen, Ignite is memory centric database and I doubt it should run query on Native persistent that is on DISK. However, for now, Ignite can use Heap memory only for heavy queries. It will be fixed in future versions via introducing memory region for SQL queries that will allow Ignite to use di

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-04-19 Thread Naveen
HI Andrey The reason I was trying to explore this feature is, I will give you an example. I have a cache with 20M records and when I run this query SELECT * FROM "AssociatedPartiesCache".ASSOCIATED_PARTIES Query took more than 200 secs and ran out of memory, here is the error thrown This I ha

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-04-05 Thread Andrey Mashenkov
Hi, >So are you saying, query execution will not have any impact on the cluster >activities like GET/PUTs in general ?? Why? Queries and get\put share same resources. So, they will affect each other. >I was thinking, if we can run these queries directly on backing store, it >may not have any impa

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-04-02 Thread Naveen
Hi ANdrew There were cases, when I just run select * from table on SQLLINE unknowingly, we could see queries getting slowed down and OOM errors. Our dev machines not very high end ones. When we deliver this solution, production support guys can run queries to debug any data related issues, during

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-04-02 Thread Andrey Mashenkov
Hi, >In this case also, since we dont have eviction in place, all the time data >is retrieved from RAM only, the only time request goes to Oracle is for >Upserts and delete. Here, all queries run on data in RAM only. Ignite just propagate updates to back store (to Oracle via CacheStore impl) with

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-04-01 Thread Naveen
HI Let me rephrase my question, guess I have conveyed my question correctly. Lets take an example Ignite cluster with backing store as RDBMS - oracle and no eviction in place. As we all know, we can run complex queries on Oracle to retrieve desired data. In this case also, since we dont have e

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-03-30 Thread Andrey Mashenkov
Hi Naveen, Ignite implements page memory concept since version 2.0 has been released. This means, Ignite reads page from disk if it is not loaded to memory and evict pages to disk to free memory. Page load and eviction work automatically. So, I can't understand what "feature like eviction" and "h

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-03-29 Thread Naveen
Lazy is something which we can use when we are expecting huge result set to minimize the memory consumption at the cost of little performance hit. However my requirement to run a query by joining couple of tables with complex where clause to debug some data related issues, this query may not return

Re: Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-03-29 Thread Andrey Mashenkov
Hi Naveen, You can try a 'lazy' flag for query. It is available from ignite-2.4 that has been released recently. See SqlFieldQuery javadoc [1] and JDBC doc [2] for details. [1] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/SqlFieldsQuery.html#setLazy-boolean- [2]

Running heavy queries on Ignite cluster on backing store directly without impacting the cluster

2018-03-28 Thread Naveen
Hi I am using ignite 2.3 with native persistence layer as backing store We do have close to half to 1 billion records in each of the tables. There are some adhoc requirements to query the tables with diffrent where conditions, columns which we use in where clause may not have indexes, which may