sql query large data set which is larger than memory

[November]

Hi

I have a large data set larger than memory.  I want to do some sql operation
on it. So I enabled swap space. But the program still run out of memory.
Below is my cache configuration code. Is there any thing wrong?

Thanks

var cache = ignite.GetOrCreateCache(new CacheConfiguration
{
Name = CacheName,
EnableSwap = true
});



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/sql-query-large-data-set-which-is-larger-than-memory-tp6219.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Create cache configuration error

[deleerhai]

Hi,
Create cache configuration error.

 
 

七月 12, 2016 10:13:19 上午 org.apache.catalina.core.ApplicationContext log
严重: StandardWrapper.Throwable
javax.cache.CacheException: Index with name 'code_idx' already exists.
at
org.apache.ignite.configuration.CacheConfiguration$TypeDescriptor.addIndex(CacheConfiguration.java:2429)
at
org.apache.ignite.configuration.CacheConfiguration.processAnnotation(CacheConfiguration.java:2340)
at
org.apache.ignite.configuration.CacheConfiguration.processAnnotationsInClass(CacheConfiguration.java:2292)
at
org.apache.ignite.configuration.CacheConfiguration.processKeyAndValueClasses(CacheConfiguration.java:2233)
at
org.apache.ignite.configuration.CacheConfiguration.setIndexedTypes(CacheConfiguration.java:1913)
at
com.czt.kbp.coupon.repository.cachestore.CouponCacheConfig.config(CouponCacheConfig.java:14)
at
com.czt.kbp.coupon.repository.cachestore.CouponCacheIniter.init(CouponCacheIniter.java:58)
at
com.czt.kbp.coupon.repository.cachestore.CouponCacheIniter.initCache(CouponCacheIniter.java:41)
at
com.czt.kbp.coupon.server.servlet.CouponInitServlet.init(CouponInitServlet.java:19)
at
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1238)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1151)
at 
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1038)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4996)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1408)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Create-cache-configuration-error-tp6218.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Issue with concurrent users on Ignite 1.6.0 ODBC

[Agneeswaran]

Hi Team,

We are running the script with 16 users concurrently on Ignite ODBC node.

Some of the user processors are interrupted and stopped in between.And also
this issue happens even with 2 user concurrent test.

The same scenario works with Ignite 1.5.0.

Please suggest, do we need to modify any configuration for concurrent test
on ODBC.

Thanks,
Agneeswaran



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Issue-with-concurrent-users-on-Ignite-1-6-0-ODBC-tp6217.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: CassandraCacheStoreFactory injection error while trying to use Cassandra as persistent store

[ghughal]

It looks like I'll have to convert all Ignite and cache configuration to XML.
Just putting cassandra config in XML doesn't seem to work.





--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/CassandraCacheStoreFactory-injection-error-while-trying-to-use-Cassandra-as-persistent-store-tp5689p6215.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Update performance

[ionut_s]

Hi,

I would like to understand which is the best solution to simulate a
transactional "update x where y" logic.

>From what I understand the transaction coordinator has to know all the
modified rows. 
Is there a way to avoid that and have the entire logic (reading filtered
data + updates) locally on all nodes which stores data for a cache?  
In this scenario the transaction coordinator is not aware of the impacted
rows.

Thanks,
Ionut



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Update-performance-tp6214.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Performance issue on Ignite ODBC API

[Igor Sapego]

Hi Agneeswaran,

Thats a great thing to hear. I'm going to change the default value to
something more appropriate.

Let us know if there are any other issues you are facing.


Best Regards,
Igor

On Mon, Jul 11, 2016 at 8:53 PM, Agneeswaran <
agneeswaran.ponnuraman...@nielsen.com> wrote:

> Hi Igor,
>
> The solution works perfectly fine.It completes in ~12 seconds ( before ~4
> minutes ).
>
> Thanks for your support.
>
> With regards,
> Agneeswaran
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Performance-issue-on-Ignite-ODBC-API-tp5908p6212.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Performance issue on Ignite ODBC API

[Agneeswaran]

Hi Igor,

The solution works perfectly fine.It completes in ~12 seconds ( before ~4
minutes ).

Thanks for your support.

With regards,
Agneeswaran



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Performance-issue-on-Ignite-ODBC-API-tp5908p6212.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Cache Partitioned Mode

[daniels]

-"it will load it from the cache store, puts into memory and returns for the
client. "

It means that  load occurrs on server node by server node loader (and not by
client node loader)and then  puts in memory on server side and returns for
client.
Doesnt it?




--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Cache-Partitioned-Mode-tp6172p6211.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Cassandra Cache Store Example

[Denis Magda]

Hi Kamal,

Please create a ticket in JIRA for that and share a link to it over there.
Hope that someone from the community will pick it up and implement. If you’re 
interested in this kind of contribution then it would be perfect.

—
Denis

> On Jul 11, 2016, at 3:05 PM, Kamal C  wrote:
> 
> Hi,
> 
> Can anyone add Cassandra CacheStore example in the examples[1] like JDBC 
> CacheStore example?
> 
> It will be useful to configure and test the feature quickly.
> 
> [1]: 
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/datagrid/store
>  
> 
> 
> --Kamal
> 
> 

Performance in case of 1 server node

[daniels]

Hi ,
I have one server node (embedded Ignite).
My ignite and cache configurations is bellow,
please help me to add or remove configs for better performance


   













 

P.S. I read Performance Tips in Documentation,it mostly related  >1 nodes.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Performance-in-case-of-1-server-node-tp6207.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Cassandra Cache Store Example

[Kamal C]

Hi,

Can anyone add Cassandra CacheStore example in the examples[1] like
JDBC CacheStore example?

It will be useful to configure and test the feature quickly.

[1]:
https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/datagrid/store

--Kamal

Re: How to stop a IgniteService gracefully?

[Kamal C]

When a node which provides the service have high CPU and heap memory usage.
Can I switch the service from that node to the other eligible node ?

On Mon, Jul 11, 2016 at 4:40 PM, Vladislav Pyatkov 
wrote:

> Hello Kamal,
>
> -  How to tell Ignite to move a clustered singleton service from one node
> to another with external scripts?
>
> What are you mean when say: "move a clustered singleton service from one
> node to another"? If singleton service was deployed on cluster group, it
> will stay until, last node leaves the cluster.
>
>  In case the cluster node on which the service was deployed crashes or
> stops, Ignite will automatically redeploy it on another node. [1]
>
> [1]: https://apacheignite.readme.io/docs/cluster-singletons
>
> On Mon, Jul 11, 2016 at 7:48 AM, Kamal C  wrote:
>
>> Sorry for polluting the mailing list.
>>
>> Ignition.stop(false); // internally stops / cancels the local Ignite
>> service.
>>
>> Can anyone answer for the 2nd question?
>>
>> --Kamal
>>
>> On Mon, Jul 11, 2016 at 10:00 AM, Kamal C  wrote:
>>
>>> Vladislav,
>>>
>>> I've applied your suggestion. Still, Ignite closes the service running
>>> in all the nodes,
>>>
>>> ClusterGroup grp =
>>> ignite.cluster().forNode(ignite.cluster().localNode());
>>> ignite.services(grp).cancel(calcService);
>>>
>>>
>>> *Exception*
>>> class org.apache.ignite.IgniteException: Failed to find deployed
>>> service: CalcService
>>> at
>>> org.apache.ignite.internal.processors.service.GridServiceProxy.invokeMethod(GridServiceProxy.java:155)
>>> at
>>> org.apache.ignite.internal.processors.service.GridServiceProxy$ProxyInvocationHandler.invoke(GridServiceProxy.java:331)
>>> at com.sun.proxy.$Proxy28.add(Unknown Source)
>>> at
>>> my.apache.ignite.examples.servicegrid.ServiceDeployer.useService(ServiceDeployer.java:63)
>>>
>>>
>>> Re-framing my question:
>>>
>>> -  Is Service.cancel() method is not a appropriate place for resource
>>> cleanups on shutdown?
>>> -  How to tell Ignite to move a clustered singleton service from one
>>> node to another with external scripts?
>>>
>>> -- Kamal
>>>
>>> On Sat, Jul 9, 2016 at 1:07 AM, Vladislav Pyatkov 
>>> wrote:
>>>
 Hello Kamal,

 Try to use Ignite.services(ClusterGroup
 
  grp)
 for local node group.


 *Ignition.ignite().services(Ignition.ignite().cluster().localNode()).cancel(serviceName);*

 On Fri, Jul 8, 2016 at 9:48 AM, Kamal C  wrote:

> Hi,
>
> I have a 3 node Ignite cluster and deployed a service using Node
> Singleton approach. The service started to run in all the 3 nodes.
>
> Whenever a node stopped, the below line gets called as I'm doing
> resource cleanups inside *IgniteService.cancel(ServiceContext
> context)* method.
>
> *Ignition.ignite().services().cancel(serviceName);*
>
> But, it cancels the service from all the nodes. How to cancel a
> locally running Ignite service gracefully?
>
> --
> Kamal
>



 --
 Vladislav Pyatkov

>>>
>>>
>>
>
>
> --
> Vladislav Pyatkov
>

Re: How to stop a IgniteService gracefully?

[Vladislav Pyatkov]

Hello Kamal,

-  How to tell Ignite to move a clustered singleton service from one node
to another with external scripts?

What are you mean when say: "move a clustered singleton service from one
node to another"? If singleton service was deployed on cluster group, it
will stay until, last node leaves the cluster.

 In case the cluster node on which the service was deployed crashes or
stops, Ignite will automatically redeploy it on another node. [1]

[1]: https://apacheignite.readme.io/docs/cluster-singletons

On Mon, Jul 11, 2016 at 7:48 AM, Kamal C  wrote:

> Sorry for polluting the mailing list.
>
> Ignition.stop(false); // internally stops / cancels the local Ignite
> service.
>
> Can anyone answer for the 2nd question?
>
> --Kamal
>
> On Mon, Jul 11, 2016 at 10:00 AM, Kamal C  wrote:
>
>> Vladislav,
>>
>> I've applied your suggestion. Still, Ignite closes the service running in
>> all the nodes,
>>
>> ClusterGroup grp = ignite.cluster().forNode(ignite.cluster().localNode());
>> ignite.services(grp).cancel(calcService);
>>
>>
>> *Exception*
>> class org.apache.ignite.IgniteException: Failed to find deployed service:
>> CalcService
>> at
>> org.apache.ignite.internal.processors.service.GridServiceProxy.invokeMethod(GridServiceProxy.java:155)
>> at
>> org.apache.ignite.internal.processors.service.GridServiceProxy$ProxyInvocationHandler.invoke(GridServiceProxy.java:331)
>> at com.sun.proxy.$Proxy28.add(Unknown Source)
>> at
>> my.apache.ignite.examples.servicegrid.ServiceDeployer.useService(ServiceDeployer.java:63)
>>
>>
>> Re-framing my question:
>>
>> -  Is Service.cancel() method is not a appropriate place for resource
>> cleanups on shutdown?
>> -  How to tell Ignite to move a clustered singleton service from one node
>> to another with external scripts?
>>
>> -- Kamal
>>
>> On Sat, Jul 9, 2016 at 1:07 AM, Vladislav Pyatkov 
>> wrote:
>>
>>> Hello Kamal,
>>>
>>> Try to use Ignite.services(ClusterGroup
>>> 
>>>  grp)
>>> for local node group.
>>>
>>>
>>> *Ignition.ignite().services(Ignition.ignite().cluster().localNode()).cancel(serviceName);*
>>>
>>> On Fri, Jul 8, 2016 at 9:48 AM, Kamal C  wrote:
>>>
 Hi,

 I have a 3 node Ignite cluster and deployed a service using Node
 Singleton approach. The service started to run in all the 3 nodes.

 Whenever a node stopped, the below line gets called as I'm doing
 resource cleanups inside *IgniteService.cancel(ServiceContext context)*
 method.

 *Ignition.ignite().services().cancel(serviceName);*

 But, it cancels the service from all the nodes. How to cancel a locally
 running Ignite service gracefully?

 --
 Kamal

>>>
>>>
>>>
>>> --
>>> Vladislav Pyatkov
>>>
>>
>>
>


-- 
Vladislav Pyatkov

Re: Cache Partitioned Mode

[Denis Magda]

Hi,

When the client executes a query like cache.get(key1) the request will be 
directed to a server node that holds a value for key1. If key1 is missing in 
the memory on the server side then it will load it from the cache store, puts 
into memory and returns for the client.

—
Denis

> On Jul 11, 2016, at 9:36 AM, daniels  wrote:
> 
> Hi,
> I use  CacheLoaderFactory(read-through) ,and do simple
> cache.get("absentKey");
> after it  CacheLoaderFactory  loades data from DB.
> Now I want know data loaded on client and seted it on server node   or  load
> occured on server node?
> 
> daniels,
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Cache-Partitioned-Mode-tp6172p6202.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Cache Partitioned Mode

[daniels]

Hi,
I use  CacheLoaderFactory(read-through) ,and do simple
cache.get("absentKey");
after it  CacheLoaderFactory  loades data from DB.
Now I want know data loaded on client and seted it on server node   or  load
occured on server node?

daniels,



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Cache-Partitioned-Mode-tp6172p6202.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Cache Partitioned Mode

[Denis Magda]

If you execute IgniteCache.loadCache(…) from the client node then in fact a 
compute task will be sent to all the server nodes that will keep cache data and 
loading will be performed directly from the server nodes.

Do you use IgniteCache.loadCache(…) or other cache loading approach from this 
page [1]?

[1] https://apacheignite.readme.io/docs/data-loading 


—
Denis
 
> On Jul 10, 2016, at 11:35 AM, daniels  wrote:
> 
> Thank you dear vkulichenko,it's so clear.
> But I have one more question 
> I have 2 server nodes(remote)and 1 client.and do get from existing cache(on
> clie t node),which key is absent in that  cache,(i have cacheloaderfactory
> for that cache)thus must work cacheLoader.
> 
> Now question on which  node will the load work?
> Whether the load will work on client node and then set in server nodes?
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/Cache-Partitioned-Mode-tp6172p6197.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: LoadCache

[Denis Magda]

> If you know cache size, good practice define it in cache configuration
>

It makes sense to switch to OFFHEAP_TIRED mode only if you are sure that all 
the caches will occupy more ~ 10 GB of Java heap. If the total size of all the 
caches is relatively small on a single cluster node then it’s ok to keep using 
ONHEAP_TIRED.

>https://apacheignite.readme.io/docs/jvm-and-system-tuning#section-detailed-garbage-collection-stats
 


—
Denis

> On Jul 10, 2016, at 1:23 AM, AndreyVel  wrote:
> 
> Hello Level D,
> 
> I suppose you have gc performance problems. 
> Read JVM and System Tuning [1], article which provides general
> recommendations on how to tune the GC and JVM. 
> 
> If you know cache size, good practice define it in cache configuration
>
>
> 
> Also better preload cahes eqt + eqt_temp before use continuous query.
> For controlling cache loading you can use CacheStoreAdapter [2]
> 
> 
> [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning
> [2] http://apacheignite.gridgain.org/v1.2/docs/data-loading
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-users.70518.x6.nabble.com/LoadCache-tp6140p6196.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.