Possible dead lock when number of jobs exceeds thread pool

[Raymond Wilson]

I am testing an aspect of a POC I have written in C# to determine how well
it performs when processing multiple tasks at one time. I’m using Ignite
2.2.



Briefly, a client node sends a request to a server node for a list of
items. Each of the items requested is send back to a listener in the client
(using Ignite messaging) and then processed (rendered onto a bitmap). Once
all parts of the request have arrived and been rendered, a wait state is
triggered allowing the request to be completed. The wait state will
terminate after 2 minutes which short cuts the request in the client node.



This works well from a functional perspective, but I ran into problems with
multithreading.



To test multi-threading, I essentially did this:



int nThreads = 4;

Parallel.For(0, nThreads, x => PerformRender());



This works well for values of nThreads of up to 7. However, once nThreads
is set more than 7 then the requests start stalling, and no progress is
made the first eight requests until the wait states start timing out at two
minutes.



Looking in the log is it common to see entries like this during the time
the requests are stalled:



 WARN  2017-10-04 13:41:57,051 252758ms
IgniteKernal%Raptor  ?  - Possible
thread pool starvation detected (no task completed in last 3ms, is
public thread pool size large enough?)

Here we see an internal monitor is warning regarding a possible deadlock or
race condition meaning progress is not being made on requests.



INFO  2017-10-04 15:47:21,182 263409ms
IgniteKernal%Raptor  ?  -

Metrics for local node (to disable set 'metricsLogFrequency' to 0)

^-- Node [id=c966b7cc, name=Raptor, uptime=00:04:10:105]

^-- H/N/C [hosts=1, nodes=7, CPUs=8]

^-- CPU [cur=0.07%, avg=1.76%, GC=0%]

^-- PageMemory [pages=0]

^-- Heap [used=103MB, free=88.58%, comm=497MB]

^-- Non heap [used=38MB, free=-1%, comm=39MB]

^-- Public thread pool [active=8, idle=0, qSize=274]

^-- System thread pool [active=0, idle=0, qSize=0]

^-- Outbound messages queue [size=0]

Here we see the pubic thread pool is set to 8 by default. This suggests
when the pool becomes saturated with requests something is stalling the
requests.



I tried increasing the number of threads in the public pool in the grid
configuration and this resulted in no stalls in requests which suggests
Ignite does not like fully committed thread pools.



Is this a known issue with Ignite?



Thanks,

Raymond.

Atomicity Mode and WAL

[John Wilson]

Hi,

What is the purpose and difference in the use of the WAL in atomic mode vs.
in transaction mode?

Thanks

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[JP]

Does this same approach will work if I have used ignite client as SQL api
instead of c# api ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[Alexey Kukushkin]

Hi JP,

Please let me first explain some basics: Ignite is implemented in Java and
presently all other APIs including .NET are "wrappers" of the Ignite JVM.
When you start your .NET node, it automatically creates an Ignite Java node
and talks to it using some thin JNI layer for .NET/Java interoperability.

Thus, any .NET Ignite node is also a Java Ignite node but, obviously, pure
Java nodes are NOT .NET nodes.

I think that explains why you see an error when you try to connect a pure
Java node to a SQL server using Microsoft .NET Entity Framework cache
store. The cache store information is transferred to the Java node fine
when you create a dynamic cache, but you get an error when Java node
actually tries to use (do "put") the .NET Entity Framework that is not
available in the Java node running on Ubuntu.

Since any node is always a Java node you can develop your
CustomerCacheStore in Java, add the JAR file to Ignite classpath and such a
cache store will work on both .NET and Java nodes. Here is an example of
how you developer a Java cache store:
https://apacheignite.readme.io/docs/3rd-party-store#section-cachestore-example.
I think you can easily modify it for your data model.

Let me know if you still have questions.

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[JP]

Hi, Alexey Kukushkin

I am confused here, can you describe me about Java cache store. 

and How to utilize that in c# ignite client...?







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cassandra failing to ReadThrough using Cache.get(key) without preloading

[Kenan Dalley]

Ok, that worked for me, too.  Such a small change that I kept overlooking. 
Thanks a lot!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[Pavel Tupitsyn]

Make sure all server nodes are started from .NET or with Apache.Ignite.exe

On Tue, Oct 3, 2017 at 5:38 PM, JP  wrote:

> ok, I got it.
>
> Can guide me how I can achieve this kind of persistence.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[JP]

ok, I got it.

Can guide me how I can achieve this kind of persistence.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite long term support (LTS) version policy?

[Dop Sun]

Hi,

I’m currently developing an application for my employer, and starting beginning 
of 2017, Ignite started at 1.8, and released 1.9 (Feb), 2.0 (Apr), 2.1 (Jul) 
and recently 2.2 (Sep), or about 2 - 3 months a version. And I can see good 
features added to every releases, together with bug fixes.

For us, we upgraded from 1.8 and then 2.0, and due to several bugs fixed in 2.1 
and 2.2, we have upgraded to 2.2. And the bug fixes, for example IGNITE-6181, 
would likely pushing us to upgrade to 2.3 when its ready before our first 
production release.

My question is:

- will there be a kind of long term support (LTS) version? By LTS, I mean there 
is a version will be considered stable, and bug fixes of future several 
releases would likely back ported for a certain period of time.

- if not today, any chance this can be considered in future?

Please kindly suggest if this should be sending to d...@ignite.apache.org 
instead.

Thanks,
Regards,
Dop

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[Pavel Tupitsyn]

Hi,

> c# as Ignite client in windows machine.
> The standalone servers in Ubuntu.
> .NET cache store

Cache store defined in C# code requires .NET on every cache node.
Your servers are on Ubuntu and are Java-only, so .NET code can't run there.

Some functionality is limited in cross-platform clusters, see
https://apacheignite-net.readme.io/docs/platform-interoperability#section-mixed-platform-clusters

Thanks,
Pavel

On Tue, Oct 3, 2017 at 4:43 PM, siva  wrote:

> Hi Alexey,
>
> Here is the code snippet,
>
>
>   client config
>   ===>
>   var cfg = new IgniteConfiguration
> {
> BinaryConfiguration = new BinaryConfiguration
> {
> NameMapper = new BinaryBasicNameMapper {
> IsSimpleName = true },
>
> }
> };
>
>
>
>
>   var customersCache = ignite.GetOrCreateCache CustomerInfo>(new
> CacheConfiguration
> {
> Name = "customer12" + tenantName,
> KeepBinaryInStore = false,
> CacheStoreFactory = new
> CustomerCacheStoreFactory(connectionString),
> ReadThrough = true,
> WriteThrough = true,
> });
>
>
>  customersCache[new Random().Next().ToString()] = new CustomerInfo
> { Name =
> tenantName, Id = Id, City = city };
>
>  here is the exception when putting into cache its throwing
> exception.
>
>
>  server config:
>  ==>
>  BinaryConfiguration bcnfg = new BinaryConfiguration();
>
> bcnfg.setNameMapper(new BinaryBasicNameMapper());
>
> IgniteConfiguration cfg  =  new
> IgniteConfiguration().setBinaryConfiguration(bcnfg);
> Ignite ignite=Ignition.start(cfg);
>
>
>
>
>  Factory implementation
>  >
>  In C#:(client)
>  --
>   [Serializable]
> class CustomerCacheStoreFactory : IFactory
> {
> string connectionString;
> public CustomerCacheStoreFactory(string connectionString)
> {
> this.connectionString = connectionString;
> }
> public ICacheStore CreateInstance()
> {
> return new CustomerCacheStore(this.connectionString);
>
> }
> }
>
> In java(server)
> ---
>  public class CustomerCacheStoreFactory implements
> Factory>{
>
> private String connectionString;
> public CustomerCacheStoreFactory(String connectionString) {
> this.connectionString=connectionString;
> }
> public CacheStore create() {
>
> return  new  CustomerCacheStore(this.connectionString);
> }
>
>
>
> }
>
> In Java and c#  Cache store where we need to write the logic to persist
> into
> sql :
> 
> ===
> public class CustomerCacheStore implements CacheStore CustomerInfo>{
>  load()
>  write()
>  ..
>  ..
>
> }
>
>
>  Model class in c# to persist
>  
>
>   class CustomerInfo
> {
> public CustomerInfo()
> {
>
> }
> public CustomerInfo(string id,string name,string city)
> {
> this.Id = id;
>
> this.Name = name;
> this.City = city;
>
> }
> [Key]
> [DatabaseGenerated(DatabaseGeneratedOption.None)]
> public string Id { get; set; }
> public string Name { get; set; }
> public string City { get; set; }
> }
>
> Model class in java
> ==
>
> public class CustomerInfo {
> final String id;
> final String name;
> final String city;
>
> public CustomerInfo(String id, String name, String city) {
> this.id= id;
> this.name = name;
> this.city = city;
> }
>
> }
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[siva]

Hi Alexey,

Here is the code snippet,


  client config
  ===>
  var cfg = new IgniteConfiguration
{
BinaryConfiguration = new BinaryConfiguration
{
NameMapper = new BinaryBasicNameMapper {
IsSimpleName = true },

}
};




  var customersCache = ignite.GetOrCreateCache(new
CacheConfiguration
{
Name = "customer12" + tenantName,
KeepBinaryInStore = false,
CacheStoreFactory = new
CustomerCacheStoreFactory(connectionString),
ReadThrough = true,
WriteThrough = true,
});


 customersCache[new Random().Next().ToString()] = new CustomerInfo { 
Name =
tenantName, Id = Id, City = city };
 
 here is the exception when putting into cache its throwing exception.
 
 
 server config:
 ==>
 BinaryConfiguration bcnfg = new BinaryConfiguration();

bcnfg.setNameMapper(new BinaryBasicNameMapper());

IgniteConfiguration cfg  =  new
IgniteConfiguration().setBinaryConfiguration(bcnfg);
Ignite ignite=Ignition.start(cfg);

 
 

 Factory implementation
 >
 In C#:(client)
 --
  [Serializable]
class CustomerCacheStoreFactory : IFactory
{
string connectionString;
public CustomerCacheStoreFactory(string connectionString)
{
this.connectionString = connectionString;
}
public ICacheStore CreateInstance()
{
return new CustomerCacheStore(this.connectionString);

}
}

In java(server)
---
 public class CustomerCacheStoreFactory implements
Factory>{

private String connectionString;
public CustomerCacheStoreFactory(String connectionString) {
this.connectionString=connectionString;
}
public CacheStore create() {

return  new  CustomerCacheStore(this.connectionString);
}



}

In Java and c#  Cache store where we need to write the logic to persist into
sql :
===
public class CustomerCacheStore implements CacheStore{
 load()
 write()
 ..
 ..
 
}
 
 
 Model class in c# to persist 
 
 
  class CustomerInfo
{
public CustomerInfo()
{

}
public CustomerInfo(string id,string name,string city)
{
this.Id = id;

this.Name = name;
this.City = city;

}
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.None)]
public string Id { get; set; }
public string Name { get; set; }
public string City { get; set; }
}

Model class in java 
==

public class CustomerInfo {
final String id;
final String name;
final String city;

public CustomerInfo(String id, String name, String city) {
this.id= id;
this.name = name;
this.city = city;
}

}





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[siva]

1. What exactly your issue is? Can you copy all the error output?

   --> we have a requirement to persist the data in SQL server  for that
written client(c#) and server(java) program.in client side implements cache
store and factory interfaces and as well as server side(java) also.

  so while inserting the data to the cache getting exception like  below

"class
org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException:
Failed to update keys (retry update if possible).: [296723062]"}


javax.cache.integration.CacheWriterException


javax.cache.integration.CacheWriterException: class
org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException:
Failed to update keys (retry update if possible).: [296723062]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1293)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2630)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1509)
at
org.apache.ignite.internal.processors.platform.cache.PlatformCache.processInStreamOutLong(PlatformCache.java:406)
at
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inStreamOutLong(PlatformTargetProxyImpl.java:65)
Caused by: class
org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException:
Failed to update keys (retry update if possible).: [296723062]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.onPrimaryError(GridNearAtomicAbstractUpdateFuture.java:397)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.onPrimaryResponse(GridNearAtomicSingleUpdateFuture.java:251)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3081)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$500(GridDhtAtomicCache.java:129)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:284)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:279)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
at java.lang.Thread.run(Thread.java:748)
Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to
update keys.
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKey(UpdateErrors.java:108)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKey(GridNearAtomicUpdateResponse.java:329)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2576)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1847)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1686)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3063)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129)
at

Re: Query performance affected by record size?

[afedotov]

Hi,

Could you please share the queries you run against the cache?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Job Listeners

[Alexey Kukushkin]

Hi Chandrika,

Is it possible for you to share your Ignite task implementation? Or are you
just running the above example I created? It looks you have some deadlock
and it is hard to guess without having the code.

Re: Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[Alexey Kukushkin]

Hi,

Please can you give more details:

   - What exactly your issue is? Can you copy all the error output?
   - Are you using Microsoft SQL server for persistence?
   - What API does your C# client use to work with Ignite with the
   persistence enabled? Is it C# API or SQL or something else?
   - If possible, can you also specify how you configured persistence.

Thanks.

Re: Job Listeners

[chandrika]

Hello Alexey,

Thanks for the valuable information, i hare tried executing a list of
dependent tasks using a DAG using session.setAttribute("COMPLETE",true), and
it is working fine as long as there are three nodes, cause there are 3 or
less parallel tasks to execute.

But when i run the same code of dependent tasks on a single node, then it
stops its execution after execution of few tasks in some time . There are no
exception as such. could u please let me know where i could have gone wrong.
Note : should i have to use waitForAttribute for the same if so could u
please guide me the usage of wiatForAttribute for completion of three
parallel tasks at every step.


thanks and regards,
chandrika



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Persistence store(MSSQL) using cross-platform c#(Ignite Client windows) and java(Ignite Server linux version)

[JP]

Hi, 
   I am trying to use cross-platform for data persistence(SQL server) using
c# and java. 

c# as Ignite client in windows machine. 
The standalone servers in Ubuntu. 

So, can anyone tell me how to implement this..? 

I am also working on this. But I am facing the issue on updating the cache
using the cache store factory configuration.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Trouble to connect to ignite cluster on kubernetes

[Anton Mushin]

Thanks for your replies!

This suggestion doesn't suit me, if I correctly understand what do 
hostNetwork=true, then I should have one ignite pod in one kubernetes node.

In one real kubernetes node I have a few ignite nodes in different namespaces, 
and I use ignite like as kubernetes service inside namespaces.

From: Denis Magda [mailto:dma...@apache.org]
Sent: Monday, October 02, 2017 10:18 PM
To: user@ignite.apache.org
Cc: Anton Mushin 
Subject: Re: Trouble to connect to ignite cluster on kubernetes

Anton,

In general, the application has to be in the same network where the cluster 
nodes are. Otherwise you might face various network related issues due to the 
specificity on how Kubernetes manages the network.

There is a ticket to cover your use case. A new IP finder should be created for 
the nodes connecting outside.
https://issues.apache.org/jira/browse/IGNITE-4161

Talking to kubernetes experts I learnt that this might not be needed if you do 
the following:
· Set hostNetwork=true in Ignite pod's YAML configuration so that the 
nodes that will be connecting from outside can establish TCP/IP connections 
with containerized Ignite nodes (pods).
· Use the same TcpDiscoveryKubernetesIpFinder for the nodes that will 
be outside of Kubernetes environment.

—
Denis

On Oct 2, 2017, at 5:45 AM, Nikolai Tikhonov 
> wrote:

Hi Anton!

For work Apache Ignite cluster requires that all nodes are able to communicate 
to each other directly. If I understood correctly kubernete starts up internal 
network interfaces and this environment looks similar that Apache Ignite works 
behind proxy. In this case you need to configure BasicAddressResolver which 
will map internal IP adresses on external host.

On Mon, Oct 2, 2017 at 12:13 PM, Anton Mushin 
> wrote:
Hi everyone!

Could you say what way for correct connect to ignite cluster on kubernetes?
I'm using ignite version 2.2.0 and try connect to cluster from my local machine 
with next configuration

private IgniteConfiguration getConfig(){
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder(false);
ipFinder.setAddresses(Arrays.asList("kuber_external_host_addr", " 
kuber_external_host_addr:forwarded_port"));

TcpCommunicationSpi commSpi=new TcpCommunicationSpi();
commSpi.setSharedMemoryPort(-1);

TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setIpFinder(ipFinder);
tcpDiscoverySpi.setNetworkTimeout(TcpDiscoverySpi.DFLT_NETWORK_TIMEOUT 
*3);

return new IgniteConfiguration()
.setDiscoverySpi(tcpDiscoverySpi)
   .setIgniteInstanceName(UUID.randomUUID().toString())
   .setCommunicationSpi(commSpi);
}

I use default configuration for nods on cluster.
Then I’m trying connect to cluster I get  error on my local machine:

[SEVERE][main][TcpDiscoverySpi] Exception on direct send: Connection refused: 
connect
java.net.ConnectException: Connection refused: connect
at 
java.net.DualStackPlainSocketImpl.waitForConnect(Native 
Method)
at 
java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1376)
at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1339)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1159)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1006)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:851)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:358)
at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1834)
at 
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:842)
at 
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1786)
   

Query performance affected by record size?

[colinc]

I have a use case where I would like to store a record that has some
QuerySqlField attributes but also a non-queriable list of child objects.
Something like this:

public class Portfolio {
@QuerySqlField(index = true)
private int portCode;
   
@QuerySqlField
private String fullName;

// Other query fields

private List positions;
}

The idea is that the cache can be queried for summary data using a
SqlFieldsQuery, or the whole object can be retrieved and processed by a
task.

I anticipated that performing a SqlFieldsQuery on indexed fields would
always be fast - as there is no need to deserialise the positions. However,
tests indicate that when the cache size exceeds 1M positions (say 1000
portfolios, 1000 positions), the queries become very slow - at least an
order of magnitude slower than would be the case with small numbers of
positions.

Is the above a valid usecase? Are we getting something wrong, or is it
necessary to split the Portfolio into two separate caches?

Thanks in advance,
Colin.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Question about number of total onheap and offheap cache entries.

[Ray]

Looks like this ticket is the answer I'm looking for.
https://issues.apache.org/jira/browse/IGNITE-6131




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Question about number of total onheap and offheap cache entries.

[Ray]

I mean in the sixth column "Size" of last chart of "cache -a" command, the
"Heap" entry count is not 0.
As the above picture shows.
 

But in the cache summary chart, the on-heap entry count is 0.
 

It's really confusing that these two statistics does not match.
Please let me know which of these is correct.

Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache size in Memory

[Gaurav Bajaj]

Adding to the same question, can I see the off heap memory usage in Visor
or Web console?
I see on-heap memory size in Web console.


On Tue, Oct 3, 2017 at 8:28 AM, Alexey Kuznetsov 
wrote:

> Hi,
>
> Take a look at documentation: https://apacheignite.readme.io/docs/
> memory-metrics
>
> You can multiply TotalAllocatedPages on page size to get an estimation.
>
> On Tue, Oct 3, 2017 at 3:35 AM, pradeepchanumolu 
> wrote:
>
>> Hi ,
>>
>> Can someone let me know how to get the stats of the cache like how large
>> (like how many MB/GB) is the cache in memory (both on and off heap)?
>>
>> Currently, I can only get number of entries in the cache using the
>> cache.size(CachePeekMode.ONHEAP), cache.size(CachePeekMode.OFFHEAP).
>>
>> cache.metrics() returns the Cache Metrics but doesn't have any methods to
>> return the total size in gb/mb of the cache in memory.
>>
>>
>>
>>
>>
>> --
>> Sent from:
>> http://apache-ignite-users.70518.x6.nabble.com/
>>
>> --
>> Alexey Kuznetsov
>> 
>>
>
>

Re: Question about number of total onheap and offheap cache entries.

[Ray]

But my on-heap entry count is still not 0 here.
As I didn't set onHeapEnabled=true, the value should be false in default,
right?
So is this a visor bug that the on-heap entry count is not 0?




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache size in Memory

[Alexey Kuznetsov]

Hi,

Take a look at documentation:
https://apacheignite.readme.io/docs/memory-metrics

You can multiply TotalAllocatedPages on page size to get an estimation.

On Tue, Oct 3, 2017 at 3:35 AM, pradeepchanumolu 
wrote:

> Hi ,
>
> Can someone let me know how to get the stats of the cache like how large
> (like how many MB/GB) is the cache in memory (both on and off heap)?
>
> Currently, I can only get number of entries in the cache using the
> cache.size(CachePeekMode.ONHEAP), cache.size(CachePeekMode.OFFHEAP).
>
> cache.metrics() returns the Cache Metrics but doesn't have any methods to
> return the total size in gb/mb of the cache in memory.
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
> --
> Alexey Kuznetsov
> 
>

Re: Question about number of total onheap and offheap cache entries.

[Dmitriy Setrakyan]

On Tue, Oct 3, 2017 at 4:19 AM, Ray  wrote:

> Hi Alexey
>
> My cache configuration is as follows.
> cacheConfig.setName("DailyAggData")
> cacheConfig.setIndexedTypes(classOf[A], classOf[B])
> cacheConfig.setSqlSchema("PUBLIC")
> aggredCacheConfig.setBackups(2)
> cacheConfig.setQueryParallelism(8)
>
> I didn't explicitly set "onHeapEnabled=true".
> So what will happen if I perform get & sql operations with
> onHeapEnabled=false?
> Will off-heap entries be brought on-heap?
>

Yes, but only to return to user. Ignite will not be caching on-heap entry,
and therefore the count should be 0.


--
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>