SQL query for nested collections and map in POJO

[Taner Ilyazov]

Hello everyone, I'm trying to find a solution for the current case:

I have generated classes, which can't be changed.

public static class CacheName {
private Long id;
private NestedClass nestedClass;
}

public static class NestedClass {
private List types;
private Map dynamicFields;
}

Write Behind slowing down performance greatly

[gigabot]

 I observed that turning on writeThrough with writeBehind slows down
performance(i.e. time data reaches listeners) greatly. I tried setting the
flush frequency to 0 and the flush size to 1 billion, and so the flushes are
not even happening yet, and still the performance is slow. I am wondering
where the slowdown is occurring? Where is the flush buffer located
(heap/off-heap/disk/etc.) and it is configurable?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Minimal config for client nodes

[ict.management.trexon]

Thank you!
My idea was to automate the configuration for the client nodes, so that the
configuration was a mirror of that of the servers, minus the unnecessary
stuff.
I'll work it out a little bit then, thanks.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Sorting result of ScanQuery

[Taner Ilyazov]

That's my concern. Because the requirements I have is persisting an Object
with a complex nested structure, which can't be changed. I mean the class
files can not be changed. Creating Data transfer objects and mapping
between them is fine. But what we want to achieve is a really high write
rate and the ability later to read data with filtering and sorting on the
values of the hash map. I'm not sure how to achieve that without too much
overhead.

On Thu, 3 Jun 2021 at 15:29, Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> You can store collections in Ignite, the challenge is they’re effectively
> invisible to SQL. In general it’s easiest to work with data in relational
> structure. Ignite isn’t a document database.
>
> On 3 Jun 2021, at 12:52, Taner Ilyazov  wrote:
>
> Okay, but since the nested object structure that I have contains a
> Map, for which the idea is to have dynamic values, I'm not
> sure how it will be handled. Do I need to create a separate table to do the
> mapping of said Map<>? Couldn't find an example mapping a query entity
> entry to a parameterized value.
>
> On Wed, 26 May 2021 at 17:01, Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
>
>> A scan isn’t ordered. As you suspect, the way to order queries in Ignite
>> is to use SQL.
>>
>> You don’t need to use annotations to define your SQL fields, indexes,
>> etc. A slightly more verbose way is to use Query Entities (indexes
>> 
>> ).
>>
>> On 26 May 2021, at 14:24, Taner Ilyazov  wrote:
>>
>> Hello everyone,
>>
>> I'm new to the community and fairly new to Apache Ignite. I have a
>> question for which I couldn't find a confirmation if it's possible or not.
>>
>> I have a use case where I need to persist a certain POJO to an ignite
>> cluster. The POJO can not be changed, so adding @SqlQueryField to it's
>> fields is not possible. Creating a data transfer object is an option, but I
>> think adding mapping from/to the actual POJO will result in too much
>> overhead, since performance requirements are really high.
>>
>> For now I'm using ScanQuery, but I could not find a way to sort the
>> result based on a field value. So my main question is if it's possible and
>> if not, what other options are there because the amount of data in question
>> is too much for sorting on client side.
>>
>> If I take the SQL approach and introduce the mapping overhead between the
>> DTO and POJO can I achieve server-side sorting on multiple nodes, keeping
>> in mind that we'll have 1 table with a huge amount of data for writing and
>> reading.
>> Co-location if I understand correctly is ensuring all related data is on
>> the same nodes, but in our case we have a single POJO which I would like
>> it's data to be separated on different nodes for performance.
>>
>>
>>
>>
>>
>>
>
>

Re: Minimal config for client nodes

[Aleksandr Shapkin]

Hello!

I don't think there is a complete list or something similar somewhere thus it’s 
better to check this manually. 

Regarding the peerClassLoading and similar configuration you might check the 
source code
https://github.com/apache/ignite/blob/b1529fbd7d42e3f3a5500a217a2b8b21acbb4189/modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java
 

#checkAttributes method.

As for other settings, I can think of security/SSL configuration for both 
client and server, also it’s better to have network or discovery timeouts to be 
the same.



> On 3 Jun 2021, at 13:10, ict.management.trexon 
>  wrote:
> 
> Hello everybody.
> I'm trying to figure out the minimum configuration to pass to the client
> nodes.
> I thought, client nodes don't need all configuration of data regions,
> configuration of memory, persistence, consistendId, etc.
> 
> I thought, once I made a minimum configuration, where at most I indicated
> the way to connect to the cluster (IP, multicast, etc.), this would be
> enough for the client nodes.
> 
> So I tested the client nodes with a minimal configuration and the server
> nodes with the full one, and it worked, but i'm going to activate
> peerClassLoadingEnabled on the server side and here even the clients who
> want to connect to that cluster go into error because they also want this
> flag set to true.
> 
> So, my question is: A node set as a client, what minimum configuration must
> it have equal to that of the server nodes?
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Sorting result of ScanQuery

[Stephen Darlington]

You can store collections in Ignite, the challenge is they’re effectively 
invisible to SQL. In general it’s easiest to work with data in relational 
structure. Ignite isn’t a document database. 

> On 3 Jun 2021, at 12:52, Taner Ilyazov  wrote:
> 
> Okay, but since the nested object structure that I have contains a 
> Map, for which the idea is to have dynamic values, I'm not 
> sure how it will be handled. Do I need to create a separate table to do the 
> mapping of said Map<>? Couldn't find an example mapping a query entity entry 
> to a parameterized value.
> 
> On Wed, 26 May 2021 at 17:01, Stephen Darlington 
> mailto:stephen.darling...@gridgain.com>> 
> wrote:
> A scan isn’t ordered. As you suspect, the way to order queries in Ignite is 
> to use SQL.
> 
> You don’t need to use annotations to define your SQL fields, indexes, etc. A 
> slightly more verbose way is to use Query Entities (indexes 
> ).
> 
>> On 26 May 2021, at 14:24, Taner Ilyazov > > wrote:
>> 
>> Hello everyone,
>> 
>> I'm new to the community and fairly new to Apache Ignite. I have a question 
>> for which I couldn't find a confirmation if it's possible or not.
>> 
>> I have a use case where I need to persist a certain POJO to an ignite 
>> cluster. The POJO can not be changed, so adding @SqlQueryField to it's 
>> fields is not possible. Creating a data transfer object is an option, but I 
>> think adding mapping from/to the actual POJO will result in too much 
>> overhead, since performance requirements are really high. 
>> 
>> For now I'm using ScanQuery, but I could not find a way to sort the result 
>> based on a field value. So my main question is if it's possible and if not, 
>> what other options are there because the amount of data in question is too 
>> much for sorting on client side.
>> 
>> If I take the SQL approach and introduce the mapping overhead between the 
>> DTO and POJO can I achieve server-side sorting on multiple nodes, keeping in 
>> mind that we'll have 1 table with a huge amount of data for writing and 
>> reading.
>> Co-location if I understand correctly is ensuring all related data is on the 
>> same nodes, but in our case we have a single POJO which I would like it's 
>> data to be separated on different nodes for performance.
>> 
>> 
>> 
> 
> 

ignite server restarted after Critical system error detected.

[xmw45688]

Hi Ignitians,

I fail to understand what causes and need your help - 
1)  When k8s sees “Critical system error”, it will restart ignite-admin
server. Restarting is fine because of the critical system error.  But what
are the causes of the critical system error?
2)  Critical system error may be corresponding to JVM held.  Still we don’t
know the reason why JVM got held.
3)  The cluster lost one ignite client node, probably due to OOME
4)  Why/how was ignite server node triggered to reload the data from
Cassandra (all data in C* tables are cached once the ignite server starts.
All SQL DML interacts with Ignite Cache which interact with Cassandra for
insert/update/delete)  If Ignite needs to rebalance the data among the
server nodes, why can't rebalance the data from one node to another?  if
even rebalancing data, why submitting invalid queries?  We are using
apache-ignite-2.8.0.20190215. 

Exceptions - 

[2021-06-02
17:09:04,005][ERROR][sys-#103562%ignite-procurant-admin-cluster%][root]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
transaction has produced runtime exception]]
... 1 more
at
com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:104)
class
org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
Committing a transaction has produced runtime exception
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:800)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:847)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:795)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.salvageTx(GridDistributedTxRemoteAdapter.java:898)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.salvageTx(IgniteTxManager.java:398)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.access$3100(IgniteTxManager.java:134)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject.onTimeout0(IgniteTxManager.java:2551)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject.access$3300(IgniteTxManager.java:2505)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$1.run(IgniteTxManager.java:2624)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6898)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot
execute this query as it might involve data filtering and thus may have
unpredictable performance. If you want to execute this query despite the
performance unpredictability, use ALLOW FILTERING
at java.lang.Thread.run(Thread.java:745)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:293)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:338)
Caused by: class org.apache.ignite.IgniteCheckedException: class
org.apache.ignite.IgniteException: Failed to execute Cassandra CQL
statement: select "id", "sourceid", "versioning", "colname", "colnewvalue",
"cololdvalue", "createdby", "ts", "op" from "admin"."user_history" where
"id"=?;
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerReload(GridCacheMapEntry.java:984)
... 13 more
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.readThrough(GridCacheMapEntry.java:619)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:701)
... 18 more
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:198)
Caused by: class org.apache.ignite.IgniteException: Failed to execute
Cassandra CQL statement: select "id", "sourceid", "versioning", "colname",
"colnewvalue", "cololdvalue", "createdby", "ts", "op" from
"admin"."user_history" where "id"=?;
Caused by: javax.cache.integration.CacheLoaderException: class
org.apache.ignite.Ignite

Re: Sorting result of ScanQuery

[Taner Ilyazov]

Okay, but since the nested object structure that I have contains a
Map, for which the idea is to have dynamic values, I'm not
sure how it will be handled. Do I need to create a separate table to do the
mapping of said Map<>? Couldn't find an example mapping a query entity
entry to a parameterized value.

On Wed, 26 May 2021 at 17:01, Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> A scan isn’t ordered. As you suspect, the way to order queries in Ignite
> is to use SQL.
>
> You don’t need to use annotations to define your SQL fields, indexes, etc.
> A slightly more verbose way is to use Query Entities (indexes
> 
> ).
>
> On 26 May 2021, at 14:24, Taner Ilyazov  wrote:
>
> Hello everyone,
>
> I'm new to the community and fairly new to Apache Ignite. I have a
> question for which I couldn't find a confirmation if it's possible or not.
>
> I have a use case where I need to persist a certain POJO to an ignite
> cluster. The POJO can not be changed, so adding @SqlQueryField to it's
> fields is not possible. Creating a data transfer object is an option, but I
> think adding mapping from/to the actual POJO will result in too much
> overhead, since performance requirements are really high.
>
> For now I'm using ScanQuery, but I could not find a way to sort the result
> based on a field value. So my main question is if it's possible and if not,
> what other options are there because the amount of data in question is too
> much for sorting on client side.
>
> If I take the SQL approach and introduce the mapping overhead between the
> DTO and POJO can I achieve server-side sorting on multiple nodes, keeping
> in mind that we'll have 1 table with a huge amount of data for writing and
> reading.
> Co-location if I understand correctly is ensuring all related data is on
> the same nodes, but in our case we have a single POJO which I would like
> it's data to be separated on different nodes for performance.
>
>
>
>
>
>

Re: Ignite throws "Failed to resolve class name" Exception

[Aleksandr Shapkin]

Hello!

It seems that you are trying to deploy DTO using peer class loading, 
unfortunately, that's not possible. Peer class loading is mostly about task 
deployment, see 
https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading

To resolve this you need to have your CacheState deployed on all nodes before 
deserialization happens or to work with cache in raw binary format.
https://ignite.apache.org/docs/latest/key-value-api/binary-objects


> On 26 May 2021, at 18:27, tsipporah22  wrote:
> 
> Hi Ilya,
> 
> Sorry I get back to you late. I have key-value classes on the server node.
> The peer class loading is enabled. I'm not getting this error consistently
> so it's hard to reproduce. Below is the code snippet that throws the error:
> 
> First I got CacheState object with tableName as the key:
> 
> public class CacheState implements Serializable {
>private static final long serialVersionUID = 1L;
> 
>@QuerySqlField(index=true)
>private String tableName;
> 
>@QuerySqlField
>private long updateVersion;
> 
> 
>public CacheState() {
>}
> 
>public CacheState(String tableName) {
>this.tableName = tableName;
>}
> 
>public String getTableName() {
>return tableName;
>}
> 
>public void setTableName(String tableName) {
>this.tableName = tableName;
>}
> 
>public long getUpdateVersion() {
>return updateVersion;
>}
> 
>public void setUpdateVersion(long updateVersion) {
>this.updateVersion = updateVersion;
>}
> 
> 
> And the error is thrown from this class:
> 
> public class WindowOptimizer {
>private final Ignite ignite;
> 
>private IgniteCache cacheStates;
> 
>public void init() {
>if (cacheStates == null) {
>cacheStates =
> ignite.cache(CacheState.class.getSimpleName().toLowerCase());
>}
>}
> }
> 
>private IgniteFuture>
> updateCacheState(IgniteCompute compute, BaselinePeriod period,
>OffsetDateTime now, WindowOptimizerCfg cfg) {
> 
>final IgniteFuture> future =
> compute.broadcastAsync(updater);
>future.listen(f -> {
>
> 
>CacheState cacheState = cacheStates.get(tableName);<--- this
> line throws Exception
> 
>
>})
>}
> 
> 
> Thanks,
> tsipporah
> 
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Minimal config for client nodes

[ict.management.trexon]

Hello everybody.
I'm trying to figure out the minimum configuration to pass to the client
nodes.
I thought, client nodes don't need all configuration of data regions,
configuration of memory, persistence, consistendId, etc.

I thought, once I made a minimum configuration, where at most I indicated
the way to connect to the cluster (IP, multicast, etc.), this would be
enough for the client nodes.

So I tested the client nodes with a minimal configuration and the server
nodes with the full one, and it worked, but i'm going to activate
peerClassLoadingEnabled on the server side and here even the clients who
want to connect to that cluster go into error because they also want this
flag set to true.

So, my question is: A node set as a client, what minimum configuration must
it have equal to that of the server nodes?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Clients got disconnected during the endurance testing

[Naveen]

HI All

We are using Ignite 2.8.1 and carrying the endurance test lasting for 7 to
12 hours.
Test ran for almost 6 hours and all of a sudden clients got disconnected and
seeing the below logs
what could be the reason for this behavior, we have enough resources like
RAM, CPU during that time


[2021-06-03 00:08:08,172][WARN ][tcp-disco-msg-worker-[8761dfbe
10.119.10.63:47500]-#2][root] Possible failure suppressed accordingly to a
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
o.a.i.IgniteException: GridWorker [name=grid-timeout-worker,
igniteInstanceName=null, finished=false, heartbeatTs=1622664488170]]]
class org.apache.ignite.IgniteException: GridWorker
[name=grid-timeout-worker, igniteInstanceName=null, finished=false,
heartbeatTs=1622664488170]
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1810)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1805)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2858)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7759)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61)


[2021-06-03 00:08:08,172][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:45160]
[2021-06-03 00:08:08,172][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:35382]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:60720]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:54156]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:55260]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:32804]
[2021-06-03 00:08:08,174][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:54822]
[2021-06-03 00:08:08,175][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:60692]
[2021-06-03 00:08:08,175][WARN
][grid-timeout-worker-#23][ClientListenerNioListener] Unable to perform
handshake within timeout [timeout=24, remoteAddr=/10.129.4.13:45316]


Thread=[name=auth-#47, id=92], state=WAITING
Locked pages = []
Locked pages log: name=auth-#47 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=checkpoint-runner-#65, id=112], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#65 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=checkpoint-runner-#66, id=113], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#66 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=checkpoint-runner-#67, id=114], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#67 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=checkpoint-runner-#68, id=115], state=WAITING
Locked pages = []
Thread=[name=checkpoint-runner-#68, id=115], state=WAITING
Locked pages = []
Locked pages log: name=checkpoint-runner-#68 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=client-connector-#72, id=123], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#72 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=client-connector-#73, id=124], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#73 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=client-connector-#74, id=125], state=BLOCKED
Locked pages = []
Locked pages log: name=client-connector-#74 time=(1622664488172, 2021-06-03
00:08:08.172)


Thread=[name=client-connector-#75, id=126],