Re: [2.11.0]: 'B+Tree is corrupted' exception in GridCacheTtlManager.expire() and PartitionsEvictManager$PartitionEvictionTask.run() on node start
probably this is the case: https://issues.apache.org/jira/browse/IGNITE-15990
>
>Hello Denis,
>Yes, as I said in the original message we do use the expiration on persistent
>caches.
>The corruptedPages_2021-11-08_11-03-21_999.txt and
>corruptedPages_2021-11-09_12-43-12_449.txt files were generated by Ignite on
>crash. They show that two different caches were affected. The first one during
>the expiration and the second (next day) during rebalance eviction. Both
>caches are persistent and use the expiration.
>I also run the diagnostic utility (IgniteWalConverter) the way it is
>recommended in the error message (output attached as diag-* files).
>Is there any usefull information in these diag-* files which can help to
>understand what and how was corruped in particular?
>***
>Generally this was a test run of new 2.11.0 version in test environment. A
>goal was to check if new version works fine with out application and also can
>be safely stopped/started for maintenance. We do that since run into the
>similar problem with 'B+Tree is corrupted' on production during eviction
>rebalance (with 2.8.1). We see two similar issues fixed in 2.11.0:
>https://issues.apache.org/jira/browse/IGNITE-12489 and
>https://issues.apache.org/jira/browse/IGNITE-14093 and consider upgrade if it
>would help. By the way fix of the IGNITE-12489 (
>https://github.com/apache/ignite/pull/8358/commits ) contains a lot changes
>with several attempts. May be it just fixes not all situations?
>Before the deactivation cluster works fine under our usual load about 5 days.
>Load is about 300 requests per second each consists of several reads and
>single write to caches with the expiration turned on. After that we stop /
>start the cluster to emulate the situation we had on production with 2.8.1
>(load from our application was stopped as well before the deactivation
>request).
>***
>Caches configurations. The first one has an affinity key and interceptor
> public CacheConfiguration
>getEdenContactHistoryCacheConfiguration() {
> CacheConfiguration cacheConfiguration =
>new CacheConfiguration<>();
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setAffinity(new RendezvousAffinityFunction(false,
>1024));
> cacheConfiguration.setBackups(1);
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> int expirationDays = appConfig.getContactHistoryEdenExpirationDays();
> cacheConfiguration
> .setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
>Duration(TimeUnit.DAYS, expirationDays)));
> cacheConfiguration.setInterceptor(new ContactHistoryInterceptor());
> return cacheConfiguration;
> }
>public class ContactHistoryKey {
> String sOfferId;
>
> @AffinityKeyMapped
> String subsIdAffinityKey;
>}
> CacheConfiguration getChannelOfferIdCache() {
> CacheConfiguration cacheConfiguration = new
>CacheConfiguration<>();
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setAffinity(new RendezvousAffinityFunction(false,
>1024));
> cacheConfiguration.setBackups(1);
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> int expirationDays = appConfig.getChannelOfferIdCacheExpirationDays();
> cacheConfiguration
> .setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
>Duration(TimeUnit.DAYS, expirationDays)));
> return cacheConfiguration;
> }
>***
>As for the other details. Not sure is it relevant or not. Deactivation was
>relatevily long and log contains a lot of warnings between 2021-11-08
>10:54:44 and 2021-11-08 10:59:33. Also there was a page locks dump at
>10:56:47,567. A lot of locks were logged for cache in the
>Batch_Campaigns_Region (region without the persistence).
>[2021-11-08
>10:56:47,560][WARN][page-lock-tracker-timeout][CacheDiagnosticManager]{}
>Threads hanged: [(sys-#99087-105322, WAITING)]
>[2021-11-08
>10:56:47,567][WARN][page-lock-tracker-timeout][CacheDiagnosticManager]{} Page
>locks dump:
>
>Thread=[name=sys-#99087, id=105322], state=WAITING
>Log overflow, size:512, headIdx=512 [structureId=7,
>pageIdpageId=281474976790149 [pageIdHex=000100013685, partId=0,
>pageIdx=79493, flags=0001]]
>Locked pages =
>[844420635168174[00020dae](r=0|w=1),844420635168175[00020daf](r=0|w=1)]
>Locked pages log: name=sys-#99087 time=(1636358127247, 2021-11-08 07:55:27.247)
>L=1 -> Write lock pageId=844420635168174,
>structureId=batch_campaign_results-p-0##CacheData [pageIdHex=00020dae,
>partId=65535, pageIdx=3502, flags=0002]
>L=2 -> Write lock pageId=844420635168175,
>structureId=batch_campaign_results-p-0##CacheData [pageIdHex=00020daf,
>partId=65535, pageIdx=3503, flags=0002]
>L=3 -> Write lock pageId=281474976716639,
>structureId=Batch_Campaigns_Region##FreeList [pageIdHex=0001175f,
>partId=0, pageIdx=5983, flags=0001]
>L=4 -> Write lock pageId=844420635243234,
RE: Ignite Cluster Config Issue
Hi,
Issue got resolved after adding below line of code.
ignite.cluster().baselineAutoAdjustEnabled(true);
ignite.cluster().baselineAutoAdjustTimeout(1);
Regards,
Gurmehar Singh
From: Gurmehar Kalra
Sent: 25 November 2021 23:08
To: andrei ; user@ignite.apache.org
Subject: RE: Ignite Cluster Config Issue
[CAUTION: This Email is from outside the Organization. Unless you trust the
sender, Don’t click links or open attachments as it may be a Phishing email,
which can steal your Information and compromise your Computer.]
Hi,
I am not deploying any explicit service.
My Question here is if I am running 2 server nodes then both should be able to
read and write to Ignite, where as in current condition only one application
is able to read from ignite .
Ignite jar are part of Spring application which are deployed on 2 different
Machines.
Below is Piece of code which is running on one machine and not on other
Lock _lock =null;
try {
_lock =
CacheManager.getLock("DSLOCK", JourneyCacheEnum.IGNITE_DS_CACHE);
_lock.lock();
ScanQuery scanQuery = new ScanQuery<>(
new
IgniteBiPredicate() {
private static final long serialVersionUID = 1L;
@Override
public boolean apply(String e1, DecisionSplitWait e2) {
Instant timeLapse = e2.getTimeLapse();
return ObjectUtils.isEmpty(timeLapse) ? true
:
timeLapse.isBefore(Instant.now());
}
});
scanQuery.setPageSize(1);
QueryCursor> cursor =
CacheManager.query(scanQuery, JourneyCacheEnum.IGNITE_DS_CACHE);
Iterator> iterator = cursor.iterator();
Entry entry = null;
if
(iterator.hasNext()) {
entry = iterator.next();
}
if
(!ObjectUtils.isEmpty(entry)) {
key
= entry.getKey();
decisionSplitWait = entry.getValue();
decisionSplitWait.setTimeLapse(Instant.now().plusSeconds(WAIT_TIME_ADDED_FOR_SCAN));
CacheManager.getAndReplace(key, decisionSplitWait,
JourneyCacheEnum.IGNITE_DS_CACHE);
}
}finally {
if(!ObjectUtils.isEmpty(_lock)) {
_lock.unlock();
}
}
Regards,
Gurmehar Singh
From: andrei mailto:aealexsand...@gmail.com>>
Sent: 25 November 2021 16:48
To: Gurmehar Kalra mailto:gurmehar.ka...@hcl.com>>;
user@ignite.apache.org
Subject: Re: Ignite Cluster Config Issue
[CAUTION: This Email is from outside the Organization. Unless you trust the
sender, Don’t click links or open attachments as it may be a Phishing email,
which can steal your Information and compromise your Computer.]
Hi,
I think you have a singleton service that has been deployed to a random Ignite
server node. Try checking your service configuration and make sure it contains
at least the following:
svcCfg.setMaxPerNodeCount(1);
svcCfg.setTotalCount(3);
With these properties, you will have one instance of the service across three
nodes. And your service was deployed as:
ig.services(ig.cluster().forServers());
Regards,
Andrei
11/25/2021 2:01 PM, Gurmehar Kalra пишет:
Hey,
Let me explain Architecture
I am running 3 Ignite server nodes ,each have some services which read and
write data from Kafka and also to
RE: Ignite Cluster Config Issue
Hi,
I am not deploying any explicit service.
My Question here is if I am running 2 server nodes then both should be able to
read and write to Ignite, where as in current condition only one application
is able to read from ignite .
Ignite jar are part of Spring application which are deployed on 2 different
Machines.
Below is Piece of code which is running on one machine and not on other
Lock _lock =null;
try {
_lock =
CacheManager.getLock("DSLOCK", JourneyCacheEnum.IGNITE_DS_CACHE);
_lock.lock();
ScanQuery scanQuery = new ScanQuery<>(
new
IgniteBiPredicate() {
private static final long serialVersionUID = 1L;
@Override
public boolean apply(String e1, DecisionSplitWait e2) {
Instant timeLapse = e2.getTimeLapse();
return ObjectUtils.isEmpty(timeLapse) ? true
:
timeLapse.isBefore(Instant.now());
}
});
scanQuery.setPageSize(1);
QueryCursor> cursor =
CacheManager.query(scanQuery, JourneyCacheEnum.IGNITE_DS_CACHE);
Iterator> iterator = cursor.iterator();
Entry entry = null;
if
(iterator.hasNext()) {
entry = iterator.next();
}
if
(!ObjectUtils.isEmpty(entry)) {
key
= entry.getKey();
decisionSplitWait = entry.getValue();
decisionSplitWait.setTimeLapse(Instant.now().plusSeconds(WAIT_TIME_ADDED_FOR_SCAN));
CacheManager.getAndReplace(key, decisionSplitWait,
JourneyCacheEnum.IGNITE_DS_CACHE);
}
}finally {
if(!ObjectUtils.isEmpty(_lock)) {
_lock.unlock();
}
}
Regards,
Gurmehar Singh
From: andrei
Sent: 25 November 2021 16:48
To: Gurmehar Kalra ; user@ignite.apache.org
Subject: Re: Ignite Cluster Config Issue
[CAUTION: This Email is from outside the Organization. Unless you trust the
sender, Don’t click links or open attachments as it may be a Phishing email,
which can steal your Information and compromise your Computer.]
Hi,
I think you have a singleton service that has been deployed to a random Ignite
server node. Try checking your service configuration and make sure it contains
at least the following:
svcCfg.setMaxPerNodeCount(1);
svcCfg.setTotalCount(3);
With these properties, you will have one instance of the service across three
nodes. And your service was deployed as:
ig.services(ig.cluster().forServers());
Regards,
Andrei
11/25/2021 2:01 PM, Gurmehar Kalra пишет:
Hey,
Let me explain Architecture
I am running 3 Ignite server nodes ,each have some services which read and
write data from Kafka and also to Ignite
all services(From all servers ) are able to read from kafka and publish it to
Ignite.(Ignite is in replication mode).
But the thread responsible for reading from Ignite are only working on Single
Node.
I have added IP in Logs to identify , which packet is read from which IP .
In logs it clearly shows that all services from all Nodes are able to write to
Ignite , but only Single Node is reading it and publishing it ahead .
Regards,
Gurmehar Singh
From: andrei
Sent: 25 November 2021 16:17
To: user@ignite.apache.org
Subject: Re: Ignite Cluster Config Iss
Re: Ignite Cluster Config Issue
Hi,
I think you have a singleton service that has been deployed to a random
Ignite server node. Try checking your service configuration and make
sure it contains at least the following:
svcCfg.setMaxPerNodeCount(1);
svcCfg.setTotalCount(3);
With these properties, you will have one instance of the service across
three nodes. And your service was deployed as:
ig.services(ig.cluster().forServers());
Regards,
Andrei
11/25/2021 2:01 PM, Gurmehar Kalra пишет:
Hey,
Let me explain Architecture
I am running 3 Ignite server nodes ,each have some services which
read and write data from Kafka and also to Ignite
all services(From all servers ) are able to read from kafka and
publish it to Ignite.(Ignite is in replication mode).
But the thread responsible for reading from Ignite are only working on
Single Node.
I have added IP in Logs to identify , which packet is read from which IP .
In logs it clearly shows that all services from all Nodes are able to
write to Ignite , but only Single Node is reading it and publishing it
ahead .
Regards,
Gurmehar Singh
*From:*andrei
*Sent:* 25 November 2021 16:17
*To:* user@ignite.apache.org
*Subject:* Re: Ignite Cluster Config Issue
[CAUTION: This Email is from outside the Organization. Unless you
trust the sender, Don’t click links or open attachments as it may be a
Phishing email, which can steal your Information and compromise your
Computer.]
Hey,
Sorry, but I don't understand what the issue you mentioned? You are
using a transaction cache and most likely using transactions. Perhaps
you are just having design problems. Tell us more about what you are
going to achieve.
Sincerely,
Andrey
11/25/2021 11:24 AM, Gurmehar Kalra пишет:
Hi,
Team stuck with Issue with in Ignite Cluster where in Only Single
Node is able to read data from Ignite ,other Nodes are able to
write data on to Ignite , data is shared with other Nodes,
Below is configuration
cache.setWriteSynchronizationMode(CacheWriteSynchronizationMode.*/FULL_ASYNC/*);
cache.setAtomicityMode(CacheAtomicityMode.*/TRANSACTIONAL/*);
cache.setCacheMode(CacheMode.*/REPLICATED/*);
cache.setGroupName("EngineGroup");
cache.setOnheapCacheEnabled(*true*);
LruEvictionPolicyFactory lruEvictionPolicyFactory=
*new*LruEvictionPolicyFactory<>(1);
lruEvictionPolicyFactory.create();
_cache__.setEvictionPolicyFactory(__lruEvictionPolicyFactory__)_;
TCP/IP Discovery Code
TcpDiscoveryVmIpFinder ipFinder= *new*TcpDiscoveryVmIpFinder();
ipFinder.setShared(*true*);
ipFinder.setAddresses(
Lists./newArrayList/(IP1,IP2,IP3));
tcpDiscoverySpi.setIpFinder(ipFinder);
All help appreciated .
Regards,
Gurmehar Singh
::DISCLAIMER::
The contents of this e-mail and any attachment(s) are confidential
and intended for the named recipient(s) only. E-mail transmission
is not guaranteed to be secure or error-free as information could
be intercepted, corrupted, lost, destroyed, arrive late or
incomplete, or may contain viruses in transmission. The e mail and
its contents (with or without referred errors) shall therefore not
attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely
those of the author and may not necessarily reflect the views or
opinions of HCL or its affiliates. Any form of reproduction,
dissemination, copying, disclosure, modification, distribution and
/ or publication of this message without the prior written consent
of authorized representative of HCL is strictly prohibited. If you
have received this email in error please delete it and notify the
sender immediately. Before opening any email and/or attachments,
please check them for viruses and other defects.
RE: Ignite Cluster Config Issue
Hey,
Let me explain Architecture
I am running 3 Ignite server nodes ,each have some services which read and
write data from Kafka and also to Ignite
all services(From all servers ) are able to read from kafka and publish it to
Ignite.(Ignite is in replication mode).
But the thread responsible for reading from Ignite are only working on Single
Node.
I have added IP in Logs to identify , which packet is read from which IP .
In logs it clearly shows that all services from all Nodes are able to write to
Ignite , but only Single Node is reading it and publishing it ahead .
Regards,
Gurmehar Singh
From: andrei
Sent: 25 November 2021 16:17
To: user@ignite.apache.org
Subject: Re: Ignite Cluster Config Issue
[CAUTION: This Email is from outside the Organization. Unless you trust the
sender, Don’t click links or open attachments as it may be a Phishing email,
which can steal your Information and compromise your Computer.]
Hey,
Sorry, but I don't understand what the issue you mentioned? You are using a
transaction cache and most likely using transactions. Perhaps you are just
having design problems. Tell us more about what you are going to achieve.
Sincerely,
Andrey
11/25/2021 11:24 AM, Gurmehar Kalra пишет:
Hi,
Team stuck with Issue with in Ignite Cluster where in Only Single Node is able
to read data from Ignite ,other Nodes are able to write data on to Ignite ,
data is shared with other Nodes,
Below is configuration
cache.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cache.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
cache.setCacheMode(CacheMode.REPLICATED);
cache.setGroupName("EngineGroup");
cache.setOnheapCacheEnabled(true);
LruEvictionPolicyFactory lruEvictionPolicyFactory = new
LruEvictionPolicyFactory<>(1);
lruEvictionPolicyFactory.create();
cache.setEvictionPolicyFactory(lruEvictionPolicyFactory);
TCP/IP Discovery Code
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setShared(true);
ipFinder.setAddresses(
Lists.newArrayList(IP1,IP2,IP3));
tcpDiscoverySpi.setIpFinder(ipFinder);
All help appreciated .
Regards,
Gurmehar Singh
::DISCLAIMER::
The contents of this e-mail and any attachment(s) are confidential and intended
for the named recipient(s) only. E-mail transmission is not guaranteed to be
secure or error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or may contain viruses in transmission.
The e mail and its contents (with or without referred errors) shall therefore
not attach any liability on the originator or HCL or its affiliates. Views or
opinions, if any, presented in this email are solely those of the author and
may not necessarily reflect the views or opinions of HCL or its affiliates. Any
form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written
consent of authorized representative of HCL is strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. Before opening any email and/or attachments, please check them for
viruses and other defects.
Re: Ignite Cluster Config Issue
Hey,
Sorry, but I don't understand what the issue you mentioned? You are
using a transaction cache and most likely using transactions. Perhaps
you are just having design problems. Tell us more about what you are
going to achieve.
Sincerely,
Andrey
11/25/2021 11:24 AM, Gurmehar Kalra пишет:
Hi,
Team stuck with Issue with in Ignite Cluster where in Only Single Node
is able to read data from Ignite ,other Nodes are able to write data
on to Ignite , data is shared with other Nodes,
Below is configuration
cache.setWriteSynchronizationMode(CacheWriteSynchronizationMode.*/FULL_ASYNC/*);
cache.setAtomicityMode(CacheAtomicityMode.*/TRANSACTIONAL/*);
cache.setCacheMode(CacheMode.*/REPLICATED/*);
cache.setGroupName("EngineGroup");
cache.setOnheapCacheEnabled(*true*);
LruEvictionPolicyFactory lruEvictionPolicyFactory=
*new*LruEvictionPolicyFactory<>(1);
lruEvictionPolicyFactory.create();
_cache__.setEvictionPolicyFactory(__lruEvictionPolicyFactory__)_;
TCP/IP Discovery Code
TcpDiscoveryVmIpFinder ipFinder= *new*TcpDiscoveryVmIpFinder();
ipFinder.setShared(*true*);
ipFinder.setAddresses(
Lists./newArrayList/(IP1,IP2,IP3));
tcpDiscoverySpi.setIpFinder(ipFinder);
All help appreciated .
Regards,
Gurmehar Singh
::DISCLAIMER::
The contents of this e-mail and any attachment(s) are confidential and
intended for the named recipient(s) only. E-mail transmission is not
guaranteed to be secure or error-free as information could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
may contain viruses in transmission. The e mail and its contents (with
or without referred errors) shall therefore not attach any liability
on the originator or HCL or its affiliates. Views or opinions, if any,
presented in this email are solely those of the author and may not
necessarily reflect the views or opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure,
modification, distribution and / or publication of this message
without the prior written consent of authorized representative of HCL
is strictly prohibited. If you have received this email in error
please delete it and notify the sender immediately. Before opening any
email and/or attachments, please check them for viruses and other defects.
Re: Ignite BinaryHeapOutputStream OutOfMemoryError
Looks like you've run out of Java heap space. Try to increase JVM memory with Xms and Xmx options. On Thu, Nov 25, 2021 at 11:07 AM y wrote: > *Hi Igniters:* > * When i using thin client binary to insert data(*BinaryObjectBuilder*). The > client has the fol**lowing error.It seems the memory > of BinaryHeapOutputStream is not enough. How can i increase the memory when > using binary to insert?* > > Caused by: java.lang.OutOfMemoryError: Java heap space > > at > org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.arrayCopy(BinaryHeapOutputStream.java:77) > > at > org.apache.ignite.internal.client.thin.TcpClientChannel.send(TcpClientChannel.java:269) > > at > org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:216) > > at > org.apache.ignite.internal.client.thin.ReliableChannel.lambda$service$1(ReliableChannel.java:166) > > at > org.apache.ignite.internal.client.thin.ReliableChannel$$Lambda$1841/187663740.apply(Unknown > Source) > > at > org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:744) > > at > org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:712) > > at > org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:165) > > at > org.apache.ignite.internal.client.thin.ReliableChannel.request(ReliableChannel.java:287) > > at > org.apache.ignite.internal.client.thin.TcpClientCache.putAll(TcpClientCache.java:316) > > *Tinayue Hu* > > > > >
Ignite Cluster Config Issue
Hi,
Team stuck with Issue with in Ignite Cluster where in Only Single Node is able
to read data from Ignite ,other Nodes are able to write data on to Ignite ,
data is shared with other Nodes,
Below is configuration
cache.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cache.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
cache.setCacheMode(CacheMode.REPLICATED);
cache.setGroupName("EngineGroup");
cache.setOnheapCacheEnabled(true);
LruEvictionPolicyFactory lruEvictionPolicyFactory = new
LruEvictionPolicyFactory<>(1);
lruEvictionPolicyFactory.create();
cache.setEvictionPolicyFactory(lruEvictionPolicyFactory);
TCP/IP Discovery Code
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setShared(true);
ipFinder.setAddresses(
Lists.newArrayList(IP1,IP2,IP3));
tcpDiscoverySpi.setIpFinder(ipFinder);
All help appreciated .
Regards,
Gurmehar Singh
::DISCLAIMER::
The contents of this e-mail and any attachment(s) are confidential and intended
for the named recipient(s) only. E-mail transmission is not guaranteed to be
secure or error-free as information could be intercepted, corrupted, lost,
destroyed, arrive late or incomplete, or may contain viruses in transmission.
The e mail and its contents (with or without referred errors) shall therefore
not attach any liability on the originator or HCL or its affiliates. Views or
opinions, if any, presented in this email are solely those of the author and
may not necessarily reflect the views or opinions of HCL or its affiliates. Any
form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written
consent of authorized representative of HCL is strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. Before opening any email and/or attachments, please check them for
viruses and other defects.
Ignite BinaryHeapOutputStream OutOfMemoryError
Hi Igniters: When i using thin client binary to insert data(BinaryObjectBuilder). The client has the following error.It seems the memory of BinaryHeapOutputStream is not enough. How can i increase the memory when using binary to insert? Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.arrayCopy(BinaryHeapOutputStream.java:77) at org.apache.ignite.internal.client.thin.TcpClientChannel.send(TcpClientChannel.java:269) at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:216) at org.apache.ignite.internal.client.thin.ReliableChannel.lambda$service$1(ReliableChannel.java:166) at org.apache.ignite.internal.client.thin.ReliableChannel$$Lambda$1841/187663740.apply(Unknown Source) at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:744) at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:712) at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:165) at org.apache.ignite.internal.client.thin.ReliableChannel.request(ReliableChannel.java:287) at org.apache.ignite.internal.client.thin.TcpClientCache.putAll(TcpClientCache.java:316) Tinayue Hu
