Re: Historical rebalance doesn't work for caches with rebalanceDelay > 0

[Dmitry Lazurkin]

Ilya, thank you for answer.

I am looking forward to dynamic cache config changes (:.

I've dirty patched CacheConfiguration. Now it always returns
rebalanceDelay as 0 if rebalanceDelay > 0. It works.

On 21.04.2021 13:33, Ilya Kasnacheev wrote:
> Hello!
>
> 1) I think that rebalanceDelay is an outdated option, now that we have
> baseline topology and baseline auto-adjust. Just set baseline
> auto-adjust to the value of rebalance delay and you will be much
> better off.
>
> 2) I'm not sure it was, but definitely not anymore.
>
> 3) I don't think so, you will have to recreate.
>
> Regards,
> -- 
> Ilya Kasnacheev
>
>
> сб, 10 апр. 2021 г. в 19:22, Dmitry Lazurkin  <mailto:dila...@gmail.com>>:
>
> Hello, folks.
>
> I have big cache with configured rebalanceMode = ASYNC,
> rebalanceDelay =
> 10_000ms. Persistence is enabled, maxWalArchiveSize = 10GB. And I
> passed
> -DIGNITE_PREFER_WAL_REBALANCE=true and 
> -DIGNITE_PDS_WAL_REBALANCE_THRESHOLD=1 to Ignite. So node should use
> historical rebalance if there is enough WAL. But it doesn't. After
> investigation I found that GridDhtPreloader#generateAssignments always
> get called with exchFut = null, and this method can't set
> histPartitions
> without exchFut. I think, that problem in
> GridCachePartitionExchangeManager
> 
> (https://github.com/apache/ignite/blob/bc24f6baf3e9b4f98cf98cc5df67fb5deb5ceb6c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePartitionExchangeManager.java#L3486).
> It doesn't call generateAssignments without forcePreload if
> rebalanceDelay is configured.
>
> Historical rebalance works after removing rebalanceDelay.
>
> - May be this is bug because I see proper usage of rebalaceDelay in
> GridDhtPartitionDemander#addAssignments?
>
> - Is this useful to have rebalanceDelay for persistent caches?
>
> - Can I turn off rebalanceDelay for existing caches?
>
> Thank you all.
>
>

Historical rebalance doesn't work for caches with rebalanceDelay > 0

[Dmitry Lazurkin]

Hello, folks.

I have big cache with configured rebalanceMode = ASYNC, rebalanceDelay =
10_000ms. Persistence is enabled, maxWalArchiveSize = 10GB. And I passed
-DIGNITE_PREFER_WAL_REBALANCE=true and 
-DIGNITE_PDS_WAL_REBALANCE_THRESHOLD=1 to Ignite. So node should use
historical rebalance if there is enough WAL. But it doesn't. After
investigation I found that GridDhtPreloader#generateAssignments always
get called with exchFut = null, and this method can't set histPartitions
without exchFut. I think, that problem in
GridCachePartitionExchangeManager
(https://github.com/apache/ignite/blob/bc24f6baf3e9b4f98cf98cc5df67fb5deb5ceb6c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePartitionExchangeManager.java#L3486).
It doesn't call generateAssignments without forcePreload if
rebalanceDelay is configured.

Historical rebalance works after removing rebalanceDelay.

- May be this is bug because I see proper usage of rebalaceDelay in
GridDhtPartitionDemander#addAssignments?

- Is this useful to have rebalanceDelay for persistent caches?

- Can I turn off rebalanceDelay for existing caches?

Thank you all.

Re: Purging a partition

[Dmitry Lazurkin]

May be: ScanQuery with partition number and cache.localClearAll for keys.

On 01.10.2020 11:47, Vladimir Pligin wrote:
> To be honest it doesn't look trivial.
> I'm not sure at the moment why one could need that.
> Could you please describe your use-case?
> It will definitely help us to design a solution somehow.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Purging a partition

[Dmitry Lazurkin]

This leads to problem with indexes.

On 30.09.2020 14:38, Ilya Kasnacheev wrote:
> Hello!
>
> Why would you want to do this?
>
> Try removing part-NNN.bin from your persistence to achieve this. Be
> sure to check in on testing first.
>
> Regards,
> -- 
> Ilya Kasnacheev
>
>
> ср, 30 сент. 2020 г. в 13:47, ashishb888  >:
>
> Is is possible to purge a Ignite partition? If yes, how can we
> achieve? and
> will it advisable to do so?
>
> BR,
> Ashish
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Preload all indexes to memory

[Dmitry Lazurkin]

Alexandr, thank you. I'll be in the loop.

On 10.09.2020 17:36, Alexandr Shapkin wrote:
> Hi,
>
> Looks like this is the part of the upcoming warmup feature [1] that should
> be published with the next Ignite release (2.10) or the upcoming GridGain
> edition (v8.7.26?) [2].
>
> Please, check the IEP-40 for the detailed explanation [3]
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-13345
> [2] -
> https://www.gridgain.com/docs/latest/release-notes/8.7.25/release-notes_8.7.25
> [3] -
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-40+Cache+warm-up
>
>
>
>
> -
> Alex Shapkin
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Preload all indexes to memory

[Dmitry Lazurkin]

Hello,

I known I can preload ordinary partitions with
IgniteCache#localPreloadPartition. But I haven't seen any possibility
for preloading indexes to memory. Now I run sql query for each indexes
to force preloading.

But I saw in wiki documentation that indexes belong to partition with
special number 65535. I tried this value with localPreloadPartition. It
works only for numbers of ordinary partitions.

May be someone knowns efficient way to preload indexes?

Thank you.

How to define compound index with some field and _key

[Dmitry Lazurkin]

Hello,

I try to define compound index with some field and _key:
create index idx_desc on mytable (somecolumn asc, _key desc);
create index idx_asc on mytable (somecolumn asc, _key asc);

Error: Column doesn't exist: _KEY (state=42000,code=3008)

Is this bug? Because I see _key column in sqlline by using !primarykeys.

I know that I can add new field to value object which contains _key
value. But this is duplication of _key. Is that the only way for achieve
what I would like?

Thank you.

How to automatic reset state of lost partitions?

[Dmitry Lazurkin]

Hello,

I have 3 nodes with backups=1 and partitionLostPolicy=READ_WRITE_SAFE.
Node-2 and node-3 own partition 2. I shutdown node-2 and node-3. After
that cluster has partition 2 in state=LOST.

Now I should call resetLostPartitions for restore state of partition 2.
For now I see only one way for automation of this process:

- Wait all nodes from baseline toplogy

- Check existence of lost partitions in cluster

- Call Ignite#resetLostPartitions

Is it good? Does someone know better solution?

Thank you.

Run task on all nodes with specified cache and throw exception if some node is offline

[Dmitry Lazurkin]

Hello.

I would like to run task on all nodes with specified cache or all
partitions of specified cache and throw exception if some node is
offline. I use this code:

private ClusterGroup getCacheClusterGroup() {
Set baselineConsitentIds = ignite
.cluster()
.currentBaselineTopology()
.stream()
.map(n -> n.consistentId())
.collect(toSet());

ClusterGroup cacheClusterGroup = ignite
.cluster()
.forCacheNodes("test_cache");

cacheClusterGroup
.nodes()
.forEach(n -> baselineConsitentIds.remove(n.consistentId()));

if (!baselineConsitentIds.isEmpty()) {
throw new IllegalStateException("Some Ignite baseline nodes are offline:
" + baselineConsitentIds);
}

return cacheClusterGroup;
}


ignite
.compute(getCacheClusterGroup())
.broadcast(job)

Is it good solution? May be someone knows something better...

Thank you.

Re: Exception on node startup: Attempted to release write lock while not holding it

[Dmitry Lazurkin]

Thank you, this is https://issues.apache.org/jira/browse/IGNITE-9303

On 3/6/19 7:11 PM, Dmitry Lazurkin wrote:
> Thank you for reply, Ilya.
>
>> Have you tried to start this on Nightly Build? Can you try that?
>
> No, on 2.7.0#20181130-sha1:256ae401. I will try Nightly Build.
>
>> If it still would not work, can you share your DB+wal files?
> I think yes, but it's 15 GB.
>
> On 3/6/19 6:59 PM, Ilya Kasnacheev wrote:
>> Hello!
>>
>> Have you tried to start this on Nightly Build? Can you try that?
>>
>> If it still would not work, can you share your DB+wal files?
>>
>> Regards,
>> -- 
>> Ilya Kasnacheev
>>
>>
>> вт, 5 мар. 2019 г. в 20:37, Dmitry Lazurkin > <mailto:dila...@gmail.com>>:
>>
>> Ignite version: 2.7.0#20181130-sha1:256ae401
>>
>>
>

Re: Exception on node startup: Attempted to release write lock while not holding it

[Dmitry Lazurkin]

Thank you for reply, Ilya.

> Have you tried to start this on Nightly Build? Can you try that?

No, on 2.7.0#20181130-sha1:256ae401. I will try Nightly Build.

> If it still would not work, can you share your DB+wal files?
I think yes, but it's 15 GB.

On 3/6/19 6:59 PM, Ilya Kasnacheev wrote:
> Hello!
>
> Have you tried to start this on Nightly Build? Can you try that?
>
> If it still would not work, can you share your DB+wal files?
>
> Regards,
> -- 
> Ilya Kasnacheev
>
>
> вт, 5 мар. 2019 г. в 20:37, Dmitry Lazurkin  <mailto:dila...@gmail.com>>:
>
> Ignite version: 2.7.0#20181130-sha1:256ae401
>
>

Re: Exception on node startup: Attempted to release write lock while not holding it

[Dmitry Lazurkin]

Ignite version: 2.7.0#20181130-sha1:256ae401

Exception on node startup: Attempted to release write lock while not holding it

[Dmitry Lazurkin]

Hello.

I have exception on node startup:

java.lang.IllegalMonitorStateException: Attempted to release write lock
while not holding it [lock=7fa721ec8610, state=0001
    at
org.apache.ignite.internal.util.OffheapReadWriteLock.writeUnlock(OffheapReadWriteLock.java:266)
    at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlockPage(PageMemoryImpl.java:1536)
    at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.writeUnlock(PageMemoryImpl.java:470)
    at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:2067)
    at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1970)
    at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:826)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:1099)
    at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:733)
    at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2667)
    at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2539)
    at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
    at java.lang.Thread.run(Thread.java:748)

WAL has record PAGE_RECORD with pageId 72340207180973833(offset=1,
flags=1, partId=265, index=1801) in fullPageId. But that record has
pageData with pageId 282613143045897(offset=0, flags=1, partId=265,
index=1801). And GridCacheDatabaseSharedManager#restoreMemory uses
pageId (tag) from record fullPageId for writeLock and uses pageId (tag)
from pageData for writeUnlock. Why are offsets different?

I saw this issue only once and don't known how to reproduce it.


Thank you.

Re: Partitions stuck in MOVING state after upgrade to 2.7

[Dmitry Lazurkin]

I think I can comment this lines in
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition:
    if (grp.walEnabled())
    ctx.wal().log(new
PartitionMetaStateRecord(grp.groupId(), id, state(), updateCounter()));

PS. https://issues.apache.org/jira/browse/IGNITE-10226

On 1/10/19 7:33 PM, dilaz03 wrote:
> There is problem with Kubernetes, because scheduler can restart Ignite node
> at any time.
>
> Thank you.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.7 Persistence

[Dmitry Lazurkin]

Some java code that helps me on node startup:

// Call for each partition in parallel
private void preloadPartition(int partition) {
    IgniteCache cache = ignite
    .cache("test_cache")
    .withKeepBinary();

    ScanQuery query = new
ScanQuery<>(partition, (k, v) -> {
    return false;
    });
    query.setLocal(true);

    try (QueryCursor> cursor =
cache.query(query)) {
    for (@SuppressWarnings("unused") Cache.Entry row  : cursor) {
    // empty
    }
    }
    }

// Call for each index
private void preloadIndex(String index) {
    // Use sql query which uses index and contains falsy-condition
}

PS. My memory region is bigger than total data size.

On 1/11/19 18:20, gweiske wrote:
> Is there a command that one can/needs to run to load the data into memory
> after restart of Ignite? The documentation suggests that at least for 2.7
> that is not necessary, and I have not found a command that would start the
> loading into memory from persistence. It looks like one can write some Java
> code, but it seems such basic functionality that I thought that there should
> be a shell command.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Partitions stuck in MOVING state after upgrade to 2.7

[Dmitry Lazurkin]

Hello.

Partitions on Ignite cluster stuck forever in MOVING state after
restart. Cluster version is 2.7.0#20181130-sha1:256ae401. I don't see
any errors in logs about moving partitions and and I don't understand
why Ignite try to move partitions. May be I can enable some additional
logs for troubleshooting this issue?

Thank you.

Re: Client stucks and doesn't connect

[Dmitry Lazurkin]

Issue with patch: https://issues.apache.org/jira/browse/IGNITE-10513

On 12/3/18 17:43, Dmitry Lazurkin wrote:
> OK. I have found steps to reproduce.
>
> For reproducing we need Ignite with slow disk. Or I known how to
> emulate slow hard disk:
>
> Add to partitions cycle in
> GridCacheDatabaseSharedManager#restorePartitionStates:
> //...
>     for (int i = 0; i < grp.affinity().partitions(); i++) {
>     try {
>     log.error("Wait");
>     Thread.sleep(1);
>     } catch (InterruptedException e) {
>     e.printStackTrace();
>     }
> //...
>
> - Now we can start server
> - Then start client
> - Wait message "Join cluster while cluster state transition is in
> progress, waiting when transition finish."
> - Kill server
> - Wait repeatable exception java.net.ConnectException: Connection
> refused (Connection refused)
> - Start server and I have 100% chance to reproduce on my computer
>
> Thank you.
>
> On 11/29/18 14:03, Stanislav Lukyanov wrote:
>>
>> Hi,
>>
>>  
>>
>> The interesting (and disappointing) part is the NPE:
>>
>> java.lang.NullPointerException: null
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:666)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1757)
>>
>>  
>>
>> Which version do you use?
>>
>> Is this reproducible? Every time?
>>
>>  
>>
>> Thanks,
>>
>> Stan
>>
>>  
>>
>>  
>>
>> *From: *Dmitry Lazurkin <mailto:dila...@gmail.com>
>> *Sent: *20 ноября 2018 г. 15:44
>> *To: *user@ignite.apache.org <mailto:user@ignite.apache.org>
>> *Subject: *Client stucks and doesn't connect
>>
>>  
>>
>> Hello.
>>
>>  
>>
>> Ignite client stops connecting to server after exception:
>>
>>  
>>
>> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>>
>> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>>
>> [/10.48.14.1:47500]
>>
>> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>>
>> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>>
>> [addr=/10.48.14.1:47500, reconnect=true,
>>
>> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>>
>> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] ERROR
>>
>> o.a.i.s.d.tcp.TcpDiscoverySpi - Exception on joining: Connection refused
>>
>> (Connection refused)
>>
>> java.net.ConnectException: Connection refused (Connection refused)
>>
>>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>>
>>     at
>>
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>>
>>     at
>>
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>>
>>     at
>>
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>>
>>     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>
>>     at java.net.Socket.connect(Socket.java:589)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1450)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:637)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>>
>>     at
>>
>> org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1408)
>>
>>     at
>>
>> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>>
>> 2018-11-19 16:00:49,258 [tcp-client-disco-r

Re: Client stucks and doesn't connect

[Dmitry Lazurkin]

OK. I have found steps to reproduce.

For reproducing we need Ignite with slow disk. Or I known how to emulate
slow hard disk:

Add to partitions cycle in
GridCacheDatabaseSharedManager#restorePartitionStates:
//...
    for (int i = 0; i < grp.affinity().partitions(); i++) {
    try {
    log.error("Wait");
    Thread.sleep(1);
    } catch (InterruptedException e) {
    e.printStackTrace();
    }
//...

- Now we can start server
- Then start client
- Wait message "Join cluster while cluster state transition is in
progress, waiting when transition finish."
- Kill server
- Wait repeatable exception java.net.ConnectException: Connection
refused (Connection refused)
- Start server and I have 100% chance to reproduce on my computer

Thank you.

On 11/29/18 14:03, Stanislav Lukyanov wrote:
>
> Hi,
>
>  
>
> The interesting (and disappointing) part is the NPE:
>
> java.lang.NullPointerException: null
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:666)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1757)
>
>  
>
> Which version do you use?
>
> Is this reproducible? Every time?
>
>  
>
> Thanks,
>
> Stan
>
>  
>
>  
>
> *From: *Dmitry Lazurkin <mailto:dila...@gmail.com>
> *Sent: *20 ноября 2018 г. 15:44
> *To: *user@ignite.apache.org <mailto:user@ignite.apache.org>
> *Subject: *Client stucks and doesn't connect
>
>  
>
> Hello.
>
>  
>
> Ignite client stops connecting to server after exception:
>
>  
>
> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>
> [/10.48.14.1:47500]
>
> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>
> [addr=/10.48.14.1:47500, reconnect=true,
>
> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>
> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] ERROR
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Exception on joining: Connection refused
>
> (Connection refused)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>
>     at
>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
>     at
>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
>     at
>
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
>     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
>     at java.net.Socket.connect(Socket.java:589)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1450)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:637)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1408)
>
>     at
>
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Failed to join to address
>
> [addr=/10.48.14.1:47500, recon=true, errs=[java.net.ConnectException:
>
> Connection refused (Connection refused)]]
>
> 2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>
> [/10.48.14.1:47500]
>
> 2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>
> [addr=/10.48.14.1:47500, reconnect=true,
>
> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>
> 2018-11-19 16:00:51,364 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Message has b

Re: Client stucks and doesn't connect

[Dmitry Lazurkin]

Hello.

I use version 2.6.0#20180710-sha1:669feacc and I can not reproduce this
issue. I have seen it at least 2 times, and I don't have full log =/ .

My cluster has lazy disk so restoration of partition states is too slow.
So I think there is race condition between ClientImpl and call of
TcpDiscoverySpi.onContextInitialized on client when it try to connect to
server which restores partition states.

PS. Sorry for duplicated messages.

On 11/29/18 14:03, Stanislav Lukyanov wrote:
>
> Hi,
>
>  
>
> The interesting (and disappointing) part is the NPE:
>
> java.lang.NullPointerException: null
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:666)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1757)
>
>  
>
> Which version do you use?
>
> Is this reproducible? Every time?
>
>  
>
> Thanks,
>
> Stan
>
>  
>
>  
>
> *From: *Dmitry Lazurkin <mailto:dila...@gmail.com>
> *Sent: *20 ноября 2018 г. 15:44
> *To: *user@ignite.apache.org <mailto:user@ignite.apache.org>
> *Subject: *Client stucks and doesn't connect
>
>  
>
> Hello.
>
>  
>
> Ignite client stops connecting to server after exception:
>
>  
>
> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>
> [/10.48.14.1:47500]
>
> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>
> [addr=/10.48.14.1:47500, reconnect=true,
>
> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>
> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] ERROR
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Exception on joining: Connection refused
>
> (Connection refused)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>
>     at
>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
>     at
>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
>     at
>
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
>     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
>     at java.net.Socket.connect(Socket.java:589)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1450)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:637)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1408)
>
>     at
>
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Failed to join to address
>
> [addr=/10.48.14.1:47500, recon=true, errs=[java.net.ConnectException:
>
> Connection refused (Connection refused)]]
>
> 2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>
> [/10.48.14.1:47500]
>
> 2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>
> [addr=/10.48.14.1:47500, reconnect=true,
>
> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>
> 2018-11-19 16:00:51,364 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Message has been sent to address
>
> [msg=TcpDiscoveryClientReconnectMessage
>
> [routerNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20,
>
> lastMsgId=c42b1bc2761-bbc61e78-97e8-49cd-844e-2dc3e8aacc68,
>
> super=TcpDiscoveryAbstractMessage [sndNodeId=null,
>
> id=48722bc2761-cd323c53-d1de-4608-8eec-e373b1f68b71,
>
> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
>
> isClient=t

Re: Client stucks and doesn't connect

[Dmitry Lazurkin]

Hello.

I use version 2.6.0#20180710-sha1:669feacc and I can not reproduce this
issue. I have seen it at least 2 times, and I don't have full log =/ .

My cluster has lazy disk so restoration of partition states is too slow.
So I think there is race condition between ClientImpl and call of
TcpDiscoverySpi.onContextInitialized on client when it try to connect to
server which restores partition states.

On 11/29/18 14:03, Stanislav Lukyanov wrote:
>
> Hi,
>
>  
>
> The interesting (and disappointing) part is the NPE:
>
> java.lang.NullPointerException: null
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:666)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1846)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1757)
>
>  
>
> Which version do you use?
>
> Is this reproducible? Every time?
>
>  
>
> Thanks,
>
> Stan
>
>  
>
>  
>
> *From: *Dmitry Lazurkin <mailto:dila...@gmail.com>
> *Sent: *20 ноября 2018 г. 15:44
> *To: *user@ignite.apache.org <mailto:user@ignite.apache.org>
> *Subject: *Client stucks and doesn't connect
>
>  
>
> Hello.
>
>  
>
> Ignite client stops connecting to server after exception:
>
>  
>
> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>
> [/10.48.14.1:47500]
>
> 2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>
> [addr=/10.48.14.1:47500, reconnect=true,
>
> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>
> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] ERROR
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Exception on joining: Connection refused
>
> (Connection refused)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>
>     at
>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
>
>     at
>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
>
>     at
>
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
>     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
>     at java.net.Socket.connect(Socket.java:589)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1450)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:637)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
>
>     at
>
> org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1408)
>
>     at
>
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
> 2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Failed to join to address
>
> [addr=/10.48.14.1:47500, recon=true, errs=[java.net.ConnectException:
>
> Connection refused (Connection refused)]]
>
> 2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
>
> [/10.48.14.1:47500]
>
> 2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
>
> [addr=/10.48.14.1:47500, reconnect=true,
>
> locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
>
> 2018-11-19 16:00:51,364 [tcp-client-disco-reconnector-#5] DEBUG
>
> o.a.i.s.d.tcp.TcpDiscoverySpi - Message has been sent to address
>
> [msg=TcpDiscoveryClientReconnectMessage
>
> [routerNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20,
>
> lastMsgId=c42b1bc2761-bbc61e78-97e8-49cd-844e-2dc3e8aacc68,
>
> super=TcpDiscoveryAbstractMessage [sndNodeId=null,
>
> id=48722bc2761-cd323c53-d1de-4608-8eec-e373b1f68b71,
>
> verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
>
> isClient=true]], addr=/10.48.14.1:47500,
>

Re: New added property with @QuerySqlField does not persist

[Dmitry Lazurkin]

Try to print putted object with new field from java code. Column in
QueryEntity is just link to binary object field. So you may be have
different names for binary field and column.

PS. I use ignite 2.6 and don't have problems with new fields/columns.

On 11/22/18 05:26, kcheng.mvp wrote:
> Thank you very much!
>
> how to handle this case in production?
>
> it's too common to add new attribute to a new cache/pojo attribute.!!!
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Client stucks and doesn't connect

[Dmitry Lazurkin]

Hello.

Ignite client stops connecting to server after exception:

2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
[/10.48.14.1:47500]
2018-11-19 16:00:49,257 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
[addr=/10.48.14.1:47500, reconnect=true,
locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] ERROR
o.a.i.s.d.tcp.TcpDiscoverySpi - Exception on joining: Connection refused
(Connection refused)
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1450)
    at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
    at
org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:637)
    at
org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:546)
    at
org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:128)
    at
org.apache.ignite.spi.discovery.tcp.ClientImpl$Reconnector.body(ClientImpl.java:1408)
    at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
2018-11-19 16:00:49,258 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Failed to join to address
[addr=/10.48.14.1:47500, recon=true, errs=[java.net.ConnectException:
Connection refused (Connection refused)]]
2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Resolved addresses from IP finder:
[/10.48.14.1:47500]
2018-11-19 16:00:51,344 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
[addr=/10.48.14.1:47500, reconnect=true,
locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
2018-11-19 16:00:51,364 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Message has been sent to address
[msg=TcpDiscoveryClientReconnectMessage
[routerNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20,
lastMsgId=c42b1bc2761-bbc61e78-97e8-49cd-844e-2dc3e8aacc68,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=48722bc2761-cd323c53-d1de-4608-8eec-e373b1f68b71,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]], addr=/10.48.14.1:47500,
rmtNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20]
2018-11-19 16:00:51,365 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Received response to join request
[addr=/10.48.14.1:47500, res=100]
2018-11-19 16:00:51,365 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Will wait before retry join.
2018-11-19 16:00:53,365 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Send join request
[addr=/10.48.14.1:47500, reconnect=true,
locNodeId=cd323c53-d1de-4608-8eec-e373b1f68b71]
2018-11-19 16:00:53,368 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Message has been sent to address
[msg=TcpDiscoveryClientReconnectMessage
[routerNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20,
lastMsgId=c42b1bc2761-bbc61e78-97e8-49cd-844e-2dc3e8aacc68,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=a8722bc2761-cd323c53-d1de-4608-8eec-e373b1f68b71,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]], addr=/10.48.14.1:47500,
rmtNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20]
2018-11-19 16:00:53,368 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Received response to join request
[addr=/10.48.14.1:47500, res=1]
2018-11-19 16:00:53,372 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Received reconnect response
[success=false, msg=TcpDiscoveryClientReconnectMessage
[routerNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20,
lastMsgId=c42b1bc2761-bbc61e78-97e8-49cd-844e-2dc3e8aacc68,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=a8722bc2761-cd323c53-d1de-4608-8eec-e373b1f68b71,
verifierNodeId=9c68d70f-883e-4a21-938b-05f3f6f98d20, topVer=0,
pendingIdx=0, failedNodes=null, isClient=true]]]
2018-11-19 16:00:53,372 [tcp-client-disco-reconnector-#5] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Grid runnable finished normally:
tcp-client-disco-reconnector-#5
2018-11-19 16:00:53,373 [tcp-client-disco-msg-worker-#4] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Failed to restore closed connection,
will try to reconnect [networkTimeout=36, joinTimeout=0, failMsg=null]
2018-11-19 16:00:53,373 [tcp-client-disco-msg-worker

Re: How to get the column names(or fieldsNames) of existing ignite sql table

[Dmitry Lazurkin]

Hello.

On 17.11.2018 13:25, siva wrote:
> Quetions:
> ===
> 1.How to get the existing table columnNames ?
>
> i am trying to use this query but its throwing exception
>
> *select COLUMN_NAME from INFORMATION_SCHEMA.COLUMNS where
> TABLE_NAME='Person'*

You need query with setLocal(true). I don't known how to do this with
thin client.

Re: Client connects to server after too long time interval (1 minute)

[Dmitry Lazurkin]

I've found. TcpDiscoveryNode try to resolve sockAddrs in readExternal
method from hostNames (first) and addrs (second). TcpDiscoverySpi fills
hostNames for node with local hostnames. And of course new client node
can resolve  only local hostname. In my case resolution of other node's
hostnames take 10 seconds for each hostname.

On 11/12/18 17:28, Dmitry Lazurkin wrote:
> After reconnect unmarshalling of TcpDiscoveryNodeAddedMessage takes 20
> seconds:
> 2018-11-12 14:10:36.105 ERROR 10 --- [-sock-reader-#3]
> o.a.ignite.marshaller.jdk.JdkMarshaller : Unmarshall 1
> 2018-11-12 14:10:36.107 ERROR 10 --- [-sock-reader-#3]
> o.a.ignite.marshaller.jdk.JdkMarshaller : Unmarshall 2
> 2018-11-12 14:10:56.262 ERROR 10 --- [-sock-reader-#3]
> o.a.ignite.marshaller.jdk.JdkMarshaller : Unmarshall 3:
> TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode [id=06960cfd-17
>
> Does class loader load many classes on initial connect?
>
> On 11/12/18 16:28, Dmitry Lazurkin wrote:
>> Hi, Andrei. Thank you for reply.
>>
>> I have found that problem is in unmarshalling:
>> 2018-11-12 13:18:24.375  INFO 10 --- [-sock-reader-#3]
>> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : 1
>> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
>> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
>> node: 28ebe679-e815-4c16-bd5c-ee320caa3019
>> 2018-11-12 13:18:24.375 ERROR 10 --- [-sock-reader-#3]
>> o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 1
>> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
>> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
>> node: 12f39501-0fba-4c1c-9c37-dc35f33fcee9
>> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
>> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
>> node: 40b74eed-a636-488f-b1a8-6d58f4bc9137
>> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
>> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
>> node: f4c370e7-cf4a-4209-84fe-42fba4a30eef
>> 2018-11-12 13:18:24.380 ERROR 10 --- [-sock-reader-#3]
>> o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 2
>> 2018-11-12 13:19:44.965 ERROR 10 --- [-sock-reader-#3]
>> o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 3:
>> TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode
>> [id=8807c1ce-1e4a-496b-9aea-66cee8d95434, addrs=[10.37.92.255],
>> sockAddrs=[client-98d86c46f-j76s5/10.37.92.255:0], discPort=0, order=0,
>> intOrder=12, lastExchangeTime=1542028704438, loc=false,
>> ver=2.6.0#20180710-sha1:669feacc, isClient=true],
>> dataPacket=org.apache.ignite.spi.discovery.tcp.internal.DiscoveryDataPacket@3b238c6e,
>> discardMsgId=null, discardCustomMsgId=null, top=[TcpDiscoveryNode
>> [id=28ebe679-e815-4c16-bd5c-ee320caa3019, addrs=[10.48.14.1],
>> sockAddrs=[ignite-1/10.48.14.1:47500], discPort=47500, order=1,
>> intOrder=1, lastExchangeTime=1542028704438, loc=false,
>> ver=2.6.0#20180710-sha1:669feacc, isClient=false], TcpDiscoveryNode
>> [id=f4c370e7-cf4a-4209-84fe-42fba4a30eef, addrs=[10.37.92.208],
>> sockAddrs=[/10.37.92.208:0], discPort=0, order=2, intOrder=2,
>> lastExchangeTime=1542028704448, loc=false,
>> ver=2.6.0#20180710-sha1:669feacc, isClient=true], TcpDiscoveryNode
>> [id=12f39501-0fba-4c1c-9c37-dc35f33fcee9, addrs=[10.37.92.205],
>> sockAddrs=[/10.37.92.205:0], discPort=0, order=5, intOrder=5,
>> lastExchangeTime=1542028714511, loc=false,
>> ver=2.6.0#20180710-sha1:669feacc, isClient=true], TcpDiscoveryNode
>> [id=40b74eed-a636-488f-b1a8-6d58f4bc9137, addrs=[10.37.92.222],
>> sockAddrs=[/10.37.92.222:0], discPort=0, order=15, intOrder=11,
>> lastExchangeTime=1542028724583, loc=false,
>> ver=2.6.0#20180710-sha1:669feacc, isClient=true]], clientTop=null,
>> gridStartTime=1542023768674, super=TcpDiscoveryAbstractMessage
>> [sndNodeId=null, id=0c5f7c70761-28ebe679-e815-4c16-bd5c-ee320caa3019,
>> verifierNodeId=28ebe679-e815-4c16-bd5c-ee320caa3019, topVer=0,
>> pendingIdx=0, failedNodes=null, isClient=true]]
>>
>> Unmarshalling TcpDiscoveryNodeAddedMessage need 1 minute...(:
>>
>> That's org.apache.ignite.marshaller.jdk.JdkMarshaller with logs:
>>     /** {@inheritDoc} */
>>     @SuppressWarnings({"unchecked"})
>>     @Override protected  T unmarshal0(InputStream in, @Nullable
>> ClassLoader clsLdr) throws IgniteCheckedException {
>>     assert in != null;
>>
>>     if (clsLdr == null)
>>     clsLdr = getClass().getClassLoader();
>>
>>     logger.error("Unmarshall 1");
>>
>>     ObjectInputStream objIn = null;
>>
>>

Re: Client connects to server after too long time interval (1 minute)

[Dmitry Lazurkin]

I've found. TcpDiscoveryNode try to resolve sockAddrs in readExternal
method from hostNames (first) and addrs (second). TcpDiscoverySpi fills
hostNames for node with local hostnames. And of course new client node
can resolve  only local hostname. In my case resolution of other node's
hostnames take 10 seconds for each hostname.

Re: Client connects to server after too long time interval (1 minute)

[Dmitry Lazurkin]

After reconnect unmarshalling of TcpDiscoveryNodeAddedMessage takes 20
seconds:
2018-11-12 14:10:36.105 ERROR 10 --- [-sock-reader-#3]
o.a.ignite.marshaller.jdk.JdkMarshaller : Unmarshall 1
2018-11-12 14:10:36.107 ERROR 10 --- [-sock-reader-#3]
o.a.ignite.marshaller.jdk.JdkMarshaller : Unmarshall 2
2018-11-12 14:10:56.262 ERROR 10 --- [-sock-reader-#3]
o.a.ignite.marshaller.jdk.JdkMarshaller : Unmarshall 3:
TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode [id=06960cfd-17

Does class loader load many classes on initial connect?

On 11/12/18 16:28, Dmitry Lazurkin wrote:
> Hi, Andrei. Thank you for reply.
>
> I have found that problem is in unmarshalling:
> 2018-11-12 13:18:24.375  INFO 10 --- [-sock-reader-#3]
> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : 1
> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
> node: 28ebe679-e815-4c16-bd5c-ee320caa3019
> 2018-11-12 13:18:24.375 ERROR 10 --- [-sock-reader-#3]
> o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 1
> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
> node: 12f39501-0fba-4c1c-9c37-dc35f33fcee9
> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
> node: 40b74eed-a636-488f-b1a8-6d58f4bc9137
> 2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
> o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
> node: f4c370e7-cf4a-4209-84fe-42fba4a30eef
> 2018-11-12 13:18:24.380 ERROR 10 --- [-sock-reader-#3]
> o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 2
> 2018-11-12 13:19:44.965 ERROR 10 --- [-sock-reader-#3]
> o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 3:
> TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode
> [id=8807c1ce-1e4a-496b-9aea-66cee8d95434, addrs=[10.37.92.255],
> sockAddrs=[client-98d86c46f-j76s5/10.37.92.255:0], discPort=0, order=0,
> intOrder=12, lastExchangeTime=1542028704438, loc=false,
> ver=2.6.0#20180710-sha1:669feacc, isClient=true],
> dataPacket=org.apache.ignite.spi.discovery.tcp.internal.DiscoveryDataPacket@3b238c6e,
> discardMsgId=null, discardCustomMsgId=null, top=[TcpDiscoveryNode
> [id=28ebe679-e815-4c16-bd5c-ee320caa3019, addrs=[10.48.14.1],
> sockAddrs=[ignite-1/10.48.14.1:47500], discPort=47500, order=1,
> intOrder=1, lastExchangeTime=1542028704438, loc=false,
> ver=2.6.0#20180710-sha1:669feacc, isClient=false], TcpDiscoveryNode
> [id=f4c370e7-cf4a-4209-84fe-42fba4a30eef, addrs=[10.37.92.208],
> sockAddrs=[/10.37.92.208:0], discPort=0, order=2, intOrder=2,
> lastExchangeTime=1542028704448, loc=false,
> ver=2.6.0#20180710-sha1:669feacc, isClient=true], TcpDiscoveryNode
> [id=12f39501-0fba-4c1c-9c37-dc35f33fcee9, addrs=[10.37.92.205],
> sockAddrs=[/10.37.92.205:0], discPort=0, order=5, intOrder=5,
> lastExchangeTime=1542028714511, loc=false,
> ver=2.6.0#20180710-sha1:669feacc, isClient=true], TcpDiscoveryNode
> [id=40b74eed-a636-488f-b1a8-6d58f4bc9137, addrs=[10.37.92.222],
> sockAddrs=[/10.37.92.222:0], discPort=0, order=15, intOrder=11,
> lastExchangeTime=1542028724583, loc=false,
> ver=2.6.0#20180710-sha1:669feacc, isClient=true]], clientTop=null,
> gridStartTime=1542023768674, super=TcpDiscoveryAbstractMessage
> [sndNodeId=null, id=0c5f7c70761-28ebe679-e815-4c16-bd5c-ee320caa3019,
> verifierNodeId=28ebe679-e815-4c16-bd5c-ee320caa3019, topVer=0,
> pendingIdx=0, failedNodes=null, isClient=true]]
>
> Unmarshalling TcpDiscoveryNodeAddedMessage need 1 minute...(:
>
> That's org.apache.ignite.marshaller.jdk.JdkMarshaller with logs:
>     /** {@inheritDoc} */
>     @SuppressWarnings({"unchecked"})
>     @Override protected  T unmarshal0(InputStream in, @Nullable
> ClassLoader clsLdr) throws IgniteCheckedException {
>     assert in != null;
>
>     if (clsLdr == null)
>     clsLdr = getClass().getClassLoader();
>
>     logger.error("Unmarshall 1");
>
>     ObjectInputStream objIn = null;
>
>     try {
>     objIn = new JdkMarshallerObjectInputStream(new
> JdkMarshallerInputStreamWrapper(in), clsLdr, clsFilter);
>
>     logger.error("Unmarshall 2");
>
>     T t = (T)objIn.readObject();
>
>     logger.error("Unmarshall 3: " + t);
>
>     return t;
>     }
> ...
>
>
> On 11/12/18 10:24, aealexsandrov wrote:
>> Hi,
>>
>> Could you please attach the XML configurations of your client and server
>> nodes and logs?
>>
>> BR,
>> Andrei
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

Re: Client connects to server after too long time interval (1 minute)

[Dmitry Lazurkin]

Hi, Andrei. Thank you for reply.

I have found that problem is in unmarshalling:
2018-11-12 13:18:24.375  INFO 10 --- [-sock-reader-#3]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : 1
2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
node: 28ebe679-e815-4c16-bd5c-ee320caa3019
2018-11-12 13:18:24.375 ERROR 10 --- [-sock-reader-#3]
o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 1
2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
node: 12f39501-0fba-4c1c-9c37-dc35f33fcee9
2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
node: 40b74eed-a636-488f-b1a8-6d58f4bc9137
2018-11-12 13:18:24.375 DEBUG 10 --- [o-msg-worker-#4]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Received metrics from unknown
node: f4c370e7-cf4a-4209-84fe-42fba4a30eef
2018-11-12 13:18:24.380 ERROR 10 --- [-sock-reader-#3]
o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 2
2018-11-12 13:19:44.965 ERROR 10 --- [-sock-reader-#3]
o.a.ignite.marshaller.jdk.JdkMarshaller  : Unmarshall 3:
TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode
[id=8807c1ce-1e4a-496b-9aea-66cee8d95434, addrs=[10.37.92.255],
sockAddrs=[client-98d86c46f-j76s5/10.37.92.255:0], discPort=0, order=0,
intOrder=12, lastExchangeTime=1542028704438, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true],
dataPacket=org.apache.ignite.spi.discovery.tcp.internal.DiscoveryDataPacket@3b238c6e,
discardMsgId=null, discardCustomMsgId=null, top=[TcpDiscoveryNode
[id=28ebe679-e815-4c16-bd5c-ee320caa3019, addrs=[10.48.14.1],
sockAddrs=[ignite-1/10.48.14.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1542028704438, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], TcpDiscoveryNode
[id=f4c370e7-cf4a-4209-84fe-42fba4a30eef, addrs=[10.37.92.208],
sockAddrs=[/10.37.92.208:0], discPort=0, order=2, intOrder=2,
lastExchangeTime=1542028704448, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], TcpDiscoveryNode
[id=12f39501-0fba-4c1c-9c37-dc35f33fcee9, addrs=[10.37.92.205],
sockAddrs=[/10.37.92.205:0], discPort=0, order=5, intOrder=5,
lastExchangeTime=1542028714511, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true], TcpDiscoveryNode
[id=40b74eed-a636-488f-b1a8-6d58f4bc9137, addrs=[10.37.92.222],
sockAddrs=[/10.37.92.222:0], discPort=0, order=15, intOrder=11,
lastExchangeTime=1542028724583, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=true]], clientTop=null,
gridStartTime=1542023768674, super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=0c5f7c70761-28ebe679-e815-4c16-bd5c-ee320caa3019,
verifierNodeId=28ebe679-e815-4c16-bd5c-ee320caa3019, topVer=0,
pendingIdx=0, failedNodes=null, isClient=true]]

Unmarshalling TcpDiscoveryNodeAddedMessage need 1 minute...(:

That's org.apache.ignite.marshaller.jdk.JdkMarshaller with logs:
    /** {@inheritDoc} */
    @SuppressWarnings({"unchecked"})
    @Override protected  T unmarshal0(InputStream in, @Nullable
ClassLoader clsLdr) throws IgniteCheckedException {
    assert in != null;

    if (clsLdr == null)
    clsLdr = getClass().getClassLoader();

    logger.error("Unmarshall 1");

    ObjectInputStream objIn = null;

    try {
    objIn = new JdkMarshallerObjectInputStream(new
JdkMarshallerInputStreamWrapper(in), clsLdr, clsFilter);

    logger.error("Unmarshall 2");

    T t = (T)objIn.readObject();

    logger.error("Unmarshall 3: " + t);

    return t;
    }
...


On 11/12/18 10:24, aealexsandrov wrote:
> Hi,
>
> Could you please attach the XML configurations of your client and server
> nodes and logs?
>
> BR,
> Andrei
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Client connects to server after too long time interval (1 minute)

[Dmitry Lazurkin]

Hello.

I need some advice. I have 1 client and 1 server. Ignite client
connection to server take ~1 minute. And I don't understand why.

In server log I see next entry:

2018-11-09 16:34:16,471 [tcp-disco-client-message-worker-#16] DEBUG
o.a.i.s.d.tcp.TcpDiscoverySpi - Redirecting message to client
[sock=Socket[addr=/10.37.92.222,port=48843,localport=47500],
locNodeId=c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f,
rmtNodeId=f4856174-de1d-4eae-ad85-80681f783955,
msg=TcpDiscoveryNodeAddFinishedMessage
[nodeId=f4856174-de1d-4eae-ad85-80681f783955,
super=TcpDiscoveryAbstractMessage
[sndNodeId=c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f,
id=a79abb8f661-c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f,
verifierNodeId=c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f, topVer=7,
pendingIdx=0, failedNodes=null, isClient=false]]]

Client log:

2018-11-09 16:35:06.585 DEBUG 10 --- [-sock-reader-#3]
o.a.i.spi.discovery.tcp.TcpDiscoverySpi  : Message has been received:
TcpDiscoveryNodeAddFinishedMessage
[nodeId=f4856174-de1d-4eae-ad85-80681f783955,
super=TcpDiscoveryAbstractMessage
[sndNodeId=c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f,
id=a79abb8f661-c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f,
verifierNodeId=c86fe162-0a55-4aaa-a2cd-f2ea1c644b5f, topVer=7,
pendingIdx=0, failedNodes=null, isClient=false]]

Difference is 50 seconds. Wireshark log entries on client and server
with TcpDiscoveryNodeAddFinishedMessage in packet bytes:

On server (10.48.14.1):
1648    16:34:16.473071    10.48.14.1    10.37.92.222    35780    TCP   
[TCP Window Full] 47500 → 48843 [ACK] Seq=180330 Ack=30711 Win=99584
Len=35712 TSval=3529221982 TSecr=2842678176
On client (10.37.92.222):
131    16:34:16.473103    10.48.14.1    10.37.92.222    35780    TCP   
[TCP Window Full] 47500 → 48843 [ACK] Seq=180330 Ack=30711 Win=99584
Len=35712 TSval=3529221982 TSecr=2842678176

I cann't understand that delay.

Thank you.

PS. This is kubernetes.

Re: How to detect sql schema changes and make update

[Dmitry Lazurkin]

I found solution. Query "select column_name, data_type from
information_schema.columns where table_schema = ? and table_name = ?" in
local mode.

On 08/16/2018 06:28 PM, Dmitry Lazurkin wrote:
> Hello.
>
> I have cache with query entity. Now I want to add new field to query
> entity of existing cache. I take query entity from existing cache and
> query entity with new fields, search new field and add it with DLL
> 'ALTER TABLE'. And now I have problem because query entity from cache
> configuration doesn't have new field. How can I make automatic schema
> update? May be exists better approach.
>
> I see interesting method QueryEntity#makePatch, but I have no idea how
> to use it.
>
> Thank you.
>
>

How to detect sql schema changes and make update

[Dmitry Lazurkin]

Hello.

I have cache with query entity. Now I want to add new field to query
entity of existing cache. I take query entity from existing cache and
query entity with new fields, search new field and add it with DLL
'ALTER TABLE'. And now I have problem because query entity from cache
configuration doesn't have new field. How can I make automatic schema
update? May be exists better approach.

I see interesting method QueryEntity#makePatch, but I have no idea how
to use it.

Thank you.