Re: Random2LruPageEvictionTracker causing hanging in our integration tests

[scottmf]

thanks Ilya,
I really have no idea why it is hanging like this.  I was even more
surprised when I looked at the code because I saw that there is throttling
in the code that should prevent this like you said.  I do see an OOM when I
turn off page eviction all together.  But when eviction is turned on it just
hangs.  Ultimately what I did was to reset my ignite cluster on random tests
after they completed so that I don't run into this anymore.

Given your response I think the next thing to do is to put in the work to
give you an environment that reproduces this.

I'll add that to the ticket when I am able to make it available for you all.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Enabling default persistence on existing ignite cluster

[Ilya Kasnacheev]

Hello!

Nice to hear!

They are also responsible for the data region on which to place those
partitions :)

Regards,
-- 
Ilya Kasnacheev


пн, 4 мая 2020 г. в 16:50, Sebastian Sindelar <
sebastian.sinde...@nexus-ips.de>:

> Hallo.
>
>
>
> Thanks for the suggestion. Just tried it and it *worked*.
>
> Never thought about the group names because I thought they had just to do
> with partition placement.
>
>
>
>
>
> Regards,
>
> *nexus / ag *
>
> *Sebastian Sindelar *
>
> Softwareentwicklung
>
> Tel.: +49 (0)561 942880
> Fax: +49 (0)561 9428877
> Support: +49 (0)561 9428899
>
> E-Mail: sebastian.sinde...@nexus-ips.de
>
> NEXUS / IPS GmbH
> Standort Kassel
> Mendelssohn-Bartholdy-Str. 17
> D-34134 Kassel
> www.nexus-ag.de | www.nexus-ips.de
>
>
> ---
> NEXUS / IPS GmbH, Irmastraße 1, D-78166 Donaueschingen
> Eingetragene Gesellschaft beim Amtsgericht Freiburg i.Br., HRB 602014
> Geschäftsführer: Stefan Born, Dirk Hübner
>
> ---
>
> Rechtlicher Hinweis:
> Diese E-Mail samt Anhängen könnte vertrauliche/rechtlich geschützte
> Informationen enthalten.
> Wenn Sie nicht der richtige Empfänger sind, informieren Sie bitte sofort
> den Absender und löschen
> diese E-Mail samt Anhängen. Das unerlaubte Kopieren oder Weitergeben
> dieser E-Mail ist nicht gestattet.
>
> Datenschutzhinweis:
> Bitte beachten Sie unsere Grundsätze der Datenverarbeitung in unserer
> Datenschutzerklärung:
> https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung
>
> Legal notice:
> This message and any attachment(s) may contain confidential/privileged
> information. If you are not the
> intended recipient, please email or telephone the sender and delete this
> message and any attachment(s) from
> your system. Any unauthorised copying, disclosure or distribution of the
> material in this email is strictly forbidden.
>
> Privacy Policy:
> Please note our principles of data processing in our privacy statement:
> https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung
>
>
>
> *Von:* Ilya Kasnacheev 
> *Gesendet:* Freitag, 24. April 2020 15:02
> *An:* user@ignite.apache.org
> *Betreff:* Re: Enabling default persistence on existing ignite cluster
>
>
>
> Hello!
>
>
>
> Have you tried supplying different group name in CollectionsConfiguration
> when creating a Queue?
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> ср, 22 апр. 2020 г. в 11:09, Sebastian Sindelar <
> sebastian.sinde...@nexus-ips.de>:
>
> Hallo.
>
> Our application uses ignite to share data between different services. We
> habe a couple of caches und queues. Currently some of the caches are
> persisted using a second data region. This works fine. A new requiredment
> is to persist the items in the queues.
>
> Because queues always use the default data region I assumed if I enable
> persistence on that region the queue content should be persistet. But this
> works just for the caches and not the queues. The queues still lose its
> content if the cluster shuts down. The log shows that persistence is
> enabled on the default region.
>
> The thing is if I reset the cluster (deleting the ignite home folder) the
> queues get persistet fine. The problem is this means a lot of manual work
> when updating clients.
>
> I tried renaming the queues, but it didn't work.
>
>
>
> Kind regards
>
> *nexus / ag *
>
> *Sebastian Sindelar *
>
> Softwareentwicklung
>
> Tel.: +49 (0)561 942880
> Fax: +49 (0)561 9428877
> Support: +49 (0)561 9428899
>
> E-Mail: sebastian.sinde...@nexus-ips.de
>
> NEXUS / IPS GmbH
> Standort Kassel
> Mendelssohn-Bartholdy-Str. 17
> D-34134 Kassel
> www.nexus-ag.de | www.nexus-ips.de
>
>
> ---
> NEXUS / IPS GmbH, Irmastraße 1, D-78166 Donaueschingen
> Eingetragene Gesellschaft beim Amtsgericht Freiburg i.Br., HRB 602014
> Geschäftsführer: Stefan Born, Dirk Hübner
>
> ---
>
> Rechtlicher Hinweis:
> Diese E-Mail samt Anhängen könnte vertrauliche/rechtlich geschützte
> Informationen enthalten.
> Wenn Sie nicht der richtige Empfänger sind, informieren Sie bitte sofort
> den Absender und löschen
> diese E-Mail samt Anhängen. Das unerlaubte Kopieren oder Weitergeben
> dieser E-Mail ist nicht gestattet.
>
> Datenschutzhinweis:
> Bitte beachten Sie unsere Grundsätze der Datenverarbeitung in unserer
> Datenschutzerklärung:
> https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung
>
> Legal notice:
> This message and any attachment(s) may contain confidential/privileged
> information. If you are not the
> intended recipient, please email or telephone the sender and delete this
> message and any attachment(s) from
> your system. Any unauthorised copying, disclosure or distribution of the
> material in this email is strictly forbidden.
>
> Privacy Policy:
> Please note our 

Re: Group eviction in progress for a very long time

[Ilya Kasnacheev]

Hello!

Can you search the previous logs for any exceptions coming from Ignite
internals? Maybe something did indeed break down.

Regards,
-- 
Ilya Kasnacheev


пн, 4 мая 2020 г. в 16:22, 18624049226 <18624049...@163.com>:

> Hi Ilya,
>
> It's hard to say why rebalancing is not over yet, judging from this log
> portion alone, but "Group eviction in progress" would probably mean that
> some partitions were successfully rebalanced and they may now be removed
> from "old" backup node.
>
> That's how I understand it,According to the visor, there are more backups
> than the primary ones.
>
>
> Are you getting the same #'s of partitions every time, or do they change?
> Any irregularities in your logs other than this one?
>
> At present, the cluster is in the state of no access, so the current log
> scrolls out the following information about every 2 seconds, basically the
> same.
>
> We have 5 nodes, 2 of which have the following logs.
>
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 3 мая 2020 г. в 04:49, 18624049226 <18624049...@163.com>:
>
>> Hi community,
>>
>> Two nodes are added to a running cluster with persistence enabled, after
>> several days, it seems that the rebalancing is not over. The log output the
>> following information about every 2 minutes:
>>
>> why?How to end the rebalancing?
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Eviction in progress [permits=0, threads=4, groups=4,
>> remainingPartsToEvict=293]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
>> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
>> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CO, grpId=64322847,
>> remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
>> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Partitions have been scheduled for eviction: [grpId=64322847,
>> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
>> 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
>> [2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>> ^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
>> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
>> ^-- CPU [cur=18%, avg=18.24%, GC=0%]
>> ^-- PageMemory [pages=36986314]
>> ^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
>> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
>> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
>> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
>> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
>> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
>> ^-- Ignite persistence [used=134048MB]
>> ^-- sysMemPlc region [used=0MB]
>> ^-- default region [used=134047MB]
>> ^-- metastoreMemPlc region [used=0MB]
>> ^-- TxLog region [used=0MB]
>> ^-- Outbound messages queue [size=0]
>> ^-- Public thread pool [active=0, idle=0, qSize=0]
>> ^-- System thread pool [active=16, idle=0, qSize=3]
>> ^-- Striped thread pool [active=0, idle=16, qSize=0]
>> [2020-04-30T21:58:49,329][INFO 
>> ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
>> Skipping checkpoint (no pages were modified)
>> [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
>> checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason=
>> 'timeout']
>> [2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>> ^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
>> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
>> ^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
>> ^-- PageMemory [pages=36986314]
>> ^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
>> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
>> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
>> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
>> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
>> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
>> ^-- Ignite persistence [used=134048MB]
>> ^-- sysMemPlc region [used=0MB]
>> ^-- default region [used=134047MB]
>> ^-- metastoreMemPlc region [used=0MB]
>> ^-- TxLog region [used=0MB]
>> ^-- Outbound messages queue [size=0]
>> ^-- Public thread pool [active=0, idle=0, qSize=0]
>> ^-- System thread pool [active=6, idle=10, qSize=0]
>> ^-- Striped thread pool [active=0, idle=16, 

AW: Enabling default persistence on existing ignite cluster

[Sebastian Sindelar]

Hallo.

Thanks for the suggestion. Just tried it and it worked.
Never thought about the group names because I thought they had just to do with 
partition placement.


Regards,

nexus / ag

Sebastian Sindelar

Softwareentwicklung

Tel.: +49 (0)561 942880
Fax: +49 (0)561 9428877
Support: +49 (0)561 9428899

E-Mail: sebastian.sinde...@nexus-ips.de 

NEXUS / IPS GmbH
Standort Kassel
Mendelssohn-Bartholdy-Str. 17
D-34134 Kassel
www.nexus-ag.de | www.nexus-ips.de

[cid:image001.jpg@01D6222B.A80C12F0]

---
NEXUS / IPS GmbH, Irmastraße 1, D-78166 Donaueschingen
Eingetragene Gesellschaft beim Amtsgericht Freiburg i.Br., HRB 602014
Geschäftsführer: Stefan Born, Dirk Hübner
---

Rechtlicher Hinweis:
Diese E-Mail samt Anhängen könnte vertrauliche/rechtlich geschützte 
Informationen enthalten.
Wenn Sie nicht der richtige Empfänger sind, informieren Sie bitte sofort den 
Absender und löschen
diese E-Mail samt Anhängen. Das unerlaubte Kopieren oder Weitergeben dieser 
E-Mail ist nicht gestattet.

Datenschutzhinweis:
Bitte beachten Sie unsere Grundsätze der Datenverarbeitung in unserer 
Datenschutzerklärung:
https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung

Legal notice:
This message and any attachment(s) may contain confidential/privileged 
information. If you are not the
intended recipient, please email or telephone the sender and delete this 
message and any attachment(s) from
your system. Any unauthorised copying, disclosure or distribution of the 
material in this email is strictly forbidden.

Privacy Policy:
Please note our principles of data processing in our privacy statement:
https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung


Von: Ilya Kasnacheev 
Gesendet: Freitag, 24. April 2020 15:02
An: user@ignite.apache.org
Betreff: Re: Enabling default persistence on existing ignite cluster

Hello!

Have you tried supplying different group name in CollectionsConfiguration when 
creating a Queue?

Regards,
--
Ilya Kasnacheev


ср, 22 апр. 2020 г. в 11:09, Sebastian Sindelar 
mailto:sebastian.sinde...@nexus-ips.de>>:

Hallo.
Our application uses ignite to share data between different services. We habe a 
couple of caches und queues. Currently some of the caches are persisted using a 
second data region. This works fine. A new requiredment is to persist the items 
in the queues.
Because queues always use the default data region I assumed if I enable 
persistence on that region the queue content should be persistet. But this 
works just for the caches and not the queues. The queues still lose its content 
if the cluster shuts down. The log shows that persistence is enabled on the 
default region.
The thing is if I reset the cluster (deleting the ignite home folder) the 
queues get persistet fine. The problem is this means a lot of manual work when 
updating clients.
I tried renaming the queues, but it didn't work.

Kind regards

nexus / ag

Sebastian Sindelar

Softwareentwicklung

Tel.: +49 (0)561 942880
Fax: +49 (0)561 9428877
Support: +49 (0)561 9428899

E-Mail: sebastian.sinde...@nexus-ips.de 

NEXUS / IPS GmbH
Standort Kassel
Mendelssohn-Bartholdy-Str. 17
D-34134 Kassel
www.nexus-ag.de | 
www.nexus-ips.de

[cid:image001.jpg@01D6222B.A80C12F0]

---
NEXUS / IPS GmbH, Irmastraße 1, D-78166 Donaueschingen
Eingetragene Gesellschaft beim Amtsgericht Freiburg i.Br., HRB 602014
Geschäftsführer: Stefan Born, Dirk Hübner
---

Rechtlicher Hinweis:
Diese E-Mail samt Anhängen könnte vertrauliche/rechtlich geschützte 
Informationen enthalten.
Wenn Sie nicht der richtige Empfänger sind, informieren Sie bitte sofort den 
Absender und löschen
diese E-Mail samt Anhängen. Das unerlaubte Kopieren oder Weitergeben dieser 
E-Mail ist nicht gestattet.

Datenschutzhinweis:
Bitte beachten Sie unsere Grundsätze der Datenverarbeitung in unserer 
Datenschutzerklärung:
https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung

Legal notice:
This message and any attachment(s) may contain confidential/privileged 
information. If you are not the
intended recipient, please email or telephone the sender and delete this 
message and any attachment(s) from
your system. Any unauthorised copying, disclosure or distribution of the 
material in this email is strictly forbidden.

Privacy Policy:
Please note our principles of data processing in our privacy statement:
https://www.nexus-ibh.de/unternehmen/datenschutzerklaerung

Re: Query timeout

[Ilya Kasnacheev]

Hello!

Turns out, "?queryTimeout=" will only be available in 2.9:
https://issues.apache.org/jira/browse/IGNITE-12462

However, you can already use (Statement)ps.setQueryTimeout(1), it works in
your case as well (checked on 2.8).

Regards,
-- 
Ilya Kasnacheev


вт, 24 мар. 2020 г. в 11:05, breathem :

> Hi, Taras.
> We try to connect to server via NetBeans 8.2 for development purposes.
> In Services tab we add Ignite 2.8.0 JDBC driver (ignite-core-2.8.0.jar) and
> choose org.apache.ignite.IgniteJdbcThinDriver.
>
> To reproduce long running query we create 2 tables:
> create table if not exists TEST
> (kField long not null, vField varchar(100) not null, rField varchar(100)
> not
> null, primary key (kField))
> WITH "template=REPLICATED, cache_name=TEST, key_type=KTEST,
> value_type=VTEST"
>
> create table if not exists TEST1
> (kField long not null, vField varchar(100) not null, rField varchar(100)
> not
> null, primary key (kField))
> WITH "template=REPLICATED, cache_name=TEST1, key_type=KTEST1,
> value_type=VTEST1"
>
> Then fill this tables with 1 000 000 random data rows, eg
> int size = 1_000_000;
>
> IgniteCallable clb = new IgniteCallable()
> {
>   @IgniteInstanceResource
>   Ignite ignite;
>
>   @Override
>   public Integer call() throws Exception
>   {
> IgniteCache test =
> ignite.cache("TEST").withKeepBinary();
>
> if (test.size(CachePeekMode.ALL) > 0)
> {
>   test.clear();
> }
>
> IgniteBinary bin = ignite.binary();
> long t;
>
> for (int i = 0; i < size; i++)
> {
>   BinaryObjectBuilder k = bin.builder("KTEST");
>   BinaryObjectBuilder v = bin.builder("VTEST");
>
>   t = System.currentTimeMillis();
>
>   k.setField("kField", t);
>   v.setField("vField", "I am value string " + t);
>   v.setField("rField", randomString(100));
>
>   test.putAsync(k.build(), v.build());
> }
>
> return null;
>   }
> };
>
> Then connect to server via NetBeans with
> jdbc:ignite:thin://192.168.1.138:10800?queryTimeout=1
>
> Then make long running query:
> select t.kField, t.vField, t1.vField from test t inner join test1 t1 on
> t.vField = t1.vField;
>
> This query in our case is executed ~73 sec and not cancelled.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Group eviction in progress for a very long time

[18624049226]

Hi Ilya,

It's hard to say why rebalancing is not over yet, judging from this 
log portion alone, but "Group eviction in progress" would probably 
mean that some partitions were successfully rebalanced and they may 
now be removed from "old" backup node.
That's how I understand it,According to the visor, there are more 
backups than the primary ones.


Are you getting the same #'s of partitions every time, or do they 
change? Any irregularities in your logs other than this one?


At present, the cluster is in the state of no access, so the current log 
scrolls out the following information about every 2 seconds, basically 
the same.


We have 5 nodes, 2 of which have the following logs.




Regards,
--
Ilya Kasnacheev


вс, 3 мая 2020 г. в 04:49, 18624049226 <18624049...@163.com 
>:


Hi community,

Two nodes are added to a running cluster with persistence enabled,
after several days, it seems that the rebalancing is not over. The
log output the following information about every 2 minutes:

why?How to end the rebalancing?

[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Eviction in progress [permits=0, threads=4, groups=4,
remainingPartsToEvict=293]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CO, grpId=64322847,
remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Partitions have been scheduled for eviction: [grpId=64322847,
grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583,
591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T21:58:32,288][INFO][grid-timeout-worker-#39][IgniteKernal]

Metrics for local node (to disable set 'metricsLogFrequency'to 0)
^-- Node [id=9ebed61d, uptime=4days, 19:34:36.448]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=18%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=16, idle=0, qSize=3]
^-- Striped thread pool [active=0, idle=16, qSize=0]

[2020-04-30T21:58:49,329][INFO][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
Skipping checkpoint (no pages were modified)
[checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms,
reason='timeout']
[2020-04-30T21:59:32,297][INFO][grid-timeout-worker-#39][IgniteKernal]

Metrics for local node (to disable set 'metricsLogFrequency'to 0)
^-- Node [id=9ebed61d, uptime=4days, 19:35:36.457]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
Eviction in progress [permits=1, threads=4, groups=4,
remainingPartsToEvict=291]

Re: REPLICATED CACHE ISSUES WHEN UPDATED THROUGH JDBC THIN CLIENT - 2.7.6

[Ilya Kasnacheev]

Hello!

This is very strange, I would assume that either you have three clusters
(each node has formed a cluster of its own) or that you have three
different caches.

Can you please elaborate what you are seeing? Can you provide logs from one
of the nodes? How do you update this cache?

Regards,
-- 
Ilya Kasnacheev


сб, 2 мая 2020 г. в 21:08, VeenaMithare :

> HI,
>
> I am running a 3 server node cluster on Ignite 2.7.6 .
>
> 1.  I have started a cache ( REPLICATED ) with native persistence enabled
> with the below configuration .
>
>
>cacheConfiguration.setRebalanceMode(CacheRebalanceMode.SYNC);
> cacheConfiguration.setCacheMode(CacheMode.REPLICATED);
>
>
> cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>
> I can see that my cache has been started using this mode :
> Started cache [name=SQL_PUBLIC_ABCDEF, id=-1243894080,
> memoryPolicyName=null, mode=REPLICATED, atomicity=TRANSACTIONAL,
> backups=2147483647, mvcc=false], encryptionEnabled=false]
>
> If i connect to one of the servers through JDBC thin client ( DBeaver ) and
> update a record in this cache - the change is visible only on the node
> where
> the update was done. The other two nodes show the old value.
>
> Kindly guide if this is a bug or I am missing some configuration .
>
> 2. Kindly note that if I create the same cache with CACHE_MODE :
> PARTITIONED
> with BACKUPS =2 - an update from the DBeaver is propogated to all the
> nodes.
>
> Isnt the configuration in 1 and 2 the same ? Why does it show different
> results ?
>
> regards,
> Veena.
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Group eviction in progress for a very long time

[Ilya Kasnacheev]

Hello!

It's hard to say why rebalancing is not over yet, judging from this log
portion alone, but "Group eviction in progress" would probably mean that
some partitions were successfully rebalanced and they may now be removed
from "old" backup node.

Are you getting the same #'s of partitions every time, or do they change?
Any irregularities in your logs other than this one?

Regards,
-- 
Ilya Kasnacheev


вс, 3 мая 2020 г. в 04:49, 18624049226 <18624049...@163.com>:

> Hi community,
>
> Two nodes are added to a running cluster with persistence enabled, after
> several days, it seems that the rebalancing is not over. The log output the
> following information about every 2 minutes:
>
> why?How to end the rebalancing?
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Eviction in progress [permits=0, threads=4, groups=4,
> remainingPartsToEvict=293]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO, grpId=64322847,
> remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Partitions have been scheduled for eviction: [grpId=64322847,
> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
> 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
> [2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
> ^-- CPU [cur=18%, avg=18.24%, GC=0%]
> ^-- PageMemory [pages=36986314]
> ^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=134048MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=134047MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=16, idle=0, qSize=3]
> ^-- Striped thread pool [active=0, idle=16, qSize=0]
> [2020-04-30T21:58:49,329][INFO 
> ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
> Skipping checkpoint (no pages were modified)
> [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
> checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason=
> 'timeout']
> [2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
> ^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
> ^-- PageMemory [pages=36986314]
> ^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=134048MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=134047MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=6, idle=10, qSize=0]
> ^-- Striped thread pool [active=0, idle=16, qSize=0]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Eviction in progress [permits=1, threads=4, groups=4,
> remainingPartsToEvict=291]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> 

Re: Ignite Upgrade 2.5.6 to 2.8.0 - data loss

[Ilya Kasnacheev]

Hello!

I would like to point out that this is an Ignite user discussion mailing
list and not a support channel; so you should not reherely your operation
on getting swift responses here.

Also, there were no such Apache Ignite release as 2.5.6.

With respect to the error that you have provided, I think it may arise if
you have different sets of indexes declared in your QueryEntity
configuration between restarts. However, this specific error seems unknown.

Can you share your QueryEntity configuration? I also recommend removing
indexes from it, one by one, to find out which one was blocking you, and
then trying to work around it.

Regards,
-- 
Ilya Kasnacheev


сб, 2 мая 2020 г. в 22:52, Sriveena Mattaparthi <
sriveena.mattapar...@ekaplus.com>:

> We have been receiving quick support till now on any queries raised.
>
> Request to please guide us in this also ASAP as we have production upgrade
> planned in couple of days.
>
>
>
> Thanks in advance.
>
>
>
> Regards,
>
> Sriveena
>
>
>
> *From:* Sriveena Mattaparthi
> *Sent:* 03 May 2020 01:21
> *To:* user@ignite.apache.org
> *Subject:* Ignite Upgrade 2.5.6 to 2.8.0 - data loss
> *Importance:* High
>
>
>
> Hi,
>
>
>
> Thank for the support so far. We have been using ignite from 4 years and
> we have no issues so far.
>
> But now, We are facing critical issue in data loss while upgrading ignite
> from 2.5.6 to 2.8.0.
>
>
>
> We tried copying the work folder from 2.5.6 to 2.8.0 and start 2.8.0
> ignite server.
>
> But the server is going down with below error.
>
> Please help us ASAP.
>
>
>
> [05:15:22] Ignite node started OK (id=233c2412)
>
> [05:15:22] Topology snapshot [ver=1, locNode=233c2412, servers=1,
> clients=0, state=INACTIVE, CPUs=8, offheap=22.0GB, heap=14.0GB]
>
> [05:15:22]   ^-- Baseline [id=0, size=1, online=1, offline=0]
>
> [05:15:22]   ^-- All baseline nodes are online, will start auto-activation
>
> [05:15:23,679][SEVERE][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
> Failed to activate node components
> [nodeId=233c2412-5483-41ad-b27f-61cbacd1610c, client=false,
> topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
>
> java.lang.IllegalStateException: Duplicate key
>
> at
> org.apache.ignite.cache.QueryEntity.checkIndexes(QueryEntity.java:233)
>
> at
> org.apache.ignite.cache.QueryEntity.makePatch(QueryEntity.java:184)
>
> at
> org.apache.ignite.internal.processors.query.QuerySchema.makePatch(QuerySchema.java:114)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1972)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$null$6a5b31b9$1(GridCacheProcessor.java:1830)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCachesIfPossible$9(GridCacheProcessor.java:1771)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$926b6886$1(GridCacheProcessor.java:1827)
>
> at
> org.apache.ignite.internal.util.IgniteUtils.lambda$null$1(IgniteUtils.java:11138)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Suppressed: java.lang.IllegalStateException: Duplicate key
>
> ... 12 more
>
> Suppressed: java.lang.IllegalStateException: Duplicate key
>
> ... 12 more
>
> Suppressed: java.lang.IllegalStateException: Duplicate key
>
> ... 12 more
>
> Suppressed: java.lang.IllegalStateException: Duplicate key
>
> ... 12 more
>
> Suppressed: java.lang.IllegalStateException: Duplicate key
>
> at
> org.apache.ignite.cache.QueryEntity.checkIndexes(QueryEntity.java:233)
>
> at
> org.apache.ignite.cache.QueryEntity.makePatch(QueryEntity.java:184)
>
> at
> org.apache.ignite.internal.processors.query.QuerySchema.makePatch(QuerySchema.java:114)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1972)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$null$6a5b31b9$1(GridCacheProcessor.java:1830)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCachesIfPossible$9(GridCacheProcessor.java:1771)
>
>
>
>
>
> Thanks,
>
> Sriveena
> “Confidentiality Notice: The contents of this email message and any
> attachments are intended solely for the addressee(s) and may contain
> confidential and/or privileged information and may be legally protected
> from disclosure. If you are not the intended recipient of this message or
> their agent, 

Re: Random2LruPageEvictionTracker causing hanging in our integration tests

[Ilya Kasnacheev]

Hello!

I can see that it did see one of this messages, but why do you think it is
stuck? For what I'm seeing, it has promptly un-stuck in one second:
2020-05-01 12:27:23.386 PDT priority='WARN' thread='Test worker'
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager@127
- Page-based evictions started. Consider increasing 'maxSize' on Data
Region configuration: DefaultDataRegion
2020-05-01 12:27:23.387 PDT priority='WARN' thread='Test worker'
org.apache.ignite.internal.processors.cache.persistence.evict.Random2LruPageEvictionTracker@127
- Too many attempts to choose data page: 5000
2020-05-01 12:27:24.374 PDT priority='TRACE' thread='Test worker'
org.apache.ignite.internal.processors.cache.GridCacheMapEntry@91 -
markObsolete0 [key=KeyCacheObjectImpl [part=-1, val=reentrant-select
count(t."id") from "sample" t where 1=1 AND t."other"=? AND
t."id"=?|bbb291c45a1-5c55-4bf1-86b3-dd8e0c410695, hasValBytes=true],
entry=29960847, clear=false]

I would maybe expect a IgniteOOM error after this, but it does not seem to
materialize. Why do you think it is stuck for good?
However, this message is throttled so it can be spending any amount of time
in that code.

I can see that you have partition map exchange at the same time, so that's
where you might be waiting.

I have also found https://issues.apache.org/jira/browse/IGNITE-12510, is
there a chance it is relevant? How large are your entries? I also think
512M is a tiny size for data region, try increasing it.

Regards,
-- 
Ilya Kasnacheev


пт, 1 мая 2020 г. в 23:08, scottmf :

> out.multipart-aa
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-aa>
>
> out.multipart-ab
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-ab>
>
> out.multipart-ac
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-ac>
>
> out.multipart-ad
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-ad>
>
> out.multipart-ae
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-ae>
>
> out.multipart-af
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-af>
>
> out.multipart-ag
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1632/out.multipart-ag>
>
>
> hi Ilya, I turned on debugging for ignite and dumped the output into a
> multipart set of files that i've attached.  Let me know if you need anymore
> info.  If needed I can try to reproduce this in a generic setting but that
> will take time.
>
> since 5MB is the limit, i had to upload the files in 5MB chunks.  To
> assemble them, put them into a directory then run 'cat * > file.gz'
>
> Answers to your questions:
>
> > Do you see any "Too many attempts to choose data page" or "Too many
> failed
> > attempts to evict page" messages in your logs?
>
> See output file
>
> > How large are your data regions
>
> we only use the default data region with default settings - 512MB
>
> > how many caches do they have?
>
> maybe 20ish?
>
> > I would expect that behavior if eviction can't find any page to evict, if
> > all data pages are evicted already and only metadata pages remain, ones
> > that cannot be evicted.
>
> Could you elaborate on this or point me to any docs?
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: JDBC Connection Pooling

[Ilya Kasnacheev]

Hello!

In streaming mode, INSERTs should not block, so you may reuse the
connection. Still, it would not hurt to benchmark both approaches, if you
can.

Regards,
-- 
Ilya Kasnacheev


сб, 2 мая 2020 г. в 01:42, narges saleh :

> Hi All,
>  If I use client connection configuration to set the number of threads for
> a JDBC connection,  and use the connection with multiple insert statements
> (with streaming set to true), to multiple different caches, are the inserts
> stacked because the connection is shared? Should I create multiple JDBC
> connections and switch them across the statements?
> thanks.
>
> On Thu, Apr 16, 2020 at 12:33 PM Evgenii Zhuravlev <
> e.zhuravlev...@gmail.com> wrote:
>
>> As I said, if you use only DataStreamer, without jdbc, just a plain,
>> key-value IgniteDataStreamer, then, you should have only one instance per
>> cache. It will give you the better performance. This one streamer can be
>> used from multiple threads.
>>
>> чт, 16 апр. 2020 г. в 09:54, narges saleh :
>>
>>> I am sorry for mixing these two up.
>>> I am asking if I were to use binaryobject builder with datastreamer in
>>> place on jdbc connection would/should I create a pool of the streamer
>>> objects. From your answers, the answer seems to be yes. thank you.
>>>
>>> On Thu, Apr 16, 2020 at 9:07 AM Evgenii Zhuravlev <
>>> e.zhuravlev...@gmail.com> wrote:
>>>
 You said that you use Binary Object Builder, so, I thought that you use
 key value API and data streamers. I don't really understand now you use
 BinaryObjectBuilder with thin JDBC client.

 >What if I have a persistent connection that sends data continuously?
 Should I hold on the instance of the streamer (for a particular cache), or
 recreate a new one once a new load of data arrives?
 If you're loading data continuously, it makes sense to store data
 streamer instance somewhere and just reuse it, avoiding recreating it each
 time. I

 >Are you saying have the data streamed to the streamer via multiple
 connections, across multiple threads?
 If you use just a simple IgniteDataStreamer, you can use it from
 multiple threads(use addData from multiple threads) to increase the
 throughput.
 Evgenii

 ср, 15 апр. 2020 г. в 12:07, narges saleh :

> Hello Evgenii,
>
> I am not sure what you mean by reuse a data streamer from multiple
> threads.  I have data being constantly "streamed" to the streamer via a
> connection. Are you saying have the data streamed to the streamer via
> multiple connections, across multiple threads?
> What if I have a persistent connection that sends data continuously?
> Should I hold on the instance of the streamer (for a particular cache), or
> recreate a new one once a new load of data arrives?
>
> On Wed, Apr 15, 2020 at 1:17 PM Evgenii Zhuravlev <
> e.zhuravlev...@gmail.com> wrote:
>
>> > Should I create a pool of data streamers (a few for each cache)?
>> If you use just KV API, it's better to have only one data streamer
>> per cache and reuse it from multiple threads - it will give you the best
>> performance.
>>
>> Evgenii
>>
>> ср, 15 апр. 2020 г. в 04:53, narges saleh :
>>
>>> Please note that in my case, the streamers are running on the server
>>> side (as part of different services).
>>>
>>> On Wed, Apr 15, 2020 at 6:46 AM narges saleh 
>>> wrote:
>>>
 So, in effect, I'll be having a pool of streamers, right?
 Would this still be the case if I am using BinaryObjectBuilder to
 build objects to stream the data to a few caches? Should I create a 
 pool of
 data streamers (a few for each cache)?
 I don't want to have to create a new object builder and data
 streamer if I am inserting to the same cache over and over.

 On Tue, Apr 14, 2020 at 11:56 AM Evgenii Zhuravlev <
 e.zhuravlev...@gmail.com> wrote:

> For each connection, on node side will be created its own
> datastreamer. I think it makes sense to try pooling for data load, 
> but you
> will need to measure everything, since the pool size depends on the 
> lot of
> things
>
> вт, 14 апр. 2020 г. в 07:31, narges saleh :
>
>> Yes, Evgenii.
>>
>> On Mon, Apr 13, 2020 at 10:06 PM Evgenii Zhuravlev <
>> e.zhuravlev...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Do you use STREAMING MODE for thin JDBC driver?
>>>
>>> Evgenii
>>>
>>> пн, 13 апр. 2020 г. в 19:33, narges saleh >> >:
>>>
 Thanks Alex. I will study the links you provided.

 I read somewhere that jdbc datasource is available via Ignite
 JDBC, (which should provide connection pooling).

 On Mon, Apr 13, 

Re: Ignite.net caching technique / recommendations

[Ilya Kasnacheev]

Hello!

Why not have different caches for these? You can control which nodes hold
data for a given cache by mean of node filters.

It is not recommended to distribute data on this basis, since load will
still be uneven between nodes (much more posts than users, for example).

It is recommended to partition data on ID or some other evenly-distributed
value.

Regards,
-- 
Ilya Kasnacheev


сб, 2 мая 2020 г. в 12:16, Sudhir Patil :

> Hello,
>
> Use case is in given cluster of machines (e.g. 3 to 4), I want to store
> all cache item with key name users on one machine, another cache item with
> key name posts on different machine etc. Thus is to distribute the load of
> cache storage hoping to get performance benefit, better cache management,
> etc..
>
> Any recommendations i.e. pros cons on this approach is welcome for me...
>
> On Thursday, April 30, 2020, Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> I think it is doable to some extent, but I would like to hear why you
>> want that.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 30 апр. 2020 г. в 13:21, Sudhir Patil :
>>
>>> Hi All,
>>>
>>> Use case is to store a cache item each on specific machine within
>>> cluster.
>>>
>>> What are recommendations for this?
>>> Is it beneficial / performant one?
>>> Is it getting away from distributed caching fundamentals & it benefits?
>>> If yes, what are reasons behind it ???
>>>
>>> Regards,
>>> SP
>>>
>>