FW: Concurrent threads updating the same cache item

[Ari Erev]

Hello,

I post an answer I received from Andrei, which completes and extends his first 
answer on the subject.

Regards,
Ari

From: Andrei Aleksandrov 
Sent: Wednesday, September 11, 2019 4:02 PM
To: Ari Erev 
Subject: Re: Concurrent threads updating the same cache item


Hi,

Ignite will execute the logic of compute task in separate Runnable instance 
that will be executed in a single thread of 
https://apacheignite.readme.io/docs/thread-pools#section-public-pool.

So one compute will be executed in one thread on the same node.

But in case if you start several distributed computes tasks and they will work 
with the same caches then yes it's possible that there will be the race between 
them and you will require to synchronize the operations.

It fully depends on your logic - should you lock on a “read then update” 
scenario or not.

BR,
Andrei


From: Andrei Aleksandrov 

Sent: Thursday, September 5, 2019 6:47 PM
To: user@ignite.apache.org
Subject: Re: Concurrent threads updating the same cache item


Hi,

When you start the compute task then it (and code from there) will be executed 
on every server (that were chosen) consistently in the single thread.

But if you will try to broadcast the same task on several servers then race 
between different tasks on different servers is possible in this case. To avoid 
it you can try to use the transactions API (to provide atomicity of 
get-update-put operation) or distributed locks for updated keys:

https://apacheignite.readme.io/docs/transactions
https://apacheignite.readme.io/docs/distributed-locks

BR,
Andrei
9/5/2019 5:41 PM, Ari Erev пишет:
Hello,

This question is related to the question by “humenius”,  with subject: “Race 
condition and conflicts during cache modifications?” – but I believe it is a 
simpler case…

When code is run on an Ignite server node (such as from a distributed compute, 
or service)  – is all access to a specific object (object with a specific key) 
– done from one (the same) specific thread?

The reason I am asking is this:
Some examples of Ignite code on GitHub and the ones that are embedded in White 
Paper articles from GridGain contain the following conceptual code 
(incrementing a value in the cache).

My_Object  obj = cache.get(key);

  obj.increment_value();

 cache.put(key, obj);


If this code is executed concurrently from more than one thread, there is a 
risk for inconsistency, as the new/incremented value may overwrite a cached 
value which is already different than it was at the time of the cache.get().
If so, should such code be synchronized (use some sort of lock)?

Thanks,
Ari


Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

Confidentiality: This communication and any attachments are intended for the 
above-named persons only and may be confidential and/or legally privileged. Any 
opinions expressed in this communication are not necessarily those of NICE 
Actimize. If this communication has come to you in error you must take no 
action based on it, nor must you copy or show it to anyone; please 
delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient shoul

RE: Connecting External Client to Containerized Cluster in Kubernetes

[vitalys]

We currently have working installation where our Ignite Server Nodes reside
on the standalone VMs and form a Cluster upon start by using shared
Configuration setup. Our Client application joins Cluster on-demand, loads
data into the Cluster and then kicks-off certain operations on the loaded
data.

We want to migrate Ignite Cluster Nodes from individual VMs to Kubernetes
and containerize them along the way. So our challenge is to have an existing
client join the Cluster while Cluster Nodes deployment platform is changed.

I have some reservations about REST client since we need to load significant
volumes of data in to the Cluster.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Data from multiple MySQL DBs

[Kurt Semba]

Hi Ilya,

Thanks for your suggestion. So what you are suggesting is:

  *   Run a Ignite single cluster which has the correct schemas
  *   Develop a tool which regularly pulls data from all these MySQL DBs and 
that uses the Ignite Data Streamer API to update the Caches in the cluster


  *   That way, all datasets from a specific schema & table from all MySQL DBs 
will end up in the same Ignite Cache and thus the data will be a UNION of all 
data from all MySQL DBs.
Kurt

From: Ilya Kasnacheev 
Sent: Wednesday, September 11, 2019 5:10 PM
To: user@ignite.apache.org
Subject: Re: Data from multiple MySQL DBs

Hello!

I think the best course will be to read all DBs in user code and use data 
streamer to populate Ignite.

Regards,
--
Ilya Kasnacheev


вт, 10 сент. 2019 г. в 18:01, Kurt Semba 
mailto:kse...@extremenetworks.com>>:
Hi all,

I need to sync data from multiple MySQL databases into Ignite.

All those MySQL databases follow the same schema / same tables but obviously 
contain different data. They are separate instances of the same DB and we want 
to pull all that data into Ignite to have a central store to query against.

What would be the best strategy to solve this?

Create a dedicated Cache for each DB instance and then find a way to create a 
UNION SQL query over all those Caches?
Find a way to define all the MySQL DBs as data sources in the Spring XML 
cluster config file and hope that Ignite pulls the same type of data from all 
DBs?

I’m open for suggestions 😊
Thanks
Kurt

RE: Connecting External Client to Containerized Cluster in Kubernetes

[Anton]

Hello Vitaly,

Unfortunately, I am not aware of any working solution right now and it looks 
that the ticket related to special discovery mechanism for external cluster 
members is not in work currently: 
https://issues.apache.org/jira/browse/IGNITE-4161

You might find this mailing thread useful if you want to try to experiment with 
this, there are some hints and clues oh how this might work: 
http://apache-ignite-users.70518.x6.nabble.com/Kubernetes-Access-Ignite-Cluster-Externally-td20532.html

However, the mentioned issue and thread are about server nodes and thick 
clients. What kind of operations are you expecting to perform on k8s Ignite 
cluster, do you really need the external client to be full-scale cluster 
member? You still have thin clients and REST that will be definitely easier to 
connect and utilize.

I would also recommend reaching out to Ignite dev list and start another 
discussion there.

Regards,
Anton

Sent from Mail for Windows 10

From: vitalys
Sent: Thursday, September 5, 2019 10:02 PM
To: user@ignite.apache.org
Subject: Connecting External Client to Containerized Cluster in Kubernetes

Hi, 

We have setup an Apache Ignite Cluster in Kubernetes ( 1 Node Per pod ) and
when we run "ignitevisorcmd.sh" from inside the Kubernetes we can see
correct topology of the Cluster ( 3 nodes ).

Our task is to run Ignite Client from outside the Kubernetes. We setup our
client config to use "KubernetesIpFinder" :





But I am wondering how does Ignite Client know how to connect to Kubernetes? 
"TcpDiscoveryKubernetesIpFinder" contains multiple properties like
"registerAddresses","masterURL" , "namespace", "accountToken". Do we need
to define these properties in our Client configuration?

And in General, what's the right approach in setting up Client connection to
the Ignite Cluster that runs in Kubernetes?

thank you in advance.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Perfomance drop

[Nikita Kuzin]

Hello!

How we can debug it to find cache name that causes it?
Some extra logging or metrics?

Скачайте Outlook для Android


From: Ilya Kasnacheev 
Sent: Wednesday, September 11, 2019 4:08:18 PM
To: user@ignite.apache.org 
Subject: Re: Perfomance drop

Hello!

I can see that there's huge unevenness with regards to striped pool usage: 
around 80% of load end up on just two striped threads,
sys-stripe-11-#12%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415% and
sys-stripe-9-#10%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415%. That's 
2/3 of all utilization of all theads.

I imagine there is a lot of contention around a very narrow set of keys. Maybe 
you update a few keys continuously which will cause locking issues. Please 
check your data partitioning and affinity.

Regards,
--
Ilya Kasnacheev


чт, 5 сент. 2019 г. в 18:12, Кузин Никита (Nikita Kuzin) 
mailto:nku...@iitdgroup.ru>>:
jfr from ignite is in attachment

_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

От: Ilya Kasnacheev 
mailto:ilya.kasnach...@gmail.com>>
Отправлено: 2 сентября 2019 г. 14:08
Кому: user@ignite.apache.org 
mailto:user@ignite.apache.org>>
Тема: Re: Perfomance drop

Hello!

If GC is fine next step is examining thread dumps and logs when a long 
operation is taking place. You can trigger it yourself by calling 
IgniteUtils.dumpThreads() in a test setting.

Regards,
--
Ilya Kasnacheev


пн, 2 сент. 2019 г. в 11:11, Кузин Никита (Nikita Kuzin) 
mailto:nku...@iitdgroup.ru>>:
Hello!

gc log for 3 days of working ignite.

There are 58 GCs with duration 100-200ms and 9 GCs with duration of 200-300ms

_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

От: Кузин Никита (Nikita Kuzin) 
mailto:nku...@iitdgroup.ru>>
Отправлено: 31 августа 2019 г. 10:26
Кому: user@ignite.apache.org 
mailto:user@ignite.apache.org>>
Тема: RE: Perfomance drop

Hello!

We collected GC log, it seems to be ok.

Long duration operations are very random events, we will try to collect.

Can the caches' eviction processes has influence on cache reading? (use of same 
threads)
_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

От: Ilya Kasnacheev 
mailto:ilya.kasnach...@gmail.com>>
Отправлено: 30 августа 2019 г. 15:49
Кому: user@ignite.apache.org 
mailto:user@ignite.apache.org>>
Тема: Re: Perfomance drop

Hello!

Have you tried collecting thread dumps/JFR/etc while such an operation is 
waiting? It could provide some clues.

Do you have GC log collected? Any long GC pauses?

Regards,
--
Ilya Kasnacheev


пт, 30 авг. 2019 г. в 14:43, Кузин Никита (Nikita Kuzin) 
mailto:nku...@iitdgroup.ru>>:
Hi!

We have application that goes to ignite node (only one node in grid).
Application and ignite are located on one machine.

And we noticed that sometimes that cache located at ignite is very slow: we 
execute method igniteCache.containsKey(key) and it can lasts 900ms, size of 
cache less than 1 entries
This cache is used mostly for read.

Persistence is off
All of JVM recommendations is applied
OS: windows server 2012 R2

_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

Re: Ignite Map-Reduce Deadlocking, Running in SYS pool

[Chris Software]

I respectfully suggest you update your public documentation ASAP, as people
(like my team) are developing new software now, using IGFS, expecting that
it will continue to be supported.  Please don't wait until you release
2.8.

On Wed, Sep 11, 2019 at 11:07 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> https://issues.apache.org/jira/browse/IGNITE-11942
>
> Vote already succeeded, there shouldn't be IGFS in 2.8.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 11 сент. 2019 г. в 16:52, Chris Software :
>
>> Ilya,
>>
>> When will an official announcement and support-drop schedule be made
>> about dropping IGFS?
>>
>> Thank you,
>>
>> Chris
>>
>> On Tue, Aug 27, 2019 at 1:47 PM Chris Software 
>> wrote:
>>
>>> I see.  Thank you.
>>>
>>> On Tue, Aug 27, 2019 at 12:30 PM Ilya Kasnacheev <
>>> ilya.kasnach...@gmail.com> wrote:
>>>
 Hello!

 This looks like a mistake. However, we're going to drop IGFS so the fix
 is unlikely to be expected.

 The recommended practical approach is to increase number of threads in
 system thread pool to large value.

 Regards,
 --
 Ilya Kasnacheev


 вт, 27 авг. 2019 г. в 00:34, Chris Software >>> >:

> Hello,
>
> I am working on a project and we have run into two related problems
> while doing Map_Reduce on Ignite Filesystem Cache.
>
> We were originally on Ignite 2.6 but upgraded to 2.7.5 in an
> unsuccessful bid to resolve the problem.
>
> We have a deadlock in our map-reduce process and have reproduced it at
> https://github.com/csteppp/ignite-deadlock-issue in
> https://github.com/csteppp/ignite-deadlock-issue/blob/master/src/test/java/testignite/MapReduceIgniteTest.java.
>
>
> Basically, if you run the test (mvn test) it will deadlock and hang.
> We have two IgfsTasks created and have set the SYS threadpool to size 2 
> for
> demonstration purposes.  Each IgfsTask sleeps and then writes to a file.
> This causes a deadlock because:
> 1.  The IgfsTask is run in the SYS pool.
> 2.  The Igfs write action uses a separate thread in the SYS pool
> 3.  Then if there are no empty threads available, the whole system
> hangs.
>
> First, shouldn't executeAsync execute the task in the PUBLIC pool?
> Using the SYS pool seems unnecessarily risky, as we found it actually 
> *locks
> up an entire cluster of many ignite nodes *when it deadlocks.  How do
> I get it to use the PUBLIC pool?  Also, since it is using the SYS pool, it
> actually seems to execute this on the client.  This is not obvious in this
> test, but in my real cluster of 30 nodes, the client seems to be doing 
> this
> work, which is a problem.
>
> Second, is it bad form to open a file within a map-reduce?  Even using
> the public pool will not solve the inherent deadlock here--that one thread
> is depending on another thread in the same thread pool.  That's an 
> inherent
> risk.  In our real process we open the file because we are performing file
> transformations in the IgfsTask, and then writing the results out to temp
> files in the cluster.  In the end, we collate all the temp files.  Is 
> there
> a better approach, or a safe way to open a file and write to it from 
> within
> a reduce?
>
> Thank you for your time!
>
> Chris
>
>
>

Re: Users named "ignite" cannot change their password?

[Andrei Aleksandrov]

Hi,

In addition to Ilia answer, please read next page:

https://apacheignite-sql.readme.io/docs/create-user#section-description

BR,
Andrei

9/4/2019 11:47 AM, Ilya Kasnacheev пишет:

Hello!

User names are case sensitive and bound by SQL rules, i.e.:

ALTER USER "ignite" WITH PASSWORD 'test123';

Regards,
--
Ilya Kasnacheev


ср, 4 сент. 2019 г. в 05:14, liyuj <18624049...@163.com 
>:


Hi,

Execute the following statement:

ALTER USER ignite WITH PASSWORD 'test123';

The error message is as follows:

SQL 错误 [1] [5]: Operation failed
[nodeId=88b03674-04a4-44cb-bd42-8f2ed1e980ff,
opId=5b656f9fc61-7cd6fa68-ee67-49d4-aee8-60958f5584af, err=class

org.apache.ignite.internal.processors.authentication.UserManagementException:

User doesn't exist [userName=IGNITE]]

The password of other users can be changed.

jdk1.8.0,gridgain-community-8.7.6

Re: Grid failure on frequent cache creation/destroying

[Abhishek Gupta (BLOOMBERG/ 731 LEX)]

Thanks Ilya! Good to have confirmation.


From: ilya.kasnach...@gmail.com At: 09/11/19 11:57:16To:  Abhishek Gupta 
(BLOOMBERG/ 731 LEX ) 
Cc:  user@ignite.apache.org
Subject: Re: Grid failure on frequent cache creation/destroying

Hello!

I'm afraid that's https://issues.apache.org/jira/browse/IGNITE-12013

There's a maillist thread attached.

Regards,
-- 
Ilya Kasnacheev


вт, 10 сент. 2019 г. в 00:33, Abhishek Gupta (BLOOMBERG/ 731 LEX) 
:

Hello,
 We have a grid of 6 nodes with a main cache. We noticed something 
interesting today where while the regular ingestion was on with the mainCache. 
We have an operational tool that created and destroys a cache 
(tempCacheByExplorerApp) using REST API on each of the 6 nodes. While doing 
this today, all the nodes hit a critical error and died. Attached is a log 
snippet from one of the nodes when this happened. 

This issue seems like the one described @ 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-2-7-0-server-node-null-pointer-exception-td28899.html
  but there's isn't any topology change as such happening here, just cache 
creation/destruction.


Appreciate your help.

-Abhi

Re: Grid failure on frequent cache creation/destroying

[Ilya Kasnacheev]

Hello!

I'm afraid that's https://issues.apache.org/jira/browse/IGNITE-12013

There's a maillist thread attached.

Regards,
-- 
Ilya Kasnacheev


вт, 10 сент. 2019 г. в 00:33, Abhishek Gupta (BLOOMBERG/ 731 LEX) <
agupta...@bloomberg.net>:

> Hello,
> We have a grid of 6 nodes with a main cache. We noticed something
> interesting today where while the regular ingestion was on with the
> mainCache. We have an operational tool that created and destroys a cache
> (tempCacheByExplorerApp) using REST API on each of the 6 nodes. While doing
> this today, all the nodes hit a critical error and died. Attached is a log
> snippet from one of the nodes when this happened.
>
> This issue seems like the one described @
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-2-7-0-server-node-null-pointer-exception-td28899.html
> but there's isn't any topology change as such happening here, just cache
> creation/destruction.
>
>
> Appreciate your help.
>
> -Abhi
>
>

Re: Data region LRU offheap algo not working

[rick_tem]

Hi, it's in the log 2.7.5...Thanks for any help here!
Thanks,
Rick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data region LRU offheap algo not working

[Michaelikus]

what is tour ignite version?



-
Russian telegram Ignite chat at https://t.me/RU_Ignite
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteCache.invoke deadlock example

[Ilya Kasnacheev]

Hello!

Yes!

You should always lock keys in the same order (they are usually sorted for
that matter).

Regards,
-- 
Ilya Kasnacheev


вт, 10 сент. 2019 г. в 22:39, Evangelos Morakis :

> Hi Andrei,
> Thanks a lot for your reply. Relatively to the dummy code I provided I
> take it you mean the following ?
>
> Cache Person personsCache=...
>
> *THREAD 1*
>
> personsCache.invoke("*personKey1*", new EntryProcessor Void>() {
>
>
> @Override public Object process(MutableEntryentry, Object... 
> args) {
>
>
> Person person= entry.getValue();
>
> entry.setValue(person.setOccupation(“foo”));
>
>
> *Some additional calculation here involving key* *personKey2 **<- - -some 
> other key?*
>
>
> return null;
>
>  }
>
>  });
>
>
> *THREAD 2*
>
> personsCache.invoke("*personKey2*", new EntryProcessor Void>() {
>
>
> @Override public Object process(MutableEntryentry, Object... 
> args) {
>
>
> Person person= entry.getValue();
>
> entry.setValue(person.setOccupation(“foo”));
>
>
> *Some additional calculation here involving key* *personKey1 <- - -some other 
> key?*
>
>
> return null;
>
>  }
>
>  });
>
>
>  The 2 threads WILL deadlock in that situation.
>
> So, could those  keys of the person cache (for example) that I have marked
> above, be the ones that are designated in the documentation (and hence in
> your explanation as well) as “other keys” ?
>
> Thanks
> Evangelos Morakis
>
>
> On 9 Sep 2019, at 20:40, Andrei Aleksandrov 
> wrote:
>
> Hello,
>
> When you use the entry processor then you lock only provided key. So when
> you tries to work with *other keys* (different from provided one) that
> are being processed somewhere in other threads then deadlock is possible
> because other thread can take lock on these *other keys* and wait for
> provided one. Otherwise, entry processor will wait for these *other keys*.
> It's typical deadlock.
>
> Sorry, I will not provide the example but hopes that my explanation is
> clear.
>
> BR,
> Andrei
> 9/7/2019 6:31 PM, Evangelos Morakis пишет:
>
>
> Dear igniters,
>
> I would like to elicit your expert
> advice in regards to how ignite differentiates
> on the use of a call to: 1)IgniteCompute.affinityRun(...)
> and
> 2)IgniteCache.invoke(...)
>
>  as far as dead locks are concerned. According to the documentation the
> main difference is that method 2 above, operates within a lock.
> Specifically the doc quotes:
> “EntryProcessors are executed atomically within a lock on the given cache
> key.”
> Now it even comes with a warning that is meant to show how it is supposed
> to be used (or conversely NOT to be used):
> “You should not access *other keys* from within the EntryProcessor logic
> as it may cause a deadlock.”
> But this phrase “*other keys*” to what kind of keys does it refer to?
> The remaining keys of the passed in cache?  For e.g. :
>  Assume a persons cache...
> Cache Person personsCache=...
>
> personsCache.invoke("personKey", new EntryProcessor() {
>
>  @Override
> public Object process(MutableEntry entry, Object... args) 
> {
>
>  Person person= entry.getValue();
>
>   entry.setValue(person.setOccupation(“foo”));
>
> return null;
>
>  }
>
>  });
>
> In other words can someone provide an example based on the above dummy
> code  that would make invoke deadlock so that I could get an understanding
> of what the documentation refers to?
>
> Thanks
>
> Evangelos Morakis
>
>

Re: Data from multiple MySQL DBs

[Ilya Kasnacheev]

Hello!

I think the best course will be to read all DBs in user code and use data
streamer to populate Ignite.

Regards,
-- 
Ilya Kasnacheev


вт, 10 сент. 2019 г. в 18:01, Kurt Semba :

> Hi all,
>
>
>
> I need to sync data from multiple MySQL databases into Ignite.
>
>
>
> All those MySQL databases follow the same schema / same tables but
> obviously contain different data. They are separate instances of the same
> DB and we want to pull all that data into Ignite to have a central store to
> query against.
>
>
>
> What would be the best strategy to solve this?
>
>
>
> Create a dedicated Cache for each DB instance and then find a way to
> create a UNION SQL query over all those Caches?
>
> Find a way to define all the MySQL DBs as data sources in the Spring XML
> cluster config file and hope that Ignite pulls the same type of data from
> all DBs?
>
>
>
> I’m open for suggestions 😊
>
> Thanks
>
> Kurt
>

Re: Ignite Map-Reduce Deadlocking, Running in SYS pool

[Ilya Kasnacheev]

Hello!

https://issues.apache.org/jira/browse/IGNITE-11942

Vote already succeeded, there shouldn't be IGFS in 2.8.

Regards,
-- 
Ilya Kasnacheev


ср, 11 сент. 2019 г. в 16:52, Chris Software :

> Ilya,
>
> When will an official announcement and support-drop schedule be made about
> dropping IGFS?
>
> Thank you,
>
> Chris
>
> On Tue, Aug 27, 2019 at 1:47 PM Chris Software 
> wrote:
>
>> I see.  Thank you.
>>
>> On Tue, Aug 27, 2019 at 12:30 PM Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> This looks like a mistake. However, we're going to drop IGFS so the fix
>>> is unlikely to be expected.
>>>
>>> The recommended practical approach is to increase number of threads in
>>> system thread pool to large value.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> вт, 27 авг. 2019 г. в 00:34, Chris Software :
>>>
 Hello,

 I am working on a project and we have run into two related problems
 while doing Map_Reduce on Ignite Filesystem Cache.

 We were originally on Ignite 2.6 but upgraded to 2.7.5 in an
 unsuccessful bid to resolve the problem.

 We have a deadlock in our map-reduce process and have reproduced it at
 https://github.com/csteppp/ignite-deadlock-issue in
 https://github.com/csteppp/ignite-deadlock-issue/blob/master/src/test/java/testignite/MapReduceIgniteTest.java.


 Basically, if you run the test (mvn test) it will deadlock and hang.
 We have two IgfsTasks created and have set the SYS threadpool to size 2 for
 demonstration purposes.  Each IgfsTask sleeps and then writes to a file.
 This causes a deadlock because:
 1.  The IgfsTask is run in the SYS pool.
 2.  The Igfs write action uses a separate thread in the SYS pool
 3.  Then if there are no empty threads available, the whole system
 hangs.

 First, shouldn't executeAsync execute the task in the PUBLIC pool?
 Using the SYS pool seems unnecessarily risky, as we found it actually 
 *locks
 up an entire cluster of many ignite nodes *when it deadlocks.  How do
 I get it to use the PUBLIC pool?  Also, since it is using the SYS pool, it
 actually seems to execute this on the client.  This is not obvious in this
 test, but in my real cluster of 30 nodes, the client seems to be doing this
 work, which is a problem.

 Second, is it bad form to open a file within a map-reduce?  Even using
 the public pool will not solve the inherent deadlock here--that one thread
 is depending on another thread in the same thread pool.  That's an inherent
 risk.  In our real process we open the file because we are performing file
 transformations in the IgfsTask, and then writing the results out to temp
 files in the cluster.  In the end, we collate all the temp files.  Is there
 a better approach, or a safe way to open a file and write to it from within
 a reduce?

 Thank you for your time!

 Chris

Re: Iterator for keys only ?

[Ilya Kasnacheev]

Hello!

You can also use ScanQuery with Transformer. Note that all key-value pairs
will still be deserialized. If your key fits in index, it would indeed be
better to use SQL.

Regards,
-- 
Ilya Kasnacheev


ср, 11 сент. 2019 г. в 15:47, Mikael :

> Hi!
>
> What is the best way to iterate over the keys only from a cache ? I
> don't want the values because these are long text strings so will slow
> down the iterator a lot.
>
> Pretty much the Ignite way of doing map.keySet().iterator();
>
> Mikael
>
>
>

Re: write-behind cache performance bottleneck

[Ilya Kasnacheev]

Hello!

Unfortunately we would need more information to answer either question.

What's `normal write of the cache block`? Can you show some code?

Regards,
-- 
Ilya Kasnacheev


пн, 9 сент. 2019 г. в 05:42, liyuj <18624049...@163.com>:

> If only put operations, with very few updates, would changing the
> writeBehindCoalescing parameter to false help improve the problem?
>
> 在 2019/9/9 上午10:31, liyuj 写道:
> > Hi community,
> >
> > When the write-behind cache is enabled, will the normal write of the
> > cache block in the process of batch writing to the third-party database?
> >
> > If the write performance of third party database is poor, does Ignite
> > have any optimization strategy?
>
>

Re: NearCaches

[Ilya Kasnacheev]

Hello!

Near cache only holds a subset of data from cache, so it's just a single
cache any way.

You can use CachePeekMode.NEAR to check if entries are cached near.

Regards,
-- 
Ilya Kasnacheev


вс, 8 сент. 2019 г. в 21:47, Кузин Никита (Nikita Kuzin) <
nku...@iitdgroup.ru>:

> Hello!
>
> I created cache on server node (
>
> ignite.getOrCreateCache(cacheConfiguration)
>
> ) and near cache on client node
> (
>
> ignite.createNearCache(nearConf)
>
> )
> , their names are the same.
> How to check that near cache is used? (in MBeans on client node i don't
> see any special "near" cache)
>
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
>

Re: How to execute an sql on a specific remote server?

[Ilya Kasnacheev]

Hello!

You can set `local=true` to execute completely local requests, and you can
connect your JDBC to NX with this option.

Regards,
-- 
Ilya Kasnacheev


вс, 8 сент. 2019 г. в 20:44, livelace :

> Hi, Andrew. I found this topic in Google.
>
> I have a case where:
>
> 1. I have two nodes (all caches are replicated) N1 and N2, NX is a client
> node .
> 2. N2 fills with small portions of data (relatively frequent).
> 3. N2 has relatively small RAM and CPU.
> 4. N1 has huge volume of RAM and many CPU cores.
> 5. NX has huge volume of RAM and many CPU cores and powerful GPU.
> 6. Network link between N1 and N2 - 1Gbps.
> 7. Network link between N2 and NX - 10Gbps.
>
> hence I need an option to query through a more powerful node (CPU, RAM,
> Network speed) in Replicated mode, not a random option.
>
> Am I able achieve this in 2.7.5 ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite Map-Reduce Deadlocking, Running in SYS pool

[Chris Software]

Ilya,

When will an official announcement and support-drop schedule be made about
dropping IGFS?

Thank you,

Chris

On Tue, Aug 27, 2019 at 1:47 PM Chris Software 
wrote:

> I see.  Thank you.
>
> On Tue, Aug 27, 2019 at 12:30 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>> Hello!
>>
>> This looks like a mistake. However, we're going to drop IGFS so the fix
>> is unlikely to be expected.
>>
>> The recommended practical approach is to increase number of threads in
>> system thread pool to large value.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> вт, 27 авг. 2019 г. в 00:34, Chris Software :
>>
>>> Hello,
>>>
>>> I am working on a project and we have run into two related problems
>>> while doing Map_Reduce on Ignite Filesystem Cache.
>>>
>>> We were originally on Ignite 2.6 but upgraded to 2.7.5 in an
>>> unsuccessful bid to resolve the problem.
>>>
>>> We have a deadlock in our map-reduce process and have reproduced it at
>>> https://github.com/csteppp/ignite-deadlock-issue in
>>> https://github.com/csteppp/ignite-deadlock-issue/blob/master/src/test/java/testignite/MapReduceIgniteTest.java.
>>>
>>>
>>> Basically, if you run the test (mvn test) it will deadlock and hang.  We
>>> have two IgfsTasks created and have set the SYS threadpool to size 2 for
>>> demonstration purposes.  Each IgfsTask sleeps and then writes to a file.
>>> This causes a deadlock because:
>>> 1.  The IgfsTask is run in the SYS pool.
>>> 2.  The Igfs write action uses a separate thread in the SYS pool
>>> 3.  Then if there are no empty threads available, the whole system
>>> hangs.
>>>
>>> First, shouldn't executeAsync execute the task in the PUBLIC pool?
>>> Using the SYS pool seems unnecessarily risky, as we found it actually *locks
>>> up an entire cluster of many ignite nodes *when it deadlocks.  How do I
>>> get it to use the PUBLIC pool?  Also, since it is using the SYS pool, it
>>> actually seems to execute this on the client.  This is not obvious in this
>>> test, but in my real cluster of 30 nodes, the client seems to be doing this
>>> work, which is a problem.
>>>
>>> Second, is it bad form to open a file within a map-reduce?  Even using
>>> the public pool will not solve the inherent deadlock here--that one thread
>>> is depending on another thread in the same thread pool.  That's an inherent
>>> risk.  In our real process we open the file because we are performing file
>>> transformations in the IgfsTask, and then writing the results out to temp
>>> files in the cluster.  In the end, we collate all the temp files.  Is there
>>> a better approach, or a safe way to open a file and write to it from within
>>> a reduce?
>>>
>>> Thank you for your time!
>>>
>>> Chris
>>>
>>>
>>>

Re: Perfomance drop

[Ilya Kasnacheev]

Hello!

I can see that there's huge unevenness with regards to striped pool usage:
around 80% of load end up on just two striped threads,
sys-stripe-11-#12%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415% and
sys-stripe-9-#10%ITLXCLKAP-server425b5ae8-214a-4ff8-b1a4-943a1513c415%.
That's 2/3 of all utilization of all theads.

I imagine there is a lot of contention around a very narrow set of keys.
Maybe you update a few keys continuously which will cause locking issues.
Please check your data partitioning and affinity.

Regards,
-- 
Ilya Kasnacheev


чт, 5 сент. 2019 г. в 18:12, Кузин Никита (Nikita Kuzin) <
nku...@iitdgroup.ru>:

> jfr from ignite is in attachment
>
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> --
> *От:* Ilya Kasnacheev 
> *Отправлено:* 2 сентября 2019 г. 14:08
> *Кому:* user@ignite.apache.org 
> *Тема:* Re: Perfomance drop
>
> Hello!
>
> If GC is fine next step is examining thread dumps and logs when a long
> operation is taking place. You can trigger it yourself by calling
> IgniteUtils.dumpThreads() in a test setting.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 2 сент. 2019 г. в 11:11, Кузин Никита (Nikita Kuzin) <
> nku...@iitdgroup.ru>:
>
> Hello!
>
> gc log for 3 days of working ignite.
>
> There are 58 GCs with duration 100-200ms and 9 GCs with duration of
> 200-300ms
>
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> --
> *От:* Кузин Никита (Nikita Kuzin) 
> *Отправлено:* 31 августа 2019 г. 10:26
> *Кому:* user@ignite.apache.org 
> *Тема:* RE: Perfomance drop
>
> Hello!
>
> We collected GC log, it seems to be ok.
>
> Long duration operations are very random events, we will try to collect.
>
> Can the caches' eviction processes has influence on cache reading? (use of
> same threads)
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> --
> *От:* Ilya Kasnacheev 
> *Отправлено:* 30 августа 2019 г. 15:49
> *Кому:* user@ignite.apache.org 
> *Тема:* Re: Perfomance drop
>
> Hello!
>
> Have you tried collecting thread dumps/JFR/etc while such an operation is
> waiting? It could provide some clues.
>
> Do you have GC log collected? Any long GC pauses?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 авг. 2019 г. в 14:43, Кузин Никита (Nikita Kuzin) <
> nku...@iitdgroup.ru>:
>
> Hi!
>
> We have application that goes to ignite node (only one node in grid).
> Application and ignite are located on one machine.
>
> And we noticed that sometimes that cache located at ignite is very slow:
> we execute method igniteCache.containsKey(key) and it can lasts 900ms, size
> of cache less than 1 entries
> This cache is used mostly for read.
>
> Persistence is off
> All of JVM recommendations is applied
> OS: windows server 2012 R2
>
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
>
>

RE: Iterator for keys only ?

[Alexandr Shapkin]

Hello!

You can try to use SqlFieldsQuery with “select _key from [CacheTable]” for that.

From: Mikael
Sent: Wednesday, September 11, 2019 3:47 PM
To: user@ignite.apache.org
Subject: Iterator for keys only ?

Hi!

What is the best way to iterate over the keys only from a cache ? I 
don't want the values because these are long text strings so will slow 
down the iterator a lot.

Pretty much the Ignite way of doing map.keySet().iterator();

Mikael

Iterator for keys only ?

[Mikael]

Hi!

What is the best way to iterate over the keys only from a cache ? I 
don't want the values because these are long text strings so will slow 
down the iterator a lot.


Pretty much the Ignite way of doing map.keySet().iterator();

Mikael

RE: exceptions in Ignite node when a thin client process ends

[Alexandr Shapkin]

Hello,

There is an open ticket https://issues.apache.org/jira/browse/IGNITE-12032 with 
more details.
In general, It’s ok to ignore this error. 
You could try to follow Lev’s recommendations with explicit connection closing.

From: Matan Levy
Sent: Thursday, August 1, 2019 9:33 AM
To: user@ignite.apache.org
Subject: exceptions in Ignite node when a thin client process ends

Hi, 

In my application I am using ODBC + Java thin clients to get data from the 
cache(stores in Ignite nodes). 
whenever a process using this thin clients is finished, I am getting in the 
node logs this exception : 

*[07:45:19,559][SEVERE][grid-nio-worker-client-listener-1-#30][ClientListenerProcessor]
 
Failed to process selector key [s 
es=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker 
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=8192 cap=8192 
], super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, 
bytesRcvd0=0, bytesSent0=0, select=true, super=GridWo 
rker [name=grid-nio-worker-client-listener-1, igniteInstanceName=null, 
finished=false, heartbeatTs=1564289118230, hashCo 
de=1829856117, interrupted=false, 
runner=grid-nio-worker-client-listener-1-#30]]], writeBuf=null, 
readBuf=null, inRecove 
ry=null, outRecovery=null, super=GridNioSessionImpl 
[locAddr=/0:0:0:0:0:0:0:1:10800, rmtAddr=/0:0:0:0:0:0:0:1:63697, cre 
ateTime=1564289116225, closeTime=0, bytesSent=1346, bytesRcvd=588, 
bytesSent0=0, bytesRcvd0=0, sndSchedTime=156428911623 
5, lastSndTime=1564289116235, lastRcvTime=1564289116235, readsPaused=false, 
filterChain=FilterChain[filters=[GridNioAsyn 
cNotifyFilter, GridNioCodecFilter [parser=ClientListenerBufferedParser, 
directMode=false]], accepted=true, markedForClos 
e=false]]] 
java.io.IOException: An existing connection was forcibly closed by the 
remote host 
        at sun.nio.ch.SocketDispatcher.read0(Native Method) 
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) 
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) 
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) 
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) 
        at 
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:11
 
04) 
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNi
 
oServer.java:2389) 
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:215
 
6) 
        at 
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
 
        at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) 
        at java.lang.Thread.run(Thread.java:748)* 



is there a soemthing I can do to avoid those exceptions? 
Thanks!  

Re: exceptions in Ignite node when a thin client process ends

[Lev555]

Hi, @matanlevy
This exception shows that your database connection wasn't correctly closed.
It probably should be fixed by executing `connection.close()` method before
process using thin clients finished.
Could you please try this and tell the result?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Complex Event Processing Using Ignite Streaming

[Ignite Enthusiast]

I am trying to build a CEP where a Complex event needs to be generated on a set 
of input events (3 Chassis hot events) over a specified time window (10 
seconds, for eg) and I am trying to evaluate Apache Ignite for this.
Are there any examples of how to do Complex Event Processing using Ignite?  The 
following wiki page hardly seems to help.
https://apacheignite.readme.io/docs/streaming--cep

Also, how does Ignite CEP implementation compare with others ? (like Apache 
Spark, Apache Flink, etc)

Re: Ignition Start - Timeout if connection is unsuccessful

[Denis Mekhanikov]

Mahesh,

There is a TcpDiscoverySpi property, that defines this behaviour:
TcpDiscoverySpi#joinTimeout [1]. So, instead of calling
Ignition.start() with a special property, you can specify this timeout
in the configuration.
If a join attempt is failed and joinTimeout is already exceeded, then
the client will stop trying.
Note, that this timeout is not strict, and the connection may take
longer before it fails in case if TcpDiscoverySpi#reconnectDelay [2]
is set or each connection attempt takes a long time.

[1] 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.html#setJoinTimeout-long-
[2] 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.html#setReconnectDelay-int-

Denis

ср, 11 сент. 2019 г. в 09:30, Mahesh Renduchintala
:
>
> Hello
>
> We are currently using Ignition.Start to get the handle for thick client.
>
> >> ignite = Ignition.start(cfg);
>
> As I understand, this API is a blocking API, unless the connection is 
> successfully established.
>
> However, in some scenarios, where the thick client is unable to connect 
> properly, it is preferable to have a timeout option as specified below.
> >> ignite = Ignition.start(cfg,  timeout);
>
> is this already available today? If not, can you take it as an enhancement 
> request for 2.8.
>
> The reason why I ask is, in some scenarios, when a thick client comes up for 
> the very first time, we see thick client making an attempt to connect to 
> ignite servers almost in an infinite loop.
> Previously, I raised this infinite loop connecting issue before.
> http://apache-ignite-users.70518.x6.nabble.com/client-reconnect-working-td28570.html
>
> regards
> mahesh
>
>
>
>