Re: Ignite .NET and TcpDiscoveryStaticIpFinder failure

[camer314]

I created a new VM and it now seems to work ok, I have this message which
would indicate everything is connected (which would be correct, 3 nodes, 2
hosts and 6 CPU total)

H/N/C [hosts=2, nodes=3, CPUs=6]

Also btw, when creating the new VM I had to install Java, I chose
jdk-8u221-windows-x64 to match my working VM. When i tried to start
Apache.Ignite.exe I was faced with the error "Failed to load jvm.dll" even
though everything seemed correct. I them came across the below issue which
recommended installing VC++ 2010 redist however ptupitsyn suggested that was
not needed from 2.4+ and i am using the latest 2.7.6. It was only after
installing the VC++ 2010 libraries that Apache.Ignite.exe was able to run.

http://apache-ignite-users.70518.x6.nabble.com/Ignite-NET-2-2-crashes-on-startup-without-any-information-td17173i20.html



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite data loss

[krkumar24061...@gmail.com]

Thanks for your reply guys. 

I am not sure if I have really solved the problem but this is how i fixed
it.  Initially i was adding the nodes to the baseline topology through code.
I have removed that and now adding the nodes thru control.sh which case i am
not loosing any data. I do not know the root cause of the issue, but its a
temp. fix that's working for me for now.

The code that i am using to set the baseline topology is  as follows:

//  logger.info("Setting the baseline topology ...");
//  
engine.cluster().setBaselineTopology(engine.cluster().forServers().nodes());
//  logger.info("Baseline topology is set");

Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

number of way segments in wal

[KR Kumar]

Hi Guys - For some weird reason, files are filling up in wal folder and
right now I have around 65,000 files and occupies almost 4.5TB disk.
Ideally it should not be more than 10 files right? Also i have disabled wal
archiving. Why is this happening? and what am i missing?

Following is my configuration regarding wal configuration







































Thanx and Regards,
KR Kumar

Re: Failed to start near cache (a cache with the same name without near cache is already started)

[akorensh]

This happens because you created a cache without a near configuration first 
then tried to create it again with a near config.

for example:
  start a server node
   then do as follows:
NearCacheConfiguration nearCfg = new
NearCacheConfiguration<>();
IgniteCache cache1 = ignite.getOrCreateCache("myCache1");
IgniteCache cache2 = ignite.getOrCreateNearCache("myCache1",
nearCfg);   <-- exception thrown here because you already created this cache

In order to remedy this put the near cache config into the original cache
config.
i.e.
CacheConfiguration cfg = new
CacheConfiguration<>("myCache1");
NearCacheConfiguration nearCfg = new
NearCacheConfiguration<>();
cfg.setNearConfiguration(nearCfg);
IgniteCache cache1 = ignite.getOrCreateCache(cfg);



more info: https://www.gridgain.com/docs/latest/developers-guide/near-cache

Thanks, Alex





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Failed to start near cache (a cache with the same name without near cache is already started)

[Hemambara]

I followed the below post
http://apache-ignite-users.70518.x6.nabble.com/Failed-to-start-near-cache-a-cache-with-the-same-name-without-near-cache-is-already-started-td18322.html
but it did not solve my problem, hence requesting for help. 

I tried two options
a) Start ignite with below xml config and then call
this.ignite.getOrCreateNearCache("MyCache", nearCfg);
b) Start ignite with below xml config and then call
this.ignite.getOrCreateCache("MyNearCache"); and then
this.ignite.getOrCreateNearCache("MyNearCache", nearCfg);

Both are giving error like "Failed to start near cache (a cache with the
same name without near cache is already started)". AS per doc, I just need
to pass existing cache name and pass near cfg to get near cache for client
alone. But either way I am getting exception. Can you provide how can I
create a near cache ONLY for client, but not on server






























--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Improving Get operation performance

[Victor]

It's 500k unique gets, spread across multiple threads. Max i tried with 30
threads.

I cant use getAll for this usecase, since it is user driven and the user
will load one record at a time. In any case i expected event the single gets
to be pretty fast as well. Given the benchmark reference -
https://www.gridgain.com/resources/blog/apacher-ignitetm-and-apacher-cassandratm-benchmarks-power-in-memory-computing

There too the code seems to be using a single get. But the throughput is
massive for 32 threads its about 120k. So now i am not sure if the numbers
listed are accurate or was the test done in a controlled setting with
additional configurations.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Improving Get operation performance

[Mikael]

Hi!

The numbers sound very low, I run on hardware close to yours (3 nodes 
(X5660*5) and 1 client), and I get way more than 1500/sec, not sure how 
much, I will have to check, but as long as you do single get's there is 
not so much you can do, each get will be one roundtrip over the network, 
and with single get's latency can have a huge impact, I modified my code 
and most of the time I cache all get's over 100ms into a getAll and that 
makes a huge impact on performance.


Not that much to change in configuration, number of backups don't have 
much impact on reads (unless you do replicated of course)


I am not sure how the traffic works but if there is only one tcp 
connection to each node you will not have much use for more than 3 
threads I would think.


Did you read 500K unique entries or the same multiple times ?

Mikael

Den 2019-11-26 kl. 21:38, skrev Victor:

I am running some comparison tests (ignite vs cassandra) to check how to
improve the performance of 'get' operation. The data is fairly
straightforward. A simple Employee Object(10 odd fields), being stored as
BinaryObject in the cache as

IgniteCache empCache;

The cache is configured with, Write Sync Mode - FULL_SYNC, Atomicity -
TRANSACTIONAL, Backup - 1 & Persistence - Enabled

Cluster config, 3 server + 1 client node. Setup on 2 machine, server machine
(Intel(R) Xeon(R) CPU X5675  @ 3.07GHz) & client machine (Intel(R) Xeon(R)
CPU X5560  @ 2.80GHz).

Client has multiple threads(configurable) making concurrent 'get' calls.
Using 'get' on purpose due to use case requirements.

For about 500k request, i am getting a throughput of about 1500/sec. Given
all of the data is in off-heap with cache hits percentage = 100%.
Interestingly with Cassandra i am getting a similar performance, with key
Cache and limited row cache.
I've tried running with 10/20/30 threads, the performance is more/less same.

Letting the defaults for most of the Data configuration. For this test i
turned the persistence off. Ideally for get's it shouldn't really matter.
The performance is the same.


Data Regions Configured:
[19:35:58]   ^-- default [initSize=256.0 MiB, maxSize=14.1 GiB,
persistence=false]

Topology snapshot [ver=4, locNode=038f99b3, servers=3, clients=1,
state=ACTIVE, CPUs=40, offheap=42.0GB, heap=63.0GB]


Additionally ran top on both the machines to check if they are hitting the
resources,
-- Server
PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
14159 root  20   0   29.7g   3.2g  15216 S  10.3  4.5   1:35.69 java
14565 root  20   0   29.4g   2.9g  15224 S   8.3  4.2   1:33.41 java
13770 root  20   0   30.0g   2.9g  15184 S   6.3  4.2   1:36.99 java

- Client
3731 root  20   0   27.8g   1.1g  15304 S 136.5  1.5   2:39.16 java

As you can see everything is well under.

Frankly, i was expecting Ignite gets to be pretty fast, given all data is in
cache. Atleast looking at this test
https://www.gridgain.com/resources/blog/apacher-ignitetm-and-apacher-cassandratm-benchmarks-power-in-memory-computing


Planning to run one more test tomorrow with no-persistence and setting near
cache (on heap) to see if it helps.

Let me know if you guys see any obvious configurations that should be set.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Simple SELECT Query giving error

[digitalchemist]

##
@DAO
public String findByKey(String keyValue) {
List keyList = new ArrayList<>();
keyList = this.igniteJdbcTemplate.query(
"SELECT KEY FROM SCHEMANAME.TABLE_NAME WHERE KEY =?",new
Object[]{keyValue},(rs, i) -> rs.getString("KEY")
);
if(keyList != null && keyList.size() >0 )
return keyList.get(0);

return Strings.EMPTY;
}
##
@Service

@Override
@Transactional(transactionManager = "igniteTxManager")
public String findByKey(String key) {
String keyValue = pairRepo.findByKey(key);
return keyValue;
}
##
@Usage

void someMethod(String text1, String text2){
String origPair = text1+text2;
String flipPair = text2+text1;

dbResponse = currencyPairService.findByCurrencyPair(origPair);
LOG.info("{} Received from DB {}", origPair, dbResponse);

if(StringUtils.isEmpty(dbResponse)) {
flipResponse = currencyPairService.findByCurrencyPair(flipPair);
LOG.info("Trying flip value for {} received from DB {} ", 
flipPair,
flipResponse);
}
}
##
@Exception
org.springframework.jdbc.UncategorizedSQLException:
PreparedStatementCallback; uncategorized SQLException for SQL [SELECT KEY
FROM SCHEMANAME.PAIR WHERE KEY = ? ]; SQL state [5]; error code [1];
javax.cache.CacheException: Failed to execute map query on remote node
[nodeId=e0840889-2ca4-431b-a123-f7f3548b6609, errMsg=Failed to execute SQL
query. General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on lookup row: RowSimple [vals=['USDPLN', null, null,
'USDPLN', null]]"; SQL statement:
SELECT
__Z0.KEY __C0_0
FROM SCHEMANAME.TABLE_NAME __Z0
WHERE __Z0.KEY = ?1 [5-197]]; nested exception is java.sql.SQLException:
javax.cache.CacheException: Failed to execute map query on remote node
[nodeId=e0840889-2ca4-431b-a123-f7f3548b6609, errMsg=Failed to execute SQL
query. General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on lookup row: RowSimple [vals=['USDPLN', null, null,
'USDPLN', null]]"; SQL statement:
SELECT
__Z0.KEY __C0_0
FROM SCHEMANAME.TABLE_NAME __Z0
WHERE __Z0.KEY = ?1 [5-197]]
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89)
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at
org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1442)
at
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:632)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:668)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:699)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:711)
at
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:762)
at com.xx.xxx.xx.Service.someMethod(ServiceDAO.java:47)

##

I am just Posting a SELECT request. The interesting part is if you see

dbResponse = currencyPairService.findByCurrencyPair(origPair);
LOG.info("{} Received from DB {}", origPair, dbResponse);

if(StringUtils.isEmpty(dbResponse)) {
flipResponse = currencyPairService.findByCurrencyPair(flipPair);
LOG.info("Trying flip value for {} received from DB {} ", flipPair,
flipResponse); }

I call two times the service, the first one works fine, and second one
throws exception




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Improving Get operation performance

[Victor]

I am running some comparison tests (ignite vs cassandra) to check how to
improve the performance of 'get' operation. The data is fairly
straightforward. A simple Employee Object(10 odd fields), being stored as
BinaryObject in the cache as

IgniteCache empCache;

The cache is configured with, Write Sync Mode - FULL_SYNC, Atomicity -
TRANSACTIONAL, Backup - 1 & Persistence - Enabled

Cluster config, 3 server + 1 client node. Setup on 2 machine, server machine
(Intel(R) Xeon(R) CPU X5675  @ 3.07GHz) & client machine (Intel(R) Xeon(R)
CPU X5560  @ 2.80GHz).

Client has multiple threads(configurable) making concurrent 'get' calls.
Using 'get' on purpose due to use case requirements.

For about 500k request, i am getting a throughput of about 1500/sec. Given
all of the data is in off-heap with cache hits percentage = 100%.
Interestingly with Cassandra i am getting a similar performance, with key
Cache and limited row cache.
I've tried running with 10/20/30 threads, the performance is more/less same.

Letting the defaults for most of the Data configuration. For this test i
turned the persistence off. Ideally for get's it shouldn't really matter.
The performance is the same.


Data Regions Configured:
[19:35:58]   ^-- default [initSize=256.0 MiB, maxSize=14.1 GiB,
persistence=false]

Topology snapshot [ver=4, locNode=038f99b3, servers=3, clients=1,
state=ACTIVE, CPUs=40, offheap=42.0GB, heap=63.0GB]


Additionally ran top on both the machines to check if they are hitting the
resources,
-- Server
PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
14159 root  20   0   29.7g   3.2g  15216 S  10.3  4.5   1:35.69 java
14565 root  20   0   29.4g   2.9g  15224 S   8.3  4.2   1:33.41 java
13770 root  20   0   30.0g   2.9g  15184 S   6.3  4.2   1:36.99 java

- Client
3731 root  20   0   27.8g   1.1g  15304 S 136.5  1.5   2:39.16 java

As you can see everything is well under.

Frankly, i was expecting Ignite gets to be pretty fast, given all data is in
cache. Atleast looking at this test 
https://www.gridgain.com/resources/blog/apacher-ignitetm-and-apacher-cassandratm-benchmarks-power-in-memory-computing

  

Planning to run one more test tomorrow with no-persistence and setting near
cache (on heap) to see if it helps.

Let me know if you guys see any obvious configurations that should be set.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[niamin]

ignite_sql_error.ignite_sql_error

  

ARInvoiceRepository.java

  

Attached is the file containing statcktrace and the repository file. The
test case is trying to executed the 
getInvoicesForOIStatement method.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[Gaurav Bajaj]

Please send exception/error message you are getting and the query you are
trying to execute.


On Tue, Nov 26, 2019, 8:04 PM niamin  wrote:

> I found what I was doing wrong. I can now load/pre-fetch from 3rd party
> store
> to cache. It looks like the data is present in in-memory cache but also
> stored in the persistent store. However, when I run a native query
> configured on my SpringData Repository I get an error that the SQL table
> mentioned in the query is not found. The same query/api was working when
> persistent store was not enabled.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite native persistence

[niamin]

I found what I was doing wrong. I can now load/pre-fetch from 3rd party store
to cache. It looks like the data is present in in-memory cache but also
stored in the persistent store. However, when I run a native query
configured on my SpringData Repository I get an error that the SQL table
mentioned in the query is not found. The same query/api was working when
persistent store was not enabled.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite native persistence

[niamin]

Thanaks Denis. However, how do or should I preload the data to disk first? Or
as Ignite fetches data from the 3rd part datastore it is expected to
distribute between RAM and disk? Is there an example project that implements
this strategy? On my POC, I have SpringBoot project that includes the Ignite
Server and I was using SpringDataRepoository to configure the Queries which
were executed by Ignite server. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Invoke and serialization

[Ilya Kasnacheev]

Hello!

No.

Actually, invoke is not going to send your code over the network more than
once.

However, invoke is going to send all bound variables together with your
closure. If it is a lambda, it will often enclose "this", potentially
sending a lot of data to server.

To avoid this, consider always using an instance of inner static class (or
top-level class) with invoke. Then you can carefully pick which context it
needs to carry with it. All the things it can get on the remote node
without sending over, should be handled in that fashion.

And no, this won't cause problems. This is the right way to do.

Regards,
-- 
Ilya Kasnacheev


сб, 23 нояб. 2019 г. в 13:46, Mikael :

> Hi!
>
> I use invoke a lot on some caches and the code is not that small, would
> it be ok to put some of the code in a static method in some other class
> so I don''t need to send it over the network all the time, so the actual
> invoke is very small and just call a static method in some other class,
> it looks like it working fine and static method should be handled as
> transient I guess, but I though I would ask so I don't run into some
> nasty problem later.
>
> Mikael
>
>
>

Re: Ignite .NET and TcpDiscoveryStaticIpFinder failure

[Ilya Kasnacheev]

Hello!

You should make Discovery identical on both nodes, pointing to both nodes
via their public address.

[11:17:08,959][INFO][tcp-disco-sock-reader-#24][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/10.0.2.5:53509, rmtPort=53509]
[11:17:08,965][INFO][tcp-disco-sock-reader-#24][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/10.0.2.5:53509, rmtPort=53509

This would point to some problem. Can you enable verbose node? ignite.sh
-v, or running with -DIGNITE_QUIET=false.

Regards,
-- 
Ilya Kasnacheev


вт, 26 нояб. 2019 г. в 14:25, camer314 :

> I dont know about that, is that maybe because I had both addresses down as
> endpoints?
>
> I have modified app.config on both machines to only point to one another:
>
> /joinTimeout="6">
> 
> 
> 10.0.2.5:47500..47502
> 
> 
> /
>
> joinTimeout="12">
> 
> 
> 10.0.2.11:47500..47502
> 
> 
> 
>
>
> Now on the #11 machine the log is a lot cleaner, it just has these messages
> repeated:
>
> [11:17:08,959][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery
> accepted incoming connection [rmtAddr=/10.0.2.5, rmtPort=53509]
> [11:17:08,959][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery
> spawning a new thread for connection [rmtAddr=/10.0.2.5, rmtPort=53509]
> [11:17:08,959][INFO][tcp-disco-sock-reader-#24][TcpDiscoverySpi] Started
> serving remote node connection [rmtAddr=/10.0.2.5:53509, rmtPort=53509]
> [11:17:08,965][INFO][tcp-disco-sock-reader-#24][TcpDiscoverySpi] Finished
> serving remote node connection [rmtAddr=/10.0.2.5:53509, rmtPort=53509
>
> But on the #5 machine the log is entirely void of any detail at all:
>
> [11:14:56,534][INFO][main][IgniteKernal] Non-loopback local IPs:
> *10.0.2.5*,
> 10.0.75.1, fe80:0:0:0:0:5efe:a00:205%net2, fe80:0:0:0:0:5efe:a00:4b01%net7,
> fe80:0:0:0:35b8:b554:51fb:2fc1%eth5, fe80:0:0:0:812a:7edf:9502:2449%eth4
> [11:14:56,534][INFO][main][IgniteKernal] Enabled local MACs:
> 00E0, 000D3A18192C, 00155D02051F
> [11:14:56,571][INFO][main][TcpDiscoverySpi] Connection check threshold is
> calculated: 2000
> [11:14:56,588][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
> [port=47500, localHost=0.0.0.0/0.0.0.0,
> locNodeId=c77edb8f-56b8-405d-b2ba-66f585a30952]
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Issue with adding nested index dynamically

[Ilya Kasnacheev]

Hello!

I think you should move this discussion over to developers list.

Regards,
-- 
Ilya Kasnacheev


вт, 26 нояб. 2019 г. в 15:01, Hemambara :

> Can anyone please check above comment and update. Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

RE: Load cache from 3rd party persistence

[Nikita Kuzin]

Ok. thank you.

_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

От: Ilya Kasnacheev 
Отправлено: 26 ноября 2019 г. 14:04
Кому: user@ignite.apache.org 
Тема: Re: Load cache from 3rd party persistence

Hello!

Unfortunately it does not seem to me that this implementation is robust enough. 
If you compare it with
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache you will 
see that it lacks pooling, support for args, etc.

However, CacheJdbcBlobStore is not an internal class. You can surely 
extend/subclass it in your code, and then instantiate the extended version 
where you have your own loadCache.

This is what I recommend, instead of waiting for a "generic" loadCache in 
Ignite code.

Regards,
--
Ilya Kasnacheev


вс, 24 нояб. 2019 г. в 11:09, Кузин Никита (Nikita Kuzin) 
mailto:nku...@iitdgroup.ru>>:
also I found the issue: https://issues.apache.org/jira/browse/IGNITE-12078
with attached PR: https://github.com/apache/ignite/pull/6783

so it seems to be done (merged) very easy:)

_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

От: Кузин Никита (Nikita Kuzin)
Отправлено: 23 ноября 2019 г. 21:22
Кому: user@ignite.apache.org 
mailto:user@ignite.apache.org>>
Тема: Load cache from 3rd party persistence

Hello!

We want to use CacheJdbcBlobStore and everything seems fine, but it have empty 
implementation of loadCache method since it extends CacheStoreAdapter with 
empty implementation of loadCache.
How we can deal with it? Why CacheJdbcBlobStore doesn't implement this method?

_
С уважением, Никита Кузин
Ведущий программист-разработчик

 Интернейшнл АйТи Дистрибьюшн

e-mail: nku...@iitdgroup.ru
тел.: 84995021375 доб. 320
моб. тел.: 79260948887
115114, Москва, Дербеневская ул., 20-27

Re: Active tasks in cluster

[Prasad Bhalerao]

Does cancelling an active task cancels the threads being used internally
for query execution (threads in query pool) or cache put/delete operation?

Thanks,
Prasad

On Sat, Nov 23, 2019 at 8:26 PM Mikael  wrote:

> The whole idea with a future is that it is a small lightweight compact
> object, and you still have Igor's suggestion:
>
> Collection>> result =
> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>
> If you would have to implement a cluster wide listening mechanism in the
> futures you would add a terrible amount of overhead to it, and you would
> cause a lot of problems, what if you try to deserialize a future on a
> computer that is in another cluster, it may not even be an Ignite
> application, what if you deserialize a future that was created 2 years ago
> and the "id" of the future is now being reused for another future that has
> nothing to do with the original one, what if you deserialize it in a
> different cluster where that id is something different and not the same you
> submitted on the other cluster, yes all these things can be handled but
> once again you would turn a small nice simple object into a complex beast.
> Den 2019-11-23 kl. 15:00, skrev Prasad Bhalerao:
>
> By member you meant the output of the thread right?
>
> If yes, can we keep the member at centralised location like an internal
> cache?
> (May be we can provide the flag if turned on then the member can be
> broadcasted to whoever is listening to it or centralised cache location)
> I am considering future as a handle to the task which can be used to
> cancel the task even if the submitter node goes down.
>
>
>
> On Sat 23 Nov, 2019, 7:21 PM Mikael 
>> Well, the thread has to set a member in the future when it is finished,
>> if you serialize the future and send it somewhere else, how is the thread
>> going to be able to tell the future it had finished ?
>> Den 2019-11-23 kl. 14:31, skrev Prasad Bhalerao:
>>
>> Can someone please explain why Active task futures can't be serialized?
>>
>> If we loose the future then we don't have the way to cancel the active
>> task if it's taking too long. I think this is important feature.
>>
>>
>>
>> Thanks,
>> Prasad
>>
>> On Thu 21 Nov, 2019, 5:16 AM Denis Magda >
>>> I think that you should broadcast another task that will simply ask
>>> every node if taskA is already running or not every time the topology
>>> changes. If the response from all the nodes is empty then you need to
>>> reschedule taskA, otherwise, you will skip this procedure.
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Wed, Nov 20, 2019 at 9:28 AM Prasad Bhalerao <
>>> prasadbhalerao1...@gmail.com> wrote:
>>>
 That means I can't do this..

 Collection>> result =
 ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());

 Is there any way to get list futures of all active tasks running on all
 nodes of the cluster?

 Thanks,
 Prasad


 On Wed 20 Nov, 2019, 10:51 PM Mikael >>>
> Hi!
>
> No you cannot serialize any future object.
>
> Mikael
>
>
> Den 2019-11-20 kl. 17:59, skrev Prasad Bhalerao:
>
> Thank you for the suggestion. I will try this.
>
> I am thinking to store the task future object in a (replicated)cache
> against a jobId. If the node goes down as described in case (b), I will 
> get
> the task's future object from this  cache using a jobId and will invoke 
> the
> get method on it.
>
> But I am not sure about this approach, whether a future object can be
> serialized and send it over the wire to another node and deserialize it 
> and
> then invoke the get API on it.
>
> I will try to implement it tomorrow.
>
> Thanks,
> Prasad
>
>
> On Wed 20 Nov, 2019, 9:06 PM Igor Belyakov  wrote:
>
>> Hi Prasad,
>>
>> I think that you can use compute().broadcast() for collecting results
>> of activeTaskFutures() from all the nodes:
>> Collection>> result =
>> ignite.compute().broadcast(() -> ignite.compute().activeTaskFutures());
>>
>> Regards,
>> Igor Belyakov
>>
>> On Wed, Nov 20, 2019 at 5:27 PM Prasad Bhalerao <
>> prasadbhalerao1...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I want to get the active tasks running in cluster (tasks running on
>>> all nodes in cluster)
>>>
>>> IgniteCompute interface has method "activeTaskFutures" which
>>> returns tasks future for active tasks started on local node.
>>>
>>> Is there anyway to get the task futures for all active tasks of
>>> whole cluster?
>>>
>>> My use case is as follows.
>>>
>>> a) The node submits the affinity task and task runs on some other
>>> node in the cluster and the node which submitted the task dies.
>>>
>>> b) The node submits the affinity task and the task runs on the same
>>> node and the same node dies.
>>>
>>> The task 

Re: Issue with adding nested index dynamically

[Hemambara]

Can anyone please check above comment and update. Thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite .NET and TcpDiscoveryStaticIpFinder failure

[camer314]

I dont know about that, is that maybe because I had both addresses down as
endpoints?

I have modified app.config on both machines to only point to one another:

/   


10.0.2.5:47500..47502


/

   


10.0.2.11:47500..47502


   


Now on the #11 machine the log is a lot cleaner, it just has these messages
repeated:

[11:17:08,959][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery
accepted incoming connection [rmtAddr=/10.0.2.5, rmtPort=53509]
[11:17:08,959][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery
spawning a new thread for connection [rmtAddr=/10.0.2.5, rmtPort=53509]
[11:17:08,959][INFO][tcp-disco-sock-reader-#24][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/10.0.2.5:53509, rmtPort=53509]
[11:17:08,965][INFO][tcp-disco-sock-reader-#24][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/10.0.2.5:53509, rmtPort=53509

But on the #5 machine the log is entirely void of any detail at all:

[11:14:56,534][INFO][main][IgniteKernal] Non-loopback local IPs: *10.0.2.5*,
10.0.75.1, fe80:0:0:0:0:5efe:a00:205%net2, fe80:0:0:0:0:5efe:a00:4b01%net7,
fe80:0:0:0:35b8:b554:51fb:2fc1%eth5, fe80:0:0:0:812a:7edf:9502:2449%eth4
[11:14:56,534][INFO][main][IgniteKernal] Enabled local MACs:
00E0, 000D3A18192C, 00155D02051F
[11:14:56,571][INFO][main][TcpDiscoverySpi] Connection check threshold is
calculated: 2000
[11:14:56,588][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port
[port=47500, localHost=0.0.0.0/0.0.0.0,
locNodeId=c77edb8f-56b8-405d-b2ba-66f585a30952]






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Re: Unable to connect to a remote apache ignite node.

[Alexandr Shapkin]

Have you tried to remove ` joinTimeout` value?  From: Ilya KasnacheevSent: Tuesday, November 26, 2019 1:48 PMTo: user@ignite.apache.orgSubject: Re: Unable to connect to a remote apache ignite node. Hello! I think you have some kind of connectivity problems between two nodes so the second node can never finish the join process. I recommend trying to specify localHost on both nodes and only keep a single non-loopback address in discovery section. Regards,-- Ilya Kasnacheev  пт, 22 нояб. 2019 г. в 13:50, xabush :I am running a simple code to test Apache Ignite distributed cache. One nodeis on a server and the other one is run locally when I start a java program.public class HelloIgnite {    public static void main(String[] args) {        System.out.println("Hello Ignite");        Ignite ignite = Ignition.start("example-cache.xml");        // get or create cache        IgniteCache cache =ignite.getOrCreateCache("testCache");        // put some cache elements        for(int i = 1; i <= 100; i++){            cache.put(i, Integer.toString(i));        }        // get them from the cache and write to the console        for(int i =1; i<= 100; i++){            System.out.println("Cache get:"+ cache.get(i));        }        ignite.close();    }} The above is the simple code I am testing. And here is my ignite confighttp://www.springframework.org/schema/beans"       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"       xsi:schemaLocation="        http://www.springframework.org/schema/beans        http://www.springframework.org/schema/beans/spring-beans.xsd">    class="org.apache.ignite.configuration.IgniteConfiguration">                            class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">                                    class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">                                                                                    127.0.0.1:47500..47509                                SERVER_IP:47500..47509                                                                                                                                I start node on the server using ignite.sh -p example-cache.xml first andthen run the above code locally. However, instead of putting values to thecache and printing them, when I run the code, it is stuck showing thismessage:/ [13:40:46] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler[tryStop=false, timeout=0, super=AbstractFailureHandler[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,SYSTEM_CRITICAL_OPERATION_TIMEOUT[13:40:48] Message queue limit is set to 0 which may lead to potential OOMEswhen running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due tomessage queues growth on sender and receiver sides.[13:40:49] Security status [authentication=off, tls/ssl=off]/On the node I started on the server, It is repeatedly printing the followingto the console:/[11:48:28] Topology snapshot [ver=56, locNode=cd55466e, servers=2,clients=0, state=ACTIVE, CPUs=40, offheap=28.0GB, heap=34.0GB][11:48:28] Topology snapshot [ver=57, locNode=cd55466e, servers=1,clients=0, state=ACTIVE, CPUs=32, offheap=25.0GB, heap=30.0GB][11:48:34] Joining node doesn't have encryption data[node=02e2f398-b5b8-4044-83dd-53755e28f94e]/However, when I stop the server node, the local node starts to run properlyand print the values.How can I resolve this? what I am doing wrong here?--Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 

Re: Load cache from 3rd party persistence

[Ilya Kasnacheev]

Hello!

Unfortunately it does not seem to me that this implementation is robust
enough. If you compare it with
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache you
will see that it lacks pooling, support for args, etc.

However, CacheJdbcBlobStore is not an internal class. You can surely
extend/subclass it in your code, and then instantiate the extended version
where you have your own loadCache.

This is what I recommend, instead of waiting for a "generic" loadCache in
Ignite code.

Regards,
-- 
Ilya Kasnacheev


вс, 24 нояб. 2019 г. в 11:09, Кузин Никита (Nikita Kuzin) <
nku...@iitdgroup.ru>:

> also I found the issue: https://issues.apache.org/jira/browse/IGNITE-12078
> with attached PR: https://github.com/apache/ignite/pull/6783
>
> so it seems to be done (merged) very easy:)
>
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
> --
> *От:* Кузин Никита (Nikita Kuzin)
> *Отправлено:* 23 ноября 2019 г. 21:22
> *Кому:* user@ignite.apache.org 
> *Тема:* Load cache from 3rd party persistence
>
> Hello!
>
> We want to use CacheJdbcBlobStore and everything seems fine, but it have
> empty implementation of loadCache method since it extends CacheStoreAdapter
> with empty implementation of loadCache.
> How we can deal with it? Why CacheJdbcBlobStore doesn't implement this
> method?
>
> _
> С уважением, Никита Кузин
> Ведущий программист-разработчик
>
> * Интернейшнл АйТи Дистрибьюшн*
>
> e-mail: nku...@iitdgroup.ru
> тел.: 84995021375 доб. 320
> моб. тел.: 79260948887
> 115114, Москва, Дербеневская ул., 20-27
>

Re: Ignite .NET and TcpDiscoveryStaticIpFinder failure

[Ilya Kasnacheev]

Hello!

Something is not right here:
sockAddrs=[camyakoubCPU/10.0.2.5:47500,
/10.0.75.1:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
host.docker.internal/10.0.2.11:47500],

Why does node think that it is both 10.0.2.5 and 10.0.2.11? Aren't those
supposed to be different addresses?

Please provide full network configuration from both nodes.

Regards,
-- 
Ilya Kasnacheev


вт, 26 нояб. 2019 г. в 11:32, camer314 :

> I just realised this is the same situation that is logged here  here
> <
> http://apache-ignite-users.70518.x6.nabble.com/Unable-to-connect-to-a-remote-apache-ignite-node-tt30367.html>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Unable to connect to a remote apache ignite node.

[Ilya Kasnacheev]

Hello!

I think you have some kind of connectivity problems between two nodes so
the second node can never finish the join process. I recommend trying to
specify localHost on both nodes and only keep a single non-loopback address
in discovery section.

Regards,
-- 
Ilya Kasnacheev


пт, 22 нояб. 2019 г. в 13:50, xabush :

> I am running a simple code to test Apache Ignite distributed cache. One
> node
> is on a server and the other one is run locally when I start a java
> program.
>
> public class HelloIgnite {
> public static void main(String[] args) {
> System.out.println("Hello Ignite");
> Ignite ignite = Ignition.start("example-cache.xml");
> // get or create cache
> IgniteCache cache =
> ignite.getOrCreateCache("testCache");
> // put some cache elements
> for(int i = 1; i <= 100; i++){
> cache.put(i, Integer.toString(i));
> }
> // get them from the cache and write to the console
> for(int i =1; i<= 100; i++){
> System.out.println("Cache get:"+ cache.get(i));
> }
> ignite.close();
>
> }
> }
>
>
>  The above is the simple code I am testing. And here is my ignite config
>
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
>  class="org.apache.ignite.configuration.IgniteConfiguration">
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> 
>
> 
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
>
> 127.0.0.1:47500..47509
> SERVER_IP:47500..47509
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> I start node on the server using ignite.sh -p example-cache.xml first and
> then run the above code locally. However, instead of putting values to the
> cache and printing them, when I run the code, it is stuck showing this
> message:
>
> / [13:40:46] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler
> [tryStop=false, timeout=0, super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT
> [13:40:48] Message queue limit is set to 0 which may lead to potential
> OOMEs
> when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
> message queues growth on sender and receiver sides.
> [13:40:49] Security status [authentication=off, tls/ssl=off]/
>
> On the node I started on the server, It is repeatedly printing the
> following
> to the console:
>
> /[11:48:28] Topology snapshot [ver=56, locNode=cd55466e, servers=2,
> clients=0, state=ACTIVE, CPUs=40, offheap=28.0GB, heap=34.0GB]
> [11:48:28] Topology snapshot [ver=57, locNode=cd55466e, servers=1,
> clients=0, state=ACTIVE, CPUs=32, offheap=25.0GB, heap=30.0GB]
> [11:48:34] Joining node doesn't have encryption data
> [node=02e2f398-b5b8-4044-83dd-53755e28f94e]/
>
> However, when I stop the server node, the local node starts to run properly
> and print the values.
>
> How can I resolve this? what I am doing wrong here?
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite .NET and TcpDiscoveryStaticIpFinder failure

[camer314]

I just realised this is the same situation that is logged here  here

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite .NET and TcpDiscoveryStaticIpFinder failure

[camer314]

I am using 2 Azure virtual machines, both part of the same VNET. The network
security policy is applied at the network level and allows all traffic
in/out from the VNET.

I am trying to run a cluster across both machines. Seeing as Azure does not
support broadcast I am using a static IP list, which are the private IPs of
my two machines like such:

   


10.0.2.5:47500..47509
10.0.2.11:47500..47509




There is not much going on in the way of error messages but the connection
from 11 -> 5 must keep dropping or some other issue is preventing that node
joining.

On the 11 machine I have these log entries, indicating its found the other
node, but then seems to immediately lose the connection?

/[06:50:17,775][INFO][tcp-disco-msg-worker-#2][GridEncryptionManager]
Joining node doesn't have encryption data
[node=41518d17-a16a-48a0-9656-cd3d2b6e0042]
[06:50:17,800][INFO][tcp-disco-msg-worker-#2][TcpDiscoverySpi] New next node
[newNext=TcpDiscoveryNode [id=41518d17-a16a-48a0-9656-cd3d2b6e0042,
addrs=[0:0:0:0:0:0:0:1, 10.0.2.5, 10.0.75.1, 127.0.0.1],
sockAddrs=[camyakoubCPU/10.0.2.5:47500, /10.0.75.1:47500,
/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
host.docker.internal/10.0.2.11:47500], discPort=47500, order=0, intOrder=2,
lastExchangeTime=1574751017748, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=false]]
[06:50:22,779][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery
accepted incoming connection [rmtAddr=/10.0.2.5, rmtPort=52234]
[06:50:22,779][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery
spawning a new thread for connection [rmtAddr=/10.0.2.5, rmtPort=52234]
[06:50:22,780][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Started
serving remote node connection [rmtAddr=/10.0.2.5:52234, rmtPort=52234]
[06:50:22,788][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Finished
serving remote node connection [rmtAddr=/10.0.2.5:52234, rmtPort=52234
[06:50:22,805][WARNING][tcp-disco-msg-worker-#2][TcpDiscoverySpi] Failed to
send message to next node [msg=TcpDiscoveryNodeAddedMessage
[node=TcpDiscoveryNode [id=41518d17-a16a-48a0-9656-cd3d2b6e0042,
addrs=[0:0:0:0:0:0:0:1, 10.0.2.5, 10.0.75.1, 127.0.0.1],
sockAddrs=[camyakoubCPU/10.0.2.5:47500, /10.0.75.1:47500,
/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
host.docker.internal/10.0.2.11:47500], discPort=47500, order=0, intOrder=2,
lastExchangeTime=1574751017748, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=false],
dataPacket=o.a.i.spi.discovery.tcp.internal.DiscoveryDataPacket@8b3b66a,
discardMsgId=null, discardCustomMsgId=null, top=null, clientTop=null,
gridStartTime=1574750964272, super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=aa79676ae61-e3b1d49a-a023-435a-961c-13394c08ad0b,
verifierNodeId=e3b1d49a-a023-435a-961c-13394c08ad0b, topVer=0, pendingIdx=0,
failedNodes=null, isClient=false]], next=TcpDiscoveryNode
[id=41518d17-a16a-48a0-9656-cd3d2b6e0042, addrs=[0:0:0:0:0:0:0:1, 10.0.2.5,
10.0.75.1, 127.0.0.1], sockAddrs=[camyakoubCPU/10.0.2.5:47500,
/10.0.75.1:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
host.docker.internal/10.0.2.11:47500], discPort=47500, order=0, intOrder=2,
lastExchangeTime=1574751017748, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=false], errMsg=Failed to send message to next node
[msg=TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode
[id=41518d17-a16a-48a0-9656-cd3d2b6e0042, addrs=[0:0:0:0:0:0:0:1, 10.0.2.5,
10.0.75.1, 127.0.0.1], sockAddrs=[camyakoubCPU/10.0.2.5:47500,
/10.0.75.1:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
host.docker.internal/10.0.2.11:47500], discPort=47500, order=0, intOrder=2,
lastExchangeTime=1574751017748, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,
isClient=false],
dataPacket=o.a.i.spi.discovery.tcp.internal.DiscoveryDataPacket@8b3b66a,
discardMsgId=null, discardCustomMsgId=null, top=null, clientTop=null,
gridStartTime=1574750964272, super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=aa79676ae61-e3b1d49a-a023-435a-961c-13394c08ad0b,
verifierNodeId=e3b1d49a-a023-435a-961c-13394c08ad0b, topVer=0, pendingIdx=0,
failedNodes=null, isClient=false]], next=ClusterNode
[id=41518d17-a16a-48a0-9656-cd3d2b6e0042, order=0, addr=[0:0:0:0:0:0:0:1,
10.0.2.5, 10.0.75.1, 127.0.0.1], daemon=false]]]
[06:50:22,806][WARNING][tcp-disco-msg-worker-#2][TcpDiscoverySpi] Local node
has detected failed nodes and started cluster-wide procedure. To speed up
failure detection please see 'Failure Detection' section under javadoc for
'TcpDiscoverySpi'
[06:50:22,812][INFO][disco-event-worker-#42][GridDiscoveryManager] Added new
node to topology: TcpDiscoveryNode [id=41518d17-a16a-48a0-9656-cd3d2b6e0042,
addrs=[0:0:0:0:0:0:0:1, 10.0.2.5, 10.0.75.1, 127.0.0.1],
sockAddrs=[camyakoubCPU/10.0.2.5:47500, /10.0.75.1:47500,
/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500,
host.docker.internal/10.0.2.11:47500], discPort=47500, order=2, intOrder=2,
lastExchangeTime=1574751017748, loc=false, ver=2.7.6#20190911-sha1:21f7ca41,