Re: How does Apache Ignite distribute???

[yann . blazart]

Or simply give an attribute with a specific value to the nodes with memory and 
only create caches on those nodes.

Envoyé de mon iPhone

> Le 31 oct. 2019 à 17:19, Stephen Darlington  
> a écrit :
> 
> The direct answer to your question is: implement your own 
> org.apache.ignite.cache.affinity.AffinityFunction.
> 
> But that’s hard to do correctly. You’d probably be better running, say, two 
> copies of Ignite on the machines with more memory.
> 
> Regards,
> Stephen
> 
>> On 31 Oct 2019, at 15:25, BorisBelozerov  wrote:
>> 
>> I want to give more data for some chosen nodes (because these computer have
>> more memory to save data), but I don't know how to do it. 
>> How can I choose node, and how can I distribute more data for chosen nodes?
>> Thank you!!
>> 
>> 
>> 
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> 
> 

Re: Re:

[yann Blazart]

Another way could be to give some attributes values to this node, then
create caches only on these node, no?

Le jeu. 31 oct. 2019 à 08:58, BorisBelozerov  a
écrit :

> I want to distribute more data to some nodes because they have more memory
> to
> save data
> Can you give me some example code for it? Thanks!!!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re:

[yann Blazart]

Hello, of my tries, with 40Gb of input data, data is not equally
partitioned, documentation say that you can have 5 to 10% of differences. I
saw 30%

Le dim. 27 oct. 2019 à 09:56, Andrey Dolmatov  a
écrit :

> Did you try to implement custom affinity function to control distribution?
>
>
> On Fri, Oct 25, 2019, 05:50 ARSENAL 1886  wrote:
>
>> Normally, Apache Ignite distribute data equally to nodes
>> How can I control this distribution?
>> I want to distribute more data to some nodes, and less data to other nodes
>> I also want to choose node to store cache
>> Thank you!!
>>
>>
>>
>> Sent from Mail  for
>> Windows 10
>>
>>
>>
>

Re: Drop index do not release memory used ?

[yann Blazart]

Yes the data pages in reuse list are reused for new objects.

If you drop table, you get back all pages used in reuse list.

But it seems that it's not true for drop indexes

Le sam. 26 oct. 2019 à 16:18, Andrey Dolmatov  a
écrit :

> Ignite believe that Ignite will reuse that memory. But it is a question,
> does Ignite reuse index data blocks for data blocks.
>
> On Fri, Oct 25, 2019, 15:28 yann.blaz...@externe.bnpparibas.com <
> yann.blaz...@externe.bnpparibas.com> wrote:
>
>> Hello all.
>>
>> If you can remember, I found a way to compute the real size of memory
>> used in offheap, using the reuselist size.
>>
>> As I'm facing some limits on my hardware, I'm trying to optimize my
>> memory consumption, in pure memory, no persistence on hdd or ssd.
>>
>> For that as I have to execute plenty of request on my stored data, I saw
>> that indexes consumes a lot of memory.
>>
>> To improve that, in my algorithm I tried to create tables with only pk,
>> no indexes at first.
>>
>> Then before each request, I tried to create indexes , execute request,
>> then drop indexes.
>>
>> What I see, is that drop index do not release memory...
>>
>> Everything is release only when we drop table.
>>
>> Is this normal  ?
>>
>>
>> Thnaks and regards.
>> This message and any attachments (the "message") is
>> intended solely for the intended addressees and is confidential.
>> If you receive this message in error,or are not the intended
>> recipient(s),
>> please delete it and any copies from your systems and immediately notify
>> the sender. Any unauthorized view, use that does not comply with its
>> purpose,
>> dissemination or disclosure, either whole or partial, is prohibited.
>> Since the internet
>> cannot guarantee the integrity of this message which may not be reliable,
>> BNP PARIBAS
>> (and its subsidiaries) shall not be liable for the message if modified,
>> changed or falsified.
>> Do not print this message unless it is necessary, consider the
>> environment.
>>
>>
>> --
>>
>> Ce message et toutes les pieces jointes (ci-apres le "message")
>> sont etablis a l'intention exclusive de ses destinataires et sont
>> confidentiels.
>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>> avertir
>> immediatement l'expediteur. Toute lecture non autorisee, toute
>> utilisation de
>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>> toute
>> publication, totale ou partielle, est interdite. L'Internet ne permettant
>> pas d'assurer
>> l'integrite de ce message electronique susceptible d'alteration, BNP
>> Paribas
>> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
>> dans l'hypothese
>> ou il aurait ete modifie, deforme ou falsifie.
>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>
>>

Re: How to know memory used by a cache or a set

[yann Blazart]

Ok, but I can't create dynamically a data region ? Because each time I
receive a new file to process, I create a cachegroup to handle it, then I
clean everything.

Le jeu. 13 juin 2019 à 13:28, Alex Plehanov  a
écrit :

> Hello,
>
> It's a known issue [1]. Now you can get cache group size via JMX only if
> persistence is used.
> If persistence is not used you can get allocated size only for data region
> (but you can have more then one data region and assign cache groups to data
> regions any way you want)
>
> [1] : https://issues.apache.org/jira/browse/IGNITE-8517
>
> ср, 12 июн. 2019 г. в 16:00, yann.blaz...@externe.bnpparibas.com <
> yann.blaz...@externe.bnpparibas.com>:
>
>> Hello, I'm back.
>>
>> Well, I need to get memory used by each execution of my process, so I put
>> all involved caches into the same cacheGroup.
>>
>> If I use the CacheGroupMetricsBean, the size gave to me is 0 !
>> If I enable persistence on DataRegion, I get size, but I don't want to
>> use the persistence enabled.
>>
>> Is it a bug ?
>>
>> regards.
>>
>> > Le 27 mai 2019 à 11:09, ibelyakov  a écrit :
>> >
>> > Hi,
>> >
>> > Did you turn on cache metrics for your data region?
>> >
>> > To turn the metrics on, use one of the following approaches:
>> > 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region
>> you
>> > want to collect the metrics for.
>> > 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
>> > special JMX bean.
>> >
>> > More information regarding cache metrics available here:
>> > https://apacheignite.readme.io/docs/cache-metrics
>> >
>> > Regards,
>> > Igor
>> >
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>> This message and any attachments (the "message") is
>> intended solely for the intended addressees and is confidential.
>> If you receive this message in error,or are not the intended
>> recipient(s),
>> please delete it and any copies from your systems and immediately notify
>> the sender. Any unauthorized view, use that does not comply with its
>> purpose,
>> dissemination or disclosure, either whole or partial, is prohibited.
>> Since the internet
>> cannot guarantee the integrity of this message which may not be reliable,
>> BNP PARIBAS
>> (and its subsidiaries) shall not be liable for the message if modified,
>> changed or falsified.
>> Do not print this message unless it is necessary, consider the
>> environment.
>>
>>
>> --
>>
>> Ce message et toutes les pieces jointes (ci-apres le "message")
>> sont etablis a l'intention exclusive de ses destinataires et sont
>> confidentiels.
>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>> avertir
>> immediatement l'expediteur. Toute lecture non autorisee, toute
>> utilisation de
>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>> toute
>> publication, totale ou partielle, est interdite. L'Internet ne permettant
>> pas d'assurer
>> l'integrite de ce message electronique susceptible d'alteration, BNP
>> Paribas
>> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
>> dans l'hypothese
>> ou il aurait ete modifie, deforme ou falsifie.
>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>
>

Re: Insert into select OOM exception on java heap

[yann Blazart]

It's an insert into select. We made "meta" tables to allow doing other
selects.

Or can I do a lazy select then batch insert you mean ?

Le jeu. 30 mai 2019 à 18:15, Ilya Kasnacheev  a
écrit :

> Hello!
>
> I think it would make better sense to mark already updated entries, update
> in batches until no unmarked entries left.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 30 мая 2019 г. в 19:14, yann Blazart :
>
>> Hmmm. Can I use limit and offset ?
>>
>> Doing limit 1 by example and continue while insert  ount = 1 ???
>>
>>
>>
>> Le jeu. 30 mai 2019 à 17:57, Ilya Kasnacheev 
>> a écrit :
>>
>>> Hello!
>>>
>>> I'm afraid you will have to split this query into smaller ones. Ignite
>>> doesn't really have lazy insert ... select, so the result set will have to
>>> be held in heap for some time.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> чт, 30 мая 2019 г. в 18:36, yann Blazart :
>>>
>>>> Hello,  we have 6 nodes configured with 3Gb heap, 30Gb offheap.
>>>>
>>>> We store lot's of data in some partitioned tables, then we are
>>>> executing some "insert into select... join..." using SqlQueryField (or
>>>> SqlQueryFieldEx).
>>>>
>>>> With tables of 5000 000 lines, we ran in a OOM error, even with lazy
>>>> set to true and skipOnReduceTable.
>>>>
>>>> How can we handle this please ?
>>>>
>>>> Regards.
>>>>
>>>

Re: Insert into select OOM exception on java heap

[yann Blazart]

Hmmm. Can I use limit and offset ?

Doing limit 1 by example and continue while insert  ount = 1 ???



Le jeu. 30 mai 2019 à 17:57, Ilya Kasnacheev  a
écrit :

> Hello!
>
> I'm afraid you will have to split this query into smaller ones. Ignite
> doesn't really have lazy insert ... select, so the result set will have to
> be held in heap for some time.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 30 мая 2019 г. в 18:36, yann Blazart :
>
>> Hello,  we have 6 nodes configured with 3Gb heap, 30Gb offheap.
>>
>> We store lot's of data in some partitioned tables, then we are executing
>> some "insert into select... join..." using SqlQueryField (or
>> SqlQueryFieldEx).
>>
>> With tables of 5000 000 lines, we ran in a OOM error, even with lazy set
>> to true and skipOnReduceTable.
>>
>> How can we handle this please ?
>>
>> Regards.
>>
>

Insert into select OOM exception on java heap

[yann Blazart]

Hello,  we have 6 nodes configured with 3Gb heap, 30Gb offheap.

We store lot's of data in some partitioned tables, then we are executing
some "insert into select... join..." using SqlQueryField (or
SqlQueryFieldEx).

With tables of 5000 000 lines, we ran in a OOM error, even with lazy set to
true and skipOnReduceTable.

How can we handle this please ?

Regards.

Re: How to know memory used by a cache or a set

[yann Blazart]

Hello.  I'm almost sure I already did that. I will check tomorrow.

Thks

Le lun. 27 mai 2019 à 11:09, ibelyakov  a écrit :

> Hi,
>
> Did you turn on cache metrics for your data region?
>
> To turn the metrics on, use one of the following approaches:
> 1. Set DataRegionConfiguration.setMetricsEnabled(true) for every region you
> want to collect the metrics for.
> 2. Use the DataRegionMetricsMXBean.enableMetrics() method exposed by a
> special JMX bean.
>
> More information regarding cache metrics available here:
> https://apacheignite.readme.io/docs/cache-metrics
>
> Regards,
> Igor
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

How to know memory used by a cache or a set

[yann Blazart]

Hello !

Because I have multiple process that compute files in ignite H2 database,
I really need to know how memory have been used by caches and sets,
but the persistence is not activated (performance) and when I try to use
the MxBean for CacheGroup, it send me back a 0.

Is there any solution ??

Thanks.

Re: Apache ignite rewrite badly SQL ???

[yann Blazart]

Hello, I answer from another mail, but it's me :)

Well, using SqlQueryFieldEx I'm able to enforceJoinOrder.

Now on my Qualification env, rewriting some requests, it's good now.
Thanks for your help !

Now I have to find an algorithm now to optimize my requests following sql
requests ! :)

Le jeu. 23 mai 2019 à 10:49, Roman Kondakov  a écrit :

> Hi!
>
> Is your data collocated? To perform fast joins in the distributed systems
> tables should be collocated on the join keys [1] to avoid network latency.
>
> Also Ignite's SQL optimizer currently is not able to select the best join
> order. If your query is slow you should choose the join order manually
> using enforceJoinOrder flag [2].
>
> [1] https://apacheignite.readme.io/docs/affinity-collocation
>
> [2]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/SqlFieldsQuery.html#setEnforceJoinOrder-boolean-
>
> --
> Kind Regards
> Roman Kondakov
>
> On 21.05.2019 20:38, yann.blaz...@externe.bnpparibas.com wrote:
>
> Hello all , I have a big problem.
>
> I have  a lot of tabes in my ignite cluster and one request take 10
> minutes executed as a SQLFieldQuery, but only 153ms from DataGrip in jdbc.
>
> I explain, I have to change some names in request to not divulgate
> informations.
>
>
> I have 3 tables  :
>
> TC with 557 000 lines
> TD with 3753 lines
> TS with 1500 lines.
>
>
> I want to execute this request :
>
> select  *fromTC co inner join TD dd
> on dd.eid = co.eid
> and dd.mid = co.mid inner join TS sch on sch.eid = dd.eid  and sch.mid = 
> dd.mid;
>
>
> On datagrip 153ms.
>
> When I look into logs, I see that request has been rewrited like that :
>
> SELECT*FROM TC CO__Z0
> /*CONTRACT.__SCAN_ */INNER JOIN TD DD__Z1
> /* batched:unicast TD_3: EID = CO__Z0.EIDAND MID = CO__Z0.MID 
> */ON 1=1/* WHERE (DD__Z1.EID = CO__Z0.EID)AND (DD__Z1.MID = 
> CO__Z0.MID)*/INNER JOIN TS SCH__Z2
> /* batched:unicast TS_2: EID = DD__Z1.EIDAND MID = DD__Z1.MID 
> */ON 1=1WHERE ((SCH__Z2.EID = DD__Z1.EID)
> AND (SCH__Z2.MID = DD__Z1.MID))
> AND ((DD__Z1.EID = CO__Z0.EID)
> AND (DD__Z1.MID = CO__Z0.MID))
>
>
> Why these (On 1=1 ) ???
>
> If I understand, this make a scalar product of my lines that explain the
> 10 minutes ! Why this is rewrited like that ???
>
> Thanks for your help, regards
>
> This message and any attachments (the "message") is
> intended solely for the intended addressees and is confidential.
> If you receive this message in error,or are not the intended recipient(s),
> please delete it and any copies from your systems and immediately notify
> the sender. Any unauthorized view, use that does not comply with its
> purpose,
> dissemination or disclosure, either whole or partial, is prohibited. Since
> the internet
> cannot guarantee the integrity of this message which may not be reliable,
> BNP PARIBAS
> (and its subsidiaries) shall not be liable for the message if modified,
> changed or falsified.
> Do not print this message unless it is necessary, consider the environment.
>
>
> --
>
> Ce message et toutes les pieces jointes (ci-apres le "message")
> sont etablis a l'intention exclusive de ses destinataires et sont
> confidentiels.
> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
> merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
> immediatement l'expediteur. Toute lecture non autorisee, toute utilisation
> de
> ce message qui n'est pas conforme a sa destination, toute diffusion ou
> toute
> publication, totale ou partielle, est interdite. L'Internet ne permettant
> pas d'assurer
> l'integrite de ce message electronique susceptible d'alteration, BNP
> Paribas
> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
> dans l'hypothese
> ou il aurait ete modifie, deforme ou falsifie.
> N'imprimez ce message que si necessaire, pensez a l'environnement.
>
>

Re: Strange behavior with tables partitioned and replicated, and indexes on join

[yann Blazart]

Hello, I must apologize, it's my bad. Indexes weren't created

Shame on me and thanks for help :)



Le jeu. 18 avr. 2019 à 16:18, Maxim.Pudov  a écrit :

> By default Ignite thinks that data is collocated. You have to tell Ignite
> that it is not. For example, by adding a parameter to JDBC connection:
> distributedJoins=true or by using this method
> SqlQueryFields.setDistributedJoins [1] in java API.
>
> [1]
>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/SqlFieldsQuery.html#setDistributedJoins-boolean-
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Strange behavior with tables partitioned and replicated, and indexes on join

[yann Blazart]

Sorry for short answer.

Document just say that if the join condition is not pk or affinity, it
broadcast request instead of unicast.

In my case, i use indexed columns, but I receive exception.

Regards

Le mer. 17 avr. 2019 à 18:36, yann Blazart  a
écrit :

> Hi. Thanks for answer.
>
> But it doesn't explain my case
>
> Le mer. 17 avr. 2019 à 16:56, Maxim.Pudov  a écrit :
>
>> Hi, check out this article:
>>
>> https://apacheignite-sql.readme.io/docs/distributed-joins#section-non-collocated-joins
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Strange behavior with tables partitioned and replicated, and indexes on join

[yann Blazart]

Hi. Thanks for answer.

But it doesn't explain my case

Le mer. 17 avr. 2019 à 16:56, Maxim.Pudov  a écrit :

> Hi, check out this article:
>
> https://apacheignite-sql.readme.io/docs/distributed-joins#section-non-collocated-joins
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Strange behavior with tables partitioned and replicated, and indexes on join

[yann Blazart]

Hello, I really need help.
First, sorry but I can’t share code because of my boss.

I have two tables, normally partitioned, but I can’t collocate data (simply 
because other tables are related to using other columns).
On these tables I have stetted my indexes, everything is done using create 
table and create index scripts.

Then when I execute my request, a left join, on only one node, ignite say to me 
that my join do not use indexes while raising an exception, wich  is wrong 
because I created them.
The most strange thing is that if I set the second table as replicated instead 
of partitioned, it works !

I precise that my data are inserted using binary objects.

Is anybody have an idea ?

Thanks in advance.

Some questions about store and distributed processing

[yann Blazart]

Hello all !

I will have to use Ignite because I think it's the best solution to all my
concerns, but I have a few question.

I have to process very big json files (200GB), with lot of objects of
different type generated from it.
These objects, I will have to do multiple controls on it (with a dsl), and
check  unicity, and in the end do some complexe join request between them.

So for complexe request, ok, I store all in partitionned tables, and do
request, easy.

But for all the dsl rule ton apply on each object, It could be very nice if
it can be applied when it's stored on the node, instead of doing it when I
read the file, I mean :

cache.putAll(mymap);

then something on node to say :  new Entry listener -> execute dsl rules.

I think I can gain lot of processing time like that. But is it possible ?

I checked the doc, but I only see ways to 1st store all then run dsl rule
on all node.

Thanks in advance ;)

Regards

Insert performance

[yann Blazart]

Hello,

I'm doing tests to get the best performance whily using Ignite.

I've to insert a lot of data, then doing some requests.

First, in my tests, I'm using objects with @SqlFields annotations,
cache.putAll is faster than IgniteDataStream, is it normal ? I'm using
cache.putAll to send 5000 objects at the same time.

Second, when I'm doing my insertion test, 30 millions of objects, using two
servers node and one client node on same machine, I raise 500 insert per
milliseconds, wich is excellent !

But if I use two differents machines, threw my network, performances go
down to 200 inserts per milliseconds. But my network is far away to be busy
(only 80Mb/s used on 200Mb/s available on simple test).

Is there any reason ?

Thanks in advance,

Regards.

Hem, registerUserClassDescriptor

[Yann BLAZART]

Hello, me ; again...

I continue to replace Hazelcast by ignite in my code.

I have to use Queues, and sometime, tere some things I don't understand in the 
Ignite code.
Well, In my case, I have multiple objects to push in Queues.
Each of its have its own Queues.But this object are "primary keys" of others, 
declared as inner class with same class name :
Example :

* com.test.ObjectA$PK

* com.test.ObjectB$PK

These classes are different, but for Ignite there are the same.

Let me explain, in the code, when I add the object in the Q,  the code is going 
threw BinaryContext.java :

private BinaryClassDescriptor registerUserClassDescriptor(Class cls, boolean 
deserialize) {
boolean registered;

String typeName = typeName(cls.getName());

typeName remove everything and get "PK" only.

So some lines after in the code I got a duplicate ID exception because of same  
hashcode.

So I had to rename It and extract its to not be inner class.

And more of that only the "simpleName" is taken (substring with lastindexof 
'.')... So we can't have class with same name in different package stored in 
Queues.

It sould be a good idea to get the fullName of the class, No ? Why do this type 
of things ?

Regards.


This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

Stupide question about Queue ?

[Yann BLAZART]

Hello,

In hazelcast we can persist Queues in Database, this  help me for example when 
consumer are down to no have OOM exception.

But I don't see this possibility in Ignite docs ?



This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

RE: Ignite "bugs" ?

[Yann BLAZART]

Gotcha



This is this part : 

TcpDiscoverySpi discoverySpi=new TcpDiscoverySpi();
TcpDiscoveryJdbcIpFinder ipFinder=new TcpDiscoveryJdbcIpFinder();
ipFinder.setDataSource(dataSource);
discoverySpi.setIpFinder(ipFinder);
config.setDiscoverySpi(discoverySpi);

When I remove it, and use real tcp discovery, it works.

-Original Message-
From: vkulichenko [mailto:valentin.kuliche...@gmail.com] 
Sent: jeudi 14 janvier 2016 04:15
To: user@ignite.apache.org
Subject: RE: Ignite "bugs" ?

Yann,

It sounds like you have some misconfiguration for discovery. Can you also 
provide your configuration file?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-bugs-tp2534p2550.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

RE: Ignite "bugs" ?

[Yann BLAZART]

Hello. I'm not using config file I'm using java code to configure : 


   @Produces
@ApplicationScoped
public Ignite produceInstance() {
LOG.info("** Prepare Ignite ");

IgniteConfiguration config = new IgniteConfiguration();
config.setClientMode(false);
config.setLifecycleBeans(e -> igniteEventRaiser.fire(e));

CacheConfiguration glblCacheConfig=new CacheConfiguration();
glblCacheConfig.setStartSize(100*1024*1024);

CacheConfiguration cacheConfigFeedsFlow = 
getCacheConfiguration(GridService.FEEDS_FLOW_MAP_KEY, feedsFlowMapStore);

CacheConfiguration 
cacheConfigFeedsFlowRaw=getCacheConfiguration(GridService.FEEDS_FLOW_RAW_MAP_KEY,feedsFlowRawMapStore);

CacheConfiguration 
cacheConfigFeedsFlowSource=getCacheConfiguration(GridService.FEEDS_FLOW_SOURCE_MAP_KEY,feedsFlowSourceMapStore);



config.setCacheConfiguration(glblCacheConfig,cacheConfigFeedsFlow,cacheConfigFeedsFlowRaw,cacheConfigFeedsFlowSource);

TcpDiscoverySpi discoverySpi=new TcpDiscoverySpi();
TcpDiscoveryJdbcIpFinder ipFinder=new TcpDiscoveryJdbcIpFinder();
ipFinder.setDataSource(dataSource);
discoverySpi.setIpFinder(ipFinder);
config.setDiscoverySpi(discoverySpi);

TransactionConfiguration txConfig=new TransactionConfiguration();
txConfig.setTxManagerFactory(()->transactionManager);
config.setTransactionConfiguration(txConfig);

Hashtable backup = new Hashtable<>();
Hashtable hashtableFromSystemProperties = 
System.getProperties();
Enumeration keys = hashtableFromSystemProperties.keys();
while ( keys.hasMoreElements() ) {
Object key =keys.nextElement();
if ( !  (hashtableFromSystemProperties.get(key) instanceof  String) 
) {
LOG.info("*** Prepare ignite : remove {} from 
System.properties",key);
backup.put(key,hashtableFromSystemProperties.get(key));
hashtableFromSystemProperties.remove(key);
}
}
Ignite ignite = Ignition.start(config);

hashtableFromSystemProperties.putAll(backup);
LOG.info("*** Prepare ignite : restore key into from 
System.properties");
return ignite;
}



@NotNull
private CacheConfiguration getCacheConfiguration(String name, 
GenericFeedsFlowJPACacheStore cacheStore) {
CacheConfiguration config=new CacheConfiguration(name);
config.setCacheStoreFactory(new FixedCacheStoreFactory(cacheStore));
config.setWriteThrough(true);
config.setReadThrough(true);
config.setCacheMode(CacheMode.PARTITIONED);
config.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
return config;
}


-Original Message-
From: vkulichenko [mailto:valentin.kuliche...@gmail.com] 
Sent: jeudi 14 janvier 2016 04:15
To: user@ignite.apache.org
Subject: RE: Ignite "bugs" ?

Yann,

It sounds like you have some misconfiguration for discovery. Can you also 
provide your configuration file?

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Ignite-bugs-tp2534p2550.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

RE: Ignite "bugs" ?

[Yann BLAZART]

I will

From: Alexey Kuznetsov [mailto:akuznet...@gridgain.com]
Sent: mercredi 13 janvier 2016 16:42
To: user@ignite.apache.org
Subject: Re: Ignite "bugs" ?

Yann,

Take into account that this mail list does not allow binary attachments,
 you should upload them on some public service (like dropbox or any) and 
provide a link.


--
Alexey Kuznetsov
GridGain Systems
www.gridgain.com


This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

RE: Ignite "bugs" ?

[Yann BLAZART]

I will do ASAP. I have some support case for now.

Thanks for your help :)

Regarding the System.properties, I've made my own patch in my code by filtering 
System.properties hashtable before calling ignite start, then I restore 
values... Ugly, but working.

From: Denis Magda [mailto:dma...@gridgain.com]
Sent: mercredi 13 janvier 2016 16:26
To: user@ignite.apache.org
Subject: Re: Ignite "bugs" ?

Hi Yann,

Could you provide thread dumps from all the nodes? It should help to see the 
place that "locket" the execution.

Regards,
Denis
On 1/13/2016 4:23 PM, Yann BLAZART wrote:
Hello everybody.

I'm currently evaluation Ignite vs Hazelcast on a poc.

I'm facing some issues.
I'm coding some integration/unit test using the ApplicationComposer of TomEE 
7.0.0. I have no problem with Hazelcast on it.


* The first one is when I'm start Ignite in the applicationComposer 
(with a CDI @Produces), Ignite complain about some System.properties that are 
not String :


it try to "serialize" System.properties by using 
System.getProperties().store(new PrintWriter(sw));



In fact, TomEE (as other framework like hibernate) use System.properties to 
store prop objects.



So I made something to remove this properties before Ignite start and recover 
its after. Well. Pehraps it will be nice to change the way to "serialize" 
System.properties.



* The second problem is strange, very strange. If I'm making a call to 
Ignite.start in @Before or using a @Inject in the test class (that use the 
@Produce that make the call to start(), everything is ok. But if the start is 
called in the @Test method (so after ApplicationComposer has made some things), 
Ignite is "locked".

o   Precisely in IgniteKernel.java:917  :



// Start discovery manager last to make sure that grid is fully initialized.
startManager(discoMgr);



The call to  this method never exit.



Any idea to help me to understand ? Anybody has tried to use Ignite with EE or 
CDI ?



Regards






This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified.
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

RE: Ignite "bugs" ?

[Yann BLAZART]

Well, TomEE is using System.properties to store its IVmContext… I can’t make 
change all open source projects on this point, even if I’m agree with you.

I have only one Ignite instance started in this case. This is really strange.

From: Matt Hoffman [mailto:m...@mhoffman.org]
Sent: mercredi 13 janvier 2016 15:31
To: user@ignite.apache.org
Subject: Re: Ignite "bugs" ?

Using System.properties() to store non-strings is really poor behavior. It 
violates the contract of the System.setProperty and System.getProperty, as well 
as the contract of the Properties object itself, from right at the top of the 
javadoc:  "Each key and its corresponding value in the property list is a 
string."
I've hit something like that in Hibernate before, years ago; I'm surprised they 
haven't fixed it yet. It's a long-standing bug in Hibernate. Not to say Ignite 
shouldn't have a workaround for badly-behaved libraries that do things like 
that, but it's definitely a Hibernate bug.

Someone else will have to talk the locking behavior you're seeing. Are you 
starting Ignite more than once in parallel in that case?

On Wed, Jan 13, 2016 at 8:23 AM, Yann BLAZART 
mailto:yann.blaz...@externe.bnpparibas.com>>
 wrote:
Hello everybody.

I’m currently evaluation Ignite vs Hazelcast on a poc.

I’m facing some issues.
I’m coding some integration/unit test using the ApplicationComposer of TomEE 
7.0.0. I have no problem with Hazelcast on it.


• The first one is when I’m start Ignite in the applicationComposer 
(with a CDI @Produces), Ignite complain about some System.properties that are 
not String :


it try to "serialize" System.properties by using 
System.getProperties().store(new PrintWriter(sw));



In fact, TomEE (as other framework like hibernate) use System.properties to 
store prop objects.



So I made something to remove this properties before Ignite start and recover 
its after. Well. Pehraps it will be nice to change the way to “serialize” 
System.properties.



• The second problem is strange, very strange. If I’m making a call to 
Ignite.start in @Before or using a @Inject in the test class (that use the 
@Produce that make the call to start(), everything is ok. But if the start is 
called in the @Test method (so after ApplicationComposer has made some things), 
Ignite is “locked”.

o   Precisely in IgniteKernel.java:917  :



// Start discovery manager last to make sure that grid is fully initialized.
startManager(discoMgr);



The call to  this method never exit.



Any idea to help me to understand ? Anybody has tried to use Ignite with EE or 
CDI ?



Regards






This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified.
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Ignite "bugs" ?

[Yann BLAZART]

Hello everybody.

I'm currently evaluation Ignite vs Hazelcast on a poc.

I'm facing some issues.
I'm coding some integration/unit test using the ApplicationComposer of TomEE 
7.0.0. I have no problem with Hazelcast on it.


* The first one is when I'm start Ignite in the applicationComposer 
(with a CDI @Produces), Ignite complain about some System.properties that are 
not String :


it try to "serialize" System.properties by using 
System.getProperties().store(new PrintWriter(sw));



In fact, TomEE (as other framework like hibernate) use System.properties to 
store prop objects.



So I made something to remove this properties before Ignite start and recover 
its after. Well. Pehraps it will be nice to change the way to "serialize" 
System.properties.



* The second problem is strange, very strange. If I'm making a call to 
Ignite.start in @Before or using a @Inject in the test class (that use the 
@Produce that make the call to start(), everything is ok. But if the start is 
called in the @Test method (so after ApplicationComposer has made some things), 
Ignite is "locked".

o   Precisely in IgniteKernel.java:917  :



// Start discovery manager last to make sure that grid is fully initialized.
startManager(discoMgr);



The call to  this method never exit.



Any idea to help me to understand ? Anybody has tried to use Ignite with EE or 
CDI ?



Regards







This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the 
internet 
cannot guarantee the integrity of this message which may not be reliable, BNP 
PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed 
or falsified. 
Do not print this message unless it is necessary,consider the environment.

--

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas 
d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans 
l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.