Re: JVM warnings during Java 11 startup
Dmitriy, Thank you for details. Yes it looks really troublesome to start Ignite development with Java 9+. I am not happy to say that but even more hacky workarounds could help us. In short term. I will try to do an exercise of setting up development environment from scratch to feel the real situation. сб, 2 мар. 2019 г. в 20:39, Dmitriy Pavlov : > > In a standalone mode user have to specify options. He or she may use > application templates to simplify launch just like > https://github.com/apache/ignite/tree/master/examples#running-examples-on-jdk-91011 > but > it is anyway required. > > As a user in the past (and currently) I may say: Ignite-based (or any other > grid-based) soft is often developed in IDEA in the same process with server > nodes. Ignite.sh (or another grid node launcher) is used in prod, but not > for dev. > > сб, 2 мар. 2019 г. в 10:01, Павлухин Иван : > > > Dmitriy, > > > > I am a little bit confused by statement that "user should specify > > options". I thought that we can setup all needed variables in our > > startup scripts. Or are you talking about library case? > > > > And answering the question. Yes, more options will be needed. > > --add-exports makes not exported packages from some module visible to > > other specified modules. > > --add-opens allows reflective access to non-public members in other > > module packages (they call it "deep reflection"). > > > > Of course we should check everything carefully during implementation. > > > > пт, 1 мар. 2019 г. в 22:26, Dmitriy Pavlov : > > > > > > Would it require a user to specify more options or use longer parameter > > > values if we change from -add-exports to -add-opens? > > > > > > Stanislav, could you prepare PR to demonstrate the idea. PR will probably > > > answer to all questions. > > > > > > пт, 1 мар. 2019 г. в 15:09, Павлухин Иван : > > > > > > > Dmitriy, > > > > > > > > > Would add opens require user to set up a longer line? > > > > > > > > I am not sure that got that question right. Could you please elaborate? > > > > > > > > пт, 1 мар. 2019 г. в 09:39, Dmitriy Pavlov : > > > > > > > > > > Would add opens require user to set up a longer line? > > > > > > > > > > пт, 1 мар. 2019 г., 9:01 Павлухин Иван : > > > > > > > > > > > By the way. It is interesting that Unsafe class is not the source > > of > > > > > > warnings. I checked (with java 10) that there is no warnings issued > > > > > > when Unsafe instance is accessed with good old reflective way. > > > > > > > > > > > > пт, 1 мар. 2019 г. в 08:05, Павлухин Иван : > > > > > > > > > > > > > > Here is how our friends from Hazelcast tackle the problem [1]. > > Using > > > > > > > --add-opens as Stan proposed looks like a good option in short > > > > > > > perspective. By the way to determine all illegally accessed > > packages > > > > > > > we can use --illegal-access=warn, which does not deny operation > > but > > > > > > > only prints a warning. I guess we can found the majority of > > packages > > > > > > > by analyzing TC logs. > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > [1] > > > > > > > > > > > > https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/1988aa92075eb2a60328e28b3a5933a7c7cbc347/hz-java9.png > > > > > > > > > > > > > > пт, 1 мар. 2019 г. в 03:04, Denis Magda : > > > > > > > > > > > > > > > > Flags/hacks is not a way to go for serious and mature projects > > like > > > > > > Ignite. > > > > > > > > We should switch to another mode - how to replace Unsafe > > > > completely. > > > > > > > > > > > > > > > > - > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 28, 2019 at 4:32 AM Dmitriy Pavlov < > > dpav...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > may become unavailable in any time. > > > > > > > > > > > > > > > > > > I remember this was discussed in 2013 that Unsafe will be > > removed > > > > > > soon. But > > > > > > > > > nothing is changed yet, so I hope unsafe will be available > > for a > > > > > > long time > > > > > > > > > (with flags/hacks/etc). > > > > > > > > > > > > > > > > > > > > > > > > > > > чт, 28 февр. 2019 г. в 09:11, Petr Ivanov < > > mr.wei...@gmail.com>: > > > > > > > > > > > > > > > > > > > According to warning message, there are no options at all, > > as > > > > > > Unsafe may > > > > > > > > > > become unavailable in any time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 27 Feb 2019, at 22:53, Denis Magda > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > It's fine as long as the project can be launched. I would > > > > better > > > > > > start > > > > > > > > > > > looking for Unsafe alternatives as the next step. We > > can't > > > > live > > > > > > with it > > > > > > > > > > > forever, the time to phase it out has come :) > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 27, 2019
[jira] [Created] (IGNITE-11465) Multiple client leave/join events may wipe affinity assignment history and cause transactions fail
Ivan Rakov created IGNITE-11465:
---
Summary: Multiple client leave/join events may wipe affinity
assignment history and cause transactions fail
Key: IGNITE-11465
URL: https://issues.apache.org/jira/browse/IGNITE-11465
Project: Ignite
Issue Type: Bug
Reporter: Ivan Rakov
Assignee: Ivan Rakov
Fix For: 2.8
We keep history of GridAffinityAssignmentCache#MAX_HIST_SIZE affinity
assignments, however flood of client joins/leaves may wipe it out entirely and
cause fail/hang of transaction that was started before the flood:
{code:java}
if (cache == null || cache.topologyVersion().compareTo(topVer) > 0)
{
throw new IllegalStateException("Getting affinity for topology
version earlier than affinity is " +
"calculated [locNode=" + ctx.discovery().localNode() +
", grp=" + cacheOrGrpName +
", topVer=" + topVer +
", head=" + head.get().topologyVersion() +
", history=" + affCache.keySet() +
']');
}
{code}
History is limited in order to prevent JVM heap overflow. At the same time,
only "server event" affinity assignments are heavy: "client event" assignments
are just shallow copies of "server event" assignments.
I suggest to limit history by the number of "server event" assignments.
Also, consider the provided fix, I don't see any need to keep 500 items in
history. I changed history size to 40.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: JVM warnings during Java 11 startup
In a standalone mode user have to specify options. He or she may use application templates to simplify launch just like https://github.com/apache/ignite/tree/master/examples#running-examples-on-jdk-91011 but it is anyway required. As a user in the past (and currently) I may say: Ignite-based (or any other grid-based) soft is often developed in IDEA in the same process with server nodes. Ignite.sh (or another grid node launcher) is used in prod, but not for dev. сб, 2 мар. 2019 г. в 10:01, Павлухин Иван : > Dmitriy, > > I am a little bit confused by statement that "user should specify > options". I thought that we can setup all needed variables in our > startup scripts. Or are you talking about library case? > > And answering the question. Yes, more options will be needed. > --add-exports makes not exported packages from some module visible to > other specified modules. > --add-opens allows reflective access to non-public members in other > module packages (they call it "deep reflection"). > > Of course we should check everything carefully during implementation. > > пт, 1 мар. 2019 г. в 22:26, Dmitriy Pavlov : > > > > Would it require a user to specify more options or use longer parameter > > values if we change from -add-exports to -add-opens? > > > > Stanislav, could you prepare PR to demonstrate the idea. PR will probably > > answer to all questions. > > > > пт, 1 мар. 2019 г. в 15:09, Павлухин Иван : > > > > > Dmitriy, > > > > > > > Would add opens require user to set up a longer line? > > > > > > I am not sure that got that question right. Could you please elaborate? > > > > > > пт, 1 мар. 2019 г. в 09:39, Dmitriy Pavlov : > > > > > > > > Would add opens require user to set up a longer line? > > > > > > > > пт, 1 мар. 2019 г., 9:01 Павлухин Иван : > > > > > > > > > By the way. It is interesting that Unsafe class is not the source > of > > > > > warnings. I checked (with java 10) that there is no warnings issued > > > > > when Unsafe instance is accessed with good old reflective way. > > > > > > > > > > пт, 1 мар. 2019 г. в 08:05, Павлухин Иван : > > > > > > > > > > > > Here is how our friends from Hazelcast tackle the problem [1]. > Using > > > > > > --add-opens as Stan proposed looks like a good option in short > > > > > > perspective. By the way to determine all illegally accessed > packages > > > > > > we can use --illegal-access=warn, which does not deny operation > but > > > > > > only prints a warning. I guess we can found the majority of > packages > > > > > > by analyzing TC logs. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > [1] > > > > > > > > > https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/1988aa92075eb2a60328e28b3a5933a7c7cbc347/hz-java9.png > > > > > > > > > > > > пт, 1 мар. 2019 г. в 03:04, Denis Magda : > > > > > > > > > > > > > > Flags/hacks is not a way to go for serious and mature projects > like > > > > > Ignite. > > > > > > > We should switch to another mode - how to replace Unsafe > > > completely. > > > > > > > > > > > > > > - > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 28, 2019 at 4:32 AM Dmitriy Pavlov < > dpav...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > > > > > > may become unavailable in any time. > > > > > > > > > > > > > > > > I remember this was discussed in 2013 that Unsafe will be > removed > > > > > soon. But > > > > > > > > nothing is changed yet, so I hope unsafe will be available > for a > > > > > long time > > > > > > > > (with flags/hacks/etc). > > > > > > > > > > > > > > > > > > > > > > > > чт, 28 февр. 2019 г. в 09:11, Petr Ivanov < > mr.wei...@gmail.com>: > > > > > > > > > > > > > > > > > According to warning message, there are no options at all, > as > > > > > Unsafe may > > > > > > > > > become unavailable in any time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 27 Feb 2019, at 22:53, Denis Magda > > > > wrote: > > > > > > > > > > > > > > > > > > > > It's fine as long as the project can be launched. I would > > > better > > > > > start > > > > > > > > > > looking for Unsafe alternatives as the next step. We > can't > > > live > > > > > with it > > > > > > > > > > forever, the time to phase it out has come :) > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 27, 2019 at 9:53 AM Dmitriy Pavlov < > > > > > dpav...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Sure, we could try this option. > > > > > > > > > >> > > > > > > > > > >> ср, 27 февр. 2019 г. в 19:16, Ilya Kasnacheev < > > > > > > > > > ilya.kasnach...@gmail.com>: > > > > > > > > > >> > > > > > > > > > >>> Hello! > > > > > > > > > >>> > > > > > > > > > >>> I wonder if we could try to redirect output to null, > > > initialize > > > > > > > > > >> GridUnsafe > > > > > > > > > >>> and then bring output back :) > > > > > > > > > >>> > > > > > > > > > >>> Regards, > > > > > > > >
Re: Tests for ML using binary builds
Yes, sure. Ignite ML algorithms actively use data sending between nodes and in several cases uses per class loading mechanism. I want to exclude failures when algorithms use unserializable data or try to send lambdas with big context etc. >From this point of view we can just run ML examples on a little cluster where servers is started from binary build. сб, 2 мар. 2019 г. в 08:39, Павлухин Иван : > Hi Alexey, > > Could you please share some background? What problem are you solving > with running tests against binary builds? Perhaps, we need something > similar for other Ignite sub-projects as well. > > пт, 1 мар. 2019 г. в 19:04, Алексей Платонов : > > > > Hello, Igniters! > > I would like to create several tests for ML algorithms using binary > builds. > > These tests should work in this way: > > 1) Get last master (or user-defined branch) from git repository; > > 2) Build Ignite with a release profile and create binary build; > > 3) Run several Ignite instances from binary build; > > 4) Run examples or synthetic tests with a training of ML algorithms and > > inference; > > 5) Accumulate fails statistics on some board. > > > > Currently, I'm working with own open repository in git that contains > > scripts for Docker and Travis as the prototype. I want to complete these > > tests and contribute them to Ignite. > > > > Should I adapt such tests for TC after prototype complete or Travis can > be > > reused? Maybe such a process was created for other Ignite modules and I > can > > use it for ML. What do you think? > > > > Best regards > > Alexey Platonov. > > > > -- > Best regards, > Ivan Pavlukhin >
[MTCGA]: new failures in builds [3020618] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *Recently contributed test failed in master cache.IgniteCacheLockPartitionOnAffinityRunTxCacheOpTest https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=3741199923370523332=%3Cdefault%3E=testDetails *Recently contributed test failed in master baseline.IgniteBaselineLockPartitionOnAffinityRunTxCacheTest https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7575422310291368820=%3Cdefault%3E=testDetails *Recently contributed test failed in master baseline.IgniteBaselineLockPartitionOnAffinityRunAtomicCacheTest https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3750933139143423230=%3Cdefault%3E=testDetails Changes may lead to failure were done by - akuznetsov https://ci.ignite.apache.org/viewModification.html?modId=875001 - sergey.chugunov https://ci.ignite.apache.org/viewModification.html?modId=874985 - isapego https://ci.ignite.apache.org/viewModification.html?modId=874986 - vozerov https://ci.ignite.apache.org/viewModification.html?modId=874922 - alexey.goncharuk https://ci.ignite.apache.org/viewModification.html?modId=874948 - ppozerov https://ci.ignite.apache.org/viewModification.html?modId=875014 - kbolyand https://ci.ignite.apache.org/viewModification.html?modId=874993 - dpavlov https://ci.ignite.apache.org/viewModification.html?modId=874945 - dpavlov https://ci.ignite.apache.org/viewModification.html?modId=874960 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 15:56:28 02-03-2019
