Re: [DISCUSS] Missed (non-suited) tests
Hi! Yes, now it's a part of the Travis check, and there is already a first successful build [1]. But I think it's also required to run the check on TC too, along with jobs for checking licenses, code style, and core inspections. [1] https://travis-ci.com/github/apache/ignite/builds/216363067 On Fri, Feb 5, 2021 at 7:13 PM Ilya Kasnacheev wrote: > Hello! > > I have merged it to master! > > I wonder what happens next. It will run as a part of travis check? Do we > also need to add it as a TC suite? > > Regards, > -- > Ilya Kasnacheev > > > ср, 3 февр. 2021 г. в 18:50, Ilya Kasnacheev : > > > Hello! > > > > Code mostly looks good, I have added a minor request. I will check how it > > works and then we may commit. > > > > + zaleslaw@gmail.com > > > > Can you please check whether the new ML suites make sense? > > math/distances/DistancesTestSuite.java > > naivebayes/NaiveBayesTestSuite.java > > > > Would we need to add them to some TC runs? > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > пн, 25 янв. 2021 г. в 22:07, Max Timonin : > > > >> Hi, Ilya! > >> > >> I made a fix to the check. Now it aggregates info about tests and suites > >> from all modules and then validates it. Could you please review the PR > >> [1]? > >> > >> I tried to move some tests between modules, but unfortunately it still > >> looks like spaghetti. So I reverted all changes to testsuites (new and > >> splitted suites) and reworked the check. > >> > >> [1] https://github.com/apache/ignite/pull/8367 > >> > >> On Mon, Dec 28, 2020 at 2:03 PM Ilya Kasnacheev < > >> ilya.kasnach...@gmail.com> > >> wrote: > >> > >> > Hello! > >> > > >> > You could try to move these tests as .java files between modules in a > >> > separate commit. I think I could review it. > >> > > >> > Regards, > >> > -- > >> > Ilya Kasnacheev > >> > > >> > > >> > пт, 25 дек. 2020 г. в 17:19, Max Timonin : > >> > > >> > > Hi! > >> > > > >> > > Ilya thanks for the reply! I agree that it's a valid case when a > test > >> is > >> > > part of multiple suites in different modules. But it is definitely a > >> bug > >> > > that the test is written in a module where it can't be run at all > and > >> > aimed > >> > > to run within different modules (core tests in core that require > h2). > >> I > >> > > propose to fix this issue. > >> > > > >> > > I'm going to check all such tests and move them to the right module. > >> As I > >> > > can see there are about 100 such test classes, but I hope that most > of > >> > them > >> > > follow only a few patterns. > >> > > > >> > > On Fri, Dec 25, 2020 at 2:58 PM Ivan Daschinsky < > ivanda...@gmail.com> > >> > > wrote: > >> > > > >> > > > Hi! > >> > > > >> I'm not sure that we should assume every test is only run from > >> one > >> > > test > >> > > > suite. One test may be run from different test suites located in > >> > > different > >> > > > modules. > >> > > > Agree. For example, if we introduce this limitation, zk suites > will > >> be > >> > > > broken. > >> > > > > >> > > > > >> > > > пт, 25 дек. 2020 г. в 14:48, Ilya Kasnacheev < > >> > ilya.kasnach...@gmail.com > >> > > >: > >> > > > > >> > > > > Hello! > >> > > > > > >> > > > > Sorry for the long wait. > >> > > > > > >> > > > > I'm not sure that we should assume every test is only run from > one > >> > test > >> > > > > suite. One test may be run from different test suites located in > >> > > > different > >> > > > > modules. > >> > > > > > >> > > > > I wonder if we can drop this requirement, check all the modules > >> > > > > transitively for used/unused tests. > >> > > > > > >> > > > > Regards, > >> > > > > -- > >> > > > > Ilya Kasnacheev > >> > > > > > >> > > > > > >> > > > > ср, 2 дек. 2020 г. в 18:23, Max Timonin < > timonin.ma...@gmail.com > >> >: > >> > > > > > >> > > > > > Hi, > >> > > > > > > >> > > > > > I don't think so. It looks like a bug that tests fail if one > >> runs > >> > > them > >> > > > > > within their module (actually, what is the goal of this > test?). > >> The > >> > > > check > >> > > > > > showed us this problem, there is no need to fix the check. > >> > > > > > > >> > > > > > Currently I see two ways: > >> > > > > > 1. Find the right module for every misplaced test. There are > 104 > >> > > tests, > >> > > > > > maybe just move them all to the target module? If TeamCity > runs > >> > them > >> > > > > within > >> > > > > > the indexing module only is there a reason to have a test in > the > >> > core > >> > > > > > module at all? > >> > > > > > 2. Back to my previous proposal - create fake suites within a > >> > module, > >> > > > > then > >> > > > > > replace test classes in a target suite with the single class > of > >> the > >> > > > fake > >> > > > > > suite. > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > On Wed, Dec 2, 2020 at 5:38 PM Ilya Kasnacheev < > >> > > > > ilya.kasnach...@gmail.com> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hello! > >> > > > > > > > >> > > > > > > I think this means that we should aband
Re[4]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.
Hello Ilya ! >Hello! > >In the test GridDifferentLocalDeploymentSelfTest, I can see that some count >of internal data structure yanked from a class field is not equal expected >(without the fix). GridDifferentLocalDeploymentSelfTest checks that inner deployment machinery are not «leak» the only test without fix will generate huge number of SharedDeployment instances which will lead to jvm metaspace oom. Previous algorithm just replace old execution implementation with new one what led to new ClassLoaderId generation each time classloader was changed in client node. The second IgniteExplicitImplicitDeploymentSelfTest demonstrates correct concurrent job calling. >But I don't see why it is a problem. Do you have some reproducer where bad >things (such as task failure, deadlock, critical error) happen? > >Regards, >-- >Ilya Kasnacheev > > >пт, 5 февр. 2021 г. в 16:00, Zhenya Stanilovsky < arzamas...@mail.ru >: > >> >> Ilya, as previously agreed, ticket [1], examples of concurrent tests you >> can find here GridDifferentLocalDeploymentSelfTest and here >> IgniteExplicitImplicitDeploymentSelfTest >> < >> https://github.com/apache/ignite/pull/8759/files#diff-802af604a112f46b1282431673f3b143da5f2bfc4da70dbc8f0cbc93a3ed0f80 >> >, >> TC in progress. >> >> thanks ! >> >> [1] https://issues.apache.org/jira/browse/IGNITE-14131 >> >> >> >> Hello! >> >> Please publish it. I don't see why not. >> >> Regards, >> >> >> >> >> >> >> -- >> >> >> >>
Apache Board Report, Feb 10
Hi Igniters, Feb 10 is the due date for submitting the next board report. I would like to submit it a bit earlier, by the end of the day on Feb 9. Here is the first version of the report (at the end of the email). Should we add here something important? What do you think if it worths mentioning/requires board attention: schema first discussions? IEP-61? Extensions release? Anything related to the 3.0 development approach? Sincerely, Dmitriy Pavlov ## Description: The mission of Ignite is the creation and maintenance of software related to High-performance, integrated, and distributed In-Memory Database and Caching Platform providing in-memory data caching, partitioning, processing, and querying components. ## Issues: There are no issues requiring board attention ## Membership Data: Apache Ignite was founded 2015-08-19 (5 years ago) There are currently 56 committers and 35 PMC members in this project. The Committer-to-PMC ratio is 8:5. Community changes, past quarter: - Alex Plehanov was added to the PMC on 2020-11-18 - Ivan Bessonov was added as a committer on 2021-01-19 - Nikita Amelchev was added as a committer on 2021-01-21 ## Project Activity: Development - Apache Ignite community voted to remove MVCC public API. Test and test suites on the TeamCity are under discussion. - 3.0.0-alpha1 was released on 2021-01-11, this release is the very first build made from standalone branch Ignite 3.0, such releases should help in early adoption and testing of 3.0 - 2.9.1 was released on 2020-12-28 (latest stable) - this release patches 2.9.0 and mostly contains fixes - 2.10.0 release is being prepared Events and conferences - Community members and Ignite users continue to run talks at online meetups ## Community Health: - dev@, user@ list traffic, and code contributions are almost the same as in the past quarter - Issues, PRs related activity have increased (from +14% to +49%)
[MTCGA]: new failures in builds [5855057] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *Test with high flaky rate in master IgniteProjectionStartStopRestartSelfTest.testStopNodes https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=1179745277331816127&branch=%3Cdefault%3E&tab=testDetails Changes may lead to failure were done by - maksim timonin https://ci.ignite.apache.org/viewModification.html?modId=915931 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 22:44:48 07-02-2021
Re[4]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.
As previously agreed, ticket [1], examples of concurrent tests you can find
here GridDifferentLocalDeploymentSelfTest and here
IgniteExplicitImplicitDeploymentSelfTest , TC in progress. Fabric — possibly i
mention incorrect word here, more clear it need to be — factory )
I suppose additional tests would be helpful.
thanks !
[1] https://issues.apache.org/jira/browse/IGNITE-14131
>Zhenya,
>
>Could you clarify what you mean by "one instance is shared between numerous
>of fabric"? What is the exact scenario and what are the implications of
>running multiple compute tasks in parallel? A code sample demonstrating the
>scenario might be helpful as well.
>
>-Val
>
>On Fri, Feb 5, 2021 at 12:58 PM Vladislav Pyatkov < vldpyat...@gmail.com >
>wrote:
>
>> Hi Zhenya,
>>
>> I don't understand your proposal without a patch.
>> I see that you want to mark as @Depricate three methods in IgniteCompute
>> interface, but what will you give instead of?
>>
>> It seems to me, this interface can invoke some job without contains a class
>> locally.
>> How will this be supported in your mind?
>>
>> On Thu, Jan 28, 2021 at 6:58 PM Ilya Kasnacheev < ilya.kasnach...@gmail.com
>> >
>> wrote:
>>
>> > Hello!
>> >
>> > Please publish it. I don't see why not.
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > чт, 28 янв. 2021 г. в 14:28, Zhenya Stanilovsky
>> < arzamas...@mail.ru.invalid
>> > >:
>> >
>> > >
>> > >
>> > > Hi Ilya , of course it contains in my PR (i don`t know if it shout be
>> > > published before this discussion will be finished).
>> > > Little changes from single thread into multiple, for example here [1]
>> > will
>> > > highlight a problem, or i can just publish my PR.
>> > >
>> > > [1]
>> > >
>> >
>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/IgniteExplicitImplicitDeploymentSelfTest.java#L221
>> > >
>> > > >
>> > > >>
>> > > >>>Hello!
>> > > >>>
>> > > >>>Do you have some kind of reproducer which demonstrates the issue?
>> > > >>>
>> > > >>>Regards,
>> > > >>>--
>> > > >>>Ilya Kasnacheev
>> > > >>>
>> > > >>>
>> > > >>>чт, 28 янв. 2021 г. в 10:32, Zhenya Stanilovsky <
>> > > arzamas...@mail.ru.invalid
>> > > :
>> > > >>>
>> > >
>> > > Hello Igniters !
>> > > In the process of Ignite usage i found that some part of Compute
>> > > functionality are thread unsafe and seems was designed with such
>> > > limitations initially.
>> > > Example : one (client, but it doesn`t matter at all) instance is
>> > > shared between numerous of fabric, all of them calls something
>> like
>> > :
>> > > IgniteCompute#execute(ComputeTask, T)
>> > > or
>> > > IgniteCompute#execute(java.lang.Class>,
>> > T)
>> > > and appropriate «async» methods — what kind of instance will be
>> > > called is
>> > > nondeterministic for now and as a confirmation of my words — i
>> found
>> > > no
>> > > tests covered multi thread usage of Computing i also found nothing
>> > on
>> > > documentation page [1].
>> > > We have all necessary info for correct processing of such cases:
>> > > from initiator (ignite.compute(...) starter) side we have Class or
>> > it
>> > > instance and appropriate class loader which will be wired by class
>> > > loader
>> > > id from execution side.
>> > > I create a fix and seems all work perfectly well besides one
>> place,
>> > > this
>> > > functionality :
>> > > /**
>> > > * Executes given task within the cluster group. For step-by-step
>> > > explanation of task execution process
>> > > * refer to {@link ComputeTask} documentation.
>> > > *
>> > > * If task for given name has not been deployed yet, then {@code
>> > > taskName}
>> > > will be
>> > > * used as task class name to auto-deploy the task (see {@link
>> > > #localDeployTask(Class, ClassLoader)} method).
>> > > */
>> > > public R execute(String taskName, T arg) throws
>> > > IgniteException;
>> > > and attendant
>> > > /**
>> > > * Finds class loader for the given class.
>> > > *
>> > > * @param rsrcName Class name or class alias to find class loader
>> > for.
>> > > * @return Deployed class loader, or {@code null} if not deployed.
>> > > */
>> > > public DeploymentResource findResource(String rsrcName);
>> > > is thread unsafe by default, no guarantee that concurrent call of
>> > > localDeployTask and execute will bring expected result.
>> > > My proposal is to deprecate (or probably annotate [2], as a
>> minimal
>> > > — additionally document it) this methods and to append additional
>> :
>> > > public DeploymentResource findResource(String rsrcName,
>> ClassLoader
>> > > clsLdr);
>> > > Only one problem i can observe here, if someone creates new class
>> > > loaders
>> > > and appropriate class instances in loop (i don`t know the purpose)
>> > a
