Re: [TC] Examples suite issues
Hey Peter, Anton, Are any of you able to help Nickolay with [1]? - Denis On Mon, Jun 3, 2019 at 6:17 AM Nikolay Izhikov wrote: > Hello, Igniters. > > We have several issues with Examples suite on TC > > 1. IgniteExamplesSparkSelfTestSuite seems configure, bu not runs on suite > execution. [1] > > 2. Spark Data Frame examples doesn't work in master. Seems some classpath > issues [2] > > I will resolve issue #2. > Can anyone help me with #1? > > > [1] > https://ci.ignite.apache.org/viewLog.html?buildId=4027327=IgniteTests24Java8_Examples=buildParameters > [2] https://issues.apache.org/jira/browse/IGNITE-11881 >
Re: Make the TeamCity console quiet.
Hi Igniters, For older versions, where the needed configuration is absent you may found the following behavior of tests SPI [ tests 1 ] JDK8 IgniteSpiTestSuite: TestSuite$1.warning Cache (Restarts) 1 [ tests 1 ] JDK8 IgniteCacheRestartTestSuite: TestSuite$1.warning If you found such issue consider unchecking `Ignite Quiet` mode for tests run for an older version. >From the configuration point of view, you may set build parameter reverse.dep.*.IGNITE_LOGGING_OPTS to value: -DIGNITE_QUIET=false Sincerely, Dmitriy Pavlov пт, 19 апр. 2019 г. в 12:59, Maksim Stepachev : > Run params example: > > https://issues.apache.org/jira/secure/attachment/12966470/quiet-console-checkbox.png > > пт, 19 апр. 2019 г. в 12:55, Dmitriy Pavlov : > > > Hi Maxim, > > > > Apache Lists removes images from emails. Could you please add this image > > somewhere and share a link? > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 19 апр. 2019 г. в 12:52, Maksim Stepachev < > maksim.stepac...@gmail.com > > >: > > > > > Hi, > > > The checkbox was added into the "Run Custom Build" window (Quiet > console > > > output *). Quiet is active by default. > > > > > > [image: quite-console-checkbox.png] > > > https://issues.apache.org/jira/browse/IGNITE-11736 > > > > > > > > > вт, 9 апр. 2019 г. в 11:06, Павлухин Иван : > > > > > >> Maksim, > > >> > > >> I like such approach. If nobody objects let's go ahead and try it out. > > >> > > >> вт, 9 апр. 2019 г. в 10:38, Maksim Stepachev < > > maksim.stepac...@gmail.com > > >> >: > > >> > > > >> > Ivan, > > >> > > > >> > 1. It'll some selector which changes ENV parameter for build run > with > > >> quiet > > >> > by default. > > >> > 2. Yes, the additional log config will be added for TC. > > >> > > > >> > > > >> > пн, 8 апр. 2019 г. в 17:08, Павлухин Иван : > > >> > > > >> > > Hi Maxim, > > >> > > > > >> > > Thank you for continuing with that initiative. Could you please > > >> > > elaborate on following: > > >> > > 1. What changes are required on TC? > > >> > > 2. Will everything work without changes on a developer's local > > >> machine? > > >> > > > > >> > > пн, 8 апр. 2019 г. в 13:13, Maksim Stepachev < > > >> maksim.stepac...@gmail.com>: > > >> > > > > > >> > > > Hi, everyone! > > >> > > > > > >> > > > I want to push the topic up and discuss a final solution about > the > > >> quiet > > >> > > > console. > > >> > > > > > >> > > > The suggestion above: > > >> > > > 1. Rollover will be locked. > > >> > > > Pros: Only one big file in an archive. > > >> > > > Cons: Max size of the file isn't limited. > > >> > > > 2. Run all will contain a parameter for switch off the quiet > mode. > > >> > > > 3. New config: log4j-tc-test.xml for TeamCity environment. > > >> > > > > > >> > > > чт, 14 февр. 2019 г. в 15:51, Павлухин Иван < > vololo...@gmail.com > > >: > > >> > > > > > >> > > > > Maksim, > > >> > > > > > > >> > > > > I tried configuration with quite console and unlimited log > file > > >> on my > > >> > > > > machine. Naively configured in log4j-test.xml it works not so > > >> fine in > > >> > > > > development environment because log file is appended by each > > test > > >> > > > > execution. Console redirection works better in development > > >> > > > > environment. Should we have separate log config for TC? > > >> > > > > > > >> > > > > By the way log redirection does not work well for every test > in > > >> > > > > development environment. Sometimes it seems that it could be > > >> easier to > > >> > > > > work with file. Sometimes IDEA shrinks console logs. Igniters, > > do > > >> you > > >> > > > > have ideas how to work with logs produced by tests more > > >> convenient? > > >> > > > > > > >> > > > > пн, 11 февр. 2019 г. в 17:29, Eduard Shangareev < > > >> > > > > eduard.shangar...@gmail.com>: > > >> > > > > > > > >> > > > > > Maksim, > > >> > > > > > > > >> > > > > > sounds good. > > >> > > > > > > > >> > > > > > 1) Can we just turn off log rollover? > > >> > > > > > 2) I believe, that we need the ability to override this by > > >> setting > > >> > > > > > parameter for Run All. > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > On Mon, Feb 11, 2019 at 4:12 PM Maksim Stepachev < > > >> > > > > maksim.stepac...@gmail.com> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > Ivan, > > >> > > > > > > > > >> > > > > > > Yes. It happens because we use the RollingFileAppender > for a > > >> file > > >> > > > > logging. > > >> > > > > > > This appender has the next properties: > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > Can I increase MaxFileSize by 200MB or more? I suppose > these > > >> limits > > >> > > > > were > > >> > > > > > > added for cases when the test writes a log. It makes the > > >> guarantee > > >> > > that > > >> > > > > > > logs less than 10*10 = 100MBs. But it's wrong if it writes > > >> into the > > >> > > > > console > > >> > > > > > > too. > > >> > > > > > > > > >> > > > > > > пн, 11 февр. 2019
Re: {DISCUSSION] Cluster read-only mode.
Sergey, Do we have an IEP for this feature? What should happen when on an active cluster with put operations if we receive a read-only state change request? How we guarantee that all cache operations delivered (or not yet) to backups are not rejected by applied read-only request? I haven't found such tests in your PR. I've downloaded your branch and run some tests locally. I've tried DataStreamer cache loads (allowOverwrite mode false) with a concurrent cluster change state request to read-only mode and I've got strange behaviour. My test scenario was: 1) Start DataStremer cache load; 2) change cluster to read-only state; 3) change state back to normal; When the state has been changed to `read-only` I've flooded with a lot of `Failed to perform cache operation (cluster is in read-only mode)` errors, but when I've reverted the state back the DataStreamer continue its load without any error. I think we should not allow such behaviour. We should cancel the DataStreamer task or allow it to be finished. On Fri, 31 May 2019 at 13:00, Sergey Antonov wrote: > > Hello, Zhenya, Maxim! > > Thank you for your replies! > > >> Should we also allow writes to the DistributedMetaStorage and if not why? > Yes. DistributedMetastorage available for updates with enabled read-only > mode. I added test about it to ClusterReadOnlyModeSelfTest > > >> What's the purpose for ignite-sys-cache updates still be available ? > ignite-sys-cache is using in the different subcomponents, for example, > security. > > чт, 30 мая 2019 г. в 20:30, Zhenya Stanilovsky : > > > hi, Sergey. > > What's the purpose for ignite-sys-cache updates still be available ? > > > > thanks ! > > > > > Hello Igniters! > > > > > > I'm working on cluster read-only mode [1] feature. In this mode cluster > > > will be available only for read operations, all data modification > > > operations in user caches will be rejected > > > with ClusterReadOnlyModeCheckedException. This feature could be helpfull > > > for maintenance works (control.sh idle_verify/validate_indexes). > > > > > > A few points about cluster read-only mode: > > > 1) Read-only mode could be enabled on active cluster only. > > > 2) Read-only mode doens't store on PDS (i.e. after cluster restart > > > enabled > > > read-only mode will be forgotten) > > > 3) Updates to ignite-sys-cache will be available with enabled read-only > > > mode. > > > > > > More informartion about implementation you could find in PR [2]. > > > > > > What do you think about this feature? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-11256 > > > [2] https://github.com/apache/ignite/pull/6423 > > > > > -- > BR, Sergey Antonov
Re: [ML] IGNITE-9978 Compound Naive Bayes Pull Request
Hi Alexey, thank you. Hi Ravil, 4 blockers from TC bot seems to be unrelated to the change. Please check only that contributed files have appropriate licenses. I see failure here https://ci.ignite.apache.org/viewLog.html?buildId=4003665=IgniteTests24Java8_LicensesHeaders=artifacts_IgniteTests24Java8=%3Cdefault%3E Sincerely, Dmitriy Pavlov пн, 3 июн. 2019 г. в 18:30, Alexey Zinoviev : > I will have a look this week, currently it has conflicts with master > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Re: Why TDE doc says that memory encryption is supported?
Guys, I updated the page. -Artem On 02.06.2019 22:20, Nikolay Izhikov wrote: Hello, Denis. Ignite doesn't encrypt memory page as long as it retains in memory. Page encrypted when it is written to the disk. пт, 31 мая 2019 г. в 19:39, Denis Magda : Nikolay, Do we really support encryption for pages in RAM? That's what I found in the docs: Ignite uses JDK-provided encryption algorithms: "AES/CBC/PKCS5Padding" for WAL records encryption and *"AES/CBC/NoPadding" for memory page encryption.* Shouldn't we remove the highlighted from the docs? - Denis
Re: [ML] IGNITE-9978 Compound Naive Bayes Pull Request
I will have a look this week, currently it has conflicts with master -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Code inspection
Hello, Ivan.
> For me fields without extra empty lines between them look fine.
This is violation of Ignite code style, so we should fix it for now.
If we decide to change code style, we can change check.
> Can/should we enforce no empty lines there?
You can look into checkstyle documentation for it.
For now, I don't know about such checks.
В Пн, 03/06/2019 в 16:11 +0300, Павлухин Иван пишет:
> Hi Nikolay,
>
> Great, another one nice step to clean code.
>
> I scrolled down the commit [1] and have couple of observations:
> 1. According to our code style agreements each class member should be
> separated by 1 empty line. As I see there were many places where it
> was not followed for class fields. E.g. empty lines were added for a
> code snippet below. For me fields without extra empty lines between
> them look fine. Of course, it is a question of our agreements. So, we
> can either permit no empty lines for fields or enforce empty line for
> each class member.
> private static class FlaggedCacheOperationCallable implements
> Callable, Serializable {
> /** */
> private static final long serialVersionUID = 0L;
> /** */
> private final String cacheName;
> /** */
> private final Set cacheFlags;
>
> 2. Also I noticed an empty line between a method signature and a first
> statement in the method in some places, e.g. [2]. Can/should we
> enforce no empty lines there?
>
> And one aside observation (in the same file), it has class fields
> without a javadoc [3]. Do not our code check jobs enforce javadoc for
> fields?
>
> [1]
> https://github.com/apache/ignite/commit/d63f4d3569dcb387394d367a7f00aaf35f1b288e
> [2]
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/util/io/GridReversedLinesFileReader.java#L154
> [3]
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/util/io/GridReversedLinesFileReader.java#L35
>
> вс, 2 июн. 2019 г. в 22:00, Nikolay Izhikov :
> >
> > Hello, Igniters.
> >
> > 1. I've added "EmptyLineSeparator" inspection, according to our code style
> > [1]
> > 2. I've fixed all current code style violations.
> >
> > Please, see my commit [2]
> >
> > You may want to merge these changes to your private repos.
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Whitespacesandemptylines
> > [2]
> > https://github.com/apache/ignite/commit/d63f4d3569dcb387394d367a7f00aaf35f1b288e
> >
> >
> >
> >
> > вт, 23 апр. 2019 г. в 10:45, Vyacheslav Daradur :
> >
> > > Also, I excluded "IntelliJ IDEA Inspections" from RunAll and marked it
> > > as "~[Excluded]".
> > >
> > > On Tue, Apr 23, 2019 at 12:25 AM Vyacheslav Daradur
> > > wrote:
> > > >
> > > > Maxim, I merged your changes to master.
> > > >
> > > > Also, I've created a new build plan "Check Code Style" on TC [1] and
> > > > included in RunAll build.
> > > > The report of check-style attaches in artifacts once build is finished.
> > > >
> > > > Please check that it works as expected once again and announce new
> > > > requirements in a separate thread to avoid confusion of community.
> > > >
> > > > cc Petr, Pavel (JFYI)
> > > >
> > > > [1]
> > >
> > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_CheckCodeStyle=buildTypeBranches
> > > >
> > > > On Sun, Apr 21, 2019 at 10:18 PM Vyacheslav Daradur
> > > >
> > >
> > > wrote:
> > > > >
> > > > > Maxim,
> > > > >
> > > > > I left some comments in Jira, let's resolve them and I'll assist you
> > > > > with merge and TC configuring.
> > > > >
> > > > > On Fri, Apr 19, 2019 at 5:50 PM Maxim Muzafarov
> > >
> > > wrote:
> > > > > >
> > > > > > Vyacheslav,
> > > > > >
> > > > > > Thank you for your interest in making Ignite coding style better.
> > > > > >
> > > > > > The short answer is - there are no different checkstyle
> > > > > > configurations. One for the JetBrains Inspections, and one the
> > > > > > Checkstyle plugin. This is a completely different approach for
> > > > > > checking the Ignites source code.
> > > > > >
> > > > > > Currently, we have two different configurations for the JetBrains
> > >
> > > IDEA
> > > > > > Inspection check:
> > > > > > - ignite\.idea\inspectionProfiles\Project_Default.xml - this is
> > > > > > default on the IDE level and used silently by every developer whos
> > > > > > checkout Ignite project (it remains)
> > > > > > - ignite\idea\ignite_inspections_teamcity.xml - this is the
> > > > > > configuration of the inspection for the TC suite (it will be
> > > > > > deleted)
> > > > > > It's unobvious to maintain both of them. Previously we've planned to
> > > > > > fix all the inspection rules one by one and add them one by one to
> > >
> > > the
> > > > > > TC inspection configuration file (something like storing the
> > > > > > intermediate result), but it didn't happen cause the inspection TC
> > > > > > suite got broken after migration to 2018
Re: Code inspection
Hi Nikolay,
Great, another one nice step to clean code.
I scrolled down the commit [1] and have couple of observations:
1. According to our code style agreements each class member should be
separated by 1 empty line. As I see there were many places where it
was not followed for class fields. E.g. empty lines were added for a
code snippet below. For me fields without extra empty lines between
them look fine. Of course, it is a question of our agreements. So, we
can either permit no empty lines for fields or enforce empty line for
each class member.
private static class FlaggedCacheOperationCallable implements
Callable, Serializable {
/** */
private static final long serialVersionUID = 0L;
/** */
private final String cacheName;
/** */
private final Set cacheFlags;
2. Also I noticed an empty line between a method signature and a first
statement in the method in some places, e.g. [2]. Can/should we
enforce no empty lines there?
And one aside observation (in the same file), it has class fields
without a javadoc [3]. Do not our code check jobs enforce javadoc for
fields?
[1]
https://github.com/apache/ignite/commit/d63f4d3569dcb387394d367a7f00aaf35f1b288e
[2]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/util/io/GridReversedLinesFileReader.java#L154
[3]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/util/io/GridReversedLinesFileReader.java#L35
вс, 2 июн. 2019 г. в 22:00, Nikolay Izhikov :
>
> Hello, Igniters.
>
> 1. I've added "EmptyLineSeparator" inspection, according to our code style
> [1]
> 2. I've fixed all current code style violations.
>
> Please, see my commit [2]
>
> You may want to merge these changes to your private repos.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Whitespacesandemptylines
> [2]
> https://github.com/apache/ignite/commit/d63f4d3569dcb387394d367a7f00aaf35f1b288e
>
>
>
>
> вт, 23 апр. 2019 г. в 10:45, Vyacheslav Daradur :
>
> > Also, I excluded "IntelliJ IDEA Inspections" from RunAll and marked it
> > as "~[Excluded]".
> >
> > On Tue, Apr 23, 2019 at 12:25 AM Vyacheslav Daradur
> > wrote:
> > >
> > > Maxim, I merged your changes to master.
> > >
> > > Also, I've created a new build plan "Check Code Style" on TC [1] and
> > > included in RunAll build.
> > > The report of check-style attaches in artifacts once build is finished.
> > >
> > > Please check that it works as expected once again and announce new
> > > requirements in a separate thread to avoid confusion of community.
> > >
> > > cc Petr, Pavel (JFYI)
> > >
> > > [1]
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_CheckCodeStyle=buildTypeBranches
> > >
> > > On Sun, Apr 21, 2019 at 10:18 PM Vyacheslav Daradur
> > wrote:
> > > >
> > > > Maxim,
> > > >
> > > > I left some comments in Jira, let's resolve them and I'll assist you
> > > > with merge and TC configuring.
> > > >
> > > > On Fri, Apr 19, 2019 at 5:50 PM Maxim Muzafarov
> > wrote:
> > > > >
> > > > > Vyacheslav,
> > > > >
> > > > > Thank you for your interest in making Ignite coding style better.
> > > > >
> > > > > The short answer is - there are no different checkstyle
> > > > > configurations. One for the JetBrains Inspections, and one the
> > > > > Checkstyle plugin. This is a completely different approach for
> > > > > checking the Ignites source code.
> > > > >
> > > > > Currently, we have two different configurations for the JetBrains
> > IDEA
> > > > > Inspection check:
> > > > > - ignite\.idea\inspectionProfiles\Project_Default.xml - this is
> > > > > default on the IDE level and used silently by every developer whos
> > > > > checkout Ignite project (it remains)
> > > > > - ignite\idea\ignite_inspections_teamcity.xml - this is the
> > > > > configuration of the inspection for the TC suite (it will be deleted)
> > > > > It's unobvious to maintain both of them. Previously we've planned to
> > > > > fix all the inspection rules one by one and add them one by one to
> > the
> > > > > TC inspection configuration file (something like storing the
> > > > > intermediate result), but it didn't happen cause the inspection TC
> > > > > suite got broken after migration to 2018 version.
> > > > >
> > > > > Now it seems to me, that it is better to use the best open source
> > > > > practices like checkstyle plugin (380K usages on github repositories)
> > > > > rather than proprietary software. So, we will:
> > > > > - keep IDE level inspection configuration (the Project_Default.xml
> > config);
> > > > > - add a new checkstyle plugin configuration file (checkstyle.xml
> > > > > config) which will be used simultaneously for checking code style on
> > > > > build procedure and for the IDE-checkstyle plugin;
> > > > >
> > > > > On Wed, 17 Apr 2019 at 21:23, Vyacheslav Daradur <
> > daradu...@gmail.com> wrote:
> > > > > >
> > > > > > Maxim,
> > > > > >
> > > > > > I looked through
[jira] [Created] (IGNITE-11889) Add security permissions cluster state change operations
Sergey Antonov created IGNITE-11889: --- Summary: Add security permissions cluster state change operations Key: IGNITE-11889 URL: https://issues.apache.org/jira/browse/IGNITE-11889 Project: Ignite Issue Type: Bug Components: security Reporter: Sergey Antonov We should add security permissions for cluster state change operations: * Cluster activation/deactivation. * Set/change baseline topology. * Enable/disable read-only mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [ML] Deployment of user-defined preprocessors
It's an amazing idea! I faced this problem several times and it's really annoying, I'd be glad to this problem fixed. Best regards, Anton Dmitriev. -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
