Re: NPE exception in KMeansTrainer
Thank you, I think I've found this bug (or related to this) here
https://issues.apache.org/jira/browse/IGNITE-9239
It will be delivered in 2.7 (Currently it's in master branch).
To be sure 100% that the bug is closed, could @DocDVZ provide an approach
of cache populating?
I mean this cache
IgniteCache dataCache = ignite.cache(storageName);
Thank you.
вт, 21 авг. 2018 г. в 10:11, Denis Magda :
> Hey, ML experts,
>
> Here is an ML issue reported. Please have a look.
>
> --
> Denis
>
> -- Forwarded message -
> From: DocDVZ
> Date: Mon, Aug 20, 2018 at 10:53 AM
> Subject: NPE exception in KMeansTrainer
> To:
>
>
> Hello,
>
> Since I'm new to data science, I'm not really sure if it's a bug or wrong
> incoming data, so I decided to ask here for advice before submitting a
> ticket. I tried to apply Kmeans algorithm on my bag-of-words data with ~8k
> features. So I copy-pasted some lines from example:
>
> IgniteCache dataCache =
> ignite.cache(storageName);
> KMeansTrainer trainer = new KMeansTrainer().withSeed(1234L);
> KMeansModel mdl = trainer.fit(
> ignite,
> dataCache,
> (k, v) -> Arrays.copyOfRange(v, 1, v.length),
> (k, v) -> v[0]
> );
>
> But this leads to a NullPointerException in KMeansTrainer.class:
>
> Caused by: java.lang.NullPointerException
> at
> org.apache.ignite.ml
>
> .clustering.kmeans.KMeansTrainer.lambda$initClusterCentersRandomly$4dba08e1$1(KMeansTrainer.java:190)
> at
>
> org.apache.ignite.ml.dataset.impl.cache.CacheBasedDataset.computeForAllPartitions(CacheBasedDataset.java:158)
> at
>
> org.apache.ignite.ml.dataset.impl.cache.CacheBasedDataset.compute(CacheBasedDataset.java:122)
> at org.apache.ignite.ml.dataset.Dataset.compute(Dataset.java:102)
> at org.apache.ignite.ml.dataset.Dataset.compute(Dataset.java:156)
> at
> org.apache.ignite.ml
>
> .clustering.kmeans.KMeansTrainer.initClusterCentersRandomly(KMeansTrainer.java:186)
> at
> org.apache.ignite.ml
> .clustering.kmeans.KMeansTrainer.fit(KMeansTrainer.java:86)
>
>
> at line:
>
> List rndPnts = dataset.compute(data -> {
> List rndPnt = new ArrayList<>();
> rndPnt.add(data.getRow(new
> Random(seed).nextInt(data.rowSize(;
> return rndPnt;
> }, (a, b) -> a == null ? b : Stream.concat(a.stream(),
> b.stream()).collect(Collectors.toList()));
>
> The reducer receives null value for b and since there's no check for null,
> b.stream() leads to NPE. Ignite version is 2.6. This seems like a bug for
> me, is there any ways to workaround this issue?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Fwd: NPE exception in KMeansTrainer
Hey, ML experts,
Here is an ML issue reported. Please have a look.
--
Denis
-- Forwarded message -
From: DocDVZ
Date: Mon, Aug 20, 2018 at 10:53 AM
Subject: NPE exception in KMeansTrainer
To:
Hello,
Since I'm new to data science, I'm not really sure if it's a bug or wrong
incoming data, so I decided to ask here for advice before submitting a
ticket. I tried to apply Kmeans algorithm on my bag-of-words data with ~8k
features. So I copy-pasted some lines from example:
IgniteCache dataCache = ignite.cache(storageName);
KMeansTrainer trainer = new KMeansTrainer().withSeed(1234L);
KMeansModel mdl = trainer.fit(
ignite,
dataCache,
(k, v) -> Arrays.copyOfRange(v, 1, v.length),
(k, v) -> v[0]
);
But this leads to a NullPointerException in KMeansTrainer.class:
Caused by: java.lang.NullPointerException
at
org.apache.ignite.ml
.clustering.kmeans.KMeansTrainer.lambda$initClusterCentersRandomly$4dba08e1$1(KMeansTrainer.java:190)
at
org.apache.ignite.ml.dataset.impl.cache.CacheBasedDataset.computeForAllPartitions(CacheBasedDataset.java:158)
at
org.apache.ignite.ml.dataset.impl.cache.CacheBasedDataset.compute(CacheBasedDataset.java:122)
at org.apache.ignite.ml.dataset.Dataset.compute(Dataset.java:102)
at org.apache.ignite.ml.dataset.Dataset.compute(Dataset.java:156)
at
org.apache.ignite.ml
.clustering.kmeans.KMeansTrainer.initClusterCentersRandomly(KMeansTrainer.java:186)
at
org.apache.ignite.ml
.clustering.kmeans.KMeansTrainer.fit(KMeansTrainer.java:86)
at line:
List rndPnts = dataset.compute(data -> {
List rndPnt = new ArrayList<>();
rndPnt.add(data.getRow(new
Random(seed).nextInt(data.rowSize(;
return rndPnt;
}, (a, b) -> a == null ? b : Stream.concat(a.stream(),
b.stream()).collect(Collectors.toList()));
The reducer receives null value for b and since there's no check for null,
b.stream() leads to NPE. Ignite version is 2.6. This seems like a bug for
me, is there any ways to workaround this issue?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Unknown known issue on cache rebalancing delayed
Hi Maxim, I have some issues with a cluster with rebalance delay enabled, but need to check more -- if I find it's related I'll share.Just wanted to make sure it's not an issue anymore from someone working on rebalancing. We should remove that comment then, it looks scary :) -- Roman Shtykh On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov wrote: Hello Roman, Did you faced with real issue of delayed rebalance or it's just only for your personal interest? If yes, please, share details and we will try to help you. As for this comment I don't think he is actual. That change was in 2015. Much has changed within rebalance process since that time. I've uncommented it and rechecked with thatcache configuration and haven't seen any failed tests or issues. Probably, that problem was about cache in SYNC mode does not start util it loads all data from other nodes. But currently delayed rebalance works the same way as IgniteCache#rebalance(), so you can `setRebalanceDelay` to `-1` and call it manually to check. On Mon, 20 Aug 2018 at 11:19 Roman Shtykh wrote: Igniters, I have found "Known issue, possible deadlock in case of low priority cache rebalancing delayed" comment in GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please explain when using rebalance delay can be an issue and why? -- Roman -- -- Maxim Muzafarov
[GitHub] ignite pull request #4381: Adding experimental support for Intel Optane DC P...
GitHub user mulugetam reopened a pull request: https://github.com/apache/ignite/pull/4381 Adding experimental support for Intel Optane DC Persistent Memory Ignite, when persistence mode is enabled, stores data and indexes on disk. To minimize the latency of disks, several tuning options can be applied. Setting the page size of a memory region to match the page size of the underlying storage, using a separate disk for the WAL, and using production-level SSDs are just a few of them [https://apacheignite.readme.io/docs/durable-memory-tuning#section-native-persistence-related-tuning]. A persistent memory store with low latency and high capacity offers a viable alternative to disks. In light of this, this patch introduces an experimental support for Intel Optane DC Persistent Memory (aka 3DXPoint, AEP) based on our Low Level Persistent Library (LLPL). The patch eliminates the checkpoint process and the WAL to propagate changes to disk and maintain durability. Instead, we rely on a transactional memory region/block defined by LLPL to store both data and indexes into a persistent heap that is restored back upon application reboots. LLPL: https://github.com/pmem/LLPL/ Examples: examples/src/main/java/org/apache/ignite/examples/aep/ You can merge this pull request into a Git repository by running: $ git pull https://github.com/mulugetam/ignite persistent-memory Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4381.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4381 commit 6fd30099004d880c32cf7282819079e97795a83d Author: Mulugeta Mammo Date: 2018-08-20T23:18:58Z Adds experimental support for Intel DC Persistent Memory. ---
[GitHub] ignite pull request #4381: Adding experimental support for Intel Optane DC P...
Github user mulugetam closed the pull request at: https://github.com/apache/ignite/pull/4381 ---
Re: Table Names in Spark Catalog
Stuart, Nikolay, I see that the 'Table' class (returned by listTables method) has a 'database' field. Can we use this one to report schema name? In any case, I think we should look into how this is done in data source implementations for other databases. Any relational database has a notion of schema, and I'm sure Spark integrations take this into account somehow. -Val On Mon, Aug 20, 2018 at 6:12 AM Nikolay Izhikov wrote: > Hello, Stuart. > > Personally, I think we should change current tables naming and return > table in form of `schema.table`. > > Valentin, could you share your opinion? > > > В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет: > > Igniters, > > > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I are > > discussing whether to introduce a change which may impact backwards > > compatibility; Nikolay suggested we take the discussion to this list. > > > > Ignite implements a custom Spark catalog which provides an API by which > > Spark users can list the tables which are available in Ignite which can > be > > queried via Spark SQL. Currently that table name list includes just the > > names of the tables, but IGNITE-9228 is introducing a change which allows > > optional prefixing of schema names to table names to disambiguate > multiple > > tables with the same name in different schemas. For the "list tables" API > > we therefore have two options: > > > > 1. List the tables using both their table names and schema-qualified > table > > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are the > same > > underlying table. This retains backwards compatibility with users who > > expect "myTable" to appear in the catalog. > > 2. List the tables using only their schema-qualified names. This > eliminates > > duplication of names in the catalog but will potentially break > > compatibility with users who expect the table name in the catalog. > > > > With either option we will allow for Spark SQL SELECT statements to use > > either table name or schema-qualified table names, this change would > purely > > impact the API which is used to list available tables. > > > > Any opinions would be welcome. > > > > Thanks, > > Stuart. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9228 > > [2] https://github.com/apache/ignite/pull/4551
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Dmitriy Due to significant improvement and to reduce the number supported modes/options would be good to convert the data at the moment of upgrade. On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan wrote: > Sergey, that was precisely my comment in the ticket: > > Can we add this option without breaking compatibility with previous > page/storage formats? If not, then this should support both implementation. > The default should be the new fastest implementation, but if we encounter > the older, slower one, then we should print out a warning in the log and > automatically switch to the older implementation. > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > wrote: > > > Hi Igniters > > > > I suppose that'll break compatibility for LFS (PDS). > > > > Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x > to > > 3.0? > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > dsetrak...@apache.org > > > > > wrote: > > > > > I commented in the ticket: https://issues.apache.org/ > > > jira/browse/IGNITE-9272 > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > release, > > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > > > D. > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > I have checked the benchmark and it shows great performance boost on > my > > > > laptop! > > > > > > > > +1 for this change. > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > dpavlov@gmail.com> > > > > wrote: > > > > > > > > > Hi Evgeniy, > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > Ignite? > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > : > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > Thanks! > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > > > >difference? > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > dpavlov@gmail.com > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> It depends. > > > > > > >> > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > > store > > > > > write > > > > > > >> are mostly about IO speed. > > > > > > >> > > > > > > >> In the same time, it can make the huge impact on machines with > > > fast > > > > IO > > > > > > >> and > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > > Alexey > > > > it > > > > > > >> could > > > > > > >> benefit performance because we save CPU. Later we can use it's > > > power > > > > > in > > > > > > a > > > > > > >> more efficient manner (e.g. with compression). > > > > > > >> > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > > yzhda...@apache.org > > > >: > > > > > > >> > > > > > > >> > Guys, what time in % does crc calculation take in WAL > logging > > > > > process? > > > > > > >> > > > > > > > >> > --Yakov > > > > > > >> > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > > dpavlov@gmail.com > > > > > >: > > > > > > >> > > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > > >> > > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > > bypassing > > > > > > >> > > implementation differences to keep compatibility? > > > > > > >> > > > > > > > > >> > > Sincerely, > > > > > > >> > > Dmitriy Pavlov > > > > > > >> > > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > > plehanov.a...@gmail.com >: > > > > > > >> > > > > > > > > >> > > > Hello, Igniters! > > > > > > >> > > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > > moreover > > > > > > new > > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > > Since > > > we > > > > > > >> moved > > > > > > >> to > > > > > > >> > > > java8, perhaps we really can get performance boost by > > using > > > > > > >> standard > > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > > >> > > > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements > > the > > > > > same > > > > > > >> > > algorithm > > > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses > the > > > > same > > > > > > >> > > polynomial > > > > > > >> > > > and the same initial value. The only difference is final > > xor > > > > > mask > > > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > > > convert > > > > > > >> from > > > > > > >> > > > PureJavaCrc32 > > > > > > >> > > > to standard CRC32 and vice versa, using this expression: > > > crc32 > > > > > ^= > > > > > > >> > > > 0x > > > > > > >> > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Igniters, let's come back to the idea from Alex Plehanov. We can try to provide custom xor mask at the end of CRC calc. We can do a compatible solution using fast implementation, can't we? Sincerely, Dmitriy Pavlov вт, 21 авг. 2018 г. в 0:04, Dmitriy Setrakyan : > Sergey, that was precisely my comment in the ticket: > > Can we add this option without breaking compatibility with previous > page/storage formats? If not, then this should support both implementation. > The default should be the new fastest implementation, but if we encounter > the older, slower one, then we should print out a warning in the log and > automatically switch to the older implementation. > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > wrote: > > > Hi Igniters > > > > I suppose that'll break compatibility for LFS (PDS). > > > > Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x > to > > 3.0? > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > dsetrak...@apache.org > > > > > wrote: > > > > > I commented in the ticket: https://issues.apache.org/ > > > jira/browse/IGNITE-9272 > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > release, > > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > > > D. > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > I have checked the benchmark and it shows great performance boost on > my > > > > laptop! > > > > > > > > +1 for this change. > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > dpavlov@gmail.com> > > > > wrote: > > > > > > > > > Hi Evgeniy, > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > Ignite? > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > : > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > Thanks! > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > > > >difference? > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > dpavlov@gmail.com > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> It depends. > > > > > > >> > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > > store > > > > > write > > > > > > >> are mostly about IO speed. > > > > > > >> > > > > > > >> In the same time, it can make the huge impact on machines with > > > fast > > > > IO > > > > > > >> and > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > > Alexey > > > > it > > > > > > >> could > > > > > > >> benefit performance because we save CPU. Later we can use it's > > > power > > > > > in > > > > > > a > > > > > > >> more efficient manner (e.g. with compression). > > > > > > >> > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > > yzhda...@apache.org > > > >: > > > > > > >> > > > > > > >> > Guys, what time in % does crc calculation take in WAL > logging > > > > > process? > > > > > > >> > > > > > > > >> > --Yakov > > > > > > >> > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > > dpavlov@gmail.com > > > > > >: > > > > > > >> > > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > > >> > > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > > bypassing > > > > > > >> > > implementation differences to keep compatibility? > > > > > > >> > > > > > > > > >> > > Sincerely, > > > > > > >> > > Dmitriy Pavlov > > > > > > >> > > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > > plehanov.a...@gmail.com >: > > > > > > >> > > > > > > > > >> > > > Hello, Igniters! > > > > > > >> > > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > > moreover > > > > > > new > > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > > Since > > > we > > > > > > >> moved > > > > > > >> to > > > > > > >> > > > java8, perhaps we really can get performance boost by > > using > > > > > > >> standard > > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > > >> > > > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements > > the > > > > > same > > > > > > >> > > algorithm > > > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses > the > > > > same > > > > > > >> > > polynomial > > > > > > >> > > > and the same initial value. The only difference is final > > xor > > > > > mask > > > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > > > convert > > > > > > >> from > > > > > > >> > > > PureJavaCrc32 > > > > > > >> > > > to standard CRC32 and vice versa, using this expression: > > > crc32 > > > > > ^= >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Sergey, that was precisely my comment in the ticket: Can we add this option without breaking compatibility with previous page/storage formats? If not, then this should support both implementation. The default should be the new fastest implementation, but if we encounter the older, slower one, then we should print out a warning in the log and automatically switch to the older implementation. On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov wrote: > Hi Igniters > > I suppose that'll break compatibility for LFS (PDS). > > Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x to > 3.0? > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan > > wrote: > > > I commented in the ticket: https://issues.apache.org/ > > jira/browse/IGNITE-9272 > > > > It if can integrate it correctly, according to my comment, in 2.7 > release, > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > D. > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > eduard.shangar...@gmail.com> wrote: > > > > > Hi, > > > > > > I have checked the benchmark and it shows great performance boost on my > > > laptop! > > > > > > +1 for this change. > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov > > > wrote: > > > > > > > Hi Evgeniy, > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > : > > > > > > > > > I fill the ticket, bench code attached there. > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > Thanks! > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > > >difference? > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > dpavlov@gmail.com > > > > > > > > > > >wrote: > > > > > > > > > > > >> It depends. > > > > > >> > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > store > > > > write > > > > > >> are mostly about IO speed. > > > > > >> > > > > > >> In the same time, it can make the huge impact on machines with > > fast > > > IO > > > > > >> and > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > Alexey > > > it > > > > > >> could > > > > > >> benefit performance because we save CPU. Later we can use it's > > power > > > > in > > > > > a > > > > > >> more efficient manner (e.g. with compression). > > > > > >> > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > yzhda...@apache.org > > >: > > > > > >> > > > > > >> > Guys, what time in % does crc calculation take in WAL logging > > > > process? > > > > > >> > > > > > > >> > --Yakov > > > > > >> > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > dpavlov@gmail.com > > > > >: > > > > > >> > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > >> > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > bypassing > > > > > >> > > implementation differences to keep compatibility? > > > > > >> > > > > > > > >> > > Sincerely, > > > > > >> > > Dmitriy Pavlov > > > > > >> > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > plehanov.a...@gmail.com >: > > > > > >> > > > > > > > >> > > > Hello, Igniters! > > > > > >> > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > moreover > > > > > new > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > Since > > we > > > > > >> moved > > > > > >> to > > > > > >> > > > java8, perhaps we really can get performance boost by > using > > > > > >> standard > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > >> > > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements > the > > > > same > > > > > >> > > algorithm > > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses the > > > same > > > > > >> > > polynomial > > > > > >> > > > and the same initial value. The only difference is final > xor > > > > mask > > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > > convert > > > > > >> from > > > > > >> > > > PureJavaCrc32 > > > > > >> > > > to standard CRC32 and vice versa, using this expression: > > crc32 > > > > ^= > > > > > >> > > > 0x > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > > > > >> > eduard.shangar...@gmail.com > > > > > >> > > >: > > > > > >> > > > > > > > > >> > > > > Evgeniy, > > > > > >> > > > > > > > > > >> > > > > Could you share benchmark code? And please share what > > > version > > > > of > > > > > >> JVM > > > > > >> > > > > you have used. > > > > > >> > > > > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > > > >> < arzamas...@mail.ru.invalid > > > > > >> > > > > > > >> > > > > wrote: > > > > > >> > > > > > > > > > >> >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi Igniters I suppose that'll break compatibility for LFS (PDS). Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x to 3.0? On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan wrote: > I commented in the ticket: https://issues.apache.org/ > jira/browse/IGNITE-9272 > > It if can integrate it correctly, according to my comment, in 2.7 release, > it would be great. Otherwise, let's plan this change for 3.0 release. > > D. > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > eduard.shangar...@gmail.com> wrote: > > > Hi, > > > > I have checked the benchmark and it shows great performance boost on my > > laptop! > > > > +1 for this change. > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov > > wrote: > > > > > Hi Evgeniy, > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > : > > > > > > > I fill the ticket, bench code attached there. > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > Thanks! > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > >difference? > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > dpavlov@gmail.com > > > > > > > > >wrote: > > > > > > > > > >> It depends. > > > > >> > > > > >> CRC is a CPU-intensive operation, while WAL logging and page store > > > write > > > > >> are mostly about IO speed. > > > > >> > > > > >> In the same time, it can make the huge impact on machines with > fast > > IO > > > > >> and > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey > > it > > > > >> could > > > > >> benefit performance because we save CPU. Later we can use it's > power > > > in > > > > a > > > > >> more efficient manner (e.g. with compression). > > > > >> > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org > >: > > > > >> > > > > >> > Guys, what time in % does crc calculation take in WAL logging > > > process? > > > > >> > > > > > >> > --Yakov > > > > >> > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > dpavlov@gmail.com > > > >: > > > > >> > > > > > >> > > Hi Alex, thank you for this idea. > > > > >> > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > bypassing > > > > >> > > implementation differences to keep compatibility? > > > > >> > > > > > > >> > > Sincerely, > > > > >> > > Dmitriy Pavlov > > > > >> > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > plehanov.a...@gmail.com >: > > > > >> > > > > > > >> > > > Hello, Igniters! > > > > >> > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > moreover > > > > new > > > > >> > > > "update" method, which use ByteBuffer was introduced. Since > we > > > > >> moved > > > > >> to > > > > >> > > > java8, perhaps we really can get performance boost by using > > > > >> standard > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > >> > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the > > > same > > > > >> > > algorithm > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses the > > same > > > > >> > > polynomial > > > > >> > > > and the same initial value. The only difference is final xor > > > mask > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > convert > > > > >> from > > > > >> > > > PureJavaCrc32 > > > > >> > > > to standard CRC32 and vice versa, using this expression: > crc32 > > > ^= > > > > >> > > > 0x > > > > >> > > > > > > > >> > > > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > > > >> > eduard.shangar...@gmail.com > > > > >> > > >: > > > > >> > > > > > > > >> > > > > Evgeniy, > > > > >> > > > > > > > > >> > > > > Could you share benchmark code? And please share what > > version > > > of > > > > >> JVM > > > > >> > > > > you have used. > > > > >> > > > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > > >> < arzamas...@mail.ru.invalid > > > > >> > > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > > > I think it would break backward compatibility, as > Nikolay > > > > >> mentioned > > > > >> > > > above > > > > >> > > > > > we would take exception here: > > > > >> > > > > > > > > > >> > > > > > [1] > > > > >> > > > > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > > >> > > > > cache/persistence/file/FilePageStore.java#L372 > > > > >> > > > > > > > > > >> > > > > > thats why i question for community thoughts here. > > > > >> > > > > > > > > > >> > > > > > > Hi Evgeniy, > > > > >> > > > > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 > > implementation > > > > >> > change? > > > > >> > > > > > > > > > > >> >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
I commented in the ticket: https://issues.apache.org/jira/browse/IGNITE-9272 It if can integrate it correctly, according to my comment, in 2.7 release, it would be great. Otherwise, let's plan this change for 3.0 release. D. On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < eduard.shangar...@gmail.com> wrote: > Hi, > > I have checked the benchmark and it shows great performance boost on my > laptop! > > +1 for this change. > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov > wrote: > > > Hi Evgeniy, > > > > Thank you. I see that the ticket is unassigned. > > > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > : > > > > > I fill the ticket, bench code attached there. > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > Thanks! > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > >difference? > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > dpavlov@gmail.com > > > > > > >wrote: > > > > > > > >> It depends. > > > >> > > > >> CRC is a CPU-intensive operation, while WAL logging and page store > > write > > > >> are mostly about IO speed. > > > >> > > > >> In the same time, it can make the huge impact on machines with fast > IO > > > >> and > > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey > it > > > >> could > > > >> benefit performance because we save CPU. Later we can use it's power > > in > > > a > > > >> more efficient manner (e.g. with compression). > > > >> > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >: > > > >> > > > >> > Guys, what time in % does crc calculation take in WAL logging > > process? > > > >> > > > > >> > --Yakov > > > >> > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com > > >: > > > >> > > > > >> > > Hi Alex, thank you for this idea. > > > >> > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing > > > >> > > implementation differences to keep compatibility? > > > >> > > > > > >> > > Sincerely, > > > >> > > Dmitriy Pavlov > > > >> > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > plehanov.a...@gmail.com >: > > > >> > > > > > >> > > > Hello, Igniters! > > > >> > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > moreover > > > new > > > >> > > > "update" method, which use ByteBuffer was introduced. Since we > > > >> moved > > > >> to > > > >> > > > java8, perhaps we really can get performance boost by using > > > >> standard > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > >> > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the > > same > > > >> > > algorithm > > > >> > > > as java.lang.zip.CRC32. These two implementations uses the > same > > > >> > > polynomial > > > >> > > > and the same initial value. The only difference is final xor > > mask > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > convert > > > >> from > > > >> > > > PureJavaCrc32 > > > >> > > > to standard CRC32 and vice versa, using this expression: crc32 > > ^= > > > >> > > > 0x > > > >> > > > > > > >> > > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > > >> > eduard.shangar...@gmail.com > > > >> > > >: > > > >> > > > > > > >> > > > > Evgeniy, > > > >> > > > > > > > >> > > > > Could you share benchmark code? And please share what > version > > of > > > >> JVM > > > >> > > > > you have used. > > > >> > > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > >> < arzamas...@mail.ru.invalid > > > >> > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > I think it would break backward compatibility, as Nikolay > > > >> mentioned > > > >> > > > above > > > >> > > > > > we would take exception here: > > > >> > > > > > > > > >> > > > > > [1] > > > >> > > > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > >> > > > > cache/persistence/file/FilePageStore.java#L372 > > > >> > > > > > > > > >> > > > > > thats why i question for community thoughts here. > > > >> > > > > > > > > >> > > > > > > Hi Evgeniy, > > > >> > > > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 > implementation > > > >> > change? > > > >> > > > > > > > > > >> > > > > > > Sincerely, > > > >> > > > > > > Dmitriy Pavlov > > > >> > > > > > > > > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > >> > > > > > > < arzamas...@mail.ru.invalid >: > > > >> > > > > > > > > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like > > > >> PureJavaCrc32 > > > >> > has > > > >> > > > > > >> performance problems in compatible with zip.CRC32. > > > >> > > > > > >> > > > >> > > > > > >> Benchmark Mode Cnt Score Error Units > > > >> > > > > > >>
Re: QueryDetailMetrics for cache-less SQL queries
Hi Evgeny,
Do you propose to move into IgniteMetrics absolutely all Ignite metrics or
just dataRegionMetrics, dataStorageMetrics and queryDetailMetrics? I think
you can't move all metrics into one place. Pluggable components and
different SPI implementations may have their own metric sets, and perhaps
it's not such a good idea to try to fit them in one common fixed interface.
2018-08-20 18:14 GMT+03:00 Evgenii Zhuravlev :
> As for now, metrics are accumulated for cache only when the query is run
> directly over this cache, for example, using ignite.cache("Some
> cache").sqlFieldsQuery("select ... from .."). When a query is started using
> other APIs, it doesn't detect cache, to which this table belongs and
> doesn't save any metrics.
>
>
>
> 2018-08-17 16:44 GMT+03:00 Vladimir Ozerov :
>
> > Query is not executed on specific cache. It is executed on many caches.
> >
> > On Fri, Aug 17, 2018 at 6:10 AM Dmitriy Setrakyan >
> > wrote:
> >
> > > But internally the SQL query still runs on some cache, no? What happens
> > to
> > > the metrics accumulated on that cache?
> > >
> > > D.
> > >
> > > On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov
> > > wrote:
> > >
> > > > Dima,
> > > >
> > > > "cache-less" means that SQL executed directly on SQL engine.
> > > >
> > > > In previous version of Ignite we execute queries via cache:
> > > >
> > > > ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
> > > >
> > > > In current Ignite we can execute query directly without using cache
> as
> > > > "gateway".
> > > >
> > > > And if we execute query directly, metrics not update.
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Evgeny, what is a "cache-less" SQL query?
> > > > >
> > > > > D.
> > > > >
> > > > > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > > > > e.zhuravlev...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Hi Igniters,
> > > > > >
> > > > > > I've started to work on adding QueryDetailMetrics for cache-less
> > SQL
> > > > > > queries(issue https://issues.apache.org/jira/browse/IGNITE-6677)
> > and
> > > > > found
> > > > > > that it's required to change API. I don't think that adding
> methods
> > > > like
> > > > > > queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to
> > > > Ignite
> > > > > > class is a good idea. So, I see 2 possible solutions here:
> > > > > >
> > > > > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> > > > > > Ignite(like dataRegionMetrics and dataStorageMetrics) and add a
> new
> > > > > > metric "queryDetailMetrics" to it. Of course, old methods will be
> > > > > > deprecated.
> > > > > >
> > > > > > 2. Finally create Ignite.sql() API, which was already discussed
> > here:
> > > > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > > > > > and place "queryDetailMetrics" metric there. Here is the ticket
> for
> > > > this
> > > > > > change: https://issues.apache.org/jira/browse/IGNITE-4701
> > > > > >
> > > > > > Personally, I think that the second solution looks better in this
> > > case,
> > > > > > however, moving dataRegionMetrics and dataStorageMetrics to
> > > > > > ignite.matrics() is still a good idea - IMO, Ignite class is not
> > the
> > > > > right
> > > > > > place for them - we shouldn't change our main API class so often.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Thank you,
> > > > > > Evgenii
> > > > > >
> > > > >
> > > > > --
> > > > > Alexey Kuznetsov
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Apache Ignite 2.7 release
Hi Nikolay, Thanks for being the release manager! I am getting a bit lost in all these tickets. Can we specify some high-level tickets, that are not plain bug fixes, which will be interesting for the community to notice? For example, here are some significant tasks that the community is either working on or has been working on: - Node.JS client - Python client - Transactional SQL (MVCC) - service grid stabilization - SQL memory utilization improvements - more? Can you please solicit status from the community for these tasks? D. On Mon, Aug 20, 2018 at 11:22 AM, Nikolay Izhikov wrote: > Hello, Igniters. > > I'm release manager of Apache Ignite 2.7. > > It's time to start discussion of release. [1] > > Current code freeze date is September, 30. > If you have any objections - please, responsd to this thread. > > [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7
Apache Ignite 2.7 release
Hello, Igniters. I'm release manager of Apache Ignite 2.7. It's time to start discussion of release. [1] Current code freeze date is September, 30. If you have any objections - please, responsd to this thread. [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7 signature.asc Description: This is a digitally signed message part
[GitHub] ignite pull request #4567: IGNITE-9220 Uncomment tests from internal test su...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4567 ---
[jira] [Created] (IGNITE-9335) IgniteOffheapReadWriteLockSelfTest.testTagIdUpdateContinuous() sometimes hangs on a barrier
Ilya Kasnacheev created IGNITE-9335: --- Summary: IgniteOffheapReadWriteLockSelfTest.testTagIdUpdateContinuous() sometimes hangs on a barrier Key: IGNITE-9335 URL: https://issues.apache.org/jira/browse/IGNITE-9335 Project: Ignite Issue Type: Bug Reporter: Ilya Kasnacheev Assignee: Alexey Goncharuk I am trying to uncomment IgniteOffheapReadWriteLockSelfTest but testTagIdUpdateContinuous() deadlocks every now and when on barr.await(); Please fix it and un-fail. Note that I also suggest changing wait time from 30 to 5 seconds per test, is that OK? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4568: IGNITE-9315: Eviction meta from near cache may ap...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4568 ---
[GitHub] ignite pull request #4583: IGNITE-8829
GitHub user NSAmelchev opened a pull request: https://github.com/apache/ignite/pull/4583 IGNITE-8829 TcpCommunicationSpi: fix configuration properties You can merge this pull request into a Git repository by running: $ git pull https://github.com/NSAmelchev/ignite ignite-8829 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4583.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4583 commit cfb27276cdd5c68862bb77ccfb78d62c9e5ad23e Author: NSAmelchev Date: 2018-08-20T15:55:31Z Refactoring TcpCommunicationSpi ---
[jira] [Created] (IGNITE-9334) Web Console should set sqlEscapeAll in generated sources when needed
Ilya Kasnacheev created IGNITE-9334: --- Summary: Web Console should set sqlEscapeAll in generated sources when needed Key: IGNITE-9334 URL: https://issues.apache.org/jira/browse/IGNITE-9334 Project: Ignite Issue Type: Bug Reporter: Ilya Kasnacheev As witnessed by userlist discussion. When some column or table names contain special characters, sqlEscapeAll=true is needed. But, Web Console doesn't know to set this parameter to true in generated server and client XMLs. My expectation, either set it to true if any of identifiers is not isJavaIdentifier, or just always set it to true. sqlEscapeAll should also be documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9333) Adding page with simple statistic for last 50 runs.
Eduard Shangareev created IGNITE-9333: - Summary: Adding page with simple statistic for last 50 runs. Key: IGNITE-9333 URL: https://issues.apache.org/jira/browse/IGNITE-9333 Project: Ignite Issue Type: Sub-task Reporter: Eduard Shangareev Ok, I am proposing to add a new page which would be named "Statistic". It should show last 50 "Run All" for master, the columns: ||Id||Total tests||Failed tests||Ignored tests|| Muted tests||Total issues (count and the list of TC configurations with links) || Total run time || Also, we need to calculate the statistic. Totals (all unique tests with issues), average issues/fails per run, median, max, min. Total issues = Exit codes + JVM Crashes + OOMs + other issues which caused TC configuration fail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Unknown known issue on cache rebalancing delayed
Hello Roman, Did you faced with real issue of delayed rebalance or it's just only for your personal interest? If yes, please, share details and we will try to help you. As for this comment I don't think he is actual. That change was in 2015. Much has changed within rebalance process since that time. I've uncommented it and rechecked with that cache configuration and haven't seen any failed tests or issues. Probably, that problem was about cache in SYNC mode does not start util it loads all data from other nodes. But currently delayed rebalance works the same way as IgniteCache#rebalance(), so you can `setRebalanceDelay` to `-1` and call it manually to check. On Mon, 20 Aug 2018 at 11:19 Roman Shtykh wrote: > Igniters, > I have found "Known issue, possible deadlock in case of low priority cache > rebalancing delayed" comment in > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please explain > when using rebalance delay can be an issue and why? > > -- Roman > -- -- Maxim Muzafarov
[jira] [Created] (IGNITE-9332) [TC Helper] Add statistic page
Eduard Shangareev created IGNITE-9332: - Summary: [TC Helper] Add statistic page Key: IGNITE-9332 URL: https://issues.apache.org/jira/browse/IGNITE-9332 Project: Ignite Issue Type: Task Reporter: Eduard Shangareev So, we are starting work on adding statistic information to our TC Helper. It should make easier to get an insight into what to do next or what needs to be fixed right now. Also, it should show how we are progressing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: QueryDetailMetrics for cache-less SQL queries
As for now, metrics are accumulated for cache only when the query is run
directly over this cache, for example, using ignite.cache("Some
cache").sqlFieldsQuery("select ... from .."). When a query is started using
other APIs, it doesn't detect cache, to which this table belongs and
doesn't save any metrics.
2018-08-17 16:44 GMT+03:00 Vladimir Ozerov :
> Query is not executed on specific cache. It is executed on many caches.
>
> On Fri, Aug 17, 2018 at 6:10 AM Dmitriy Setrakyan
> wrote:
>
> > But internally the SQL query still runs on some cache, no? What happens
> to
> > the metrics accumulated on that cache?
> >
> > D.
> >
> > On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov
> > wrote:
> >
> > > Dima,
> > >
> > > "cache-less" means that SQL executed directly on SQL engine.
> > >
> > > In previous version of Ignite we execute queries via cache:
> > >
> > > ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
> > >
> > > In current Ignite we can execute query directly without using cache as
> > > "gateway".
> > >
> > > And if we execute query directly, metrics not update.
> > >
> > >
> > >
> > >
> > > On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >
> > > wrote:
> > >
> > > > Evgeny, what is a "cache-less" SQL query?
> > > >
> > > > D.
> > > >
> > > > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > > > e.zhuravlev...@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Igniters,
> > > > >
> > > > > I've started to work on adding QueryDetailMetrics for cache-less
> SQL
> > > > > queries(issue https://issues.apache.org/jira/browse/IGNITE-6677)
> and
> > > > found
> > > > > that it's required to change API. I don't think that adding methods
> > > like
> > > > > queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to
> > > Ignite
> > > > > class is a good idea. So, I see 2 possible solutions here:
> > > > >
> > > > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> > > > > Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
> > > > > metric "queryDetailMetrics" to it. Of course, old methods will be
> > > > > deprecated.
> > > > >
> > > > > 2. Finally create Ignite.sql() API, which was already discussed
> here:
> > > > > http://apache-ignite-developers.2346864.n4.nabble.
> > > > > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > > > > and place "queryDetailMetrics" metric there. Here is the ticket for
> > > this
> > > > > change: https://issues.apache.org/jira/browse/IGNITE-4701
> > > > >
> > > > > Personally, I think that the second solution looks better in this
> > case,
> > > > > however, moving dataRegionMetrics and dataStorageMetrics to
> > > > > ignite.matrics() is still a good idea - IMO, Ignite class is not
> the
> > > > right
> > > > > place for them - we shouldn't change our main API class so often.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Thank you,
> > > > > Evgenii
> > > > >
> > > >
> > > > --
> > > > Alexey Kuznetsov
> > > >
> > > >
> > >
> >
>
[jira] [Created] (IGNITE-9331) Add thin client tests for custom TLS protocols
Stanislav Lukyanov created IGNITE-9331: -- Summary: Add thin client tests for custom TLS protocols Key: IGNITE-9331 URL: https://issues.apache.org/jira/browse/IGNITE-9331 Project: Ignite Issue Type: Bug Reporter: Stanislav Lukyanov IGNITE-6167 added an ability to specify a custom TLS protocol for Ignite connections. It includes tests for regular Ignite node and Java Thin client. Need to add tests for other thin clients: JDBC, ODBC, .Net, Node.js, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Release policy updates
Yes, let’s just remove md5. Will you create the ticket and handle this for 2.7? Denis On Monday, August 20, 2018, Anton Vinogradov wrote: > Denis, > > Currently we provide md5 and sha512 [1]. > Should we just get rid of md5? > > [1] https://www.apache.org/dist/ignite/2.6.0/ > > сб, 18 авг. 2018 г. в 3:51, Denis Magda : > >> Peter, Anton V, Igniters, >> >> The board communicated the following release policy changes: >> -- for new releases : >> -- you MUST supply a SHA-256 and/or SHA-512 file >> -- you SHOULD NOT supply MD5 or SHA-1 files >> >> Are we good? More details are below. >> >> >> >> >> *2 Release Dist Policy Changes (Q? us...@infra.apache.org) >> --- >> >> The Release Distribution Policy[1] changed regarding checksum files. >> See under "Cryptographic Signatures and Checksums Requirements" [2]. >> >> Note that "MUST", "SHOULD", "SHOULD NOT" are technical terms ; >> not just emphasized words ; for an explanation see RFC-2119 [3]. >> >> Old policy : >> >> -- SHOULD supply a SHA checksum file >> -- SHOULD NOT supply a MD5 checksum file >> >> New policy : >> >> -- SHOULD supply a SHA-256 and/or SHA-512 checksum file >> -- SHOULD NOT supply MD5 or SHA-1 checksum files >> >> Why this change ? >> >> -- Like MD5, SHA-1 is too broken ; we should move away from it. >> >> Impact for PMCs : >> >> -- for new releases : >> -- you MUST supply a SHA-256 and/or SHA-512 file >> -- you SHOULD NOT supply MD5 or SHA-1 files >> >> -- for past releases : >> -- you are not required to change anything ; >> -- it would be nice if you fixed your dist area ; >> start with : cleanup ; rename .sha's ; remove .md5's >> >
[GitHub] ignite pull request #4582: Ignite 6167
GitHub user alamar opened a pull request: https://github.com/apache/ignite/pull/4582 Ignite 6167 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-6167 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4582.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4582 commit b7202b3eb44c682d5dfa51303f5f840f38000e0e Author: mcherkasov Date: 2018-07-23T14:47:42Z Added ssl parameters for ssl context configuration. commit a7143c3f98a2f3614dd101a8a47f18fe6c84 Author: mcherkasov Date: 2018-07-26T15:28:43Z Added java doc. Added test. Fixed code style. commit 5c0c10d57f3e10b5ad734d172c90070cc73152ff Author: mcherkasov Date: 2018-07-26T17:03:49Z Fixed code style. commit d7887548db95f5c87231d35fb0f9be3a61a89f9a Author: mcherkasov Date: 2018-07-30T12:51:13Z SSLParameters are replaced with String[] which allows only to set enabled cipher suites. commit 11b4840b793d52a7ab7e4a9e9530bfa1bc731fae Author: mcherkasov Date: 2018-07-30T15:03:05Z SSLParameters are replaced with String[] which allows only to set enabled cipher suites. commit 1816523f3cffc567f5bd38b2c6cf6d7b66bd55ba Author: mcherkasov Date: 2018-07-31T13:04:43Z Arrays are replaced with varargs. commit abe6592ddba81fd41d1810bf6c18132de2ba9a96 Author: mcherkasov Date: 2018-08-15T13:40:46Z Fixed code style. commit b6022eb8d48fa38d79864d3f411157508f31db84 Author: mcherkasov Date: 2018-08-15T13:41:26Z Added negative tests for non existent protocol and cipher suites commit 9f384baac3adc9f05eb12bf72210d0ef28da0ed2 Author: mcherkasov Date: 2018-08-17T17:36:14Z The test added to suite. commit 7883b8190c8859a2b3ec4c68d7b9142a71e86d6f Author: mcherkasov Date: 2018-08-19T21:30:12Z Added test for thin client. ---
[jira] [Created] (IGNITE-9330) Multiple CacheMetricsManageTest tests are failing
Ilya Kasnacheev created IGNITE-9330: --- Summary: Multiple CacheMetricsManageTest tests are failing Key: IGNITE-9330 URL: https://issues.apache.org/jira/browse/IGNITE-9330 Project: Ignite Issue Type: Bug Reporter: Ilya Kasnacheev testCacheManagerStatisticsEnable and testClusterApiClearStatistics fail every so often, such as in https://ci.ignite.apache.org/viewLog.html?buildId=1676464 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9329) Positive and negative tests for SSL ciphers and protocols
Ilya Kasnacheev created IGNITE-9329: --- Summary: Positive and negative tests for SSL ciphers and protocols Key: IGNITE-9329 URL: https://issues.apache.org/jira/browse/IGNITE-9329 Project: Ignite Issue Type: Test Reporter: Ilya Kasnacheev Assignee: Mikhail Cherkasov We should check that SSL client and JDBC client can connect to cluster with configured cipher and protocol restrictions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9328) IgniteDevOnlyLogTest.testDevOnlyQuietMessage() fails to write.
Ilya Kasnacheev created IGNITE-9328: --- Summary: IgniteDevOnlyLogTest.testDevOnlyQuietMessage() fails to write. Key: IGNITE-9328 URL: https://issues.apache.org/jira/browse/IGNITE-9328 Project: Ignite Issue Type: Bug Reporter: Ilya Kasnacheev Assignee: Stanislav Lukyanov After I have re-enabled it in IGNITE-9220 it started failing. I have also migrated it to multiJvm. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4580: Need to change merge metadata behavior during new...
GitHub user zstan opened a pull request: https://github.com/apache/ignite/pull/4580 Need to change merge metadata behavior during new schema registration. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-14109 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4580.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4580 commit 58cc6f1b0ef16b48eabcaaaffc0d9944411650cc Author: Evgeny Stanilovskiy Date: 2018-08-20T13:27:29Z Need to change merge metadata behavior during new schema registration. ---
[GitHub] ignite pull request #4579: IGNITE-9327 stop the node if IgniteNeedReconnectE...
GitHub user akalash opened a pull request: https://github.com/apache/ignite/pull/4579 IGNITE-9327 stop the node if IgniteNeedReconnectException happened You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9327 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4579.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4579 commit aed7d76983798abb76972d92d1a8f63fa8a93762 Author: Anton Kalashnikov Date: 2018-08-20T13:18:24Z IGNITE-9327 stop the node if IgniteNeedReconnectException happened ---
[jira] [Created] (IGNITE-9327) Client Nodes hangs because client reconnect not handled
Anton Kalashnikov created IGNITE-9327: - Summary: Client Nodes hangs because client reconnect not handled Key: IGNITE-9327 URL: https://issues.apache.org/jira/browse/IGNITE-9327 Project: Ignite Issue Type: Test Reporter: Anton Kalashnikov Assignee: Anton Kalashnikov Reproduced by IgniteCacheClientReconnectTest#testClientInForceServerModeStopsOnExchangeHistoryExhaustion If IgniteNeedReconnectException happend we should stop not if reconnect doens't supported. But after https://issues.apache.org/jira/browse/IGNITE-8673 we became to ignore this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Table Names in Spark Catalog
Hello, Stuart. Personally, I think we should change current tables naming and return table in form of `schema.table`. Valentin, could you share your opinion? В Пн, 20/08/2018 в 10:04 +0100, Stuart Macdonald пишет: > Igniters, > > While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I are > discussing whether to introduce a change which may impact backwards > compatibility; Nikolay suggested we take the discussion to this list. > > Ignite implements a custom Spark catalog which provides an API by which > Spark users can list the tables which are available in Ignite which can be > queried via Spark SQL. Currently that table name list includes just the > names of the tables, but IGNITE-9228 is introducing a change which allows > optional prefixing of schema names to table names to disambiguate multiple > tables with the same name in different schemas. For the "list tables" API > we therefore have two options: > > 1. List the tables using both their table names and schema-qualified table > names (eg. [ "myTable", "mySchema.myTable" ]) even though they are the same > underlying table. This retains backwards compatibility with users who > expect "myTable" to appear in the catalog. > 2. List the tables using only their schema-qualified names. This eliminates > duplication of names in the catalog but will potentially break > compatibility with users who expect the table name in the catalog. > > With either option we will allow for Spark SQL SELECT statements to use > either table name or schema-qualified table names, this change would purely > impact the API which is used to list available tables. > > Any opinions would be welcome. > > Thanks, > Stuart. > > [1] https://issues.apache.org/jira/browse/IGNITE-9228 > [2] https://github.com/apache/ignite/pull/4551 signature.asc Description: This is a digitally signed message part
Re: TC: Compilation error: modules\platforms\cpp\jni\project\vs\jni.vcxproj
Thank you! On Mon, Aug 20, 2018 at 2:53 PM Igor Sapego wrote: > > Oh, you are talking of Java 8 suite. > > There is an issue with configuration of the suite. > Peter currently working on it. > > Best Regards, > Igor > > > On Mon, Aug 20, 2018 at 2:42 PM Igor Sapego wrote: >> >> Yeah, I know. >> >> There is no 32-bit JDK for Java 9 on Windows. >> >> So, I guess, we have no other choice but to stop supporting of >> 32 bit version client for Java 9 and later. >> Best Regards, >> Igor >> >> >> On Sun, Aug 19, 2018 at 12:30 PM Vyacheslav Daradur >> wrote: >>> >>> Hi, Igniters! >>> >>> I found that TC build plan named "Platform C++ (Windows x86)" failed >>> with compilation error in master branch, with the following error: >>> >>> java.obj error LNK2001: unresolved external symbol >>> __imp__JNI_CreateJavaVM@12 >>> C:\BuildAgent\work\ae48a463bd611b52\modules\platforms\cpp\project\vs\Win32\Release\ignite.jni.dll >>> error LNK1120: 1 unresolved externals >>> >>> Igor, could you have a look, please? >>> >>> >>> -- >>> Best Regards, Vyacheslav D. -- Best Regards, Vyacheslav D.
[jira] [Created] (IGNITE-9326) IgniteCacheFailedUpdateResponseTest hangs in master
Alexey Goncharuk created IGNITE-9326: Summary: IgniteCacheFailedUpdateResponseTest hangs in master Key: IGNITE-9326 URL: https://issues.apache.org/jira/browse/IGNITE-9326 Project: Ignite Issue Type: Test Reporter: Alexey Goncharuk Assignee: Alexey Goncharuk The test started to hang after IGNITE-8926 was merged. The reason is that entry processor result started to be lazily serialized during the message send, which results in a failure handler invocation. However, the test checks that the exception is rethrown to a user. One of the possible fixes is to marshal the result after the topology lock is released. Muting test in master for now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: TC: Compilation error: modules\platforms\cpp\jni\project\vs\jni.vcxproj
Oh, you are talking of Java 8 suite. There is an issue with configuration of the suite. Peter currently working on it. Best Regards, Igor On Mon, Aug 20, 2018 at 2:42 PM Igor Sapego wrote: > Yeah, I know. > > There is no 32-bit JDK for Java 9 on Windows. > > So, I guess, we have no other choice but to stop supporting of > 32 bit version client for Java 9 and later. > Best Regards, > Igor > > > On Sun, Aug 19, 2018 at 12:30 PM Vyacheslav Daradur > wrote: > >> Hi, Igniters! >> >> I found that TC build plan named "Platform C++ (Windows x86)" failed >> with compilation error in master branch, with the following error: >> >> java.obj error LNK2001: unresolved external symbol >> __imp__JNI_CreateJavaVM@12 >> >> C:\BuildAgent\work\ae48a463bd611b52\modules\platforms\cpp\project\vs\Win32\Release\ignite.jni.dll >> error LNK1120: 1 unresolved externals >> >> Igor, could you have a look, please? >> >> >> -- >> Best Regards, Vyacheslav D. >> >
[GitHub] ignite pull request #4560: IGNITE-9297 Review ML examples javadocs as of rel...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4560 ---
Re: TC: Compilation error: modules\platforms\cpp\jni\project\vs\jni.vcxproj
Yeah, I know. There is no 32-bit JDK for Java 9 on Windows. So, I guess, we have no other choice but to stop supporting of 32 bit version client for Java 9 and later. Best Regards, Igor On Sun, Aug 19, 2018 at 12:30 PM Vyacheslav Daradur wrote: > Hi, Igniters! > > I found that TC build plan named "Platform C++ (Windows x86)" failed > with compilation error in master branch, with the following error: > > java.obj error LNK2001: unresolved external symbol > __imp__JNI_CreateJavaVM@12 > > C:\BuildAgent\work\ae48a463bd611b52\modules\platforms\cpp\project\vs\Win32\Release\ignite.jni.dll > error LNK1120: 1 unresolved externals > > Igor, could you have a look, please? > > > -- > Best Regards, Vyacheslav D. >
[GitHub] ignite pull request #4563: IGNITE-9294 Implement bounds for StandAloneWALIte...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4563 ---
[jira] [Created] (IGNITE-9325) Add to all command-line utility ability to make sound after complite execution
ARomantsov created IGNITE-9325: -- Summary: Add to all command-line utility ability to make sound after complite execution Key: IGNITE-9325 URL: https://issues.apache.org/jira/browse/IGNITE-9325 Project: Ignite Issue Type: Improvement Components: examples Affects Versions: 2.5 Reporter: ARomantsov Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi, I have checked the benchmark and it shows great performance boost on my laptop! +1 for this change. On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov wrote: > Hi Evgeniy, > > Thank you. I see that the ticket is unassigned. > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > Sincerely, > Dmitriy Pavlov > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > : > > > I fill the ticket, bench code attached there. > > https://issues.apache.org/jira/browse/IGNITE-9272 > > Thanks! > > > > > > >Has anyone else run the benchmark and reproduced the performance > > >difference? > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < dpavlov@gmail.com > > > > >wrote: > > > > > >> It depends. > > >> > > >> CRC is a CPU-intensive operation, while WAL logging and page store > write > > >> are mostly about IO speed. > > >> > > >> In the same time, it can make the huge impact on machines with fast IO > > >> and > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey it > > >> could > > >> benefit performance because we save CPU. Later we can use it's power > in > > a > > >> more efficient manner (e.g. with compression). > > >> > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >: > > >> > > >> > Guys, what time in % does crc calculation take in WAL logging > process? > > >> > > > >> > --Yakov > > >> > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com > >: > > >> > > > >> > > Hi Alex, thank you for this idea. > > >> > > > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing > > >> > > implementation differences to keep compatibility? > > >> > > > > >> > > Sincerely, > > >> > > Dmitriy Pavlov > > >> > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > plehanov.a...@gmail.com >: > > >> > > > > >> > > > Hello, Igniters! > > >> > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover > > new > > >> > > > "update" method, which use ByteBuffer was introduced. Since we > > >> moved > > >> to > > >> > > > java8, perhaps we really can get performance boost by using > > >> standard > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > >> > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the > same > > >> > > algorithm > > >> > > > as java.lang.zip.CRC32. These two implementations uses the same > > >> > > polynomial > > >> > > > and the same initial value. The only difference is final xor > mask > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily convert > > >> from > > >> > > > PureJavaCrc32 > > >> > > > to standard CRC32 and vice versa, using this expression: crc32 > ^= > > >> > > > 0x > > >> > > > > > >> > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > >> > eduard.shangar...@gmail.com > > >> > > >: > > >> > > > > > >> > > > > Evgeniy, > > >> > > > > > > >> > > > > Could you share benchmark code? And please share what version > of > > >> JVM > > >> > > > > you have used. > > >> > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > >> < arzamas...@mail.ru.invalid > > >> > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > I think it would break backward compatibility, as Nikolay > > >> mentioned > > >> > > > above > > >> > > > > > we would take exception here: > > >> > > > > > > > >> > > > > > [1] > > >> > > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > >> > > > > cache/persistence/file/FilePageStore.java#L372 > > >> > > > > > > > >> > > > > > thats why i question for community thoughts here. > > >> > > > > > > > >> > > > > > > Hi Evgeniy, > > >> > > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 implementation > > >> > change? > > >> > > > > > > > > >> > > > > > > Sincerely, > > >> > > > > > > Dmitriy Pavlov > > >> > > > > > > > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > >> > > > > > > < arzamas...@mail.ru.invalid >: > > >> > > > > > > > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like > > >> PureJavaCrc32 > > >> > has > > >> > > > > > >> performance problems in compatible with zip.CRC32. > > >> > > > > > >> > > >> > > > > > >> Benchmark Mode Cnt Score Error Units > > >> > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > >> > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± > 3746712.210 > > >> > ns/op > > >> > > > > > >> > > >> > > > > > >> thoughts? > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > -- > > Евгений Станиловский > > >
[GitHub] ignite pull request #4571: IGNITE-9259: change log message
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4571 ---
[GitHub] ignite pull request #4573: IGNITE-9232 Remove commented method GridCacheUtil...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4573 ---
Re: Release policy updates
Denis, Currently we provide md5 and sha512 [1]. Should we just get rid of md5? [1] https://www.apache.org/dist/ignite/2.6.0/ сб, 18 авг. 2018 г. в 3:51, Denis Magda : > Peter, Anton V, Igniters, > > The board communicated the following release policy changes: > -- for new releases : > -- you MUST supply a SHA-256 and/or SHA-512 file > -- you SHOULD NOT supply MD5 or SHA-1 files > > Are we good? More details are below. > > > > > *2 Release Dist Policy Changes (Q? us...@infra.apache.org) > --- > > The Release Distribution Policy[1] changed regarding checksum files. > See under "Cryptographic Signatures and Checksums Requirements" [2]. > > Note that "MUST", "SHOULD", "SHOULD NOT" are technical terms ; > not just emphasized words ; for an explanation see RFC-2119 [3]. > > Old policy : > > -- SHOULD supply a SHA checksum file > -- SHOULD NOT supply a MD5 checksum file > > New policy : > > -- SHOULD supply a SHA-256 and/or SHA-512 checksum file > -- SHOULD NOT supply MD5 or SHA-1 checksum files > > Why this change ? > > -- Like MD5, SHA-1 is too broken ; we should move away from it. > > Impact for PMCs : > > -- for new releases : > -- you MUST supply a SHA-256 and/or SHA-512 file > -- you SHOULD NOT supply MD5 or SHA-1 files > > -- for past releases : > -- you are not required to change anything ; > -- it would be nice if you fixed your dist area ; > start with : cleanup ; rename .sha's ; remove .md5's >
[jira] [Created] (IGNITE-9324) MVCC: support explicit locks
Vladimir Ozerov created IGNITE-9324: --- Summary: MVCC: support explicit locks Key: IGNITE-9324 URL: https://issues.apache.org/jira/browse/IGNITE-9324 Project: Ignite Issue Type: Task Components: mvcc Reporter: Vladimir Ozerov We need to evaluate and understand whether explicit lock support is needed for MVCC. Most probably the answer would be "no". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9323) MVCC: support cache interceptors
Vladimir Ozerov created IGNITE-9323: --- Summary: MVCC: support cache interceptors Key: IGNITE-9323 URL: https://issues.apache.org/jira/browse/IGNITE-9323 Project: Ignite Issue Type: Task Components: mvcc Reporter: Vladimir Ozerov We need to support cache interceptor API for mvcc-enabled caches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Table Names in Spark Catalog
Igniters, While reviewing the changes for IGNITE-9228 [1,2], Nikolay and I are discussing whether to introduce a change which may impact backwards compatibility; Nikolay suggested we take the discussion to this list. Ignite implements a custom Spark catalog which provides an API by which Spark users can list the tables which are available in Ignite which can be queried via Spark SQL. Currently that table name list includes just the names of the tables, but IGNITE-9228 is introducing a change which allows optional prefixing of schema names to table names to disambiguate multiple tables with the same name in different schemas. For the "list tables" API we therefore have two options: 1. List the tables using both their table names and schema-qualified table names (eg. [ "myTable", "mySchema.myTable" ]) even though they are the same underlying table. This retains backwards compatibility with users who expect "myTable" to appear in the catalog. 2. List the tables using only their schema-qualified names. This eliminates duplication of names in the catalog but will potentially break compatibility with users who expect the table name in the catalog. With either option we will allow for Spark SQL SELECT statements to use either table name or schema-qualified table names, this change would purely impact the API which is used to list available tables. Any opinions would be welcome. Thanks, Stuart. [1] https://issues.apache.org/jira/browse/IGNITE-9228 [2] https://github.com/apache/ignite/pull/4551
[GitHub] ignite pull request #4569: IGNITE-9316 Analyse and improve code coverage in ...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4569 ---
[jira] [Created] (IGNITE-9322) MVCC: implement deadlock detector
Vladimir Ozerov created IGNITE-9322: --- Summary: MVCC: implement deadlock detector Key: IGNITE-9322 URL: https://issues.apache.org/jira/browse/IGNITE-9322 Project: Ignite Issue Type: Task Components: mvcc Reporter: Vladimir Ozerov Deadlocks are not uncommon during SQL execution. We need to implement distributed deadlock detection protocol for MVCC. Essentially, nodes should exchange some map of tx wait lists, and try to find a loop. If loop is found, then one of problematic transactions should be rolled back. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9321) MVCC: support cache events
Vladimir Ozerov created IGNITE-9321: --- Summary: MVCC: support cache events Key: IGNITE-9321 URL: https://issues.apache.org/jira/browse/IGNITE-9321 Project: Ignite Issue Type: Task Components: mvcc Reporter: Vladimir Ozerov Fix For: 2.7 Currently cache events are not fired for MVCC caches. Need to decide what to do with it. Possible options: # Fix events (will require separate data structure and incur possibly serious overhead) # Do not fire events at all # Introduce new events which will not require additional data structures (e.g. "entry_changed" which may be fired even if transaction is rolled back in the end) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Ticket review checklist
Hi Dmitriy, I agree with you about lambdas. For me they are quite useful and I believe that this language feature is a solid and well proven part of modern Java. I still feel that current statement in our guidelines should be rephrased. But if others are ok with it then let's keep it as is. 2018-08-16 16:47 GMT+03:00 Dmitriy Pavlov : > Hi Ivan, > > Unfortunately, the review checklist does not work as well as it could. I > hope the situation will change in the nearest future, I think we should > come back to this idea and encourage contributors and reviewers to use the > list. > > As for lambda's: some Igniters feel confident about it, and some Igniters > don't. My opinion it is perfectly ok to use it if usage is local node only, > is there is no chance lambda is serialized to the network. If there is such > chance it is better to avoid it. > > Sincerely, > Dmitriy Pavlov > > чт, 16 авг. 2018 г. в 12:09, Павлухин Иван : > > > Vladimir, > > > > First of all, statements in Java 8 section [1] looks kind of prohibitive > > for me. When a new contributor see words "preferred" and "avoided in most > > cases" he most likely will not use such features (like I did). If a > > statement is not prohibitive in practice it could be at least rephrased. > > > > A bit about expressiveness. I written a code during working on a real > > ticket. The case is quite common in Ignite codebase. You can find example > > with couple of approaches in snippet [2]. For me approach with lambdas is > > expressive, compact and simple. > > > > What do you think? > > > > [1] > > > > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines# > CodingGuidelines-Java8 > > [2] https://gist.github.com/pavlukhin/92701277f66f8901a7feda6283a5a299 > > > > 2018-08-16 11:21 GMT+03:00 Vladimir Ozerov : > > > > > Hi Ivan, > > > > > > From what I see we do not restrict contributors to use lambdas and > > streams. > > > Document states that plain collections and anonymous classes are > > > *preferred*. This is not obligatory requirement, and it seems > reasonable > > to > > > me, because when developing complex projects at times it is better to > > have > > > more expressive code, than less non-obvious code which makes dozens > > > operations in a single string. > > > > > > Or may be there are any other statements in the checklist which > prevents > > > users from using Java 8 features? > > > > > > Vladimir. > > > > > > On Tue, Aug 14, 2018 at 7:16 PM ipavlukhin > wrote: > > > > > > > Hi Igniters, > > > > > > > > I would like to refresh review checklist a little bit. Currently it > [1] > > > > contains section against lambda Lambda expressions and Stream API. As > > > > far as I know it is not true anymore. Currently both features have > > > > theirs usage in core module. What is a state of affairs for a > subject? > > > > Are there some well-known cases where e.g. lambdas are not > applicable? > > > > Should we document it? > > > > > > > > I personally think that we could delete entire Java 8 section from > > > > checklist (and Java 5 as well). I understand that every tool should > be > > > > used judiciously but I doubt that all cases can be covered in short > > > > checklist. > > > > > > > > [1] > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/ > Coding+Guidelines# > > > CodingGuidelines-Java8 > > > > > > > > > > > > On 2018/07/09 20:53:42, Dmitry Pavlov wrote: > > > > > I also tend to agree about updating checklist> > > > > > > > > > > About suite timeouts, I suspect there is one problem introduced > > > > recently> > > > > > within 3 days, which caused this mass timeouts.> > > > > > > > > > > I hope Igniters will find out reason soon. In relation to compute > we > > > > have> > > > > > only 2 possible cause:> > > > > > Ivan Daschinskiy (idaschinskiy) 2 files IGNITE-8869 Fixed> > > > > > PartitionsExchangeOnDiscoveryHistoryOverflowTest hanging> > > > > > Signed-off-by: Andrey Gura ···> > > > > > > > > > > Dmitriy Govorukhin (dgovorukhin) 12 files IGNITE-8827 Disable WAL > > > > during> > > > > > apply updates on recovery> > > > > > > > > > > I guess if we fix this reason we will fix 10 suites more> > > > > > References:> > > > > > > > > > > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId= > > > IgniteTests24Java8_ComputeGrid=buildTypeHistoryList_ > > > IgniteTests24Java8=%3Cdefault%3E> > > > > > > > > > > > > > > > > > > > > > > > пн, 9 июл. 2018 г. в 22:29, Anton Vinogradov :> > > > > > > > > > > > Sounds reasonable.> > > > > > > I've satrted Data Structures suite hang investigation [1].> > > > > > >> > > > > > > Igniters, especially commiters,> > > > > > > I know, you're busy, but it will be a great help to the project > in > > > > case you> > > > > > > fix at least one hang per person.> > > > > > >> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-8783> > > > > > >> > > > > > > пн, 9 июл. 2018 г. в 19:24, Maxim Muzafarov :> > > > > > >> > > > > > > > Hi
[jira] [Created] (IGNITE-9320) MVCC: finalize configuration
Vladimir Ozerov created IGNITE-9320:
---
Summary: MVCC: finalize configuration
Key: IGNITE-9320
URL: https://issues.apache.org/jira/browse/IGNITE-9320
Project: Ignite
Issue Type: Task
Components: mvcc
Reporter: Vladimir Ozerov
Fix For: 2.7
We need to find a way to configure MVCC caches. Currently this is a global
setting, which is not very convenient. Proposed solution:
# Introduce new \{{CacheAtomicityMode.TRANSACTIONAL_MVCC}}
# Do not allow to change cache this mode during restart (when persistence is
enabled)
# Do not allow transactions between \{{TRANSACTIONAL}} and
\{{TRANSACTIONAL_MVCC}} caches
# Add limitation to cache group - all caches within a group should have the
same mode
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Unknown known issue on cache rebalancing delayed
Igniters, I have found "Known issue, possible deadlock in case of low priority cache rebalancing delayed" comment in GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please explain when using rebalance delay can be an issue and why? -- Roman
Re: [MTCGA]: new failures in builds [1678707] needs to be handled
Hello everyone. Looks like I found cause of problem. It is https://issues.apache.org/jira/browse/IGNITE-8926. Ilya Lantukh could you please recheck your changes. You can easily reproduce problem locally by IgniteCacheFailedUpdateResponseTest#testInvokeAtomic. I also rerun all tests on your branch(ignite-8926), for recheck other possible problems. -- Best regards, Anton Kalashnikov 18.08.2018, 13:38, "Dmitriy Pavlov" : > Hi Igniters, > > I apologize for a number of notifications, it should be investigated and > fixed in TC Bot in case the same suite is constantly timed out. > > It is some constantly occurring timeout in Cache 4, started to occur after > some from these changes > biryukovvitaliy92, xxtern, dkarachentsev, stkuzma, kaa.dev, plehanov.alex, > kaa.dev, ilantukh, av, vinokurov.pasha > > The test probably causing hang is > IgniteCacheFailedUpdateResponseTest.testInvokeTx > (last started) > > Folks, please check your PRs Cache 4 status in RunAlls. > > I have to disable notifications from Bot to avoid spam. > > Sincerely, > Dmitriy Pavlov > > сб, 18 авг. 2018 г. в 13:22, : > >> Hi Ignite Developer, >> >> I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. >> I hope you can help. >> >> * New Critical Failure in master Cache 4 >> >> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Cache4=%3Cdefault%3E=buildTypeStatusDiv >> No changes in build >> >> - If your changes can led to this failure(s), please create issue >> with label MakeTeamCityGreenAgain and assign it to you. >> -- If you have fix, please set ticket to PA state and write to dev >> list fix is ready >> -- For case fix will require some time please mute test and set >> label Muted_Test to issue >> - If you know which change caused failure please contact change >> author directly >> - If you don't know which change caused failure please send >> message to dev list to find out >> Should you have any questions please contact dpav...@apache.org or write >> to dev.list >> Best Regards, >> MTCGA.Bot >> Notification generated at Sat Aug 18 13:22:35 MSK 2018
[jira] [Created] (IGNITE-9319) CacheAsyncOperationsFailoverTxTest.testPutAllAsyncFailover is flaky in master.
Alexei Scherbakov created IGNITE-9319: - Summary: CacheAsyncOperationsFailoverTxTest.testPutAllAsyncFailover is flaky in master. Key: IGNITE-9319 URL: https://issues.apache.org/jira/browse/IGNITE-9319 Project: Ignite Issue Type: Bug Reporter: Alexei Scherbakov Fix For: 2.7 https://ci.ignite.apache.org/viewLog.html?buildId=1688647=queuedBuildOverviewTab https://ci.ignite.apache.org/viewLog.html?buildId=1688542=queuedBuildOverviewTab -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4577: IGNITE-8251 Reduce testPageEviction run time
GitHub user udaykale opened a pull request: https://github.com/apache/ignite/pull/4577 IGNITE-8251 Reduce testPageEviction run time You can merge this pull request into a Git repository by running: $ git pull https://github.com/udaykale/ignite IGNITE-8251 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4577 commit 8d3021205018312347cc171b960481eca3fab1ed Author: uday Date: 2018-08-20T06:58:01Z IGNITE-8251 Reduce testPageEviction run time ---
