Re: QueryDetailMetrics for cache-less SQL queries

[Dmitriy Setrakyan]

But internally the SQL query still runs on some cache, no? What happens to
the metrics accumulated on that cache?

D.

On Thu, Aug 16, 2018, 18:51 Alexey Kuznetsov  wrote:

> Dima,
>
> "cache-less" means that SQL executed directly on SQL engine.
>
> In previous version of Ignite we execute queries via cache:
>
> ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")
>
> In current Ignite we can execute query directly without using cache as
> "gateway".
>
> And if we execute query directly, metrics not update.
>
>
>
>
> On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan 
> wrote:
>
> > Evgeny, what is a "cache-less" SQL query?
> >
> > D.
> >
> > On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> > e.zhuravlev...@gmail.com
> > > wrote:
> >
> > > Hi Igniters,
> > >
> > > I've started to work on adding QueryDetailMetrics for cache-less SQL
> > > queries(issue https://issues.apache.org/jira/browse/IGNITE-6677) and
> > found
> > > that it's required to change API. I don't think that adding methods
> like
> > > queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to
> Ignite
> > > class is a good idea. So, I see 2 possible solutions here:
> > >
> > > 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> > > Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
> > > metric "queryDetailMetrics" to it. Of course, old methods will be
> > > deprecated.
> > >
> > > 2. Finally create Ignite.sql() API, which was already discussed here:
> > > http://apache-ignite-developers.2346864.n4.nabble.
> > > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > > and place "queryDetailMetrics" metric there. Here is the ticket for
> this
> > > change: https://issues.apache.org/jira/browse/IGNITE-4701
> > >
> > > Personally, I think that the second solution looks better in this case,
> > > however, moving dataRegionMetrics and dataStorageMetrics to
> > > ignite.matrics() is still a good idea - IMO, Ignite class is not the
> > right
> > > place for them - we shouldn't change our main API class so often.
> > >
> > > What do you think?
> > >
> > > Thank you,
> > > Evgenii
> > >
> >
> > --
> > Alexey Kuznetsov
> >
> >
>

Re: Wrong off-heap size is reported for a node

[Dmitriy Setrakyan]

Is there a blocker ticket for 2.7?

On Thu, Aug 16, 2018, 19:59 Denis Magda  wrote:

> Igniters,
>
> Was troubleshooting an Ignite deployment today and couldn't find out from
> the logs what was the actual off-heap space used.
>
> Those were the given memory resoures (Ignite 2.6):
>
> [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology
> snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*,
> heap=24.0GB]
>
> And that weird stuff was reported by the node (pay attention to the last
> line):
>
> [2018-08-16 15:45:50,211][INFO
>
> ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257]
> ^-- H/N/C [hosts=1, nodes=1, CPUs=64]
> ^-- CPU [cur=0.03%, avg=5.54%, GC=0%]
> ^-- PageMemory [pages=6997377]
> ^-- Heap [used=9706MB, free=61.18%, comm=22384MB]
>* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always
> the same!*
>
> Had to change the code by using dataRegion.getPhysicalMemoryPages() to find
> out that actual off-heap usage size was
> >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB
>
> Let's fix this issue in 2.7, I proposed a new format. Please review and
> share your thoughts:
> https://issues.apache.org/jira/browse/IGNITE-9305
>
> --
> Denis
>

Wrong off-heap size is reported for a node

[Denis Magda]

Igniters,

Was troubleshooting an Ignite deployment today and couldn't find out from
the logs what was the actual off-heap space used.

Those were the given memory resoures (Ignite 2.6):

[2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology
snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*,
heap=24.0GB]

And that weird stuff was reported by the node (pay attention to the last
line):

[2018-08-16 15:45:50,211][INFO
][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257]
^-- H/N/C [hosts=1, nodes=1, CPUs=64]
^-- CPU [cur=0.03%, avg=5.54%, GC=0%]
^-- PageMemory [pages=6997377]
^-- Heap [used=9706MB, free=61.18%, comm=22384MB]
   * ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always
the same!*

Had to change the code by using dataRegion.getPhysicalMemoryPages() to find
out that actual off-heap usage size was
>>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB

Let's fix this issue in 2.7, I proposed a new format. Please review and
share your thoughts:
https://issues.apache.org/jira/browse/IGNITE-9305

--
Denis

[jira] [Created] (IGNITE-9305) Wrong off-heap size is reported for a node

[Denis Magda (JIRA)]

Denis Magda created IGNITE-9305:
---

 Summary: Wrong off-heap size is reported for a node
 Key: IGNITE-9305
 URL: https://issues.apache.org/jira/browse/IGNITE-9305
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.6
Reporter: Denis Magda
 Fix For: 2.7


Was troubleshooting an Ignite deployment today and couldn't find out from the 
logs what was the actual off-heap space used. 

Those were the given memory resoures (Ignite 2.6):

{code}
[2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology snapshot 
[ver=1, servers=1, clients=0, CPUs=64, offheap=30.0GB, heap=24.0GB]
{code}

And that weird stuff was reported by the node (pay attention to the last line):

{code}
[2018-08-16 15:45:50,211][INFO 
][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257]
^-- H/N/C [hosts=1, nodes=1, CPUs=64]
^-- CPU [cur=0.03%, avg=5.54%, GC=0%]
^-- PageMemory [pages=6997377]
^-- Heap [used=9706MB, free=61.18%, comm=22384MB]
^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always the 
same!
{code}

Had to change the code by using {code}dataRegion.getPhysicalMemoryPages(){code} 
to find out that actual off-heap usage size was 
{code}
>>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB
{code}


The logs have to report the following instead:
{code}
 ^-- Off-heap {Data Region 1} [used={dataRegion1.getPhysicalMemorySize()}, 
free=X%, comm=dataRegion1.maxSize()]
 ^-- Off-heap {Data Region 2} [used={dataRegion2.getPhysicalMemorySize()}, 
free=X%, comm=dataRegion2.maxSize()]
{code}

If Ignite persistence is enabled then the following extra lines have to be 
added to see the disk used space:
{code}
 ^-- Off-heap {Data Region 1}: used={dataRegion1.getTotalAllocatedSize() - 
dataRegion1.getPhysicalMemorySize()}
 ^-- Off-heap {Data Region 2} [used={dataRegion2.getTotalAllocatedSize() - 
dataRegion2.getPhysicalMemorySize()}]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: QueryDetailMetrics for cache-less SQL queries

[Alexey Kuznetsov]

Dima,

"cache-less" means that SQL executed directly on SQL engine.

In previous version of Ignite we execute queries via cache:

ignite.cache("Some cache").sqlFieldsQuery("select ... from ..")

In current Ignite we can execute query directly without using cache as
"gateway".

And if we execute query directly, metrics not update.




On Fri, Aug 17, 2018 at 4:21 AM Dmitriy Setrakyan 
wrote:

> Evgeny, what is a "cache-less" SQL query?
>
> D.
>
> On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev <
> e.zhuravlev...@gmail.com
> > wrote:
>
> > Hi Igniters,
> >
> > I've started to work on adding QueryDetailMetrics for cache-less SQL
> > queries(issue https://issues.apache.org/jira/browse/IGNITE-6677) and
> found
> > that it's required to change API. I don't think that adding methods like
> > queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to Ignite
> > class is a good idea. So, I see 2 possible solutions here:
> >
> > 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> > Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
> > metric "queryDetailMetrics" to it. Of course, old methods will be
> > deprecated.
> >
> > 2. Finally create Ignite.sql() API, which was already discussed here:
> > http://apache-ignite-developers.2346864.n4.nabble.
> > com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> > and place "queryDetailMetrics" metric there. Here is the ticket for this
> > change: https://issues.apache.org/jira/browse/IGNITE-4701
> >
> > Personally, I think that the second solution looks better in this case,
> > however, moving dataRegionMetrics and dataStorageMetrics to
> > ignite.matrics() is still a good idea - IMO, Ignite class is not the
> right
> > place for them - we shouldn't change our main API class so often.
> >
> > What do you think?
> >
> > Thank you,
> > Evgenii
> >
>
> --
> Alexey Kuznetsov
>
>

[GitHub] ignite pull request #4511: IGNITE-9031: SpringCacheManager throws AssertionE...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4511


---

Re: Spark SQL Table Name Resolution

[Dmitriy Setrakyan]

Stuart, can you please move the ticket into PATCH_AVAILABLE state? You need
to click "Submit Patch" button in Jira.

D.

On Wed, Aug 15, 2018 at 10:22 AM, Stuart Macdonald 
wrote:

> Here's the initial pull request for this issue, please review and let me
> know your feedback. I had to combine the two approaches to enable this to
> work for both standard .read() where we can add the schema option, and
> catalog-based selects where we use schemaName.tableName. Happy to discuss
> on a call if this isn't clear.
>
> https://github.com/apache/ignite/pull/4551
>
> On Thu, Aug 9, 2018 at 2:32 PM, Stuart Macdonald 
> wrote:
>
> > Hi Nikolay, yes would be happy to - will likely be early next week. I’ll
> > go with the approach of adding a new optional field to the Spark data
> > source provider unless there are any objections.
> >
> > Stuart.
> >
> > > On 9 Aug 2018, at 14:20, Nikolay Izhikov  wrote:
> > >
> > > Stuart, do you want to work on this ticket?
> > >
> > > В Вт, 07/08/2018 в 11:13 -0700, Stuart Macdonald пишет:
> > >> Thanks Val, here’s the ticket:
> > >>
> > >> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-9228
> > >>  > IGNITE-9228?filter=allopenissues>
> > >>
> > >> (Thanks for correcting my terminology - I work mostly with the
> > traditional
> > >> CacheConfiguration interface where I believe each cache occupies its
> own
> > >> schema.)
> > >>
> > >> Stuart.
> > >>
> > >> On 7 Aug 2018, at 18:34, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>
> > >> wrote:
> > >>
> > >> Stuart,
> > >>
> > >> Two tables can have same names only if they are located in different
> > >> schemas. Said that, sdding schema name support makes sense to me for
> > sure.
> > >> We can implement this using either separate SCHEMA_NAME parameter, or
> > >> similar to what you suggested in option 3 but with schema name instead
> > of
> > >> cache name.
> > >>
> > >> Please feel free to create a ticket.
> > >>
> > >> -Val
> > >>
> > >> On Tue, Aug 7, 2018 at 9:32 AM Stuart Macdonald 
> > wrote:
> > >>
> > >> Hello Igniters,
> > >>
> > >>
> > >> The Ignite Spark SQL interface currently takes just “table name” as a
> > >>
> > >> parameter which it uses to supply a Spark dataset with data from the
> > >>
> > >> underlying Ignite SQL table with that name.
> > >>
> > >>
> > >> To do this it loops through each cache and finds the first one with
> the
> > >>
> > >> given table name [1]. This causes issues if there are multiple tables
> > >>
> > >> registered in different caches with the same table name as you can
> only
> > >>
> > >> access one of those caches from Spark. Is the right thing to do here:
> > >>
> > >>
> > >> 1. Simply not support such a scenario and note in the Spark
> > documentation
> > >>
> > >> that table names must be unique?
> > >>
> > >> 2. Pass an extra parameter through the Ignite Spark data source which
> > >>
> > >> optionally specifies the cache name?
> > >>
> > >> 3. Support namespacing in the existing table name parameter, ie
> > >>
> > >> “cacheName.tableName”?
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Stuart.
> > >>
> > >>
> > >> [1]
> > >>
> > >>
> > >> https://github.com/apache/ignite/blob/ca973ad99c6112160a305df05be945
> > 8e29f88307/modules/spark/src/main/scala/org/apache/ignite/
> > spark/impl/package.scala#L119
> >
>

Re: QueryDetailMetrics for cache-less SQL queries

[Dmitriy Setrakyan]

Evgeny, what is a "cache-less" SQL query?

D.

On Thu, Aug 16, 2018 at 6:36 AM, Evgenii Zhuravlev  wrote:

> Hi Igniters,
>
> I've started to work on adding QueryDetailMetrics for cache-less SQL
> queries(issue https://issues.apache.org/jira/browse/IGNITE-6677) and found
> that it's required to change API. I don't think that adding methods like
> queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to Ignite
> class is a good idea. So, I see 2 possible solutions here:
>
> 1. Create IgniteMetrics(ignite.metrics()) and move metrics from
> Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
> metric "queryDetailMetrics" to it. Of course, old methods will be
> deprecated.
>
> 2. Finally create Ignite.sql() API, which was already discussed here:
> http://apache-ignite-developers.2346864.n4.nabble.
> com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
> and place "queryDetailMetrics" metric there. Here is the ticket for this
> change: https://issues.apache.org/jira/browse/IGNITE-4701
>
> Personally, I think that the second solution looks better in this case,
> however, moving dataRegionMetrics and dataStorageMetrics to
> ignite.matrics() is still a good idea - IMO, Ignite class is not the right
> place for them - we shouldn't change our main API class so often.
>
> What do you think?
>
> Thank you,
> Evgenii
>

Re: StandaloneWalRecordsIterator: support iteration from custom pointer

[Dmitriy Govorukhin]

Ivan,

I implemented this issue, please review my changes.
https://reviews.ignite.apache.org/ignite/review/IGNT-CR-729

On Thu, Aug 16, 2018 at 3:09 PM Ivan Rakov  wrote:

> Thanks for your comments!
> I've created a ticket: https://issues.apache.org/jira/browse/IGNITE-9294
>
> Best Regards,
> Ivan Rakov
>
> On 15.08.2018 21:31, Dmitriy Setrakyan wrote:
> > Agree, this should be a great performance boost.
> >
> > On Wed, Aug 15, 2018 at 10:17 AM, Dmitriy Pavlov 
> > wrote:
> >
> >> Hi Ivan,
> >>
> >> I agree that providing WAL pointer is the better option. Initially,
> >> Standalone WAL iterator was developed for debugging utility, so a set of
> >> files was perfectly OK.
> >>
> >> Sincerely,
> >> Dmitriy Palov
> >>
> >> ср, 15 авг. 2018 г. в 20:06, Ivan Rakov :
> >>
> >>> Igniters,
> >>>
> >>> Right now we are developing WAL shipping process for our internal
> >>> purposes and we use StandaloneWalRecordsIterator to read WAL files from
> >>> custom destination. We have bumped into a problem - iterator can be
> >>> constructed from set of files and dirs, but there's no option to pass
> >>> WAL pointer to the iterator factory class to start iteration with. It
> >>> can be worked around (by filtering all records prior to needed
> pointer),
> >>> but I think it would be handy to add such option to
> >>> IgniteWalIteratorFactory API.
> >>>
> >>> What do you think?
> >>>
> >>> --
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>>
>
>

[GitHub] ignite pull request #4563: IGNITE-9294 Implement bounds for StandAloneWALIte...

[DmitriyGovorukhin]

GitHub user DmitriyGovorukhin opened a pull request:

https://github.com/apache/ignite/pull/4563

IGNITE-9294 Implement bounds for StandAloneWALIterator



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9294

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4563.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4563


commit 6c8e731ca75b2c628a3fc40c60501d374145db13
Author: Dmitriy Govorukhin 
Date:   2018-08-16T20:51:13Z

IGNITE-9294 Implement bound for StandAloneIterator. Replay WAL 
form/to/between. + test




---

[jira] [Created] (IGNITE-9304) AssertionError while updating ATOMIC cache while topology changes

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-9304:
-

 Summary: AssertionError while updating ATOMIC cache while topology 
changes
 Key: IGNITE-9304
 URL: https://issues.apache.org/jira/browse/IGNITE-9304
 Project: Ignite
  Issue Type: Bug
Reporter: Eduard Shangareev


I have reproduced it with 
CacheContinuousQueryFailoverAbstractSelfTest#testMultiThreadedFailover.

{code}
java.lang.AssertionError: Assertion error on search row: 
org.apache.ignite.internal.processors.cache.tree.SearchRow@257eba3
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1655)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1306)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:381)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1789)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2471)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1930)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1772)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1662)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3111)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:133)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:269)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:264)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at 
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1$2$1.run(GridCacheIoManager.java:275)
at 
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Invalid version for inner update 
[isNew=false, entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=2, 
super=GridDistributedCacheEntry [super=CacheObjectImpl [val=null, 
hasValBytes=true]CacheObjectImpl [val=null, hasValBytes=true], 
ver=GridCacheVersion [topVer=145812931, order=1534332978541, nodeOrder=37], 
hash=2, extras=null, flags=0]]], newVer=GridCacheVersion [topVer=145812931, 
order=1534332978538, nodeOrder=43]]
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.versionCheck(GridCacheMapEntry.java:5397)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4778)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4562)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3104)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2998)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1744)
at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1622)
... 20 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9303) PageSnapshot can contain wrong pageId tag when not dirty page is recycling

[Aleksey Plekhanov (JIRA)]

Aleksey Plekhanov created IGNITE-9303:
-

 Summary: PageSnapshot can contain wrong pageId tag when not dirty 
page is recycling
 Key: IGNITE-9303
 URL: https://issues.apache.org/jira/browse/IGNITE-9303
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Aleksey Plekhanov


When page is recycling (for example in {{BPlusTree.Remove#freePage()}} -> 
{{DataStructure#recyclePage()}}) tag of {{pageId}} is modified, but original 
{{pageId}} is passed to {{writeUnlock()}} method and this passed {{pageId}} is 
stored to PageSnapshot WAL record.
This bug may lead to errors in WAL applying during crash recovery.

Reproducer (ignite-indexing module must be in classpath):

{code:java}
public class WalFailReproducer extends AbstractWalDeltaConsistencyTest {
/** {@inheritDoc} */
@Override protected boolean checkPagesOnCheckpoint() {
return true;
}

/**
 *
 */
public final void testPutRemoveCacheDestroy() throws Exception {
CacheConfiguration ccfg = new 
CacheConfiguration<>("cache0");
ccfg.setIndexedTypes(Integer.class, Integer.class);

IgniteEx ignite = startGrid(0);

ignite.cluster().active(true);

IgniteCache cache0 = ignite.getOrCreateCache(ccfg);

for (int i = 0; i < 5_000; i++)
cache0.put(i, i);

forceCheckpoint();

for (int i = 1_000; i < 4_000; i++)
cache0.remove(i);

forceCheckpoint();

stopAllGrids();
}
}

{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Code inspection

[Dmitriy Pavlov]

Hi Pavel,

Thank you for noticing and bringing it here. I've fixed TC failure.

Sincerely,
Dmitriy Pavlov

чт, 16 авг. 2018 г. в 0:10, Pavel Pereslegin :

> Hello Igniters.
>
> It seems that "idea/ignite_inspections.xml" should be excluded from
> "check-licenses" maven profile, because "_Licenses Headers_"
> configuration always fails now [1] on TeamCity.
>
> [1]
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_LicensesHeaders=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E
> ср, 15 авг. 2018 г. в 20:49, Dmitriy Pavlov :
> >
> > I've updated wiki page
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-C.CodeInspection
> > with
> > reference to settings.xml placement in the project.
> >
> > It is only advice, so I hope you don't mind having this reference.
> >
> > ср, 15 авг. 2018 г. в 16:45, Dmitriy Pavlov :
> >
> > > Hi Maxim,
> > >
> > > Thank you for stepping in. I've committed the first version here
> > > 'idea/ignite_inspections.xml'. We can move it to project default later
> when
> > > all inspection problems are fixed.
> > > Commit:
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=ignite.git;a=commit;h=3e0f04edf7cc0aa1631fbd1b9af1e9b87b697eb1
> > >
> > >
> > > Igniters,
> > >
> > > you can enable this profile using the following steps: Inspections
> > > (icon)->Configure inspections->(settings button)->Import
> Profile->select
> > > file and import.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 14 авг. 2018 г. в 16:31, Maxim Muzafarov :
> > >
> > >> Dmitry and other Igniters,
> > >>
> > >> Previously you has suggested to commit `Code Inspections` into Ignite
> > >> codebase.
> > >> It makes sense for me. I think it's the easiest way to share this
> profile
> > >> among community
> > >> members and this inspection can be set as for the project level.
> > >> So, I suggest:
> > >>
> > >> 1) According to Jetbrains documentation [1] the inspection profile
> can be
> > >> placed to
> > >> `/.idea/inspectionProfiles` with name `Project_Default.xml`
> > >> (hope most of us using IDEA for development)
> > >> This allows enable this profile automatically on per project level and
> > >> will
> > >> simplify
> > >> development process according to rules accepted by our community.
> > >>
> > >> 2) I can file tickets and do some of them to fix inspection failures
> which
> > >> Alexey mentioned
> > >> earlier. Hope other members of community will help me with it.
> > >>
> > >> 3) I think `Inspections (Core)` TeamCity can be triggered as nightly
> build
> > >> as it takes more
> > >> than 4 hours. Suppose, inspection build in each PR is not the best
> way in
> > >> our case. New run
> > >> here [2].
> > >>
> > >> 4) We can tune our MTCGA.Bot to notify members with new inspection
> > >> failures
> > >> added by them.
> > >>
> > >>
> > >> Also, I've taken Alexey's inspection file as an example, I’ve checked
> what
> > >> we already discussed
> > >> previously (e.g. `Anonymous class can be converted to lambda` should
> be
> > >> disabled by default)
> > >> and added these additional rules to it:
> > >>  - General | Line is longer than allowed by code style
> > >>  - Java | Code maturity | Call to 'printStackTrace()'
> > >>  - Java | Code style issues | Unnecessary 'null' check before
> 'equals()'
> > >> call
> > >>
> > >> If we decide to proceed I will attach this file to JIRA.
> > >>
> > >>
> > >> [1] https://www.jetbrains.com/help/idea/code-inspection.html
> > >> [2]
> > >>
> > >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsCore=buildTypeStatusDiv_IgniteTests24Java8=pull%2F3710%2Fhead
> > >>
> > >>
> > >> On Tue, 14 Aug 2018 at 16:19 Dmitriy Pavlov 
> > >> wrote:
> > >>
> > >> > Hi Dmitriy Govoruknin,
> > >> >
> > >> > It seems there is a lack of volunteers to apply code inspections to
> our
> > >> CI
> > >> > & patch submission process. Probably we could come back to your
> > >> > initial idea about setting up inspection locally.
> > >> >
> > >> > Could you commit or share your IDEA inspection settings? I could
> apply
> > >> it
> > >> > at least on my machine and remove odd warning types one by one.
> What do
> > >> you
> > >> > think?
> > >> >
> > >> > Sincerely,
> > >> > Dmitriy Pavlov
> > >> >
> > >> > сб, 4 авг. 2018 г. в 1:22, Dmitriy Pavlov :
> > >> >
> > >> > > Ideally, I should not asking for people to solve something for me.
> > >> > >
> > >> > > I just want this idea did not disappear from our radar.
> > >> > >
> > >> > > пт, 3 авг. 2018 г. в 23:47, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >> >:
> > >> > >
> > >> > >> On Fri, Aug 3, 2018 at 7:49 AM, Dmitriy Pavlov <
> > >> dpavlov@gmail.com>
> > >> > >> wrote:
> > >> > >>
> > >> > >> >
> > >> > >> > I understand it is not so Apache-way from my side to ask
> > >> volunteers to
> > >> > >> do
> > >> > >> > some things (instead of contributing it by myself).
> > >> > >>
> > >> > >>
> > >> > >> Dmitriy, I am 

Re: [MTCGA]: new failures in builds [1662343] needs to be handled

[Dmitriy Pavlov]

Hi,

I've failed https://issues.apache.org/jira/browse/IGNITE-9301 and I'm
trying to disable test failure.

Help with .Net is really appreciated. I think it should not be a difficult
issue.

Sincerely,
Dmitriy Pavlov

чт, 16 авг. 2018 г. в 2:17, Vyacheslav Daradur :

> Hi, Alex, please have a look, it may be interesting for you.
>
> AFAIK, extending of interface *ICompute* with a new method
> *WithNoResultCache* should be enough to fix the *ApiParity* test. It
> may require changes in XSD scheme.
>
> But it would be great if you'll find time to fully implement a new method.
>
> On Wed, Aug 15, 2018 at 8:22 PM Dmitriy Pavlov 
> wrote:
> >
> > Hi Igniters,
> >
> > Failure related to recently merged change,
> > https://issues.apache.org/jira/browse/IGNITE-6284
> >
> > I know how to disable this test failure, but probably we need method
> > 'compute.withNoResultCache()' in .NET.
> >
> > What do you think? Who can help with adding such method to .NET API?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 15 авг. 2018 г. в 20:19, :
> >
> > > Hi Ignite Developer,
> > >
> > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be
> addressed.
> > > I hope you can help.
> > >
> > >  *New test failure in master ComputeParityTest.TestCompute
> > >
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8894704366690255803=%3Cdefault%3E=testDetails
> > >  Changes may led to failure were done by
> > >  - eshangareev
> > >
> http://ci.ignite.apache.org/viewModification.html?modId=828247=false
> > >  - lipkovich.mikhail
> > >
> http://ci.ignite.apache.org/viewModification.html?modId=828246=false
> > >
> > > - If your changes can led to this failure(s), please create
> issue
> > > with label MakeTeamCityGreenAgain and assign it to you.
> > > -- If you have fix, please set ticket to PA state and write to
> dev
> > > list fix is ready
> > > -- For case fix will require some time please mute test and set
> > > label Muted_Test to issue
> > > - If you know which change caused failure please contact change
> > > author directly
> > > - If you don't know which change caused failure please send
> > > message to dev list to find out
> > > Should you have any questions please contact dpav...@apache.org or
> write
> > > to dev.list
> > > Best Regards,
> > > MTCGA.Bot
> > > Notification generated at Wed Aug 15 20:19:28 MSK 2018
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.
>

[GitHub] ignite pull request #4562: IGNITE-9302 Java Thin Clients TC configuration ha...

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/4562

IGNITE-9302 Java Thin Clients TC configuration hangs because of no ti…

…meout on tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9302

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4562.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4562


commit 0bcd88ba6f75f52d63f11b2b0a38fb103713b823
Author: EdShangGG 
Date:   2018-08-16T16:46:15Z

IGNITE-9302 Java Thin Clients TC configuration hangs because of no timeout 
on tests




---

[jira] [Created] (IGNITE-9302) Java Thin Clients TC configuration hangs because of no timeout on tests

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-9302:
-

 Summary: Java Thin Clients TC configuration hangs because of no 
timeout on tests
 Key: IGNITE-9302
 URL: https://issues.apache.org/jira/browse/IGNITE-9302
 Project: Ignite
  Issue Type: Bug
Reporter: Eduard Shangareev
Assignee: Eduard Shangareev


https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_JavaThinClient=buildTypeHistoryList_IgniteTests24Java8=__all_branches__



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9301) Support method compute withNoResultCache in .Net

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-9301:
--

 Summary: Support method compute withNoResultCache in .Net
 Key: IGNITE-9301
 URL: https://issues.apache.org/jira/browse/IGNITE-9301
 Project: Ignite
  Issue Type: Task
  Components: platforms
Reporter: Dmitriy Pavlov


During https://issues.apache.org/jira/browse/IGNITE-6284 implementation new 
method was added -
org.apache.ignite.IgniteCompute#withNoResultCache

but this method was not supported in .NET API version.

It is required to add correct support to .NET



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: This morning I added ssl support to...

[Dmitriy Setrakyan]

On Thu, Aug 16, 2018 at 8:39 AM, Paul Anderson  wrote:

> Wonderful... I also have Netdata integration... (producing netdata charts
> straight from ignite) if you are interested...
>
> http://my-netdata.io


Yes, we are interested very much.

[GitHub] ignite pull request #4561: IGNITE-896 Configuration inconsistency

[zzzadruga]

GitHub user zzzadruga opened a pull request:

https://github.com/apache/ignite/pull/4561

IGNITE-896 Configuration inconsistency



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zzzadruga/ignite IGNITE-896

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4561.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4561


commit 39226745ce2a620d34efe31b5bcbd54508786375
Author: zzzadruga 
Date:   2018-08-14T17:06:22Z

IGNITE-896 Add evictionFilterFactory field, getter and setter. Mark 
evictionFilter as @deprecated.

commit 93eaaccabdc8c778969cfb20dc3bdf60699fb7df
Author: zzzadruga 
Date:   2018-08-14T17:08:33Z

IGNITE-896 Add Factory for IgfsEvictionFilter

commit 43e4696603e863b668eeb683e62fb5046a34ce1d
Author: zzzadruga 
Date:   2018-08-14T17:10:23Z

IGNITE-896 Refactor IgfsHelper

commit bf7697d1b13d3e22f5fa284a08ed4dd9ce74bf1b
Author: zzzadruga 
Date:   2018-08-14T17:11:56Z

IGNITE-896 Refactor GridCacheEvictionManager

commit ed38c977801d7b3fa2da3d69d9c8a329dad9ca93
Author: Nikolai Kulagin 
Date:   2018-08-15T12:10:48Z

IGNITE-896 Add topValidatorFactory field, getter and setter. Mark 
topValidator as @deprecated

commit c31db024012bd9a3dacb3294397a0a277909d1e8
Author: Nikolai Kulagin 
Date:   2018-08-16T12:36:34Z

IGNITE-896 Fix some tests

commit f0119237cbcdcc4001fecd1cf2aafb782c26e475
Author: Nikolai Kulagin 
Date:   2018-08-16T12:58:25Z

IGNITE-896 Fix CacheGroupContext

commit 97894930038b9c007238cb4af0ecae056f117173
Author: Nikolai Kulagin 
Date:   2018-08-16T13:02:12Z

IGNITE-896 Apply patch [IGNITE-9005, PR 4373] + fix for 
evictionFilterFactory

commit d56bb7ad893cdd5a70b1059f36fcd7a96d8b9928
Author: Nikolai Kulagin 
Date:   2018-08-16T13:08:49Z

IGNITE-896 Fix GridCacheAttributes

commit a2731af7475281fabe2289a18a1ee1db26aa3111
Author: Nikolai Kulagin 
Date:   2018-08-16T13:11:16Z

IGNITE-896 Fix VisorCacheConfiguration

commit d21485da5d05fe2872563ab8af3680f4dc6b3510
Author: Nikolai Kulagin 
Date:   2018-08-16T13:20:35Z

IGNITE-896 Fix ConfigVariations

commit bee2788441dddc0b07f00b03ad32ccc210413b7c
Author: Nikolai Kulagin 
Date:   2018-08-16T15:52:29Z

IGNITE-896 Add serialVersionUID in IgfsEvictionFilterFactory

commit 485eb16340a8aaef162e391f652d95d5399868d7
Author: Nikolai Kulagin 
Date:   2018-08-16T15:53:02Z

IGNITE-896 Fix ClusterCachesInfo




---

[GitHub] ignite pull request #4518: IGNITE-9227 Fast reply on single message if excha...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4518


---

Re: This morning I added ssl support to...

[Paul Anderson]

Wonderful... I also have Netdata integration... (producing netdata charts
straight from ignite) if you are interested...

http://my-netdata.io


On Thu, 16 Aug 2018, 17:35 Dmitriy Pavlov,  wrote:

> Hi Paul,
>
> I've added you to the list of contributors, so you can now assign the issue
> to yourself.
>
> Being a contributor does not require you to participate any more than you
> already do.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 16 авг. 2018 г. в 17:05, Paul Anderson :
>
> > IGNITE-9298
> >
> > On Thu, 16 Aug 2018, 15:35 Nikolay Izhikov,  wrote:
> >
> > > Let's do it.
> > >
> > > Please, send a ticket number to this thread.
> > >
> > > В Чт, 16/08/2018 в 15:22 +0200, Paul Anderson пишет:
> > > > We needed here so I just did it... Very simple will raise a ticket
> and
> > > add
> > > > patch to it...
> > > >
> > > > On Thu, 16 Aug 2018, 15:21 Nikolay Izhikov, 
> > wrote:
> > > >
> > > > > Hello, Paul.
> > > > >
> > > > > Is there a ticket you trying to resolve?
> > > > > Yes, please, provide the patch, so community can take a look.
> > > > >
> > > > > В Чт, 16/08/2018 в 15:03 +0200, Paul Anderson пишет:
> > > > > > org.apache.ignite.internal.commandline.CommandHandler
> (control.sh)
> > > > > >
> > > > > > Can I send you a diff rather than becoming a contributer for the
> > > moment.
> >
>

Re: This morning I added ssl support to...

[Dmitriy Pavlov]

Hi Paul,

I've added you to the list of contributors, so you can now assign the issue
to yourself.

Being a contributor does not require you to participate any more than you
already do.

Sincerely,
Dmitriy Pavlov

чт, 16 авг. 2018 г. в 17:05, Paul Anderson :

> IGNITE-9298
>
> On Thu, 16 Aug 2018, 15:35 Nikolay Izhikov,  wrote:
>
> > Let's do it.
> >
> > Please, send a ticket number to this thread.
> >
> > В Чт, 16/08/2018 в 15:22 +0200, Paul Anderson пишет:
> > > We needed here so I just did it... Very simple will raise a ticket and
> > add
> > > patch to it...
> > >
> > > On Thu, 16 Aug 2018, 15:21 Nikolay Izhikov, 
> wrote:
> > >
> > > > Hello, Paul.
> > > >
> > > > Is there a ticket you trying to resolve?
> > > > Yes, please, provide the patch, so community can take a look.
> > > >
> > > > В Чт, 16/08/2018 в 15:03 +0200, Paul Anderson пишет:
> > > > > org.apache.ignite.internal.commandline.CommandHandler (control.sh)
> > > > >
> > > > > Can I send you a diff rather than becoming a contributer for the
> > moment.
>

Re: Metrics for MVCC caches

[Павлухин Иван]

Hi Alex,

Thank you for referring spec. It really gives strict point how JCache
metrics should be counted.
So, I will start implementation of metrics for MVCC caches according to
spec and existing behavior.

As for improper "removals" calculation, I filed a ticket
https://issues.apache.org/jira/browse/IGNITE-9300

2018-08-10 17:48 GMT+03:00 Alex Plehanov :

> Hi, Ivan
>
> Looks like a bug. Metrics counting described at JSR107 (JCache)
> specification. It says, that "cache removals" metric should be incremented
> if remove() method returns true. Since there are no matching keys in the
> first remove() call should be found, remove() should return false and
> "cache removals" metric should not be affected.
>
> 2018-08-10 12:56 GMT+03:00 Павлухин Иван :
>
> > Hi,
> >
> > Dmitriy thanks for note, I will keep metrics IEP in mind.
> >
> > Roman it sounds quite a straightforward approach. Moreover current cache
> > transactions follow it: there could be a number of invisible actions in
> > private workspace (e.g. creating and then deleting the same entry) but
> only
> > final changes are metered.
> >
> > Also, I observed strange behavior for TRANSACTIONAL cache operations. For
> > following case metrics surprised me (numbers in comments):
> >
> > public void testPutRemove() throws Exception {
> > cache.remove(1);
> > System.out.println(cache.metrics().getCacheRemovals()); // 1
> >
> > cache.put(1, "a");
> >
> > cache.remove(1);
> > System.out.println(cache.metrics().getCacheRemovals()); // 2
> > }
> >
> > While for ATOMIC cache I see 0 and 1 which looks much better. And similar
> > effect is observed for "replace" operations.
> >
> > Is it a bug? Do we have some document describing how metrics should be
> > counted?
> >
> > 2018-08-08 21:11 GMT+03:00 Dmitriy Setrakyan :
> >
> > > I think we should be updating the IEP with metrics specifications in
> > > parallel.
> > >
> > > D.
> > >
> > > On Wed, Aug 8, 2018, 05:32 Roman Kondakov 
> > > wrote:
> > >
> > > > Hi Ivan!
> > > >
> > > > In my opinion we need to preserve the essence of the metrics: if we
> > > > didn't consider dirty writes as updates before MVCC implementation,
> we
> > > > also shouldn't count these writes in MVCC metrics implementation too.
> > > > So, I think we need to count only committed entries. But we can count
> > > > this dirty writes as additional metrics.
> > > >
> > > > Also additional metrics concerning MVCC could be:
> > > >
> > > > - Average count of the active transactions per snapshot
> > > >
> > > > - Average quantity of versions per key
> > > >
> > > >
> > > > --
> > > >
> > > > Roman Kondakov
> > > >
> > > >
> > > > On 07.08.2018 17:17, Павлухин Иван wrote:
> > > > > Hi Igniters,
> > > > >
> > > > > I am working on cache metrics support for caches with enabled MVCC.
> > As
> > > > you
> > > > > may know, during MVCC transaction execution entry versions are
> > written
> > > to
> > > > > storage right away (without deferring until commit). So, it is not
> > > > obvious
> > > > > for me if we should update writes count right away or defer it
> until
> > > > > commit. On one hand, MVCC entry version write is not "true" write
> > until
> > > > > commit. On the other hand, it definitely changes the storage. How
> do
> > we
> > > > > interpret write metrics in Ignite philosophy?
> > > > >
> > > > > Also, it seems that bunch of MVCC specific metrics could be
> > > introduced. I
> > > > > would like to collect some thoughts about it. Which such metrics
> come
> > > to
> > > > > your mind?
> > > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>



-- 
Best regards,
Ivan Pavlukhin

[GitHub] ignite pull request #4554: IGNITE-9258: NodeJS multiple clients fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4554


---

Re: Peer review: Victory over Patch Available debt

[Dmitriy Setrakyan]

Great idea!

Review is not only meant for the committers, but for everyone in the
community. Committers are only responsible for final reviews and merges.

D.

On Thu, Aug 16, 2018 at 7:29 AM, Dmitriy Pavlov 
wrote:

> Hi Igniters,
>
> For a long time review queue in the community is quite long. Ticket count
> is around 80-100 for last year. I would like to make the proposal, which
> should help the community with this aspect. Being reviewer for some tickets
> I found a number of tickets from new community members with not checked
> tests, not linked dev. list discussion, empty description, code style
> violations etc. It is not easy to finalize the review in such case.
>
> Every Igniter could make way too needed contribution to Apache Ignite by
> educating newcomers how to make ticket, patch, and discussion as perfect as
> it is humanly possible.
>
> I propose any Igniter (regardless of his role and experience: PMC,
> committer or contributor) will comment patches of other members in JIRA to
> help newcomers make contribution compliant with
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute  I
> know a number of community members already do so.
>
> Every contributor could find any ticket he or she likes in
> https://cwiki.apache.org/confluence/display/IGNITE/
> Issues+waiting+for+review
>
>
> The contributor can check compliance using
> https://cwiki.apache.org/confluence/display/IGNITE/Review+Checklist
> and copy-paste review checklist items to the ticket with his or her visa
> about each item. Each contribution can have an infinite number of visas.
>
> Looks good visa are also helpful, but I would like to go forward and
> formalize this process. Please share your vision. I would be happy to run a
> short webinar in case you have questions. I will update how to contribute
> shortly if we agree.
>
> Sincerely,
> Dmitriy Pavlov
>

[GitHub] ignite pull request #4505: IGNITE-9196 Fix memory leak in MapNodeResults

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4505


---

[jira] [Created] (IGNITE-9300) Removal count metric is calculated improperly for transactional cache

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-9300:
--

 Summary: Removal count metric is calculated improperly for 
transactional cache
 Key: IGNITE-9300
 URL: https://issues.apache.org/jira/browse/IGNITE-9300
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Ivan Pavlukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Peer review: Victory over Patch Available debt

[Dmitriy Pavlov]

Hi Igniters,

For a long time review queue in the community is quite long. Ticket count
is around 80-100 for last year. I would like to make the proposal, which
should help the community with this aspect. Being reviewer for some tickets
I found a number of tickets from new community members with not checked
tests, not linked dev. list discussion, empty description, code style
violations etc. It is not easy to finalize the review in such case.

Every Igniter could make way too needed contribution to Apache Ignite by
educating newcomers how to make ticket, patch, and discussion as perfect as
it is humanly possible.

I propose any Igniter (regardless of his role and experience: PMC,
committer or contributor) will comment patches of other members in JIRA to
help newcomers make contribution compliant with
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute  I
know a number of community members already do so.

Every contributor could find any ticket he or she likes in
https://cwiki.apache.org/confluence/display/IGNITE/Issues+waiting+for+review


The contributor can check compliance using
https://cwiki.apache.org/confluence/display/IGNITE/Review+Checklist
and copy-paste review checklist items to the ticket with his or her visa
about each item. Each contribution can have an infinite number of visas.

Looks good visa are also helpful, but I would like to go forward and
formalize this process. Please share your vision. I would be happy to run a
short webinar in case you have questions. I will update how to contribute
shortly if we agree.

Sincerely,
Dmitriy Pavlov

[jira] [Created] (IGNITE-9299) .NET: Support dependency injection for user-defined classes that are instantiated by Ignite

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-9299:
--

 Summary: .NET: Support dependency injection for user-defined 
classes that are instantiated by Ignite
 Key: IGNITE-9299
 URL: https://issues.apache.org/jira/browse/IGNITE-9299
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn


Ignite instantiates some of user-defined classes like Services, Compute Jobs, 
and so on.
There is no way to inject user-specific dependencies into these objects.

See StackOverflow question for more details:
https://stackoverflow.com/questions/51872358/dependency-injection-in-apache-ignite-net-service

In java there is {{@ SpringResource}}, we need to come up with a similar 
solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: This morning I added ssl support to...

[Paul Anderson]

IGNITE-9298

On Thu, 16 Aug 2018, 15:35 Nikolay Izhikov,  wrote:

> Let's do it.
>
> Please, send a ticket number to this thread.
>
> В Чт, 16/08/2018 в 15:22 +0200, Paul Anderson пишет:
> > We needed here so I just did it... Very simple will raise a ticket and
> add
> > patch to it...
> >
> > On Thu, 16 Aug 2018, 15:21 Nikolay Izhikov,  wrote:
> >
> > > Hello, Paul.
> > >
> > > Is there a ticket you trying to resolve?
> > > Yes, please, provide the patch, so community can take a look.
> > >
> > > В Чт, 16/08/2018 в 15:03 +0200, Paul Anderson пишет:
> > > > org.apache.ignite.internal.commandline.CommandHandler (control.sh)
> > > >
> > > > Can I send you a diff rather than becoming a contributer for the
> moment.

[jira] [Created] (IGNITE-9298) control.sh does not support SSL (org.apache.ignite.internal.commandline.CommandHandler)

[Paul Anderson (JIRA)]

Paul Anderson created IGNITE-9298:
-

 Summary: control.sh does not support SSL 
(org.apache.ignite.internal.commandline.CommandHandler)
 Key: IGNITE-9298
 URL: https://issues.apache.org/jira/browse/IGNITE-9298
 Project: Ignite
  Issue Type: Bug
  Components: clients
Affects Versions: 2.6
Reporter: Paul Anderson
 Attachments: Arguments.patch, CommandHandler.patch

We required SSL on the connector port and to use control.sh to work with the 
baseline configuration.

This morning I added support, see attached patches against 2.6.0 for 

org/apache/ignite/internal/commandline/CommandHandler.java

org/apache/ignite/internal/commandline/Arguments.java

No tests, no docs.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4560: IGNITE-9297 Review ML examples javadocs as of rel...

[oignatenko]

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/4560

IGNITE-9297 Review ML examples javadocs as of release 2.6



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9267

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4560.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4560


commit 926b679caa7aa3af1eb21a8621ae99a0c8cab2c7
Author: Oleg Ignatenko 
Date:   2018-08-08T16:42:36Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit b98854bc84aea69f3c0881249cecc89046738e78
Author: Oleg Ignatenko 
Date:   2018-08-10T15:12:54Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit a6f5f54a950a395c9c7addd5c07e189f29229628
Author: Oleg Ignatenko 
Date:   2018-08-10T18:21:52Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 504f9e21fc8ffbd3fb16214762d681621f66b5aa
Author: Oleg Ignatenko 
Date:   2018-08-10T18:43:14Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 0a673b19b4486ad4b4bcd771bf2c52a69aafd3ff
Author: Oleg Ignatenko 
Date:   2018-08-10T18:44:23Z

Merge branch 'master-ml' into ml-examples-review

commit 5fc0f526977478ad8b450b339bb1178ce10479f5
Author: Oleg Ignatenko 
Date:   2018-08-11T18:40:05Z

Merge branch 'master-ml' into ml-examples-review

commit 2d9b3dc6d8ac47a00c1c4d5820ac48e9545b37e1
Author: Oleg Ignatenko 
Date:   2018-08-13T11:58:20Z

Merge branch 'master-ml' into ml-examples-review

commit 7d4113f3b4f1d57e32a77c74816e50a150592851
Author: Oleg Ignatenko 
Date:   2018-08-13T16:19:36Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit f97ef2e9340a37d5c57c1eef9fd3a0804ba9cedb
Author: Oleg Ignatenko 
Date:   2018-08-13T17:00:24Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 0520d8ab79816fd0d2045e3a534497137b3f1635
Author: Oleg Ignatenko 
Date:   2018-08-14T13:34:18Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 315190332ee7e2d3810e80436131eaa8263cf9c1
Author: Oleg Ignatenko 
Date:   2018-08-14T15:20:52Z

ml examples review
- fixes for code style warnings (mostly duplication and javadocs)
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit ea07f1e8793297e38bdf078ab76f0718bde277a8
Author: Oleg Ignatenko 
Date:   2018-08-14T15:22:13Z

Merge branch 'master-ml' into ml-examples-review

commit e7b8837df6a8d84d099b8ada7a9d97a3aff818d7
Author: Oleg Ignatenko 
Date:   2018-08-14T17:06:55Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit efa7489f799778ea590a94d5ab8da7e404b66ad2
Author: Oleg Ignatenko 
Date:   2018-08-14T17:19:15Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit b801edaf07c09168819800da22f021140fc5f17b
Author: Oleg Ignatenko 
Date:   2018-08-14T17:24:17Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 4e3ea9f1b8ca6848f7395121f18b428180ba4284
Author: Oleg Ignatenko 
Date:   2018-08-14T17:50:58Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 7578fc0bf753d2f65fa08aef3697a9a2876fe69a
Author: Oleg Ignatenko 
Date:   2018-08-15T12:47:49Z

ml examples review
- fixes for code style warnings
-- verified with diffs overview, rebuild and executing the examples

commit 536573ac555cf6ab7478f58ed443277ae029d76a
Author: Oleg Ignatenko 
Date:   2018-08-15T12:52:27Z

ml examples review
- formatting cleanup
-- verified with diffs overview

commit 9105b1bf2939666eb7cc2c878160daec991a45ff
Author: Oleg Ignatenko 
Date:   2018-08-15T12:54:28Z

Merge branch 'master-ml' into ml-examples-review

commit b17b0d87a27f708b34d7f617ef0c3e0e4100ce0a
Author: Oleg Ignatenko 
Date:   2018-08-15T15:16:02Z

ml examples review
- wip
-- verified with diffs overview, executing the example and studying code 
and execution logs

commit 19089728d9ef5cccdd16fd5d7e7fc33ac53c183d
Author: Oleg Ignatenko 
Date:   2018-08-15T16:06:20Z

Merge branch 

Re: Ticket review checklist

[Dmitriy Pavlov]

Hi Ivan,

Unfortunately, the review checklist does not work as well as it could. I
hope the situation will change in the nearest future, I think we should
come back to this idea and encourage contributors and reviewers to use the
list.

As for lambda's: some Igniters feel confident about it, and some Igniters
don't. My opinion it is perfectly ok to use it if usage is local node only,
is there is no chance lambda is serialized to the network. If there is such
chance it is better to avoid it.

Sincerely,
Dmitriy Pavlov

чт, 16 авг. 2018 г. в 12:09, Павлухин Иван :

> Vladimir,
>
> First of all, statements in Java 8 section [1] looks kind of prohibitive
> for me. When a new contributor see words "preferred" and "avoided in most
> cases" he most likely will not use such features (like I did). If a
> statement is not prohibitive in practice it could be at least rephrased.
>
> A bit about expressiveness. I written a code during working on a real
> ticket. The case is quite common in Ignite codebase. You can find example
> with couple of approaches in snippet [2]. For me approach with lambdas is
> expressive, compact and simple.
>
> What do you think?
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Java8
> [2] https://gist.github.com/pavlukhin/92701277f66f8901a7feda6283a5a299
>
> 2018-08-16 11:21 GMT+03:00 Vladimir Ozerov :
>
> > Hi Ivan,
> >
> > From what I see we do not restrict contributors to use lambdas and
> streams.
> > Document states that plain collections and anonymous classes are
> > *preferred*. This is not obligatory requirement, and it seems reasonable
> to
> > me, because when developing complex projects at times it is better to
> have
> > more expressive code, than less non-obvious code which makes dozens
> > operations in a single string.
> >
> > Or may be there are any other statements in the checklist which prevents
> > users from using Java 8 features?
> >
> > Vladimir.
> >
> > On Tue, Aug 14, 2018 at 7:16 PM ipavlukhin  wrote:
> >
> > > Hi Igniters,
> > >
> > > I would like to refresh review checklist a little bit. Currently it [1]
> > > contains section against lambda Lambda expressions and Stream API. As
> > > far as I know it is not true anymore. Currently both features have
> > > theirs usage in core module. What is a state of affairs for a subject?
> > > Are there some well-known cases where e.g. lambdas are not applicable?
> > > Should we document it?
> > >
> > > I personally think that we could delete entire Java 8 section from
> > > checklist (and Java 5 as well). I understand that every tool should be
> > > used judiciously but I doubt that all cases can be covered in short
> > > checklist.
> > >
> > > [1]
> > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#
> > CodingGuidelines-Java8
> > >
> > >
> > > On 2018/07/09 20:53:42, Dmitry Pavlov  wrote:
> > >  > I also tend to agree about updating checklist>
> > >  >
> > >  > About suite timeouts, I suspect there is one problem introduced
> > > recently>
> > >  > within 3 days, which caused this mass timeouts.>
> > >  >
> > >  > I hope Igniters will find out reason soon. In relation to compute we
> > > have>
> > >  > only 2 possible cause:>
> > >  > Ivan Daschinskiy (idaschinskiy) 2 files IGNITE-8869 Fixed>
> > >  > PartitionsExchangeOnDiscoveryHistoryOverflowTest hanging>
> > >  > Signed-off-by: Andrey Gura  ···>
> > >  >
> > >  > Dmitriy Govorukhin (dgovorukhin) 12 files IGNITE-8827 Disable WAL
> > > during>
> > >  > apply updates on recovery>
> > >  >
> > >  > I guess if we fix this reason we will fix 10 suites more>
> > >  > References:>
> > >  >
> > >
> > > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> > IgniteTests24Java8_ComputeGrid=buildTypeHistoryList_
> > IgniteTests24Java8=%3Cdefault%3E>
> > >
> > >
> > >  >
> > >  >
> > >  > пн, 9 июл. 2018 г. в 22:29, Anton Vinogradov :>
> > >  >
> > >  > > Sounds reasonable.>
> > >  > > I've satrted Data Structures suite hang investigation [1].>
> > >  > >>
> > >  > > Igniters, especially commiters,>
> > >  > > I know, you're busy, but it will be a great help to the project in
> > > case you>
> > >  > > fix at least one hang per person.>
> > >  > >>
> > >  > > [1] https://issues.apache.org/jira/browse/IGNITE-8783>
> > >  > >>
> > >  > > пн, 9 июл. 2018 г. в 19:24, Maxim Muzafarov :>
> > >  > >>
> > >  > > > Hi Igniters,>
> > >  > > >>
> > >  > > > Let's back to discussion of review checklist. Can we add more>
> > >  > > clarification>
> > >  > > > about running all suites on TeamCity?>
> > >  > > >>
> > >  > > > My suggestion is: “All test suites MUST be run on TeamCity [3]
> > > before>
> > >  > > merge>
> > >  > > > to master, there MUST NOT be any test failures * and any
> > > tests\suites>
> > >  > > with>
> > >  > > > “execution timeouts” *. Not important test failures should be
> > > muted and>
> > >  > > > handled according to [4] process.”>
> > >  > > >>
> > >  > > > As you can see we have 

[GitHub] ignite pull request #4484: IGNITE-9186: Omit datastreamer per-thread buffers...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4484


---

[jira] [Created] (IGNITE-9297) Review ML examples javadocs as of release 2.6

[Oleg Ignatenko (JIRA)]

Oleg Ignatenko created IGNITE-9297:
--

 Summary: Review ML examples javadocs as of release 2.6
 Key: IGNITE-9297
 URL: https://issues.apache.org/jira/browse/IGNITE-9297
 Project: Ignite
  Issue Type: Task
  Components: examples, ml
Affects Versions: 2.6
Reporter: Oleg Ignatenko


There are many ML examples added and changed in latest releases, and some of 
them have rather brief javadocs and may lack useful details.

Review the examples and where needed improve javadocs, preferably using some 
well established example as a reference (eg 
[ComputeClientBinaryTaskExecutionExample|https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/binary/computegrid/ComputeClientBinaryTaskExecutionExample.java]).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9296) Stopping node by Failure Handler hangs up

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-9296:
--

 Summary:  Stopping node by Failure Handler hangs up
 Key: IGNITE-9296
 URL: https://issues.apache.org/jira/browse/IGNITE-9296
 Project: Ignite
  Issue Type: Bug
Reporter: Sergey Kosarev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

QueryDetailMetrics for cache-less SQL queries

[Evgenii Zhuravlev]

Hi Igniters,

I've started to work on adding QueryDetailMetrics for cache-less SQL
queries(issue https://issues.apache.org/jira/browse/IGNITE-6677) and found
that it's required to change API. I don't think that adding methods like
queryDetailMetrics, resetQueryDetailMetrics, as in IgniteCache to Ignite
class is a good idea. So, I see 2 possible solutions here:

1. Create IgniteMetrics(ignite.metrics()) and move metrics from
Ignite(like dataRegionMetrics and dataStorageMetrics) and add a new
metric "queryDetailMetrics" to it. Of course, old methods will be
deprecated.

2. Finally create Ignite.sql() API, which was already discussed here:
http://apache-ignite-developers.2346864.n4.nabble.com/Rethink-native-SQL-API-in-Apache-Ignite-2-0-td14335.html
and place "queryDetailMetrics" metric there. Here is the ticket for this
change: https://issues.apache.org/jira/browse/IGNITE-4701

Personally, I think that the second solution looks better in this case,
however, moving dataRegionMetrics and dataStorageMetrics to
ignite.matrics() is still a good idea - IMO, Ignite class is not the right
place for them - we shouldn't change our main API class so often.

What do you think?

Thank you,
Evgenii

Re: This morning I added ssl support to...

[Nikolay Izhikov]

Let's do it.

Please, send a ticket number to this thread.

В Чт, 16/08/2018 в 15:22 +0200, Paul Anderson пишет:
> We needed here so I just did it... Very simple will raise a ticket and add
> patch to it...
> 
> On Thu, 16 Aug 2018, 15:21 Nikolay Izhikov,  wrote:
> 
> > Hello, Paul.
> > 
> > Is there a ticket you trying to resolve?
> > Yes, please, provide the patch, so community can take a look.
> > 
> > В Чт, 16/08/2018 в 15:03 +0200, Paul Anderson пишет:
> > > org.apache.ignite.internal.commandline.CommandHandler (control.sh)
> > > 
> > > Can I send you a diff rather than becoming a contributer for the moment.

signature.asc
Description: This is a digitally signed message part

Re: This morning I added ssl support to...

[Paul Anderson]

We needed here so I just did it... Very simple will raise a ticket and add
patch to it...

On Thu, 16 Aug 2018, 15:21 Nikolay Izhikov,  wrote:

> Hello, Paul.
>
> Is there a ticket you trying to resolve?
> Yes, please, provide the patch, so community can take a look.
>
> В Чт, 16/08/2018 в 15:03 +0200, Paul Anderson пишет:
> > org.apache.ignite.internal.commandline.CommandHandler (control.sh)
> >
> > Can I send you a diff rather than becoming a contributer for the moment.

Re: This morning I added ssl support to...

[Nikolay Izhikov]

Hello, Paul.

Is there a ticket you trying to resolve?
Yes, please, provide the patch, so community can take a look.

В Чт, 16/08/2018 в 15:03 +0200, Paul Anderson пишет:
> org.apache.ignite.internal.commandline.CommandHandler (control.sh)
> 
> Can I send you a diff rather than becoming a contributer for the moment.

signature.asc
Description: This is a digitally signed message part

This morning I added ssl support to...

[Paul Anderson]

org.apache.ignite.internal.commandline.CommandHandler (control.sh)

Can I send you a diff rather than becoming a contributer for the moment.

Re: StandaloneWalRecordsIterator: support iteration from custom pointer

[Ivan Rakov]

Thanks for your comments!
I've created a ticket: https://issues.apache.org/jira/browse/IGNITE-9294

Best Regards,
Ivan Rakov

On 15.08.2018 21:31, Dmitriy Setrakyan wrote:

Agree, this should be a great performance boost.

On Wed, Aug 15, 2018 at 10:17 AM, Dmitriy Pavlov 
wrote:


Hi Ivan,

I agree that providing WAL pointer is the better option. Initially,
Standalone WAL iterator was developed for debugging utility, so a set of
files was perfectly OK.

Sincerely,
Dmitriy Palov

ср, 15 авг. 2018 г. в 20:06, Ivan Rakov :


Igniters,

Right now we are developing WAL shipping process for our internal
purposes and we use StandaloneWalRecordsIterator to read WAL files from
custom destination. We have bumped into a problem - iterator can be
constructed from set of files and dirs, but there's no option to pass
WAL pointer to the iterator factory class to start iteration with. It
can be worked around (by filtering all records prior to needed pointer),
but I think it would be handy to add such option to
IgniteWalIteratorFactory API.

What do you think?

--
Best Regards,
Ivan Rakov

[jira] [Created] (IGNITE-9294) StandaloneWalRecordsIterator: support iteration from custom pointer

[Ivan Rakov (JIRA)]

Ivan Rakov created IGNITE-9294:
--

 Summary: StandaloneWalRecordsIterator: support iteration from 
custom pointer
 Key: IGNITE-9294
 URL: https://issues.apache.org/jira/browse/IGNITE-9294
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Reporter: Ivan Rakov


StandaloneWalRecordsIterator can be constructed from set of files and dirs, but 
there's no option to pass WAL pointer to the iterator factory class to start 
iteration with. It can be worked around (by filtering all records prior to 
needed pointer), but also it would be handy to add such option to 
IgniteWalIteratorFactory API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [MTCGA]: new failures in builds [1667472] needs to be handled

[Dmitriy Pavlov]

Hi Folks,

It seems it is a flaky failure, which unluckily failed 3 times in a row.

Sincerely,
Dmitriy Pavlov

чт, 16 авг. 2018 г. в 13:34, :

> Hi Ignite Developer,
>
> I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> I hope you can help.
>
>  *New test failure in master
> TcpCommunicationSpiSkipMessageSendTest.testClientSegmented
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3225493048753223945=%3Cdefault%3E=testDetails
>  No changes in build
>
> - If your changes can led to this failure(s), please create issue
> with label MakeTeamCityGreenAgain and assign it to you.
> -- If you have fix, please set ticket to PA state and write to dev
> list fix is ready
> -- For case fix will require some time please mute test and set
> label Muted_Test to issue
> - If you know which change caused failure please contact change
> author directly
> - If you don't know which change caused failure please send
> message to dev list to find out
> Should you have any questions please contact dpav...@apache.org or write
> to dev.list
> Best Regards,
> MTCGA.Bot
> Notification generated at Thu Aug 16 13:34:41 MSK 2018
>

Re: how ignite c++ node set baselinetopology

[Igor Sapego]

Filed a ticket: [1]

[1] - https://issues.apache.org/jira/browse/IGNITE-9293

Best Regards,
Igor


On Thu, Aug 16, 2018 at 3:28 AM Denis Magda  wrote:

> Igor,
>
> I think it's reasonable to add the baseline topology APIs to C++. Can you
> plan this for the nearest releases?
>
> Pave, do you have time to do the same for .Net?
>
> --
> Denis
>
> -- Forwarded message -
> From: aealexsandrov 
> Date: Tue, Aug 14, 2018 at 3:23 AM
> Subject: Re: how ignite c++ node set baselinetopology
> To: 
>
>
> Hi,
>
> C++ API doesn't contain the methods to update the baseline. So, in this
> case, you can use Java API for it:
>
> Add the code that will listen to the EVT_NODE_JOINED event
>
> private final IgniteEx ignite;
>
> ignite.events().localListen(event -> {
> DiscoveryEvent e = (DiscoveryEvent)event;
>
> final long topVer = e.topologyVersion();
>
> ignite.cluster().setBaselineTopology(topVer);
>
> return true;
> }, EventType.EVT_NODE_JOINED);
>
> The whole example you can see here:
>
>
> https://apacheignite.readme.io/docs/baseline-topology#section-triggering-rebalancing-programmatically
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

[jira] [Created] (IGNITE-9293) CPP: Add API to set baseline topology for C++

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-9293:
---

 Summary: CPP: Add API to set baseline topology for C++ 
 Key: IGNITE-9293
 URL: https://issues.apache.org/jira/browse/IGNITE-9293
 Project: Ignite
  Issue Type: New Feature
  Components: platforms
Affects Versions: 2.6
Reporter: Igor Sapego
 Fix For: 2.7


We need to add API for C++ to set baseline topology.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [Distributed SQL] Do we have a plan to implement QuadTree index?

[Alexey Zinoviev]

Vladimir, yes I remember this discussion with Yuri. Of course I couldn't
estimate the changes to add new index type as you and thank you for your
consultation.

Maybe it's best approach to continue with my draft where it's implemented
via ML specific data structures over the cache.

2018-08-16 14:16 GMT+06:00 Vladimir Ozerov :

> Hi Alex,
>
> From what I see there is some interest to K-D indexes in SQL world.
> Postgres has it. MySQL users asks about it from time to time, and usually
> use some simpler spatial solutions ask MySQL lacks this index type. We
> definitely may consider integrating it with SQL, but this would be big
> enough feature, requiring changes in almost all SQL components.
>
> For this reason I would put SQL case aside for now and focus on
> implementation for ML. I had a talk with Yuriy Babak some time ago and
> explained how new index type can be added to the product. May be Yuriy has
> some thoughts on how to do that properly with respect to ML roadmap and
> plans.
>
> On Thu, Aug 16, 2018 at 6:21 AM Alexey Zinoviev 
> wrote:
>
> > Sorry, for the delay, dear Pavel and Denis.
> >
> > Yes, I need a kind of indexing to improve KNN algorithms during training
> > the model.
> >
> > In my draft solution I've implemented building of
> > https://en.wikipedia.org/wiki/K-d_tree
> >  on the training data set.
> > It collects the information about data distributed in our specific ML
> > Datasets (distributed by nodes over Ignite cache)
> >
> > Pavel, you ask me any questions and I've prepared answers.
> >
> > 1) Should be this index in-memory only or you want to persist it?
> > *Maybe it should be persisted (to reuse that for next predictions)*
> >
> > 2) In general index can be implemented in two ways per-partition and
> > per-node.
> > *Thank you for your explanation. Of course the per-node is better.*
> >
> > 3) I think K-d tree is preferable
> > *You are absolutely right, it should be K-d tree*
> >
> > 4) Could you please share use cases you're trying to solve with QuadTree?
> > With
> > close to real data and examples of queries?
> >
> > I need to solve *k-nearest neighbors search task *on a set of vectors
> with
> > unique keys presented in Ignite Cache (training set),
> > during the training phase I'm going to build a temporary index as a
> KD-Tree
> > (based on distance between vectors).
> >
> > The distance metric is a parameter too.
> >
> > After that, in prediction phase the *k-nearest neighbors search task *is
> > solved by brute-force iteration over all vectors to find the *k-nearest
> > neighbors.*
> > I'd like to improve the search part with queries to index to extract
> > closest vectors.
> >
> > Of course, it's kind of experiment, and maybe it's bad idea to patch SQL
> > internals to solve this private task, but as you mentioned it can be a
> good
> > point of collaboration between ML and SQL components.
> >
> > Can I get one of the implemented indices as a primary example and
> implement
> > it by myself?
> > Could you recommend something as an start point for me?
> >
> > Thanks for your help.
> >
> >
> >
> >
> > 2018-08-04 11:18 GMT+06:00 Denis Magda :
> >
> > > Alexey, are you working on some new ML/DL APIs/algorithms? Please
> > elaborate
> > > what you'd like to add to Ignite.
> > >
> > > --
> > > Denis
> > >
> > > On Wed, Aug 1, 2018 at 3:10 PM Pavel Kovalenko 
> > wrote:
> > >
> > > > Hello Alexey,
> > > >
> > > > It's not so difficult to implement new type of indexing of data, but
> if
> > > you
> > > > want to reach performance in distributed environment you need to have
> > > > strong knowledge of a data you're indexing and what kind of queries
> you
> > > > want to execute.
> > > > Should be this index in-memory only or you want to persist it?
> > > > In case of persistence your index should fit our page memory model
> > > > requirements.
> > > > In both cases your index should be ready to work in concurrent
> > > environment.
> > > >
> > > > In general index can be implemented in two ways per-partition and
> > > per-node.
> > > > Per-partition may be efficient if you have a lot of points (x,y)
> > > > representing a big one, e.g. image. In this case it's required that
> all
> > > of
> > > > these points will be in one partition that query e.g. makes images
> > > > intersection will execute in one node. But if you have multiple
> images,
> > > > your index will contain also another points from other object and
> will
> > > > overload it.
> > > > Per-node may be efficient if you have a lot of points (x,y) that are
> > > > independent of each other, that you will use it as spatial e.g.. But
> in
> > > > this case, I think K-d tree is preferable as it can be used in more
> > wide
> > > > way.
> > > >
> > > > Could you please share use cases you're trying to solve with
> QuadTree?
> > > With
> > > > close to real data and examples of queries? Because now the question
> is
> > > too
> > > > abstract and it's hard to understand how it 

Re: [Distributed SQL] Do we have a plan to implement QuadTree index?

[Alexey Zinoviev]

Ok, many thanks, will look at geospatial index

2018-08-16 14:10 GMT+06:00 Alexey Goncharuk :

> Alexey,
>
> I am not sure if it will be a proper fir for you, but I think it worth a
> try.
>
> Apache Ignite has an option to index geospatial data using third-party
> libraries (note that it is not included in the default Ignite build, the
> module is activated via the lgpl profile). The index is located in
> Ignite-geospatial module and uses an r-tree implementation underneath. One
> downside of this module is that the geospatial index is not supported for
> the Ignite native persistence yet.
>
> Hope this helps!
> --AG
>
> чт, 16 авг. 2018 г. в 6:21, Alexey Zinoviev :
>
> > Sorry, for the delay, dear Pavel and Denis.
> >
> > Yes, I need a kind of indexing to improve KNN algorithms during training
> > the model.
> >
> > In my draft solution I've implemented building of
> > https://en.wikipedia.org/wiki/K-d_tree
> >  on the training data set.
> > It collects the information about data distributed in our specific ML
> > Datasets (distributed by nodes over Ignite cache)
> >
> > Pavel, you ask me any questions and I've prepared answers.
> >
> > 1) Should be this index in-memory only or you want to persist it?
> > *Maybe it should be persisted (to reuse that for next predictions)*
> >
> > 2) In general index can be implemented in two ways per-partition and
> > per-node.
> > *Thank you for your explanation. Of course the per-node is better.*
> >
> > 3) I think K-d tree is preferable
> > *You are absolutely right, it should be K-d tree*
> >
> > 4) Could you please share use cases you're trying to solve with QuadTree?
> > With
> > close to real data and examples of queries?
> >
> > I need to solve *k-nearest neighbors search task *on a set of vectors
> with
> > unique keys presented in Ignite Cache (training set),
> > during the training phase I'm going to build a temporary index as a
> KD-Tree
> > (based on distance between vectors).
> >
> > The distance metric is a parameter too.
> >
> > After that, in prediction phase the *k-nearest neighbors search task *is
> > solved by brute-force iteration over all vectors to find the *k-nearest
> > neighbors.*
> > I'd like to improve the search part with queries to index to extract
> > closest vectors.
> >
> > Of course, it's kind of experiment, and maybe it's bad idea to patch SQL
> > internals to solve this private task, but as you mentioned it can be a
> good
> > point of collaboration between ML and SQL components.
> >
> > Can I get one of the implemented indices as a primary example and
> implement
> > it by myself?
> > Could you recommend something as an start point for me?
> >
> > Thanks for your help.
> >
> >
> >
> >
> > 2018-08-04 11:18 GMT+06:00 Denis Magda :
> >
> > > Alexey, are you working on some new ML/DL APIs/algorithms? Please
> > elaborate
> > > what you'd like to add to Ignite.
> > >
> > > --
> > > Denis
> > >
> > > On Wed, Aug 1, 2018 at 3:10 PM Pavel Kovalenko 
> > wrote:
> > >
> > > > Hello Alexey,
> > > >
> > > > It's not so difficult to implement new type of indexing of data, but
> if
> > > you
> > > > want to reach performance in distributed environment you need to have
> > > > strong knowledge of a data you're indexing and what kind of queries
> you
> > > > want to execute.
> > > > Should be this index in-memory only or you want to persist it?
> > > > In case of persistence your index should fit our page memory model
> > > > requirements.
> > > > In both cases your index should be ready to work in concurrent
> > > environment.
> > > >
> > > > In general index can be implemented in two ways per-partition and
> > > per-node.
> > > > Per-partition may be efficient if you have a lot of points (x,y)
> > > > representing a big one, e.g. image. In this case it's required that
> all
> > > of
> > > > these points will be in one partition that query e.g. makes images
> > > > intersection will execute in one node. But if you have multiple
> images,
> > > > your index will contain also another points from other object and
> will
> > > > overload it.
> > > > Per-node may be efficient if you have a lot of points (x,y) that are
> > > > independent of each other, that you will use it as spatial e.g.. But
> in
> > > > this case, I think K-d tree is preferable as it can be used in more
> > wide
> > > > way.
> > > >
> > > > Could you please share use cases you're trying to solve with
> QuadTree?
> > > With
> > > > close to real data and examples of queries? Because now the question
> is
> > > too
> > > > abstract and it's hard to understand how it should be implemented to
> > > reach
> > > > good results.
> > > >
> > > >
> > > > 2018-08-01 16:45 GMT+03:00 Alexey Zinoviev :
> > > >
> > > > > Hi, Igniters.
> > > > >
> > > > > Currently I'm working on different math stuff over the Apache
> Ignite
> > > and
> > > > in
> > > > > a few tasks I need to implement in memory something like this
> > > > >
> > > > > 

[MTCGA]: new failures in builds [1667472] needs to be handled

[dpavlov . tasks]

Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New test failure in master 
TcpCommunicationSpiSkipMessageSendTest.testClientSegmented 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3225493048753223945=%3Cdefault%3E=testDetails
 No changes in build

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dpav...@apache.org or write to 
dev.list 
Best Regards,
MTCGA.Bot 
Notification generated at Thu Aug 16 13:34:41 MSK 2018 

Re: IEP-22: Direct Data Load proposal

[Vladimir Ozerov]

Dima,

By "out of question" I meant that 3rd party persistence should work out of
the box when IEP-22 is ready. No changes should be required there.

As far as persistence vs memory, most probably yes, there might be some
differences. Specifically, when data load starts and persistence is
enabled, we will bypass free lists and write data to new blocks. This way,
overall data will need more pages than when loaded in normal mode. This is
a kind of trade-off you face when loading speed is important (at the very
least Oracle works this way, most probably other vendors does the same).
But this approach may be not applicable for in-memory mode, where total
number of pages is limited, and we do not want to hit page eviction.

To summarize - some optimizations which are applicable for persistent mode
will not be applicable for in-memory.

Vladimir.

On Thu, Aug 16, 2018 at 11:41 AM Dmitriy Setrakyan 
wrote:

> On Thu, Aug 16, 2018 at 1:24 AM, Vladimir Ozerov 
> wrote:
>
> > Hi Denis,
> >
> > This IEP is mostly about how we work with our own indexes and pages. So
> 3rd
> > party DB is out of question.
> >
>
> Why? I think 3rd party DB will be supported automatically with CacheStore.
> However, do we need to do something different for memory-only vs.
> memory+disk?
>
> D.
>

[GitHub] ignite pull request #4559: Ignite 9280

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4559

Ignite 9280



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-9280

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4559.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4559


commit b74b4a645a13fea214937652bc4523bb0340995e
Author: Andrey Kuznetsov 
Date:   2018-08-15T12:15:33Z

IGNITE-9280 Compressing WAL archive segments ASAP.

commit 099d67410ee19e68b2dc3c335d3497b1f37c4696
Author: Andrey Kuznetsov 
Date:   2018-08-15T15:52:02Z

IGNITE-9280 Added WAL segment compaction completion event.

commit 72bbecd98c19b1cda2c605b19f2b1ba2e7867d18
Author: Andrey Kuznetsov 
Date:   2018-08-15T17:42:44Z

IGNITE-9280 Exposed last WAL segment compaction time to public API.

commit ea6a2e1249695cd3d838eb7d01367d21feaed821
Author: Andrey Kuznetsov 
Date:   2018-08-16T10:28:34Z

Merge branch 'master' into ignite-9280




---

[GitHub] ignite pull request #4558: Ignite 9053

[anton-vinogradov]

GitHub user anton-vinogradov opened a pull request:

https://github.com/apache/ignite/pull/4558

Ignite 9053



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/ignite ignite-9053

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4558.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4558


commit 2f03e6147e9ed61119fc98a499959b5956cded96
Author: Anton Vinogradov 
Date:   2018-08-13T15:48:40Z

IGNITE-9053 testReentrantLockConstantTopologyChangeNonFailoverSafe can hang 
in case of broken tx

Signed-off-by: Anton Vinogradov 

commit d32ee24be3a3d81a50d0ffa93b5b7e026fe9a956
Author: Anton Vinogradov 
Date:   2018-08-14T11:43:04Z

IGNITE-9053 testReentrantLockConstantTopologyChangeNonFailoverSafe can hang 
in case of broken tx

Signed-off-by: Anton Vinogradov 

commit 1c09137e9be1167a8c7fdf2f80c5674f5d6101d4
Author: Anton Vinogradov 
Date:   2018-08-14T13:56:43Z

IGNITE-9053 testReentrantLockConstantTopologyChangeNonFailoverSafe can hang 
in case of broken tx

Signed-off-by: Anton Vinogradov 

commit 3540e1c440fa2361fc63de88de4a60be3ec17781
Author: Anton Vinogradov 
Date:   2018-08-15T13:39:33Z

IGNITE-9053 testReentrantLockConstantTopologyChangeNonFailoverSafe can hang 
in case of broken tx

Signed-off-by: Anton Vinogradov 

commit da3b7a0ab52b2cb451a8cb2adbabbef7c0d46086
Author: Anton Vinogradov 
Date:   2018-08-15T22:03:25Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-9053




---

Re: Page replacement policy improvements (when persistent is enabled)

[Dmitriy Pavlov]

Hi Igniters,

I feel monitoring will provide us with the necessary knowledge about how to
change.

I have no idea how region separation can help with performance. But it
could be an additional challenge for users to correctly configure separate
regions.

Sincerely,
Dmitriy Pavlov

чт, 16 авг. 2018 г. в 12:45, Dmitriy Setrakyan :

> On Thu, Aug 16, 2018 at 2:27 AM, Vladimir Ozerov 
> wrote:
>
> > Dima,
> >
> > None database I know use separate regions for index pages due to the
> reason
> > I expressed above. Instead, they split all pages into two groups - hot
> and
> > cold. With certain rules on how to move pages inside and between these
> > groups. None of these algorithms are complex enough. In fact, they are
> > pretty straightforward and battle-tested. When implemented properly it
> > doesn't matter whether the page is index page or data page. The only
> thing
> > that matter is how often it is accessed. This is critical piece that we
> > lack in the product - our policy is called "random *LRU*", while in
> reality
> > is not LRU at all.
> >
>
> Aerospike keeps index pages in memory, so there is at least one database
> that does that. I am sure if we research around, there will be more.
>
>
> > As far as index pages replacement we do not know whether this is problem
> at
> > all. We heard some complaints that it might be a problem. But we didn't
> see
> > any proofs (thanks to lack of monitoring) and even if this is a problem,
> we
> > do not understand how severe it is. May be it adds 1% overhead and can be
> > ignored for years, may be it adds 1000% overhead and must be fixed
> > immediately.
> >
>
> I remember one test case with a potential user where we had to change our
> eviction algorithm to avoid evicting index pages and because of that
> improved performance by about 10x.
>
>
> > This is sensitive piece of a product. Let's use objective data, not
> > assumptions.
> >
>
> I agree. The difference is that we need a solution in the mean time. I am
> suggesting a very straight forward approach that can be added fairly
> quickly and will solve majority performance problems associated with index
> page eviction. Once we have that, we can take our time and investigate
> further.
>

[jira] [Created] (IGNITE-9292) MVCC SQL: Race during updating TX log on near backup

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-9292:
--

 Summary: MVCC SQL: Race during updating TX log on near backup
 Key: IGNITE-9292
 URL: https://issues.apache.org/jira/browse/IGNITE-9292
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Pavlukhin
Assignee: Ivan Pavlukhin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4557: IGNITE-9278 Fix TensorFlow integration: Can't fin...

[dmitrievanthony]

GitHub user dmitrievanthony opened a pull request:

https://github.com/apache/ignite/pull/4557

IGNITE-9278 Fix TensorFlow integration: Can't find free ports in range



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9278

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4557.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4557


commit 51e9c026e447e120c1ff6f8fc76a2f3031036eee
Author: Anton Dmitriev 
Date:   2018-08-16T09:36:58Z

IGNITE-9278 Add debug output and add save after update in cluster port
manager.

commit c710424996685dfd1fe1c1c792e1c358a6c21df5
Author: Anton Dmitriev 
Date:   2018-08-16T10:04:16Z

IGNITE-9278 Remove debug output.




---

Re: Page replacement policy improvements (when persistent is enabled)

[Dmitriy Setrakyan]

On Thu, Aug 16, 2018 at 2:27 AM, Vladimir Ozerov 
wrote:

> Dima,
>
> None database I know use separate regions for index pages due to the reason
> I expressed above. Instead, they split all pages into two groups - hot and
> cold. With certain rules on how to move pages inside and between these
> groups. None of these algorithms are complex enough. In fact, they are
> pretty straightforward and battle-tested. When implemented properly it
> doesn't matter whether the page is index page or data page. The only thing
> that matter is how often it is accessed. This is critical piece that we
> lack in the product - our policy is called "random *LRU*", while in reality
> is not LRU at all.
>

Aerospike keeps index pages in memory, so there is at least one database
that does that. I am sure if we research around, there will be more.


> As far as index pages replacement we do not know whether this is problem at
> all. We heard some complaints that it might be a problem. But we didn't see
> any proofs (thanks to lack of monitoring) and even if this is a problem, we
> do not understand how severe it is. May be it adds 1% overhead and can be
> ignored for years, may be it adds 1000% overhead and must be fixed
> immediately.
>

I remember one test case with a potential user where we had to change our
eviction algorithm to avoid evicting index pages and because of that
improved performance by about 10x.


> This is sensitive piece of a product. Let's use objective data, not
> assumptions.
>

I agree. The difference is that we need a solution in the mean time. I am
suggesting a very straight forward approach that can be added fairly
quickly and will solve majority performance problems associated with index
page eviction. Once we have that, we can take our time and investigate
further.

Re: Page replacement policy improvements (when persistent is enabled)

[Vladimir Ozerov]

Dima,

None database I know use separate regions for index pages due to the reason
I expressed above. Instead, they split all pages into two groups - hot and
cold. With certain rules on how to move pages inside and between these
groups. None of these algorithms are complex enough. In fact, they are
pretty straightforward and battle-tested. When implemented properly it
doesn't matter whether the page is index page or data page. The only thing
that matter is how often it is accessed. This is critical piece that we
lack in the product - our policy is called "random *LRU*", while in reality
is not LRU at all.

As far as index pages replacement we do not know whether this is problem at
all. We heard some complaints that it might be a problem. But we didn't see
any proofs (thanks to lack of monitoring) and even if this is a problem, we
do not understand how severe it is. May be it adds 1% overhead and can be
ignored for years, may be it adds 1000% overhead and must be fixed
immediately.

This is sensitive piece of a product. Let's use objective data, not
assumptions.

On Thu, Aug 16, 2018 at 12:08 PM Dmitriy Setrakyan 
wrote:

> On Thu, Aug 16, 2018 at 2:01 AM, Vladimir Ozerov 
> wrote:
>
> > Hi Dima,
> >
> > Putting index pages in separate region is wrong approach, because data
> > pages may be equally important on certain workloads, especially in
> > heap-organized databases, such as Ignite
>
>
> Never seen a use case where the data page was more important than the index
> page. I think we are getting into a hypothetical land. Most Ignite users
> are having the reverse problem - index pages get evicted in the same way as
> data pages.
>
> Currently, we are solving it in a most awkward way by trying to give index
> pages a different eviction policy. A right solution would be to stick them
> into a different region and control the eviction policy for the index
> region separately from the data region.
>
> D.
>

Re: Ticket review checklist

[Павлухин Иван]

Vladimir,

First of all, statements in Java 8 section [1] looks kind of prohibitive
for me. When a new contributor see words "preferred" and "avoided in most
cases" he most likely will not use such features (like I did). If a
statement is not prohibitive in practice it could be at least rephrased.

A bit about expressiveness. I written a code during working on a real
ticket. The case is quite common in Ignite codebase. You can find example
with couple of approaches in snippet [2]. For me approach with lambdas is
expressive, compact and simple.

What do you think?

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Java8
[2] https://gist.github.com/pavlukhin/92701277f66f8901a7feda6283a5a299

2018-08-16 11:21 GMT+03:00 Vladimir Ozerov :

> Hi Ivan,
>
> From what I see we do not restrict contributors to use lambdas and streams.
> Document states that plain collections and anonymous classes are
> *preferred*. This is not obligatory requirement, and it seems reasonable to
> me, because when developing complex projects at times it is better to have
> more expressive code, than less non-obvious code which makes dozens
> operations in a single string.
>
> Or may be there are any other statements in the checklist which prevents
> users from using Java 8 features?
>
> Vladimir.
>
> On Tue, Aug 14, 2018 at 7:16 PM ipavlukhin  wrote:
>
> > Hi Igniters,
> >
> > I would like to refresh review checklist a little bit. Currently it [1]
> > contains section against lambda Lambda expressions and Stream API. As
> > far as I know it is not true anymore. Currently both features have
> > theirs usage in core module. What is a state of affairs for a subject?
> > Are there some well-known cases where e.g. lambdas are not applicable?
> > Should we document it?
> >
> > I personally think that we could delete entire Java 8 section from
> > checklist (and Java 5 as well). I understand that every tool should be
> > used judiciously but I doubt that all cases can be covered in short
> > checklist.
> >
> > [1]
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#
> CodingGuidelines-Java8
> >
> >
> > On 2018/07/09 20:53:42, Dmitry Pavlov  wrote:
> >  > I also tend to agree about updating checklist>
> >  >
> >  > About suite timeouts, I suspect there is one problem introduced
> > recently>
> >  > within 3 days, which caused this mass timeouts.>
> >  >
> >  > I hope Igniters will find out reason soon. In relation to compute we
> > have>
> >  > only 2 possible cause:>
> >  > Ivan Daschinskiy (idaschinskiy) 2 files IGNITE-8869 Fixed>
> >  > PartitionsExchangeOnDiscoveryHistoryOverflowTest hanging>
> >  > Signed-off-by: Andrey Gura  ···>
> >  >
> >  > Dmitriy Govorukhin (dgovorukhin) 12 files IGNITE-8827 Disable WAL
> > during>
> >  > apply updates on recovery>
> >  >
> >  > I guess if we fix this reason we will fix 10 suites more>
> >  > References:>
> >  >
> >
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests24Java8_ComputeGrid=buildTypeHistoryList_
> IgniteTests24Java8=%3Cdefault%3E>
> >
> >
> >  >
> >  >
> >  > пн, 9 июл. 2018 г. в 22:29, Anton Vinogradov :>
> >  >
> >  > > Sounds reasonable.>
> >  > > I've satrted Data Structures suite hang investigation [1].>
> >  > >>
> >  > > Igniters, especially commiters,>
> >  > > I know, you're busy, but it will be a great help to the project in
> > case you>
> >  > > fix at least one hang per person.>
> >  > >>
> >  > > [1] https://issues.apache.org/jira/browse/IGNITE-8783>
> >  > >>
> >  > > пн, 9 июл. 2018 г. в 19:24, Maxim Muzafarov :>
> >  > >>
> >  > > > Hi Igniters,>
> >  > > >>
> >  > > > Let's back to discussion of review checklist. Can we add more>
> >  > > clarification>
> >  > > > about running all suites on TeamCity?>
> >  > > >>
> >  > > > My suggestion is: “All test suites MUST be run on TeamCity [3]
> > before>
> >  > > merge>
> >  > > > to master, there MUST NOT be any test failures * and any
> > tests\suites>
> >  > > with>
> >  > > > “execution timeouts” *. Not important test failures should be
> > muted and>
> >  > > > handled according to [4] process.”>
> >  > > >>
> >  > > > As you can see we have stable “Execution timeouts” for>
> >  > > > “Activate\Deactiveate Cluster” test suite since 16-th June. How
> > can we be>
> >  > > > sure in this case that new changes would not break up old
> > functionality?>
> >  > > >>
> >  > > > From my point, all new changes MUST NOT be merged to master util
> > we will>
> >  > > > fix all execution timeouts for suites. Even if current changes
> > are not>
> >  > > > related to these timeouts.>
> >  > > >>
> >  > > > [1]>
> >  > > >>
> >  > > >>
> >  > >
> >
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=
> IgniteTests24Java8_ActivateDeactivateCluster=
> buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E>
> >
> >
> >  > > >>
> >  > > >>
> >  > > > пн, 4 июн. 2018 г. в 15:56, Dmitry Pavlov :>
> >  > > >>
> >  > > > > Requirement of green TC 

[jira] [Created] (IGNITE-9291) IgniteJdbcThinDriver with SSL doesn't work for sqlline on Windows

[Sergey Kozlov (JIRA)]

Sergey Kozlov created IGNITE-9291:
-

 Summary: IgniteJdbcThinDriver with SSL doesn't work for sqlline on 
Windows
 Key: IGNITE-9291
 URL: https://issues.apache.org/jira/browse/IGNITE-9291
 Project: Ignite
  Issue Type: Bug
  Components: jdbc
Affects Versions: 2.6
Reporter: Sergey Kozlov
 Fix For: 2.7


I tried to run AI 2.6 with same configuration under Linux (Ubuntu) and Windows 
10 and then connect {sqlline} with activated SSL. 
It works fine for Linux and doesn't work for Windows.
There's no errors on server nodes but sqlline prints out following:
{noformat}

['C:/BuildAgent/work/dd4d79acf76cc870/i2test/var/suite-client/gg-pro-fab-nolgpl/bin/sqlline.bat',
 '-d', 'org.apache.ignite.IgniteJdbcThinDriver', '--verbose=true', 
'--force=true', '--showWarnings=true', '--showNestedErrs=true', '-u', 
'jdbc:ignite:thin://127.0.0.1/?sslMode=require=C:/BuildAgent/work/dd4d79acf76cc870/i2test/var/suite-client/test_modules/test_sqlline/jdbc-thin-data-region-ssl/client.jks=123456=C:/BuildAgent/work/dd4d79acf76cc870/i2test/var/suite-client/test_modules/test_sqlline/jdbc-thin-data-region-ssl/trust.jks=123456',
 '-f', 
'C:/BuildAgent/work/dd4d79acf76cc870/i2test/var/suite-client/test_modules/test_sqlline/jdbc-thin-data-region-ssl/sqlline_output.1.batch.txt']
C:/BuildAgent/work/dd4d79acf76cc870/i2test/var/suite-client/gg-pro-fab-nolgpl/bin/sqlline.bat,
 WARN: IGNITE_HOME environment variable may be pointing to wrong folder: 
C:/BuildAgent/work/dd4d79acf76cc870/i2test/var/suite-client/gg-pro-fab-nolgpl
Setting property: [force, true]
Setting property: [showWarnings, true]
Setting property: [showNestedErrs, true]
issuing: !connect jdbc:ignite:thin://127.0.0.1/?sslMode=require '' '' 
org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://127.0.0.1/?sslMode=require
Error: Failed to SSL connect to server [url=jdbc:ignite:thin://127.0.0.1:10800] 
(state=08001,code=0)
java.sql.SQLException: Failed to SSL connect to server 
[url=jdbc:ignite:thin://127.0.0.1:10800]
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinSSLUtil.createSSLSocket(JdbcThinSSLUtil.java:93)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.connect(JdbcThinTcpIo.java:217)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.start(JdbcThinTcpIo.java:159)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.start(JdbcThinTcpIo.java:134)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.ensureConnected(JdbcThinConnection.java:151)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.(JdbcThinConnection.java:140)
at 
org.apache.ignite.IgniteJdbcThinDriver.connect(IgniteJdbcThinDriver.java:157)
at sqlline.DatabaseConnection.connect(DatabaseConnection.java:156)
at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:204)
at sqlline.Commands.connect(Commands.java:1095)
at sqlline.Commands.connect(Commands.java:1001)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
at sqlline.SqlLine.dispatch(SqlLine.java:791)
at sqlline.SqlLine.initArgs(SqlLine.java:566)
at sqlline.SqlLine.begin(SqlLine.java:643)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
Caused by: javax.net.ssl.SSLHandshakeException: 
sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at 
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
at 
sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at 

Re: Page replacement policy improvements (when persistent is enabled)

[Dmitriy Setrakyan]

On Thu, Aug 16, 2018 at 2:01 AM, Vladimir Ozerov 
wrote:

> Hi Dima,
>
> Putting index pages in separate region is wrong approach, because data
> pages may be equally important on certain workloads, especially in
> heap-organized databases, such as Ignite


Never seen a use case where the data page was more important than the index
page. I think we are getting into a hypothetical land. Most Ignite users
are having the reverse problem - index pages get evicted in the same way as
data pages.

Currently, we are solving it in a most awkward way by trying to give index
pages a different eviction policy. A right solution would be to stick them
into a different region and control the eviction policy for the index
region separately from the data region.

D.

Re: Page replacement policy improvements (when persistent is enabled)

[Vladimir Ozerov]

Hi Dima,

Putting index pages in separate region is wrong approach, because data
pages may be equally important on certain workloads, especially in
heap-organized databases, such as Ignite.
At the moment we'd better focus on monitoring.to better understand usages
patterns. This would give us solid ground for further decisions.

Vladimir.

On Sat, Aug 4, 2018 at 12:06 AM Dmitriy Setrakyan 
wrote:

> Vladimir,
>
> Are we only counting timestamp of the last access? In that case, it would
> create a problem. We should also count number of times a page has been
> touched within a certain time frame, e.g. last hour or so. In this case,
> index pages would not be evicted as they get touched the most.
>
> I would also consider putting index pages into a separate memory region.
> This way you can apply a different eviction policy to the index pages or
> decide not to evict them altogether. This will also be a much simpler and
> less error-prone approach than introducing new eviction policies.
>
> D.
>
> On Fri, Aug 3, 2018 at 12:19 AM, Vladimir Ozerov 
> wrote:
>
> > Igniters,
> >
> > I heard some complaints about our page replacement algorithm that index
> > pages could be evicted from memory too often. I reviewed our current
> > implementation and looks like we have choosen very simple approach with
> > eviction of random pages, without taking in count their nature (data vs
> > index) and typical usage patterns (such as scans).
> >
> > With our heap-based architecture typical SQL query is executed as
> follows:
> > 1) Read non-leaf index pages, then in loop:
> > 2.1) Read 1 leaf index page
> > 2.2) Read several hunderds data pages
> >
> > This way index pages on average has smaller timestamp than data pages and
> > has good probabilty of being evicted.
> >
> > Another major problem is scan resistance, which doesn't seem to be
> covered
> > anyhow.
> >
> > My question is - what was the reason of choosing random-pseudo-LRU
> > algorithm instead of commonly used variation of *real* LRU (such as
> LRU-K,
> > 2Q, etc)? Did we perform any evaluation of it's effectiveness?
> >
> > I am thinking of creating new IEP to evaluate and possibly improve our
> page
> > replacement as follows:
> > 1) Implement metrics to count page cache hit/miss by page type [1]
> > 2) Implement *heat map* which can optionally be enabled to track page
> > hits/misses per page or per specific object (cache, index)
> > 3) Run heat map on typical workloads (lookups, scans, joins, etc) to get
> a
> > baseline
> > 4) Prototype several LRU-based implementation and see if they gave any
> > benefit. It makes sense to start with minor improvements to current
> > algorithm (e.g. favor index pages over data pages, play with sample size,
> > replace timestamps with read counters, etc).
> >
> > In any case the first two action items would be good addition to product
> > monitoring.
> >
> > What do you think?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-8580
> >
>

Re: IEP-22: Direct Data Load proposal

[Dmitriy Setrakyan]

On Thu, Aug 16, 2018 at 1:24 AM, Vladimir Ozerov 
wrote:

> Hi Denis,
>
> This IEP is mostly about how we work with our own indexes and pages. So 3rd
> party DB is out of question.
>

Why? I think 3rd party DB will be supported automatically with CacheStore.
However, do we need to do something different for memory-only vs.
memory+disk?

D.

[GitHub] ignite pull request #4556: IGNITE-9290: Make remove explicit locks async whe...

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/4556

IGNITE-9290: Make remove explicit locks async when node left.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9290

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4556


commit 5190f178898ca25a71cd1b7be5efec71ef39a827
Author: Andrey V. Mashenkov 
Date:   2018-08-15T09:52:08Z

Cleanup code.

Remove unused method parameter.

commit 7602100c68bd177d93a9a8d12df7ddb397efc77f
Author: Andrey V. Mashenkov 
Date:   2018-08-15T09:53:01Z

Make explicit locks removal async.




---

Re: IEP-22: Direct Data Load proposal

[Vladimir Ozerov]

Hi Denis,

This IEP is mostly about how we work with our own indexes and pages. So 3rd
party DB is out of question.

On Thu, Jun 21, 2018 at 10:38 PM Denis Magda  wrote:

> Vladimir,
>
> As I see from the IEP, this data loading technique is supposed to be used
> for deployments with Ignite persistence enabled. Is it possible to
> generalize this solution and use for pure in-memory and in-memory + 3rd
> party DB scenarios?
>
> --
> Denis
>
> On Wed, Jun 20, 2018 at 8:08 AM Vladimir Ozerov 
> wrote:
>
> > Igniters,
> >
> > Initial data load is one of the most important use cases for our product.
> > This is one the first things user try to do with Ignite. And if it takes
> > too much time, it is very likely that user will look for other solutions.
> >
> > We did good progress in this area recently. Specifically - a set of
> > internal improvements on our indexes, steaming mode for JDBC driver, COPY
> > command. But our internals are still not very efficient - every single
> > update goes through the whole set of Ignite components, such as page
> cache,
> > free-lists, BTrees, etc..
> >
> > I created IEP-22 [1]. It's goal is to implement special direct data load
> > mode which will bypass our page cache and use alternative algorithm for
> > index updates. Together with COPY command and streaming this improvement
> > will allow Ignite to load data with very high speed.
> >
> > Please review the IEP and share your comments.
> >
> > Vladimir.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-22%3A+Direct+Data+Load
> >
>

Re: Ticket review checklist

[Vladimir Ozerov]

Hi Ivan,

>From what I see we do not restrict contributors to use lambdas and streams.
Document states that plain collections and anonymous classes are
*preferred*. This is not obligatory requirement, and it seems reasonable to
me, because when developing complex projects at times it is better to have
more expressive code, than less non-obvious code which makes dozens
operations in a single string.

Or may be there are any other statements in the checklist which prevents
users from using Java 8 features?

Vladimir.

On Tue, Aug 14, 2018 at 7:16 PM ipavlukhin  wrote:

> Hi Igniters,
>
> I would like to refresh review checklist a little bit. Currently it [1]
> contains section against lambda Lambda expressions and Stream API. As
> far as I know it is not true anymore. Currently both features have
> theirs usage in core module. What is a state of affairs for a subject?
> Are there some well-known cases where e.g. lambdas are not applicable?
> Should we document it?
>
> I personally think that we could delete entire Java 8 section from
> checklist (and Java 5 as well). I understand that every tool should be
> used judiciously but I doubt that all cases can be covered in short
> checklist.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Java8
>
>
> On 2018/07/09 20:53:42, Dmitry Pavlov  wrote:
>  > I also tend to agree about updating checklist>
>  >
>  > About suite timeouts, I suspect there is one problem introduced
> recently>
>  > within 3 days, which caused this mass timeouts.>
>  >
>  > I hope Igniters will find out reason soon. In relation to compute we
> have>
>  > only 2 possible cause:>
>  > Ivan Daschinskiy (idaschinskiy) 2 files IGNITE-8869 Fixed>
>  > PartitionsExchangeOnDiscoveryHistoryOverflowTest hanging>
>  > Signed-off-by: Andrey Gura  ···>
>  >
>  > Dmitriy Govorukhin (dgovorukhin) 12 files IGNITE-8827 Disable WAL
> during>
>  > apply updates on recovery>
>  >
>  > I guess if we fix this reason we will fix 10 suites more>
>  > References:>
>  >
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ComputeGrid=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E>
>
>
>  >
>  >
>  > пн, 9 июл. 2018 г. в 22:29, Anton Vinogradov :>
>  >
>  > > Sounds reasonable.>
>  > > I've satrted Data Structures suite hang investigation [1].>
>  > >>
>  > > Igniters, especially commiters,>
>  > > I know, you're busy, but it will be a great help to the project in
> case you>
>  > > fix at least one hang per person.>
>  > >>
>  > > [1] https://issues.apache.org/jira/browse/IGNITE-8783>
>  > >>
>  > > пн, 9 июл. 2018 г. в 19:24, Maxim Muzafarov :>
>  > >>
>  > > > Hi Igniters,>
>  > > >>
>  > > > Let's back to discussion of review checklist. Can we add more>
>  > > clarification>
>  > > > about running all suites on TeamCity?>
>  > > >>
>  > > > My suggestion is: “All test suites MUST be run on TeamCity [3]
> before>
>  > > merge>
>  > > > to master, there MUST NOT be any test failures * and any
> tests\suites>
>  > > with>
>  > > > “execution timeouts” *. Not important test failures should be
> muted and>
>  > > > handled according to [4] process.”>
>  > > >>
>  > > > As you can see we have stable “Execution timeouts” for>
>  > > > “Activate\Deactiveate Cluster” test suite since 16-th June. How
> can we be>
>  > > > sure in this case that new changes would not break up old
> functionality?>
>  > > >>
>  > > > From my point, all new changes MUST NOT be merged to master util
> we will>
>  > > > fix all execution timeouts for suites. Even if current changes
> are not>
>  > > > related to these timeouts.>
>  > > >>
>  > > > [1]>
>  > > >>
>  > > >>
>  > >
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ActivateDeactivateCluster=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E>
>
>
>  > > >>
>  > > >>
>  > > > пн, 4 июн. 2018 г. в 15:56, Dmitry Pavlov :>
>  > > >>
>  > > > > Requirement of green TC for each PR is community rule, not my.>
>  > > > >>
>  > > > > I'll answer ro another question, what should we do with test
> failure:>
>  > > > > "Ideally - fix, but at least mute test and create ticket. ">
>  > > > >>
>  > > > > May be it's time to formalize Make TC Green Again process in
> details,>
>  > > so>
>  > > > > let me share my draft.>
>  > > > >>
>  > > > > If Igniter see test failure (in master, in release bracnh,
> etc), he>
>  > > > should>
>  > > > > consider following steps:>
>  > > > >>
>  > > > > - If your changes can led to this failure(s), please create issue>
>  > > with>
>  > > > > label MakeTeamCityGreenAgain and assign it to you.>
>  > > > > - If you have fix, please set ticket to PA state and write to dev>
>  > > > > list fix is ready.>
>  > > > > - For case fix will require some time please mute test and set>
>  > > > label>
>  > > > > Muted_Test to issue>
>  > > > > - If you know which change caused failure please contact change>
>  > > author>
>  > > > > directly.>
>  > > > > - If you 

[GitHub] ignite pull request #4555: IGNITE-7054 S3 IP finder: support client side enc...

[udaykale]

GitHub user udaykale opened a pull request:

https://github.com/apache/ignite/pull/4555

IGNITE-7054 S3 IP finder: support client side encryption



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/udaykale/ignite IGNITE-7054

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4555


commit 3e5c3018807ae1eb04154e71c54c399e82b9a641
Author: uday 
Date:   2018-08-16T07:32:23Z

IGNITE-7054 S3 IP finder: support client side encryption




---

Re: [Distributed SQL] Do we have a plan to implement QuadTree index?

[Vladimir Ozerov]

Hi Alex,

>From what I see there is some interest to K-D indexes in SQL world.
Postgres has it. MySQL users asks about it from time to time, and usually
use some simpler spatial solutions ask MySQL lacks this index type. We
definitely may consider integrating it with SQL, but this would be big
enough feature, requiring changes in almost all SQL components.

For this reason I would put SQL case aside for now and focus on
implementation for ML. I had a talk with Yuriy Babak some time ago and
explained how new index type can be added to the product. May be Yuriy has
some thoughts on how to do that properly with respect to ML roadmap and
plans.

On Thu, Aug 16, 2018 at 6:21 AM Alexey Zinoviev 
wrote:

> Sorry, for the delay, dear Pavel and Denis.
>
> Yes, I need a kind of indexing to improve KNN algorithms during training
> the model.
>
> In my draft solution I've implemented building of
> https://en.wikipedia.org/wiki/K-d_tree
>  on the training data set.
> It collects the information about data distributed in our specific ML
> Datasets (distributed by nodes over Ignite cache)
>
> Pavel, you ask me any questions and I've prepared answers.
>
> 1) Should be this index in-memory only or you want to persist it?
> *Maybe it should be persisted (to reuse that for next predictions)*
>
> 2) In general index can be implemented in two ways per-partition and
> per-node.
> *Thank you for your explanation. Of course the per-node is better.*
>
> 3) I think K-d tree is preferable
> *You are absolutely right, it should be K-d tree*
>
> 4) Could you please share use cases you're trying to solve with QuadTree?
> With
> close to real data and examples of queries?
>
> I need to solve *k-nearest neighbors search task *on a set of vectors with
> unique keys presented in Ignite Cache (training set),
> during the training phase I'm going to build a temporary index as a KD-Tree
> (based on distance between vectors).
>
> The distance metric is a parameter too.
>
> After that, in prediction phase the *k-nearest neighbors search task *is
> solved by brute-force iteration over all vectors to find the *k-nearest
> neighbors.*
> I'd like to improve the search part with queries to index to extract
> closest vectors.
>
> Of course, it's kind of experiment, and maybe it's bad idea to patch SQL
> internals to solve this private task, but as you mentioned it can be a good
> point of collaboration between ML and SQL components.
>
> Can I get one of the implemented indices as a primary example and implement
> it by myself?
> Could you recommend something as an start point for me?
>
> Thanks for your help.
>
>
>
>
> 2018-08-04 11:18 GMT+06:00 Denis Magda :
>
> > Alexey, are you working on some new ML/DL APIs/algorithms? Please
> elaborate
> > what you'd like to add to Ignite.
> >
> > --
> > Denis
> >
> > On Wed, Aug 1, 2018 at 3:10 PM Pavel Kovalenko 
> wrote:
> >
> > > Hello Alexey,
> > >
> > > It's not so difficult to implement new type of indexing of data, but if
> > you
> > > want to reach performance in distributed environment you need to have
> > > strong knowledge of a data you're indexing and what kind of queries you
> > > want to execute.
> > > Should be this index in-memory only or you want to persist it?
> > > In case of persistence your index should fit our page memory model
> > > requirements.
> > > In both cases your index should be ready to work in concurrent
> > environment.
> > >
> > > In general index can be implemented in two ways per-partition and
> > per-node.
> > > Per-partition may be efficient if you have a lot of points (x,y)
> > > representing a big one, e.g. image. In this case it's required that all
> > of
> > > these points will be in one partition that query e.g. makes images
> > > intersection will execute in one node. But if you have multiple images,
> > > your index will contain also another points from other object and will
> > > overload it.
> > > Per-node may be efficient if you have a lot of points (x,y) that are
> > > independent of each other, that you will use it as spatial e.g.. But in
> > > this case, I think K-d tree is preferable as it can be used in more
> wide
> > > way.
> > >
> > > Could you please share use cases you're trying to solve with QuadTree?
> > With
> > > close to real data and examples of queries? Because now the question is
> > too
> > > abstract and it's hard to understand how it should be implemented to
> > reach
> > > good results.
> > >
> > >
> > > 2018-08-01 16:45 GMT+03:00 Alexey Zinoviev :
> > >
> > > > Hi, Igniters.
> > > >
> > > > Currently I'm working on different math stuff over the Apache Ignite
> > and
> > > in
> > > > a few tasks I need to implement in memory something like this
> > > >
> > > > https://en.wikipedia.org/wiki/Quadtree
> > > >
> > > > I didn't find such index in Apache Ignite, but maybe it's under
> > > development
> > > > by somebody?
> > > >
> > > > Is it a difficult to add a new index type to our distributed SQL
> 

Re: [Distributed SQL] Do we have a plan to implement QuadTree index?

[Alexey Goncharuk]

Alexey,

I am not sure if it will be a proper fir for you, but I think it worth a
try.

Apache Ignite has an option to index geospatial data using third-party
libraries (note that it is not included in the default Ignite build, the
module is activated via the lgpl profile). The index is located in
Ignite-geospatial module and uses an r-tree implementation underneath. One
downside of this module is that the geospatial index is not supported for
the Ignite native persistence yet.

Hope this helps!
--AG

чт, 16 авг. 2018 г. в 6:21, Alexey Zinoviev :

> Sorry, for the delay, dear Pavel and Denis.
>
> Yes, I need a kind of indexing to improve KNN algorithms during training
> the model.
>
> In my draft solution I've implemented building of
> https://en.wikipedia.org/wiki/K-d_tree
>  on the training data set.
> It collects the information about data distributed in our specific ML
> Datasets (distributed by nodes over Ignite cache)
>
> Pavel, you ask me any questions and I've prepared answers.
>
> 1) Should be this index in-memory only or you want to persist it?
> *Maybe it should be persisted (to reuse that for next predictions)*
>
> 2) In general index can be implemented in two ways per-partition and
> per-node.
> *Thank you for your explanation. Of course the per-node is better.*
>
> 3) I think K-d tree is preferable
> *You are absolutely right, it should be K-d tree*
>
> 4) Could you please share use cases you're trying to solve with QuadTree?
> With
> close to real data and examples of queries?
>
> I need to solve *k-nearest neighbors search task *on a set of vectors with
> unique keys presented in Ignite Cache (training set),
> during the training phase I'm going to build a temporary index as a KD-Tree
> (based on distance between vectors).
>
> The distance metric is a parameter too.
>
> After that, in prediction phase the *k-nearest neighbors search task *is
> solved by brute-force iteration over all vectors to find the *k-nearest
> neighbors.*
> I'd like to improve the search part with queries to index to extract
> closest vectors.
>
> Of course, it's kind of experiment, and maybe it's bad idea to patch SQL
> internals to solve this private task, but as you mentioned it can be a good
> point of collaboration between ML and SQL components.
>
> Can I get one of the implemented indices as a primary example and implement
> it by myself?
> Could you recommend something as an start point for me?
>
> Thanks for your help.
>
>
>
>
> 2018-08-04 11:18 GMT+06:00 Denis Magda :
>
> > Alexey, are you working on some new ML/DL APIs/algorithms? Please
> elaborate
> > what you'd like to add to Ignite.
> >
> > --
> > Denis
> >
> > On Wed, Aug 1, 2018 at 3:10 PM Pavel Kovalenko 
> wrote:
> >
> > > Hello Alexey,
> > >
> > > It's not so difficult to implement new type of indexing of data, but if
> > you
> > > want to reach performance in distributed environment you need to have
> > > strong knowledge of a data you're indexing and what kind of queries you
> > > want to execute.
> > > Should be this index in-memory only or you want to persist it?
> > > In case of persistence your index should fit our page memory model
> > > requirements.
> > > In both cases your index should be ready to work in concurrent
> > environment.
> > >
> > > In general index can be implemented in two ways per-partition and
> > per-node.
> > > Per-partition may be efficient if you have a lot of points (x,y)
> > > representing a big one, e.g. image. In this case it's required that all
> > of
> > > these points will be in one partition that query e.g. makes images
> > > intersection will execute in one node. But if you have multiple images,
> > > your index will contain also another points from other object and will
> > > overload it.
> > > Per-node may be efficient if you have a lot of points (x,y) that are
> > > independent of each other, that you will use it as spatial e.g.. But in
> > > this case, I think K-d tree is preferable as it can be used in more
> wide
> > > way.
> > >
> > > Could you please share use cases you're trying to solve with QuadTree?
> > With
> > > close to real data and examples of queries? Because now the question is
> > too
> > > abstract and it's hard to understand how it should be implemented to
> > reach
> > > good results.
> > >
> > >
> > > 2018-08-01 16:45 GMT+03:00 Alexey Zinoviev :
> > >
> > > > Hi, Igniters.
> > > >
> > > > Currently I'm working on different math stuff over the Apache Ignite
> > and
> > > in
> > > > a few tasks I need to implement in memory something like this
> > > >
> > > > https://en.wikipedia.org/wiki/Quadtree
> > > >
> > > > I didn't find such index in Apache Ignite, but maybe it's under
> > > development
> > > > by somebody?
> > > >
> > > > Is it a difficult to add a new index type to our distributed SQL
> (from
> > > > point of view of different infrastructure issues and so on P.S I
> don't
> > > > worry the math stuff here because I've implemented it many 

[jira] [Created] (IGNITE-9290) Make remove explicit locks async when node left.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-9290:


 Summary: Make remove explicit locks async when node left.
 Key: IGNITE-9290
 URL: https://issues.apache.org/jira/browse/IGNITE-9290
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Andrew Mashenkov


GridCacheMvccManager.removeExplicitNodeLocks() run synchronously in discovery 
and exchange threads. This introduce unnecessary delays in discovery and 
exchange process.

Also, this may cause a deadlock on node stop if user transaction holds an entry 
lock and awaits some Ignite manager response (e.g. cache store or dr), as 
manager stops right after last exchange has finished so managers can't detect 
node is stopping. 

 

[1] 
[http://apache-ignite-developers.2346864.n4.nabble.com/Synchronous-tx-entries-unlocking-in-discovery-exchange-threads-td33827.html]
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9289) Document CPP thin client

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-9289:
---

 Summary: Document CPP thin client
 Key: IGNITE-9289
 URL: https://issues.apache.org/jira/browse/IGNITE-9289
 Project: Ignite
  Issue Type: Task
  Components: documentation, thin client
Reporter: Vladimir Ozerov
Assignee: Igor Sapego
 Fix For: 2.7






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9288) Document how to start Ignite under Java 9

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-9288:
---

 Summary: Document how to start Ignite under Java 9
 Key: IGNITE-9288
 URL: https://issues.apache.org/jira/browse/IGNITE-9288
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.7






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9287) Document SQL best practices and known problems

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-9287:
---

 Summary: Document SQL best practices and known problems
 Key: IGNITE-9287
 URL: https://issues.apache.org/jira/browse/IGNITE-9287
 Project: Ignite
  Issue Type: Task
  Components: documentation, sql
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.7


We have a page with common performance suggestions [1]. Need to make sure that 
all important points are mentioned there.

[1] https://apacheignite-sql.readme.io/docs/performance-and-debugging



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4554: IGNITE-9258: NodeJS multiple clients fix

[ekaterina-nbl]

GitHub user ekaterina-nbl opened a pull request:

https://github.com/apache/ignite/pull/4554

IGNITE-9258: NodeJS multiple clients fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nobitlost/ignite ignite-9258

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4554


commit 4b444dff62c04b97bc2cdb0676ad52880374
Author: ekaterina-nbl 
Date:   2018-03-20T21:12:21Z

initial implementation

commit d1e8014c80f698028d23e09cafd7ea190ac3e929
Author: ekaterina-nbl 
Date:   2018-03-20T21:32:52Z

initial implementation

commit f79229e233ffa7bff1e7c22f04749924af6d712a
Author: ekaterina-nbl 
Date:   2018-03-22T09:39:32Z

initial implementation

commit 658d60b58840080b664e02815f4ba6aac76e5804
Author: ekaterina-nbl 
Date:   2018-03-22T16:52:18Z

minor update

commit 617cef12ad72791c7e7e67179e1b0c3f7f3ae6bb
Author: ekaterina-nbl 
Date:   2018-03-25T13:33:27Z

api spec

commit d9729585f5a901977e2a2e40e86a2b715fab79fa
Author: alexey-nbl 
Date:   2018-03-25T21:27:11Z

link to api_spec added

commit ea847eba62e556fa81cbf9838ffe17af60f464e4
Author: ekaterina-nbl 
Date:   2018-03-31T22:07:50Z

error types modified

commit c2ad53fe625cc3a05155eeef318421538830
Author: ekaterina-nbl 
Date:   2018-03-31T23:41:56Z

client states

commit 6d2233864b4d891361d38a7143846570bd6c0ef6
Author: ekaterina-nbl 
Date:   2018-04-01T13:11:27Z

object types improvement

commit 52fbc5f87143da068596141cb59b17b684fd2c1f
Author: ekaterina-nbl 
Date:   2018-04-02T16:59:52Z

complex objects

commit cae5a28e7e0d610434debcc140738e2f48d061cf
Author: ekaterina-nbl 
Date:   2018-04-02T20:13:00Z

object types improvement

commit 3165405d4eae09519dc9b5f40f162eb74a9b3c5d
Author: ekaterina-nbl 
Date:   2018-04-02T21:21:26Z

client states

commit fdbb8f86b32fe4c038d38620a80921be3038f31f
Author: alexey-nbl 
Date:   2018-04-03T08:26:04Z

Ignite client states described

commit 04b946885db0ea2f6fe56a75e28302641dad5f61
Author: alexey-nbl 
Date:   2018-04-03T09:35:49Z

minor update

commit f4aaf1c3f23c82ba7974b4c571a9984748e5e82e
Author: alexey-nbl 
Date:   2018-04-03T12:47:54Z

Update ObjectType.js

commit e4c8279f4e83b3ed13383420ab3d1417b090a3fa
Author: ekaterina-nbl 
Date:   2018-04-03T13:46:19Z

minor updates + api spec

commit ed2e4ee830ca40a28dc31958665f52fab6a0bcdd
Author: alexey-nbl 
Date:   2018-04-04T14:34:52Z

Update ObjectType.js

commit 7e1666eb5df82b622a028c0ea949fa21c79e66c2
Author: alexey-nbl 
Date:   2018-04-04T15:14:00Z

Update ObjectType.js

commit 7b0270d65b597dc1b0d164e78f07c3a21c37ca67
Author: ekaterina-nbl 
Date:   2018-04-08T17:16:43Z

sql queries, key value ops

commit fe90f53fd08f77add17fbf06d014f9a9b0a11c65
Author: alexey-nbl 
Date:   2018-04-08T18:31:47Z

Update IgniteClient.js

commit 04137bf5ec3b7077e194edd0100a01bb43f7933a
Author: alexey-nbl 
Date:   2018-04-08T18:37:04Z

Update CacheConfiguration.js

commit b63ad5980718da1b0c44fa4ca138c6e85e47aef3
Author: alexey-nbl 
Date:   2018-04-08T19:11:11Z

Update ObjectType.js

commit 756b908c9dc38ae497e4d7d740f836dabed93e48
Author: alexey-nbl 
Date:   2018-04-08T22:41:54Z

Update CacheClient.js

commit e96ffee17298dd25d26a7029738132478271cf92
Author: ekaterina-nbl 
Date:   2018-04-08T23:23:33Z

object array, minor updates

commit c050e671f74232c4efc41f51c2018d08b152cbbc
Author: alexey-nbl 
Date:   2018-04-09T21:04:35Z

Update CacheClient.js

commit 25052a4e93d6fcc0b0c4789fdd5a1eb85413b5a2
Author: alexey-nbl 
Date:   2018-04-09T22:43:50Z

Update ObjectType.js

commit c516b4147c10398d4f34d78a8830dd0fcb5f28f4
Author: alexey-nbl 
Date:   2018-04-10T11:50:13Z

Update CacheClient.js

commit 0e9924a0df8a1d41718fb929698b8e4416f2efa2
Author: ekaterina-nbl 
Date:   2018-04-10T13:21:16Z

cache key value operations tests

commit e09ee0e7969bffd6f4cfd11579f6d0ff9b486c99
Author: ekaterina-nbl 
Date:   2018-04-10T13:21:50Z

Merge branch 'master' of https://github.com/nobitlost/ignite

commit 15d2abce41b4f1c9d4d5fecd44a99a5393177482
Author: alexey-nbl 
Date:   2018-04-10T14:38:57Z

Update Query.js




---

[jira] [Created] (IGNITE-9286) Redesign and Refactor UI

[Dmitriy Shabalin (JIRA)]

Dmitriy Shabalin created IGNITE-9286:


 Summary: Redesign and Refactor UI
 Key: IGNITE-9286
 URL: https://issues.apache.org/jira/browse/IGNITE-9286
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Dmitriy Shabalin
Assignee: Dmitriy Shabalin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4553: Ignite wc 585

[Dmitriyff]

GitHub user Dmitriyff opened a pull request:

https://github.com/apache/ignite/pull/4553

Ignite wc 585



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-wc-585

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4553.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4553


commit 49935620a8574ae7ab45dd4fbe3cc58ce2dfc219
Author: Dmitriy Shabalin 
Date:   2018-07-11T10:27:16Z

WC-600 redesign dialog

commit 602b4db78deb82a236b03826eec570c07d888e97
Author: Dmitriy Shabalin 
Date:   2018-07-11T10:48:39Z

WC-600 added paddings to legend

commit 8ccd452aeb5b79d63cba28c3a74474496b67382a
Author: Dmitriy Shabalin 
Date:   2018-07-25T09:57:50Z

IGNITE-8234 Add new form field typeahead

commit e1e18b0534046b05af103ced3ce5cb21835d481a
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:01:41Z

IGNITE-8234 Refactor form fieldsd

commit 2c110dc3100a479438f969dc275bcc542c6c8c4e
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:02:44Z

IGNITE-8234 Replace to new fields

commit 2c54acdf2762e55aed68586a4be64b524e5116b8
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:03:34Z

IGNITE-8234 Add to list editable support new label

commit a05336cd3226670d11908f914399e54e17c54154
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:05:31Z

IGNITE-8234 Add support new stylee

commit f3364a7492c2da621689b8ff1bb44c8713349e18
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:06:06Z

IGNITE-8234 Refactor foem fields

commit 16346ee13729c0f72e18d7930b9dcf42541d72e6
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:06:29Z

IGNITE-8234 Refactor foem fields on model edit form

commit 825848ad2854ae7ee7d11da06f4e17f2167ddf9d
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:06:54Z

IGNITE-8234 Refactor foem fields on igfs edit form

commit 272c56d45aff63456a33682b884c18e81807155d
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:07:14Z

IGNITE-8234 Refactor foem fields on cluster edit form

commit b35da9b4aba9567b8fac39ab6ba50f4b223ec7f3
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:07:26Z

IGNITE-8234 Refactor foem fields on cache edit form

commit 0d7dc636c8be1a8c90597bf8f0ac1012f2609380
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:07:51Z

IGNITE-8234 Refactor form fields on import dialog

commit 6ea61658b8ed4d6dea8a2b39184b608ca08c7b8f
Author: Dmitriy Shabalin 
Date:   2018-07-25T10:08:22Z

IGNITE-8234 Refactor form add mod to show errors

commit 33e55c20e0c3c5948a8e6c2761b6140486066603
Author: Dmitriy Shabalin 
Date:   2018-07-27T09:37:05Z

WC-585 Refactor form field size

commit e610137e5ec2740cd4d5c0fddd277302767f6cce
Author: Dmitriy Shabalin 
Date:   2018-07-27T09:38:54Z

WC-585 Wip

commit 8bd56180294c3ec2746592cd6e584672822cc416
Author: Dmitriy Shabalin 
Date:   2018-07-27T10:55:03Z

WC-617 set wide dialog

commit 27692fe952fd3efa24e147a0847da048f745842b
Author: Dmitriy Shabalin 
Date:   2018-07-27T11:11:36Z

Merge branch 'ignite-wc-600' into ignite-wc-585

commit bbde55dfea69113cb2bf9ea4c36f14b45e4c3ea3
Author: Dmitriy Shabalin 
Date:   2018-07-30T05:19:48Z

WC-585 Add dialog footer style

commit af719599efd4253bc22028b93d3f7e8651eaa5db
Author: Dmitriy Shabalin 
Date:   2018-07-30T05:25:49Z

WC-585 Remove obsolete style

commit 025b13a0cb8a6eef7adf24314181607bfe814c0b
Author: Dmitriy Shabalin 
Date:   2018-07-30T05:38:37Z

WC-585 Remove unusable field

commit 81a537ff5fc403450f81d30c6e7a05ba9d94a85b
Author: Dmitriy Shabalin 
Date:   2018-07-30T06:15:17Z

WC-585 Remove obsoleted checkbox mixin

commit 187ea966178309d6df3b8bc357e6a495f5d1c675
Author: Dmitriy Shabalin 
Date:   2018-07-30T06:18:13Z

WC-585 Replace tooltip to new mixin

commit cded111fc546949c6b8c97b85cfc90be991ee70f
Author: Dmitriy Shabalin 
Date:   2018-07-30T07:37:26Z

WC-585 Refactor radio input

commit 0deca5443756d368f302bf63de9820fdf26d4375
Author: Dmitriy Shabalin 
Date:   2018-07-30T08:25:39Z

WC-585 Replace obsoleted class

commit e128ae0d90168282b55b6f509ad0cc98932e3794
Author: Dmitriy Shabalin 
Date:   2018-07-30T09:13:34Z

WC-585 Refactor errors output

commit 8866a0c7a615413ae5219bd3aee26e4f509bb0b0
Author: Dmitriy Shabalin 
Date:   2018-07-30T11:18:49Z

WC-585 Replace obsoleted error mixin to new

commit 24b0ae6564c08664b991f9133c6a5a1a16106f53
Author: Dmitriy Shabalin 
Date:   2018-07-31T08:14:38Z

WC-585 Refactor input dialog

commit 01536840b1afa5a587140c7a817db2798beacd6e
Author: Dmitriy Shabalin 
Date:   2018-07-31T08:15:11Z

WC-585 Remove obsoleted styles

commit 3d792c218084195cbd998ca83be1e39b11f40e52
Author: Dmitriy Shabalin 
Date:   2018-07-31T08:15:38Z

WC-585 Fix modal footer




---

Re: CacheStore and ignite.close

[Alexey Goncharuk]

Ilya,

Can you please clarify what you mean by 'abandons cache store operations'?
Does it mean that a read-through/write-through op is omitted, but the
public API method returns without an error? If it is so, then this is a
bug. If a public API method finishes with an exception when read-through is
omitted, then I think it is a correct behavior.

ср, 15 авг. 2018 г. в 16:38, Ilya Kasnacheev :

> Hello!
>
> I'm working on https://issues.apache.org/jira/browse/IGNITE-9093 test fix
> Turns out, IgniteDbPutGetWithCacheStoreTest.testReadThrough fails
> sporadically because the default ignite.close() is close(cancel=true), and
> it seems that it abandons cache store operations. So not all data is
> read/written from cache store.
>
> Is it really so? Is it considered safe?
>
> I have a pull request on this topic:
> https://github.com/apache/ignite/pull/4545
>
> Regards,
>
> --
> Ilya Kasnacheev
>

Re: Synchronous tx entries unlocking in discovery\exchange threads.

[Alexey Goncharuk]

Andrey, I agree that most likely this can be done in an async way. There
are some nuances, though, because if a node leaves during an ongoing
exchange, we should remove the locks in the context of the ongoing exchange
and not wait for the next exchange event.

I will take a look at your PR shortly.

ср, 15 авг. 2018 г. в 15:57, Andrey Mashenkov :

> Hi Igniters,
>
> I've found Ignite node tries to unlock tx entries when a node left the
> grid.
> Ignite do this synchronously in
> GridCacheMvccManager.removeExplicitNodeLocks() in discovery and exchange
> threads.
>
> Looks like this can be done in ascync way.
> I've made a PR #4565 and seems there is no new test failures [1].
>
> I'm not familiar enough with exchange manager code, but looks like we can
> scan locked entries more than once per node left event.
> Also, it looks possible we can scan locked entries once for several merged
> exchange events.
>
> Thoughts? Any ideas how this can be refactoried?
>
> [1]
>
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=projectOverview_IgniteTests24Java8=pull%2F4546%2Fhead
>
> --
> Best regards,
> Andrey V. Mashenkov
>