[jira] [Created] (IGNITE-6207) Low GC inserting of new entries

[Pranas Baliuka (JIRA)]

Pranas Baliuka created IGNITE-6207:
--

 Summary: Low GC inserting of new entries
 Key: IGNITE-6207
 URL: https://issues.apache.org/jira/browse/IGNITE-6207
 Project: Ignite
  Issue Type: New Feature
Reporter: Pranas Baliuka


It would be nice to have ability to keep buffer of mutable objects and reuse 
objects as container for API.

Currently looks like there's no way to have a latch/flush checkpoint and forced 
to create new objects for Key/Value pair.

Proposed API e.g. {{forceAddOrReplace(Iterator> entry)}} (or 
even better visitor pattern) where objects serialized but owners ship stays on 
client i.e. Ignite does not keep references to the objects provide between 
different itterator.next invocation. 

Clarification:
I know it sounds weird for enterprise developers (tolerant to extremely large 
latencies e.g. 100ms), but current system can not be practically applied for 
medium latency systems in finance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6206) IgniteDataStreamer flush() is not working

[Pranas Baliuka (JIRA)]

Pranas Baliuka created IGNITE-6206:
--

 Summary: IgniteDataStreamer flush() is not working
 Key: IGNITE-6206
 URL: https://issues.apache.org/jira/browse/IGNITE-6206
 Project: Ignite
  Issue Type: Test
Reporter: Pranas Baliuka
Priority: Blocker


Dear Ignite experts,

I'm total newbie at Ignite and trying to interpret JavaDoc of 
IgniteDataStreamer and test.
Summary of the test at forum:
http://apache-ignite-users.70518.x6.nabble.com/Why-DataStreamer-flush-is-not-flushing-td16466.html

Attempt to have checkpoint to flush data to cache from IgniteDataStreamer are 
not working:
* Restricted buffers;
* Time based flushing;
* Explicit flushig.

Test project - https://github.com/pranasblk/ignite-not-flushing

Let me know if it is by design. Otherwise point to documentation explaining how 
to do flusing ...

Thanks
P



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6205) Stability testing on CI server

[Sergey Kozlov (JIRA)]

Sergey Kozlov created IGNITE-6205:
-

 Summary: Stability testing on CI server
 Key: IGNITE-6205
 URL: https://issues.apache.org/jira/browse/IGNITE-6205
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.1
Reporter: Sergey Kozlov
 Fix For: 2.3


CI server shows many sporadic failures and the communty spend significant 
efforts to fix them.
We should to avoid such cases by adding the task on teamcity where new or 
modified tests wil run 100 times. Once it will be done the "100 times run" can 
become the part/requirement of review procedure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Testing Ignite Applications Locally

[Sergey Kozlov]

Yakov, ok. I'll do that

On Mon, Aug 28, 2017 at 6:19 PM, Yakov Zhdanov  wrote:

> >As far as Maven archetype, Yakov, is the only purpose of it to load a
> project, so users can add tests to it?
>
> Yes, but not just add. We will also be providing test examples, data
> loading application sample, shell scripts, etc. This can be pretty handy
> thing to create project with all dependencies initialized and useful stuff.
>
> --Yakov
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

[GitHub] ignite pull request #2534: For testing

[ilantukh]

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/2534

For testing



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.1.4-hotfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2534


commit 1e08c3fb5c02ec8acafd71b50b6ad3b749259f1a
Author: Andrey V. Mashenkov 
Date:   2017-07-31T11:14:56Z

IGNITE-4800: Lucene query may fails with NPE. This closes #2315.

commit 3fdf453e89a7bd76dff6b6d0646e3821ea3921d5
Author: Andrey V. Mashenkov 
Date:   2017-07-31T14:32:12Z

IGNITE-4800: Lucene query may fails with NPE.
Test fixed.

commit e255a564985a12113984ec02f15a4443495b8ffc
Author: Nikolay Izhikov 
Date:   2017-08-02T08:52:44Z

ignite-5712 Context switching for optimistic transactions

commit 772d462b68c7de8517d1f61e2e05ec8eefb18eac
Author: Alexey Kuznetsov 
Date:   2017-08-03T04:55:15Z

Merge branch ignite-2.1.3 into ignite-2.1.4

commit 0f3b7ca25313083e4dc35e7842931a655abd
Author: tledkov-gridgain 
Date:   2017-08-04T08:46:14Z

IGNITE-5126: Batch support for this JDBC driver. This closes #2162.

commit d1a74a4be8744528e6ed23706174041ddb0f2618
Author: devozerov 
Date:   2017-08-04T09:04:38Z

Merge remote-tracking branch 'upstream/ignite-2.1.4' into ignite-2.1.4

commit 0b3a9a7176f5ae44a96ecf700c8147193dfbf064
Author: Igor Sapego 
Date:   2017-08-04T10:18:00Z

IGNITE-5923: ODBC: SQLGetTypeInfo now works with SQL_ALL_TYPES

(cherry picked from commit 48c914d)

commit 4e0385fbc0f50548f2da3407fdfdfe939b463c67
Author: Igor Sapego 
Date:   2017-08-04T15:34:27Z

IGNITE-5939: ODBC: SQLColAttributes now works with legacy attribute codes.

(cherry picked from commit 70ffa2c)

commit 4f02504475fd1e5cc3b9f4754856e44d20fdc1cb
Author: Alexey Kuznetsov 
Date:   2017-08-07T02:41:22Z

Merge branch ignite-2.1.3 into ignite-2.1.4.

commit b093afb8231135a4904f5fffd62f5b4c332f1d47
Author: tledkov-gridgain 
Date:   2017-08-08T09:05:36Z

IGNITE-5211: Added new constructor: QueryEntity(Class keyCls, Class 
valCls). This closes #2371. This closes #2388. This closes #2407.

commit 879f19106b22e66d5f6ea94424d961d049397410
Author: devozerov 
Date:   2017-08-08T12:16:58Z

IGNITE-5982: GridMapQueryExecutor was split into several pieces.

commit 7c77b869bc3efdf19e53cc2b064f4290fd73e2b2
Author: devozerov 
Date:   2017-08-09T08:47:58Z

IGNITE-5993: Removed unused SQL-related classes and methods (old tree 
index, snapshots, etc). This closes #2414.

commit ab18fdfcc4f6db1e54fb1f3b68ba7fbc31a7f6e7
Author: Andrey V. Mashenkov 
Date:   2017-07-14T17:14:47Z

IGNITE-5452: GridTimeoutProcessor can hang on stop. This closes #2279.

commit 580b6aa8e5a8a887397eab5c4c830ec28f45cd30
Author: Alexey Kuznetsov 
Date:   2017-08-09T10:22:54Z

IGNITE-5734 Web Console: Fixed npm dependencies.
(cherry picked from commit aeafbf1)

commit 841db65e56063605475710bc170de4aea672c31d
Author: Alexey Kuznetsov 
Date:   2017-08-09T11:55:04Z

IGNITE-5987 Added -nq (visor will not quit in batch mode) option for Visor 
Cmd.
(cherry picked from commit 8d6e842)

commit 5c2097856714a7803956d754735c68b21156019c
Author: Alexey Kuznetsov 
Date:   2017-08-11T03:25:36Z

IGNITE-5902 Implemented stop caches at once.
(cherry picked from commit ebb8765)

commit 8b2461942c18f228c0107020aa28c03711bdceda
Author: Alexey Kuznetsov 
Date:   2017-08-11T04:07:26Z

IGNITE-6012 Refactored GridJettyRestHandler.processRequest(): replace 
mapper.writeValueAsString with writeValue(stream, v).
(cherry picked from commit 3a390c8)

commit 3a7d4f4a79e7c0a23244387e3a68535375a66a87
Author: Alexey Kuznetsov 
Date:   2017-08-11T04:18:42Z

IGNITE-6013 Optimized processing response from cluster.
(cherry picked from commit b02c481)

commit 74d6ab9916b3a01c78cdf1ad86211c9fcbb2214d
Author: Alexey Goncharuk 
Date:   2017-08-14T08:08:28Z

Merge branch 'ignite-2.1.3' into ignite-2.1.4

commit fde550bac56fd0cc7c51c62a9c291dd4c3f3030c
Author: Ilya Lantukh 
Date:   2017-08-14T08:32:11Z

IGNITE-5941 - Fixed index name length restrictions. This closes #2408

commit 13f38d79b57b395e43d42a8f3c278cf48336d7c5
Author: Dmitriy Govorukhin 
Date:   

[GitHub] ignite pull request #2533: IGNITE-6122: Propagate SqlFieldsQuery.lazy proper...

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/2533

IGNITE-6122: Propagate SqlFieldsQuery.lazy property for C++



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6122

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2533


commit 6c3b358b16928fc0c51a16cc6d605894098e2d96
Author: Igor Sapego 
Date:   2017-08-28T17:07:10Z

IGNITE-6122: Propagate SqlFieldsQuery.lazy property for C++




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: How a new index is built in runtime?

[Vladimir Ozerov]

Sync.

пн, 28 авг. 2017 г. в 19:34, Denis Magda :

> Vladimir,
>
> Is this a sync or async operation?
>
> —
> Denis
>
> > On Aug 28, 2017, at 1:05 AM, Vladimir Ozerov 
> wrote:
> >
> > Denis,
> >
> > We iterate over the whole cache and build the index entry-by-entry.
> Control
> > is returned back to the user when index is ready.
> >
> > On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov 
> wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>
>

Re: How a new index is built in runtime?

[Denis Magda]

Vladimir,

Is this a sync or async operation?

—
Denis

> On Aug 28, 2017, at 1:05 AM, Vladimir Ozerov  wrote:
> 
> Denis,
> 
> We iterate over the whole cache and build the index entry-by-entry. Control
> is returned back to the user when index is ready.
> 
> On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov  wrote:
> 
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted batches
>> in a single tree operation.
>> 
>> --
>> Yakov Zhdanov
>> 

Re: [DISCUSSION] Urgent Ignite bug fix release

[Denis Magda]

Anton, thanks for stepping in! The tickets set looks complete for me!

Let’s start the vote once the fixes are merged and tests pass.

—
Denis

> On Aug 28, 2017, at 7:19 AM, Anton Vinogradov  
> wrote:
> 
> Igniters,
> 
> Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
> In this case all other issues with fixVersion = 2.2 should be moved to 2.3.
> 
> Currently, I see 835 issues with fixVersion = 2.2
> 
> Seems we should have only 4 issues with fixVersion = 2.2:
> - Change default max memory size from 80% to 20% -
> https://issues.apache.org/jira/browse/IGNITE-6182
> - Detecting low memory on ignite node startup -
> https://issues.apache.org/jira/browse/IGNITE-6003
> - Backport of improvements to checkpoint algorithm -
> https://issues.apache.org/jira/browse/IGNITE-
> - ML profile is missing in 2.1 binary release -
> https://issues.apache.org/jira/browse/IGNITE-6193
> 
> Please correct me in case I've missed something.
> 
> Ivan,
>> Let's include to the release tickets with optimizations of checkpointing
> algorithm:
>> https://issues.apache.org/jira/browse/IGNITE-6178
>> https://issues.apache.org/jira/browse/IGNITE-6033
>> https://issues.apache.org/jira/browse/IGNITE-5961
>> This will help users who are experiencing problems with slow checkpoints.
>> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
> soften "Ignite node crashed in the middle of checkpoint" message as per
> discussion <
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>> .
> 
> All of these issues should have fixVersion 2.3.
> Please create backport issue and link it to all necessary issues.
> 
> P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.
> 
> On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov 
> wrote:
> 
>> Denis,
>> 
>>> BTW, who is considered to be the release manager of this release?
>> I'll do it.
>> 
>> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor 
>> wrote:
>> 
>>> Ok, the check happens at the node start time or on NODE_JOIN event
>>> 
>>> in general it looks like:
>>> 
>>> 1) calculate expected used memory = heap max + system cache max + all
>>> custom policies max + default policy size and put it into a node attribute
>>> 
>>> 2) get total physycal memory, calculate expected safe to be used memory
>>> amount (leave 4 gb min or 20% of available memory for OS)
>>> 
>>> 3) if expected used memory + expected used memory of other nodes on the
>>> host > than safe to be used memory amount, start calculating suggestions
>>> 
>>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>>> 100mb default polycy, if available memory is less we cannot suggest
>>> anything reasonable, print warning, stop calculation.
>>> 
>>> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
>>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
>>> calculated value or 512MB minimal)
>>> 
>>> 6) check if system cache size changed, suggest default value if it's so
>>> 
>>> 7) in case 100 mb * policies count < available memory, suggest using
>>> default policy with max size equals to remaining memory (available - heap
>>> -
>>> system cache)
>>> 
>>> 8) calculate new size for each memory policy ( it's user defined size *
>>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>> 
>>> 9) print suggestions
>>> 
>>> 
>>> 
>>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan :
>>> 
 Igor, can you please describe the algorithm with all the thresholds?
 
 On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor  The suggestion here is based on initial settings, and it's so because
 there
> is no other nodes on the host in the example.
> 
> The algorithm tries to preserve the original ratio of memory policies
> keeping numbers reasonable (for example after some thresshold it will
> suggest not to use several memory policies if there is not enough of
 memory
> for all of them) and taking into consideration nodes count on the
>>> host,
> each jvm heap, needed memory for OS, etc
> 
> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :
> 
>> Looks good, but why in the example provided are we suggesting 8GB? 2
> nodes
>> with 8GB will completely exhaust the available memory. I would
>>> suggest
 6
> or
>> 7GB.
>> 
>> Also, why 100MB for default policy. Anything under 1GB seems too
>>> small.
>> 
>> Can you please comment?
>> 
>> D.
>> 
>> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
 gvvinbl...@gmail.com>
>> wrote:
>> 
>>> One more example of possible warning:
>>> 
>>> 

[GitHub] ignite pull request #2532: IGNITE-5839: Unclear exception from BinaryObjectB...

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/2532

IGNITE-5839: Unclear exception from BinaryObjectBuilder::build call when 
builder is reused.

Fix BinaryObjectBuilder reusage issue.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2532.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2532


commit 774028b0aba9dcb453ce487b9265603e84bb147f
Author: Andrey V. Mashenkov 
Date:   2017-08-28T15:18:05Z

Fix BinaryObject builder reuse issue.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2531: IGNITE-5838: Unclear exception from BinaryObjectB...

[AMashenkov]

Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/2531


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2531: IGNITE-5838: Unclear exception from BinaryObjectB...

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/2531

IGNITE-5838: Unclear exception from BinaryObjectBuilder::build call when 
builder is reused

Fix BinaryObjectBuilder reusage issue.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2531


commit 774028b0aba9dcb453ce487b9265603e84bb147f
Author: Andrey V. Mashenkov 
Date:   2017-08-28T15:18:05Z

Fix BinaryObject builder reuse issue.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Testing Ignite Applications Locally

[Yakov Zhdanov]

>As far as Maven archetype, Yakov, is the only purpose of it to load a
project, so users can add tests to it?

Yes, but not just add. We will also be providing test examples, data
loading application sample, shell scripts, etc. This can be pretty handy
thing to create project with all dependencies initialized and useful stuff.

--Yakov

[GitHub] ignite pull request #2530: Ignite 6204 - Backport optimizations of checkpoin...

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/2530

Ignite 6204 - Backport optimizations of checkpointing algorithm into 2.2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6204

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2530


commit a17683fe4bd8826f8ff5145343e55896b407400d
Author: Ivan Rakov 
Date:   2017-08-10T10:31:23Z

IGNITE-5961 - Align pages in LFS partition files to pageSize

(cherry picked from commit 3919d80)

commit b96659f54c43bed96f8b1e6c9ceb67a6dbebc0ac
Author: Alexey Goncharuk 
Date:   2017-08-10T15:13:17Z

IGNITE-5961 - Fixed pages() method for FilePageStore

(cherry picked from commit 7199037)

commit 1808393b1c870b0c089cc495f070fcb75202d65d
Author: Ivan Rakov 
Date:   2017-08-17T12:54:21Z

IGNITE-6033 Added sorted and multithreaded modes in checkpointing algorithm 
- Fixes #2441.

Signed-off-by: Alexey Goncharuk 

(cherry picked from commit 69e6f8b)

commit 6423eb535e791f7c0b81ac2dddcbf5b3a73b9f11
Author: Dmitriy Govorukhin 
Date:   2017-08-22T13:34:31Z

IGNITE-6154 fix incorrect check checkpoint pages

(cherry picked from commit 474ecb8)

commit 7b9777eae0a821571b412ef923c77d129b92189e
Author: Ivan Rakov 
Date:   2017-08-22T14:03:42Z

IGNITE-6154 also fixed check for WAL record

(cherry picked from commit fa42218)

commit fd0aa3b7453f9cd8e3451ddd0c2d14cde586e793
Author: Ivan Rakov 
Date:   2017-08-24T15:18:31Z

IGNITE-6178 Make CheckpointWriteOrder.SEQUENTIAL and checkpointingThreads=4 
default in persistent store confguration

(cherry picked from commit 6f279b0)

commit 958948d973962b7bb121fff79d5d200a7074874e
Author: Ivan Rakov 
Date:   2017-08-25T17:09:43Z

IGNITE-6183 Make "Node crashed in the middle of checkpoint" message softer 
and more informative

Signed-off-by: Andrey Gura 

(cherry picked from commit a1fab62)

commit 56a3acc905fda1176ff94bb0f5c499d278a99155
Author: Ivan Rakov 
Date:   2017-08-28T15:16:11Z

IGNITE-6204 CE fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Testing Ignite Applications Locally

[Yakov Zhdanov]

Sergey, can you please elaborate?

--Yakov

2017-08-26 23:51 GMT+03:00 Sergey Kozlov :

> The idea is great!
>
> Also I would suggest an ability to run new (modified) tests 100 times in
> loop on the CI server to make sure that they don't cause no sporadic
> failures (we can include that as part of the requirements before review)
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: How to add new build-plan in the TeamCity?

[Vyacheslav Daradur]

Hi, Aleksey!

I'd like to add a new build configuration to the exist project "ignite-2.0".

I'm talking about a module which is contained only in my PR [1] at current
time.

Path to the module: "ignite/modules/compatibility"
Path to a test-suite:
"/java/org/apache/ignite/compatibility/testsuites/IgniteCompatibilityBasicTestSuite.java"

[1] https://github.com/apache/ignite/pull/2425

2017-08-25 17:14 GMT+03:00 Aleksey Chetaev :

> Hi,
> Do you need to add new build configuration to exist project or you want add
> new project for your module?
> About what module do we speak?
>
>
> 2017-08-15 18:57 GMT+03:00 daradurvs [via Apache Ignite Developers] <
> ml+s2346864n20860...@n4.nabble.com>:
>
> > Hi Igniters!
> >
> > I am working on new test module in the Ignite project [1].
> >
> > I need check than it will work in the TeamCity [2].
> >
> > I have no permissions to create new configuration.
> >
> > How to add new build-plan in the TeamCity according to test-suite within
> > new module?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5732
> > [2] https://ci.ignite.apache.org
> >
> > --
> > Best Regards, Vyacheslav D.
> >
> >
> > --
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://apache-ignite-developers.2346864.n4.nabble.
> > com/How-to-add-new-build-plan-in-the-TeamCity-tp20860.html
> > To unsubscribe from Apache Ignite Developers, click here
> >  com/template/NamlServlet.jtp?macro=unsubscribe_by_code=1=
> YWxleC5jaGV0YWV2QGdtYWlsLmNvbXwxfC0xOTM2ODA5ODU5>
> > .
> > NAML
> >  com/template/NamlServlet.jtp?macro=macro_viewer=instant_
> html%21nabble%3Aemail.naml=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
> --
> View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/How-to-add-new-build-plan-
> in-the-TeamCity-tp20860p21316.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.




-- 
Best Regards, Vyacheslav D.

Re: [DISCUSSION] Urgent Ignite bug fix release

[Ivan Rakov]

Backport issue: https://issues.apache.org/jira/browse/IGNITE-6204

Best Regards,
Ivan Rakov

On 28.08.2017 17:19, Anton Vinogradov wrote:

Igniters,

Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
In this case all other issues with fixVersion = 2.2 should be moved to 2.3.

Currently, I see 835 issues with fixVersion = 2.2

Seems we should have only 4 issues with fixVersion = 2.2:
- Change default max memory size from 80% to 20% -
https://issues.apache.org/jira/browse/IGNITE-6182
- Detecting low memory on ignite node startup -
https://issues.apache.org/jira/browse/IGNITE-6003
- Backport of improvements to checkpoint algorithm -
https://issues.apache.org/jira/browse/IGNITE-
- ML profile is missing in 2.1 binary release -
https://issues.apache.org/jira/browse/IGNITE-6193

Please correct me in case I've missed something.

Ivan,

Let's include to the release tickets with optimizations of checkpointing

algorithm:

https://issues.apache.org/jira/browse/IGNITE-6178
https://issues.apache.org/jira/browse/IGNITE-6033
https://issues.apache.org/jira/browse/IGNITE-5961
This will help users who are experiencing problems with slow checkpoints.
Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to

soften "Ignite node crashed in the middle of checkpoint" message as per
discussion <
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html

.

All of these issues should have fixVersion 2.3.
Please create backport issue and link it to all necessary issues.

P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.

On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov 
wrote:


Denis,


BTW, who is considered to be the release manager of this release?

I'll do it.

On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor 
wrote:


Ok, the check happens at the node start time or on NODE_JOIN event

in general it looks like:

1) calculate expected used memory = heap max + system cache max + all
custom policies max + default policy size and put it into a node attribute

2) get total physycal memory, calculate expected safe to be used memory
amount (leave 4 gb min or 20% of available memory for OS)

3) if expected used memory + expected used memory of other nodes on the
host > than safe to be used memory amount, start calculating suggestions

4) Each ignite instance needs at least 512mb heap + 40mb system cache +
100mb default polycy, if available memory is less we cannot suggest
anything reasonable, print warning, stop calculation.

5) check heap size (shouldn't exceed 30% of available memory (total_memory
- reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
calculated value or 512MB minimal)

6) check if system cache size changed, suggest default value if it's so

7) in case 100 mb * policies count < available memory, suggest using
default policy with max size equals to remaining memory (available - heap
-
system cache)

8) calculate new size for each memory policy ( it's user defined size *
(remaining / (all_policies_size * nodes_cnt)); in proportion to
remaining memory, devided by nodes number on the host or 100 mb minimal)

9) print suggestions



2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan :


Igor, can you please describe the algorithm with all the thresholds?

On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor :


Looks good, but why in the example provided are we suggesting 8GB? 2

nodes

with 8GB will completely exhaust the available memory. I would

suggest

6

or

7GB.

Also, why 100MB for default policy. Anything under 1GB seems too

small.

Can you please comment?

D.

On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <

gvvinbl...@gmail.com>

wrote:


One more example of possible warning:

-
Excessive memory usage by Ignite node process (performance may

drop)

[requested=44613MB, available=15942MB].

Please tune the folowing settings as suggested:
   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB

Current settings:
   Java Heap  maxSize: 3543MB
   Java Heap initSize: 250MB
   MemoryPolicyConfiguration.initialSize for 

Re: [DISCUSSION] Urgent Ignite bug fix release

[Anton Vinogradov]

Igniters,

Seems 2.2 is a urgent bugfix release, so it should be based on 2.1,
In this case all other issues with fixVersion = 2.2 should be moved to 2.3.

Currently, I see 835 issues with fixVersion = 2.2

Seems we should have only 4 issues with fixVersion = 2.2:
- Change default max memory size from 80% to 20% -
https://issues.apache.org/jira/browse/IGNITE-6182
- Detecting low memory on ignite node startup -
https://issues.apache.org/jira/browse/IGNITE-6003
- Backport of improvements to checkpoint algorithm -
https://issues.apache.org/jira/browse/IGNITE-
- ML profile is missing in 2.1 binary release -
https://issues.apache.org/jira/browse/IGNITE-6193

Please correct me in case I've missed something.

Ivan,
> Let's include to the release tickets with optimizations of checkpointing
algorithm:
> https://issues.apache.org/jira/browse/IGNITE-6178
> https://issues.apache.org/jira/browse/IGNITE-6033
> https://issues.apache.org/jira/browse/IGNITE-5961
> This will help users who are experiencing problems with slow checkpoints.
> Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to
soften "Ignite node crashed in the middle of checkpoint" message as per
discussion <
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-close-G-stop-name-true-Change-flag-cancel-to-false-td20473.html
>.

All of these issues should have fixVersion 2.3.
Please create backport issue and link it to all necessary issues.

P.s. I've created branch ignite-2.2, currently it equals to ignite-2.1.

On Mon, Aug 28, 2017 at 4:05 PM, Anton Vinogradov 
wrote:

> Denis,
>
> > BTW, who is considered to be the release manager of this release?
> I'll do it.
>
> On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor 
> wrote:
>
>> Ok, the check happens at the node start time or on NODE_JOIN event
>>
>> in general it looks like:
>>
>> 1) calculate expected used memory = heap max + system cache max + all
>> custom policies max + default policy size and put it into a node attribute
>>
>> 2) get total physycal memory, calculate expected safe to be used memory
>> amount (leave 4 gb min or 20% of available memory for OS)
>>
>> 3) if expected used memory + expected used memory of other nodes on the
>> host > than safe to be used memory amount, start calculating suggestions
>>
>> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
>> 100mb default polycy, if available memory is less we cannot suggest
>> anything reasonable, print warning, stop calculation.
>>
>> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
>> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
>> calculated value or 512MB minimal)
>>
>> 6) check if system cache size changed, suggest default value if it's so
>>
>> 7) in case 100 mb * policies count < available memory, suggest using
>> default policy with max size equals to remaining memory (available - heap
>> -
>> system cache)
>>
>> 8) calculate new size for each memory policy ( it's user defined size *
>> (remaining / (all_policies_size * nodes_cnt)); in proportion to
>> remaining memory, devided by nodes number on the host or 100 mb minimal)
>>
>> 9) print suggestions
>>
>>
>>
>> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan :
>>
>> > Igor, can you please describe the algorithm with all the thresholds?
>> >
>> > On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor > >
>> > wrote:
>> >
>> > > The suggestion here is based on initial settings, and it's so because
>> > there
>> > > is no other nodes on the host in the example.
>> > >
>> > > The algorithm tries to preserve the original ratio of memory policies
>> > > keeping numbers reasonable (for example after some thresshold it will
>> > > suggest not to use several memory policies if there is not enough of
>> > memory
>> > > for all of them) and taking into consideration nodes count on the
>> host,
>> > > each jvm heap, needed memory for OS, etc
>> > >
>> > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :
>> > >
>> > > > Looks good, but why in the example provided are we suggesting 8GB? 2
>> > > nodes
>> > > > with 8GB will completely exhaust the available memory. I would
>> suggest
>> > 6
>> > > or
>> > > > 7GB.
>> > > >
>> > > > Also, why 100MB for default policy. Anything under 1GB seems too
>> small.
>> > > >
>> > > > Can you please comment?
>> > > >
>> > > > D.
>> > > >
>> > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
>> > gvvinbl...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > One more example of possible warning:
>> > > > >
>> > > > > -
>> > > > > Excessive memory usage by Ignite node process (performance may
>> drop)
>> > > > > [requested=44613MB, available=15942MB].
>> > > > >
>> > > > > Please tune the folowing settings as suggested:
>> > > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>> > > 

[GitHub] ignite pull request #2529: UpdateSequence fix in ClientTopology.

[ilantukh]

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/2529

UpdateSequence fix in ClientTopology.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-12688

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2529


commit 1e08c3fb5c02ec8acafd71b50b6ad3b749259f1a
Author: Andrey V. Mashenkov 
Date:   2017-07-31T11:14:56Z

IGNITE-4800: Lucene query may fails with NPE. This closes #2315.

commit 3fdf453e89a7bd76dff6b6d0646e3821ea3921d5
Author: Andrey V. Mashenkov 
Date:   2017-07-31T14:32:12Z

IGNITE-4800: Lucene query may fails with NPE.
Test fixed.

commit e255a564985a12113984ec02f15a4443495b8ffc
Author: Nikolay Izhikov 
Date:   2017-08-02T08:52:44Z

ignite-5712 Context switching for optimistic transactions

commit 772d462b68c7de8517d1f61e2e05ec8eefb18eac
Author: Alexey Kuznetsov 
Date:   2017-08-03T04:55:15Z

Merge branch ignite-2.1.3 into ignite-2.1.4

commit 0f3b7ca25313083e4dc35e7842931a655abd
Author: tledkov-gridgain 
Date:   2017-08-04T08:46:14Z

IGNITE-5126: Batch support for this JDBC driver. This closes #2162.

commit d1a74a4be8744528e6ed23706174041ddb0f2618
Author: devozerov 
Date:   2017-08-04T09:04:38Z

Merge remote-tracking branch 'upstream/ignite-2.1.4' into ignite-2.1.4

commit 0b3a9a7176f5ae44a96ecf700c8147193dfbf064
Author: Igor Sapego 
Date:   2017-08-04T10:18:00Z

IGNITE-5923: ODBC: SQLGetTypeInfo now works with SQL_ALL_TYPES

(cherry picked from commit 48c914d)

commit 4e0385fbc0f50548f2da3407fdfdfe939b463c67
Author: Igor Sapego 
Date:   2017-08-04T15:34:27Z

IGNITE-5939: ODBC: SQLColAttributes now works with legacy attribute codes.

(cherry picked from commit 70ffa2c)

commit 4f02504475fd1e5cc3b9f4754856e44d20fdc1cb
Author: Alexey Kuznetsov 
Date:   2017-08-07T02:41:22Z

Merge branch ignite-2.1.3 into ignite-2.1.4.

commit b093afb8231135a4904f5fffd62f5b4c332f1d47
Author: tledkov-gridgain 
Date:   2017-08-08T09:05:36Z

IGNITE-5211: Added new constructor: QueryEntity(Class keyCls, Class 
valCls). This closes #2371. This closes #2388. This closes #2407.

commit 879f19106b22e66d5f6ea94424d961d049397410
Author: devozerov 
Date:   2017-08-08T12:16:58Z

IGNITE-5982: GridMapQueryExecutor was split into several pieces.

commit 7c77b869bc3efdf19e53cc2b064f4290fd73e2b2
Author: devozerov 
Date:   2017-08-09T08:47:58Z

IGNITE-5993: Removed unused SQL-related classes and methods (old tree 
index, snapshots, etc). This closes #2414.

commit ab18fdfcc4f6db1e54fb1f3b68ba7fbc31a7f6e7
Author: Andrey V. Mashenkov 
Date:   2017-07-14T17:14:47Z

IGNITE-5452: GridTimeoutProcessor can hang on stop. This closes #2279.

commit 580b6aa8e5a8a887397eab5c4c830ec28f45cd30
Author: Alexey Kuznetsov 
Date:   2017-08-09T10:22:54Z

IGNITE-5734 Web Console: Fixed npm dependencies.
(cherry picked from commit aeafbf1)

commit 841db65e56063605475710bc170de4aea672c31d
Author: Alexey Kuznetsov 
Date:   2017-08-09T11:55:04Z

IGNITE-5987 Added -nq (visor will not quit in batch mode) option for Visor 
Cmd.
(cherry picked from commit 8d6e842)

commit 5c2097856714a7803956d754735c68b21156019c
Author: Alexey Kuznetsov 
Date:   2017-08-11T03:25:36Z

IGNITE-5902 Implemented stop caches at once.
(cherry picked from commit ebb8765)

commit 8b2461942c18f228c0107020aa28c03711bdceda
Author: Alexey Kuznetsov 
Date:   2017-08-11T04:07:26Z

IGNITE-6012 Refactored GridJettyRestHandler.processRequest(): replace 
mapper.writeValueAsString with writeValue(stream, v).
(cherry picked from commit 3a390c8)

commit 3a7d4f4a79e7c0a23244387e3a68535375a66a87
Author: Alexey Kuznetsov 
Date:   2017-08-11T04:18:42Z

IGNITE-6013 Optimized processing response from cluster.
(cherry picked from commit b02c481)

commit 74d6ab9916b3a01c78cdf1ad86211c9fcbb2214d
Author: Alexey Goncharuk 
Date:   2017-08-14T08:08:28Z

Merge branch 'ignite-2.1.3' into ignite-2.1.4

commit fde550bac56fd0cc7c51c62a9c291dd4c3f3030c
Author: Ilya Lantukh 
Date:   2017-08-14T08:32:11Z

IGNITE-5941 - Fixed index name length restrictions. This closes #2408

commit 13f38d79b57b395e43d42a8f3c278cf48336d7c5
Author: Dmitriy Govorukhin 

[jira] [Created] (IGNITE-6203) Valid query cannot be processed.

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-6203:
-

 Summary: Valid query cannot be processed.
 Key: IGNITE-6203
 URL: https://issues.apache.org/jira/browse/IGNITE-6203
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Alexei Scherbakov
 Fix For: 2.3


Query: select * from Integer where _KEY=? and false
Exception:
{noformat}
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to bind 
parameter [idx=1, obj=1, stmt=prep4: SELECT
__Z0._KEY,
__Z0._VAL
FROM "default".INTEGER __Z0
WHERE FALSE]
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.bindObject(IgniteH2Indexing.java:515)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.bindParameters(IgniteH2Indexing.java:1048)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.optimize(GridSqlQuerySplitter.java:1628)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:220)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFields(IgniteH2Indexing.java:1336)
... 18 more
Caused by: org.h2.jdbc.JdbcSQLException: Invalid value "1" for parameter 
"parameterIndex" [90008-195]
...
{noformat}


{noformat}
public void test() throws Exception {
try {
Ignite ignite = startGrid();

SqlFieldsQuery qry = new SqlFieldsQuery("select * from Integer 
where _KEY=? and false");

qry.setArgs(1);

FieldsQueryCursor query = 
ignite.cache(DEFAULT_CACHE_NAME).query(qry);

System.out.println(query.getAll());
} finally {
stopAllGrids();
}
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Cluster auto activation design proposal

[Dmitriy Setrakyan]

Sergey, the interface you are suggesting is internal, not external. Why
should user ever see it or care about it?

D.

On Mon, Aug 28, 2017 at 3:18 PM, Sergey Chugunov 
wrote:

> Dmitriy,
>
> It was my misunderstanding, I believe that setter is not enough and we need
> a full-fledged entity.
>
> We should also be able to check if BLTs are compatible. Interface looks
> like this and use case for this functionality is described below.
>
> interface BaselineTopology {
>Collection nodes();
>boolean isCompatibleWith(BaselineTopology blt);
> }
>
> Let's consider the following scenario:
>
>1. We have a grid with N nodes: it is up, active and has data in it. ->
>BLT #1 created.
>2. We shutdown the grid. Then divide it into two parts: Part1_grid and
>Part2_grid.
>3. We start and activate Part1_grid . Topology has changed -> BLT#2
>created.
>After that we shutdown that Part1_grid.
>4. We start and activate Part2_grid. Topology also has changed -> BLT#3
>created.
>5. Then we start Part1_grid and it's nodes try to join Part2_grid.
>
>
> If join is successful we have an undefined state of the resulting grid:
> values for the same key may (and will) differ between grid parts.
>
> So to prevent this we should keep nodes with BLT#2 from joining the grid
> with BLT#3. And we should fail nodes with an error message.
>
> Thanks,
> Sergey.
>
>
>
>
> On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan 
> wrote:
>
> > Sergey, I am still confused. What is the BaselineTopology interface in
> your
> > example? I thought that you agreed with me that we simply need a setter
> for
> > activation nodes, no?
> >
> > D.
> >
> > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > As I understand you use the term "minimalActivationNodes" as a synonym
> > for
> > > BaselineTopology concept.
> > > In that case I agree with you that we can replace both "establish*"
> > methods
> > > with a simple setter method (see below in summary).
> > >
> > > Summing up the whole discussion I see the functionality as following:
> > >
> > > New concept BaselineTopology is introduced. The main features it
> enables
> > > are:
> > >
> > >1. automatic activation of cluster;
> > >
> > >2. easy management of cluster topology changes (planned nodes
> > >maintenance, adding new nodes etc);
> > >
> > >3. eliminating of rebalancing traffic on short-term node failures.
> > >
> > >
> > > Use cases to create BLT:
> > >
> > >1. user starts up new cluster of desired number of nodes and
> activates
> > >it using existing API. BLT is created with all nodes presented in
> the
> > >cluster at the moment of activation, no API is needed;
> > >
> > >2. user prepares BLT using web-console or visor CMD tools and sets
> it
> > to
> > >the cluster. New API setter is needed:
> > >Ignite.activation().setBaselineTopology(BaselineTopology blt);
> > >
> > >3. user provides via static configuration a list of nodes that are
> > >expected to be in the cluster.
> > >User starts nodes one by one; when all preconfigured nodes are
> started
> > >cluster is activated and BLT is created.
> > >As list of nodes may be huge it is provided via separate file to
> avoid
> > >flooding main configuration.
> > >
> > >
> > > Igniters, does this description match with your understanding of
> > > functionality? If it does I'll create a set of tickets and start
> working
> > on
> > > implementation.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> > > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > I still do not see why anyone would explicitly call these 2 methods:
> > > >
> > > > *Ignite::activation::establishBaselineTopology();*
> > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > bltTop);*
> > > >
> > > > For example, if a web console, or some other admin process, want to
> > > > automatically set currently started nodes as the baseline topology,
> > > > shouldn't they just call a setter for minimalActivationNodes?
> > > >
> > > > D.
> > > >
> > > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > > admitr...@gridgain.com>
> > > > wrote:
> > > >
> > > > > API is proposed in the head of the thread by Sergey, as I
> understood:
> > > > > __
> > > > >
> > > > > API for BaselineTopology manipulation may look like this:
> > > > >
> > > > > *Ignite::activation::establishBaselineTopology();*
> > > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > > bltTop);*
> > > > >
> > > > > Both methods will establish BT and activate cluster once it is
> > > > established.
> > > > >
> > > > > The first one allows user to establish BT using current topology.
> If
> > > any
> > > > > changes happen to the topology during 

[GitHub] ignite pull request #2528: IGNITE-2845-2.1.5 Get operation might ignore entr...

[asfedotov]

GitHub user asfedotov opened a pull request:

https://github.com/apache/ignite/pull/2528

IGNITE-2845-2.1.5 Get operation might ignore entry update from 
EntryProcessor.

For CI purposes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2845

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2528.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2528


commit 84c7427a53a8e1712b1d0b763d7539c9cb844cb6
Author: Sergey Stronchinskiy 
Date:   2017-07-04T11:51:25Z

IGNITE-5532 .NET: Split CacheLinqTest into partial classes

This closes #2226

commit 64c156e9252395504af00f09d934f36b6bc21913
Author: Igor Sapego 
Date:   2017-07-04T16:42:33Z

IGNITE-5663: ODBC: Closing cursor do not reset prepared statement anymore

commit 80c95ff79f344daf1fca3f094733a24bac2a218d
Author: Igor Sapego 
Date:   2017-07-05T15:41:28Z

IGNITE-5576: Added Compute::Run() for C++

commit 836906c89dfb880ac602046c37b3a2dba3ebdc46
Author: samaitra 
Date:   2017-07-06T04:28:15Z

IGNITE-5695 FlinkIgniteSinkSelfTest is failing due to conflicting default 
test timeout and default flush frequency - Fixes #2241.

Signed-off-by: samaitra 

commit 651ffc544bbc32cded55626adcd3ed4cc74f11ce
Author: shroman 
Date:   2017-07-06T05:00:08Z

Removed unnecessary line from the comments.

commit d1d6802378d874b039f775fe787f78c507661bb2
Author: devozerov 
Date:   2017-07-07T09:36:13Z

Merge branch 'ignite-2.1'

commit 45cd87fe73db117f5148ed2006f8de8d2517bbfe
Author: mcherkasov 
Date:   2017-06-30T17:23:55Z

IGNITE-5554 ServiceProcessor may process failed reassignments in timeout 
thread

commit fa974286e8f066a8d6aa57519edf5ec7761be095
Author: Igor Sapego 
Date:   2017-07-07T13:49:15Z

IGNITE-5582: Implemented Compute::Broadcast for C++

commit 01f504ff83cc77f80d37981b5c5a15b653861bbd
Author: NSAmelchev 
Date:   2017-07-10T12:03:01Z

IGNITE-5087 Enum comparison fails after marshal-unmarshal with 
BinaryMarshaller.

commit ecfbc2c97464ad7da3f24afaaf1868a2d2fdb87e
Author: devozerov 
Date:   2017-07-11T09:17:41Z

Merge branch 'ignite-2.1'

# Conflicts:
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Apache.Ignite.Core.Tests.csproj

commit 1be9b40c37efcbf332ebeeefc865c2fe864339e7
Author: sboikov 
Date:   2017-07-11T09:42:54Z

Exchange future cleanup, added special tasks for reassign/force rebalance.

commit 8d4a0c2ca2abc17a1d54fa0d33b161531fa59b12
Author: Pavel Tupitsyn 
Date:   2017-07-11T09:49:16Z

Merge branch 'ignite-2.1'

commit bf25b5c52be044f07076c0800447230c75174db3
Author: Slava Koptilin 
Date:   2017-07-07T12:35:33Z

ignite-5562: assert statements were changed to the 'if' blocks

commit e93b28488693381fcd232de93087ab8ec1d0f5bb
Author: sboikov 
Date:   2017-07-11T11:18:52Z

ignite-5446 Only lateAffinity logic in CacheAffinitySharedManager.

commit 5c363184c80f2fd8b79f1075d1eacbf9af5369a1
Author: Denis Magda 
Date:   2017-07-11T19:20:16Z

Simplified Memory Policies Example

commit b95f76f8a0a3a7e920f78f20b3d814112fc6d522
Author: sboikov 
Date:   2017-07-12T05:47:04Z

ignite-5727 Call TcpCommunicationSpi's discovery listener first

commit 120384fca2b5f6f141207697f776d7859afa857f
Author: devozerov 
Date:   2017-07-12T06:48:51Z

Merge branch 'ignite-2.1'

commit 5394bbdeff4e9fb97d3b413bf30001ede580bdd7
Author: sboikov 
Date:   2017-07-13T10:30:59Z

Unnecessary synchronous rollback in GridDhtTxLocal.prepareAsync

commit 00c6b6c4ba00fa6577f74fc95b378737fb5a789c
Author: Alexander Menshikov 
Date:   2017-07-13T12:24:59Z

IGNITE-5567 Make benchmark Ignite.reentrantLock vs IgniteCache.lock

commit 18bdfe96a1e579371108c661e3374183c58a296d
Author: Alexey Goncharuk 
Date:   2017-07-13T12:42:30Z

Fixed NPE in tests

commit 7338445ac9c1a2343fd41cdd20785de07b727796
Author: dkarachentsev 
Date:   2017-07-13T13:00:08Z

IGNITE-5597 - Fix javadoc in Affinity and AffinityFunction for REPLICATED 
cache. This closes #2268.

commit d9ed07c67e4a4ff3a9de543cbe039ac2a48f03a0
Author: Sergey Chugunov 
Date:   2017-07-13T14:32:06Z

Functionality of muted test is debated now

commit 871d9260f3b32bed5273852dbdb74c758f73d383
Author: Sergey Chugunov 
Date:   

Re: Cluster auto activation design proposal

[Sergey Chugunov]

Dmitriy,

It was my misunderstanding, I believe that setter is not enough and we need
a full-fledged entity.

We should also be able to check if BLTs are compatible. Interface looks
like this and use case for this functionality is described below.

interface BaselineTopology {
   Collection nodes();
   boolean isCompatibleWith(BaselineTopology blt);
}

Let's consider the following scenario:

   1. We have a grid with N nodes: it is up, active and has data in it. ->
   BLT #1 created.
   2. We shutdown the grid. Then divide it into two parts: Part1_grid and
   Part2_grid.
   3. We start and activate Part1_grid . Topology has changed -> BLT#2
   created.
   After that we shutdown that Part1_grid.
   4. We start and activate Part2_grid. Topology also has changed -> BLT#3
   created.
   5. Then we start Part1_grid and it's nodes try to join Part2_grid.


If join is successful we have an undefined state of the resulting grid:
values for the same key may (and will) differ between grid parts.

So to prevent this we should keep nodes with BLT#2 from joining the grid
with BLT#3. And we should fail nodes with an error message.

Thanks,
Sergey.




On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan 
wrote:

> Sergey, I am still confused. What is the BaselineTopology interface in your
> example? I thought that you agreed with me that we simply need a setter for
> activation nodes, no?
>
> D.
>
> On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Dmitriy,
> >
> > As I understand you use the term "minimalActivationNodes" as a synonym
> for
> > BaselineTopology concept.
> > In that case I agree with you that we can replace both "establish*"
> methods
> > with a simple setter method (see below in summary).
> >
> > Summing up the whole discussion I see the functionality as following:
> >
> > New concept BaselineTopology is introduced. The main features it enables
> > are:
> >
> >1. automatic activation of cluster;
> >
> >2. easy management of cluster topology changes (planned nodes
> >maintenance, adding new nodes etc);
> >
> >3. eliminating of rebalancing traffic on short-term node failures.
> >
> >
> > Use cases to create BLT:
> >
> >1. user starts up new cluster of desired number of nodes and activates
> >it using existing API. BLT is created with all nodes presented in the
> >cluster at the moment of activation, no API is needed;
> >
> >2. user prepares BLT using web-console or visor CMD tools and sets it
> to
> >the cluster. New API setter is needed:
> >Ignite.activation().setBaselineTopology(BaselineTopology blt);
> >
> >3. user provides via static configuration a list of nodes that are
> >expected to be in the cluster.
> >User starts nodes one by one; when all preconfigured nodes are started
> >cluster is activated and BLT is created.
> >As list of nodes may be huge it is provided via separate file to avoid
> >flooding main configuration.
> >
> >
> > Igniters, does this description match with your understanding of
> > functionality? If it does I'll create a set of tickets and start working
> on
> > implementation.
> >
> > Thanks,
> > Sergey.
> >
> >
> > On Sat, Aug 19, 2017 at 5:41 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > I still do not see why anyone would explicitly call these 2 methods:
> > >
> > > *Ignite::activation::establishBaselineTopology();*
> > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > bltTop);*
> > >
> > > For example, if a web console, or some other admin process, want to
> > > automatically set currently started nodes as the baseline topology,
> > > shouldn't they just call a setter for minimalActivationNodes?
> > >
> > > D.
> > >
> > > On Fri, Aug 18, 2017 at 10:18 AM, Alexey Dmitriev <
> > admitr...@gridgain.com>
> > > wrote:
> > >
> > > > API is proposed in the head of the thread by Sergey, as I understood:
> > > > __
> > > >
> > > > API for BaselineTopology manipulation may look like this:
> > > >
> > > > *Ignite::activation::establishBaselineTopology();*
> > > > *Ignite::activation::establishBaselineTopology(BaselineTopology
> > > bltTop);*
> > > >
> > > > Both methods will establish BT and activate cluster once it is
> > > established.
> > > >
> > > > The first one allows user to establish BT using current topology. If
> > any
> > > > changes happen to the topology during establishing process, user will
> > be
> > > > notified and allowed to proceed or abort the procedure.
> > > >
> > > > Second method allows to use some monitoring'n'management tools like
> > > > WebConsole where user can prepare a list of nodes, using them create
> a
> > BT
> > > > and send to the cluster a command to finally establish it.
> > > >
> > > > From high level BaselineTopology entity contains only collection of
> > > nodes:
> > > >
> > > > *BaselineTopology {*
> > > > *  Collection 

Re: [DISCUSSION] Urgent Ignite bug fix release

[Anton Vinogradov]

Denis,

> BTW, who is considered to be the release manager of this release?
I'll do it.

On Mon, Aug 28, 2017 at 3:54 PM, Seliverstov Igor 
wrote:

> Ok, the check happens at the node start time or on NODE_JOIN event
>
> in general it looks like:
>
> 1) calculate expected used memory = heap max + system cache max + all
> custom policies max + default policy size and put it into a node attribute
>
> 2) get total physycal memory, calculate expected safe to be used memory
> amount (leave 4 gb min or 20% of available memory for OS)
>
> 3) if expected used memory + expected used memory of other nodes on the
> host > than safe to be used memory amount, start calculating suggestions
>
> 4) Each ignite instance needs at least 512mb heap + 40mb system cache +
> 100mb default polycy, if available memory is less we cannot suggest
> anything reasonable, print warning, stop calculation.
>
> 5) check heap size (shouldn't exceed 30% of available memory (total_memory
> - reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
> calculated value or 512MB minimal)
>
> 6) check if system cache size changed, suggest default value if it's so
>
> 7) in case 100 mb * policies count < available memory, suggest using
> default policy with max size equals to remaining memory (available - heap -
> system cache)
>
> 8) calculate new size for each memory policy ( it's user defined size *
> (remaining / (all_policies_size * nodes_cnt)); in proportion to
> remaining memory, devided by nodes number on the host or 100 mb minimal)
>
> 9) print suggestions
>
>
>
> 2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan :
>
> > Igor, can you please describe the algorithm with all the thresholds?
> >
> > On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor 
> > wrote:
> >
> > > The suggestion here is based on initial settings, and it's so because
> > there
> > > is no other nodes on the host in the example.
> > >
> > > The algorithm tries to preserve the original ratio of memory policies
> > > keeping numbers reasonable (for example after some thresshold it will
> > > suggest not to use several memory policies if there is not enough of
> > memory
> > > for all of them) and taking into consideration nodes count on the host,
> > > each jvm heap, needed memory for OS, etc
> > >
> > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Looks good, but why in the example provided are we suggesting 8GB? 2
> > > nodes
> > > > with 8GB will completely exhaust the available memory. I would
> suggest
> > 6
> > > or
> > > > 7GB.
> > > >
> > > > Also, why 100MB for default policy. Anything under 1GB seems too
> small.
> > > >
> > > > Can you please comment?
> > > >
> > > > D.
> > > >
> > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> > gvvinbl...@gmail.com>
> > > > wrote:
> > > >
> > > > > One more example of possible warning:
> > > > >
> > > > > -
> > > > > Excessive memory usage by Ignite node process (performance may
> drop)
> > > > > [requested=44613MB, available=15942MB].
> > > > >
> > > > > Please tune the folowing settings as suggested:
> > > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > > >   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > > > >
> > > > > Current settings:
> > > > >   Java Heap  maxSize: 3543MB
> > > > >   Java Heap initSize: 250MB
> > > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > > >   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > > >   The overall expected memory usage by all Ignite nodes on the
> host:
> > > > > 44613MB
> > > > > -
> > > > >
> > > > > Your thoughts?
> > > > >
> > > > > 2017-08-28 5:06 GMT+03:00 Denis Magda :
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > ML lib profile is missing in 2.1 release! That must be fixed and
> > > rolled
> > > > > > out in this emergency release:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > > > >
> > > > > > Oleg, Yuri, please step in and handle the issue.
> > > > > >
> > > > > > BTW, who is considered to be the release manager of this release?
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > I like the format proposed by Denis, very clear.
> > > > > > >
> > > > > > > However, I also do not understand why a user should change the
> > size
> > > > of
> > > > > > some
> > 

Re: How a new index is built in runtime?

[Vladimir Ozerov]

Should not be too hard. However, this will not change situation a lot.
Instead, we need to implement more efficient algorithm of index filling.
E.g. fill it with pre-sorted batches. We already have a ticket for that.

On Mon, Aug 28, 2017 at 3:46 PM,  wrote:

> Vova, how hard is it to make it multi-threaded?
>
> ⁣D.​
>
>
> On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov <
> voze...@gridgain.com> wrote:
> >Denis,
> >
> >We iterate over the whole cache and build the index entry-by-entry.
> >Control
> >is returned back to the user when index is ready.
> >
> >On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov 
> >wrote:
> >
> >> Of course, iteration should have an option to be run from more than 1
> >> thread. What will really help, IMO, is ability to insert presorted
> >batches
> >> in a single tree operation.
> >>
> >> --
> >> Yakov Zhdanov
> >>
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

Ok, the check happens at the node start time or on NODE_JOIN event

in general it looks like:

1) calculate expected used memory = heap max + system cache max + all
custom policies max + default policy size and put it into a node attribute

2) get total physycal memory, calculate expected safe to be used memory
amount (leave 4 gb min or 20% of available memory for OS)

3) if expected used memory + expected used memory of other nodes on the
host > than safe to be used memory amount, start calculating suggestions

4) Each ignite instance needs at least 512mb heap + 40mb system cache +
100mb default polycy, if available memory is less we cannot suggest
anything reasonable, print warning, stop calculation.

5) check heap size (shouldn't exceed 30% of available memory (total_memory
- reserved for OS memory) * 30% for all JVMs, if it exeedes, suggest just
calculated value or 512MB minimal)

6) check if system cache size changed, suggest default value if it's so

7) in case 100 mb * policies count < available memory, suggest using
default policy with max size equals to remaining memory (available - heap -
system cache)

8) calculate new size for each memory policy ( it's user defined size *
(remaining / (all_policies_size * nodes_cnt)); in proportion to
remaining memory, devided by nodes number on the host or 100 mb minimal)

9) print suggestions



2017-08-28 15:10 GMT+03:00 Dmitriy Setrakyan :

> Igor, can you please describe the algorithm with all the thresholds?
>
> On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor 
> wrote:
>
> > The suggestion here is based on initial settings, and it's so because
> there
> > is no other nodes on the host in the example.
> >
> > The algorithm tries to preserve the original ratio of memory policies
> > keeping numbers reasonable (for example after some thresshold it will
> > suggest not to use several memory policies if there is not enough of
> memory
> > for all of them) and taking into consideration nodes count on the host,
> > each jvm heap, needed memory for OS, etc
> >
> > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Looks good, but why in the example provided are we suggesting 8GB? 2
> > nodes
> > > with 8GB will completely exhaust the available memory. I would suggest
> 6
> > or
> > > 7GB.
> > >
> > > Also, why 100MB for default policy. Anything under 1GB seems too small.
> > >
> > > Can you please comment?
> > >
> > > D.
> > >
> > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <
> gvvinbl...@gmail.com>
> > > wrote:
> > >
> > > > One more example of possible warning:
> > > >
> > > > -
> > > > Excessive memory usage by Ignite node process (performance may drop)
> > > > [requested=44613MB, available=15942MB].
> > > >
> > > > Please tune the folowing settings as suggested:
> > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > > >   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > > >
> > > > Current settings:
> > > >   Java Heap  maxSize: 3543MB
> > > >   Java Heap initSize: 250MB
> > > >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > > >   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > > >   The overall expected memory usage by all Ignite nodes on the host:
> > > > 44613MB
> > > > -
> > > >
> > > > Your thoughts?
> > > >
> > > > 2017-08-28 5:06 GMT+03:00 Denis Magda :
> > > >
> > > > > Guys,
> > > > >
> > > > > ML lib profile is missing in 2.1 release! That must be fixed and
> > rolled
> > > > > out in this emergency release:
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > > >
> > > > > Oleg, Yuri, please step in and handle the issue.
> > > > >
> > > > > BTW, who is considered to be the release manager of this release?
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > I like the format proposed by Denis, very clear.
> > > > > >
> > > > > > However, I also do not understand why a user should change the
> size
> > > of
> > > > > some
> > > > > > system cache. How would a user ever know what value to put there?
> > > This
> > > > > > value should be configured by Ignite automatically.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda 
> > > > wrote:
> > > > > >
> > > > > >> Igor,
> > > > > >>
> > > > > >> Let me suggest this format.
> > > > > >>
> > > > > >> 

[jira] [Created] (IGNITE-6202) SQL: support hash join

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6202:
---

 Summary: SQL: support hash join
 Key: IGNITE-6202
 URL: https://issues.apache.org/jira/browse/IGNITE-6202
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov


Justification is the same as for IGNITE-6201 (merge joins). If join is 
equijoin, and participating attributes are not sorted in advance, then instead 
of executing nested loops, we can do the following:
1) Get the table with smaller number of entries
2) Build a hash table from target attribute to the list of matching entries
3) Iterate over larger table and perform lookup into hash table

Note that this operation might be memory-intensive in case smaller table is 
large enough still. For this reason we should provide a mechanism to avoid 
out-of-memory errors. Let's avoid spilling to disk in the first iteration, and 
try to set a kind of hard limit on how much data can be kept in memory. If we 
see that this limit cannot be satisfied, then fallback to nested loops.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: How a new index is built in runtime?

[dsetrakyan]

Vova, how hard is it to make it multi-threaded?

⁣D.​


On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov  
wrote:
>Denis,
>
>We iterate over the whole cache and build the index entry-by-entry.
>Control
>is returned back to the user when index is ready.
>
>On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov 
>wrote:
>
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted
>batches
>> in a single tree operation.
>>
>> --
>> Yakov Zhdanov
>>

[jira] [Created] (IGNITE-6201) SQL: support merge join

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6201:
---

 Summary: SQL: support merge join 
 Key: IGNITE-6201
 URL: https://issues.apache.org/jira/browse/IGNITE-6201
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov


It looks like currently the only way to join data is {{nested loops}} approach. 
While being perfectly fine for small tables, this approach is terribly 
inefficient for joins on large tables. We need to make sure that merge join 
technique is used instead if join is performed on sorted attributes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6200) org.dom4j.QName can't be serialized

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-6200:
-

 Summary: org.dom4j.QName can't be serialized
 Key: IGNITE-6200
 URL: https://issues.apache.org/jira/browse/IGNITE-6200
 Project: Ignite
  Issue Type: Bug
Reporter: Alexei Scherbakov


Exception:

{noformat}
class org.apache.ignite.binary.BinaryObjectException: Failed to marshal object 
with optimized marshaller: org.dom4j.QName@364492 [name: test namespace: 
"org.dom4j.Namespace@e20 [Namespace: prefix qq mapped to URI ""]"]
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:186)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at 
org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:248)
at 
org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82)
at 
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at 
org.apache.ignite.internal.processors.cache.MarshallerTest.test(MarshallerTest.java:160)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to serialize 
object: org.dom4j.QName@364492 [name: test namespace: "org.dom4j.Namespace@e20 
[Namespace: prefix qq mapped to URI ""]"]
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:206)
at 
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at 
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9836)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:179)
... 15 more
Caused by: java.io.IOException: java.io.NotActiveException: Not in 
writeObject() call.
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeSerializable(OptimizedObjectOutputStream.java:324)
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.write(OptimizedClassDescriptor.java:827)
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:224)
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObjectOverride(OptimizedObjectOutputStream.java:152)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:201)
... 18 more
Caused by: java.io.NotActiveException: Not in writeObject() call.
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.defaultWriteObject(OptimizedObjectOutputStream.java:684)
at org.dom4j.QName.writeObject(QName.java:239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at 
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeSerializable(OptimizedObjectOutputStream.java:318)
... 23 more
{noformat}

Reproducer:

{noformat}
public void test() throws Exception {
try {
IgniteEx ex = startGrid(0);

QName qName = new QName("test", new Namespace("qq", null), "q");

byte[] marshal = ex.configuration().getMarshaller().marshal(qName);
} finally {
stopAllGrids();
}
}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6199) SQL: DML processor should send deltas instead of full values when possible

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6199:
---

 Summary: SQL: DML processor should send deltas instead of full 
values when possible
 Key: IGNITE-6199
 URL: https://issues.apache.org/jira/browse/IGNITE-6199
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov


Currently we send full values during DML operations. Instead, we should find 
the way to send only changed attributes. Chances that it will require adding 
versions of existing entries, or even the whole MVCC, to prevent inconsistent 
updates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6198) SQL: Merge operation should group keys by affinity

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6198:
---

 Summary: SQL: Merge operation should group keys by affinity
 Key: IGNITE-6198
 URL: https://issues.apache.org/jira/browse/IGNITE-6198
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
 Fix For: 2.2


We implemented affinity-based grouping for updated batches for most DML [1]. 
But we forgot about {{MERGE}} - it use {{IgniteCache,putAll}} without proper 
grouping. Let's do that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6197) SQL: QueryIndex.setInlineSize should return this instead of void

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6197:
---

 Summary: SQL: QueryIndex.setInlineSize should return this instead 
of void
 Key: IGNITE-6197
 URL: https://issues.apache.org/jira/browse/IGNITE-6197
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
Assignee: Taras Ledkov
 Fix For: 2.2


Subj. We messed up with return type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6196) SQL: verify usability of template-based CREATE TABLE operations

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6196:
---

 Summary: SQL: verify usability of template-based CREATE TABLE 
operations
 Key: IGNITE-6196
 URL: https://issues.apache.org/jira/browse/IGNITE-6196
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
Assignee: Alexander Paschenko
 Fix For: 2.2


We do not have enough tests and examples on cache template usage for {{CREATE 
TABLE}}. I found only one test which used not template, but existing cache as a 
base for new cache. Let's add more tests with real pre-configured templates and 
extend our documentation accordingly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Ignite not friendly for Monitoring

[Vladimir Ozerov]

Dima,

Please see latest comments in the ticket [1]. There is special
specification called SQLSTATE governing what errors code are thrown from
SQL operations [2]. This is applicable to both JDBC and ODBC. Apart of from
standard code, database vendor can add it's own codes as a separate field,
or even extend error codes from the standard. However, as a first iteration
we should start respecting SQLSTATE spec without our own Ignite-specific
error codes.

[1] https://issues.apache.org/jira/browse/IGNITE-5620
[2]
https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/codes/src/tpc/db2z_sqlstatevalues.html#db2z_sqlstatevalues__code07

On Mon, Aug 28, 2017 at 3:23 PM, Dmitriy Setrakyan 
wrote:

> On Mon, Aug 28, 2017 at 1:22 AM, Vladimir Ozerov 
> wrote:
>
> > IGNITE-5620 is about error codes thrown from drivers. This is completely
> > different story, as every driver has specification with it's own specific
> > error codes. There is no common denominator.
> >
>
> Vova, I am not sure I understand. I would expect that drivers should
> provide the same SQL error codes as the underlying database. Perhaps,
> drivers have their custom codes for the errors in the driver itself, not in
> SQL.
>
> Can you please clarify?
>
>
> >
> > On Thu, Aug 17, 2017 at 11:10 PM, Denis Magda  wrote:
> >
> > > Vladimir,
> > >
> > > I would disagree. In IGNITE-5620 we’re going to introduce some constant
> > > error codes and prepare a sheet that will elaborate on every error.
> > That’s
> > > a part of bigger endeavor when the whole platform should be covered by
> > > special unique IDs for errors, warning and events.
> > >
> > > Now, we need to agree at least on the IDs range for SQL.
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 15, 2017, at 11:10 PM, Vladimir Ozerov 
> > > wrote:
> > > >
> > > > Denis,
> > > >
> > > > IGNITE-5620 is completely different thing. Let's do not mix cluster
> > > > monitoring and parser errors.
> > > >
> > > > ср, 16 авг. 2017 г. в 2:57, Denis Magda :
> > > >
> > > >> Alexey,
> > > >>
> > > >> Didn’t know that such an improvement as consistent IDs for errors
> and
> > > >> events can be used as an integration point with the DevOps tools.
> > Thanks
> > > >> for sharing your experience with us.
> > > >>
> > > >> Would you step in as a architect for this task and make out a JIRA
> > > ticket
> > > >> with all the required information.
> > > >>
> > > >> In general, we’ve already planned to do something around this
> starting
> > > >> with SQL:
> > > >> https://issues.apache.org/jira/browse/IGNITE-5620 <
> > > >> https://issues.apache.org/jira/browse/IGNITE-5620>
> > > >>
> > > >> It makes sense to consider your input before the work on IGNITE-5620
> > is
> > > >> started.
> > > >>
> > > >> —
> > > >> Denis
> > > >>
> > > >>> On Aug 15, 2017, at 10:56 AM, Alexey Kukushkin <
> > > >> alexeykukush...@yahoo.com.INVALID> wrote:
> > > >>>
> > > >>> Hi Alexey,
> > > >>> A nice thing about delegating alerting to 3rd party enterprise
> > systems
> > > >> is that those systems already deal with lots of things including
> > > >> distributed apps.
> > > >>> What is needed from Ignite is to consistently write to log files
> > (again
> > > >> that means stable event IDs, proper event granularity, no
> repetition,
> > > >> documentation). This would be 3rd party monitoring system's
> > > responsibility
> > > >> to monitor log files on all nodes, filter, aggregate, process,
> > visualize
> > > >> and notify on events.
> > > >>> How a monitoring tool would deal with an event like "node left":
> > > >>> The only thing needed from Ignite is to write an entry like below
> to
> > > log
> > > >> files on all Ignite servers. In this example 3300 identifies this
> > "node
> > > >> left" event and will never change in the future even if text
> > description
> > > >> changes:
> > > >>> [2017-09-01 10:00:14] [WARN] 3300 Node DF2345F-XCVDS4-34ETJH left
> the
> > > >> cluster
> > > >>> Then we document somewhere on the web that Ignite has event 3300
> and
> > it
> > > >> means a node left the cluster. Maybe provide documentation how to
> deal
> > > with
> > > >> it. Some examples:Oracle Web Cache events:
> > > >> https://docs.oracle.com/cd/B14099_19/caching.1012/b14046/
> > > event.htm#sthref2393MS
> > > >> SQL Server events:
> > > >> https://msdn.microsoft.com/en-us/library/cc645603(v=sql.105).aspx
> > > >>> That is all for Ignite! Everything else is handled by specific
> > > >> monitoring system configured by DevOps on the customer side.
> > > >>> Basing on the Ignite documentation similar to above, DevOps of a
> > > company
> > > >> where Ignite is going to be used will configure their monitoring
> > system
> > > to
> > > >> understand Ignite events. Consider the "node left" event as an
> > example.
> > > >>> - This event is output on every node but DevOps do not want to be
> > > >> notified many times. To address this, 

Re: Resurrect FairAffinityFunction

[Vladimir Ozerov]

Igniters,

I created a ticket for SQL exception in case of invalid combination of
affinity functions are used in query with JOINs:
https://issues.apache.org/jira/browse/IGNITE-6195

On Wed, Aug 16, 2017 at 11:59 AM, Vladimir Ozerov 
wrote:

> I am still not quite understand why we bother so much about partition
> migration while we already have RendezvousAffinityFunction? Looks like we
> tried to make "fair" be closer to "rendezvous", and then tried to make
> "rendezvous" closer to "fair", but without any success:
> - we are still not satisfied with distribution of "rendezvous", so we want
> to return "fair" back
> - and "fair" cannot co-locate caches properly, so things like SQL and
> "affinityCall" are broken for it
>
> Can we add a flag to FairAffinityFunction to control whether it should
> previous history or not? Is yes - it is not deterministic, if no - it is
> deterministic.
>
>
> On Wed, Aug 16, 2017 at 11:29 AM, Vladimir Ozerov 
> wrote:
>
>> I already explained why checking specifically for FairAffintiyFunction in
>> SQL is wrong.
>>
>> On Wed, Aug 16, 2017 at 10:00 AM, Dmitriy Setrakyan <
>> dsetrak...@apache.org> wrote:
>>
>>> Vladimir, I think you should look more carefully at what FairAffinity
>>> does
>>> and why it exists. There is no way to "fix" it not to accept previous
>>> state
>>> - it must be provided.
>>>
>>> I do not understand why we need to change anything in code when there is
>>> a
>>> simple solution being proposed here. Why not just add a validation to SQL
>>> when fair affinity is used with JOINs and be done with it?
>>>
>>> Alexey G, given that you originally wrote the FairAffinity function, can
>>> you please comment?
>>>
>>> D.
>>>
>>> On Tue, Aug 15, 2017 at 10:41 PM, Vladimir Ozerov 
>>> wrote:
>>>
>>> > Dima,
>>> >
>>> > It is not hard to implement. It is hard to reason on whether your query
>>> > will fail or not. Moreover, cache groups is an antipattern for SQL,
>>> > personally I do not want users to use it unless absolutely needed
>>> (large
>>> > topologies, large number of caches). Also take in count that the same
>>> > problem with different partition distribution holds for any two caches
>>> with
>>> > different affinity functions, so the problem is not tied to
>>> > FairAffinityFunction only.
>>> >
>>> > IMO correct fix should be as follows:
>>> > 1) Add requirement that every affinity function must provide sensible
>>> > implementation of equals/hashCode
>>> > 2) Add "boolean deterministic()" method to affinity function interface;
>>> > "true" means that function doesn't depend on any external things, such
>>> as
>>> > topology history
>>> > 3) Fail SQL if there are at least two PARTITIONED caches with
>>> different or
>>> > non-deterministic affinity functions, and distributed joins are not
>>> enabled
>>> > 4) Fix FairAffinityFunction and make it deterministic, it should not
>>> depend
>>> > on previous state.
>>> >
>>> > Makes sense?
>>> >
>>> > On Wed, Aug 16, 2017 at 3:31 AM, Dmitriy Setrakyan <
>>> dsetrak...@apache.org>
>>> > wrote:
>>> >
>>> > > On Tue, Aug 15, 2017 at 1:12 PM, Vladimir Ozerov <
>>> voze...@gridgain.com>
>>> > > wrote:
>>> > >
>>> > > > I do not like the idea as it would make it very hard to reason
>>> about
>>> > > > whether your SQL will fail or not. Let's looks at the problem from
>>> the
>>> > > > different angle. I have this question for years - why in the world
>>> > *fair*
>>> > > > affinity function, whose only ultimate goal is to provide equal
>>> > partition
>>> > > > distribution, depends on it's own previous state? Can we re-design
>>> in a
>>> > > way
>>> > > > that it depends only on partition count and current topology state?
>>> > > >
>>> > >
>>> > > Vladimir, we must know previous state, otherwise the data partitions
>>> will
>>> > > be randomly moving across the network every time a topology changes.
>>> > >
>>> > > From the SQL standpoint, you can just fail all queries that have a
>>> JOIN
>>> > > from different cache groups, if at least one of the groups is using
>>> Fair
>>> > > Affinity. I am not sure why this would be hard.
>>> > >
>>> > >
>>> > > >
>>> > > > On Thu, Aug 10, 2017 at 12:16 AM, Valentin Kulichenko <
>>> > > > valentin.kuliche...@gmail.com> wrote:
>>> > > >
>>> > > > > As far as I know, all logical caches with the same affinity
>>> function
>>> > > and
>>> > > > > node filter will end up in the same group. If that's the case, I
>>> like
>>> > > the
>>> > > > > idea. This is exactly what I was looking for.
>>> > > > >
>>> > > > > -Val
>>> > > > >
>>> > > > > On Wed, Aug 9, 2017 at 8:18 AM, Evgenii Zhuravlev <
>>> > > > > e.zhuravlev...@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Dmitriy,
>>> > > > > >
>>> > > > > > Yes, you're right. Moreover, it looks like a good practice to
>>> > combine
>>> > > > > > caches that will be used for collocated JOINs in one group
>>> since it
>>> > > > > reduces
>>> > > > > > overall 

[jira] [Created] (IGNITE-6195) SQL: Do not allow JOINs on caches with different affinity functions

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6195:
---

 Summary: SQL: Do not allow JOINs on caches with different affinity 
functions
 Key: IGNITE-6195
 URL: https://issues.apache.org/jira/browse/IGNITE-6195
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
 Fix For: 2.2


Currently it is possible to execute JOIN on non-colocated caches. No exceptions 
will appear, user just receive incorrect result. We need to detect such 
situations and throw errors instead.

*Proposed solution*
Correct SQL result is possible when either distributed joins are enabled, or 
data is co-located properly. Under *proper* co-location we mean:
1) Participating {{PARTITIONED}} caches use the same affinity function
2) This affinity function doesn't depend on it's own previous state, i.e. it 
doesn't rely on {{AffinityFunctionContext.previousAssignment}}. For instance, 
{{RendezvousAffinityFunction}} doesn't use, while {{FairAffinityFunction}} does.

As such, the following procedure should be implemented in order to determine 
whether SQL can be executed:
1) If {{distributedJoins}} are enabled - return, SQL can be executed
2) Get the list of participating caches
3) Exclude {{REPLICATED}} caches from that list
4) If all remaining caches belong to the same cache group - return, SQL can be 
executed
5) Get affinity function of the first cache
6) Check if affinity function doesn't use 
{{AffinityFunctionContext.previousAssignment}}. This could be controlled either 
through annotation, or through new method on {{AffinityFunction}} interface, 
e.g. {{boolean isDependOnPreviousState}}. If {{false}} - throw an exception
7) Check if affinity functions of all caches are equal through standard 
{{equals()}} method. If {{false}} - throw an exception.
8) Otherwise - SQL can be executed safely.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Ignite not friendly for Monitoring

[Dmitriy Setrakyan]

On Mon, Aug 28, 2017 at 1:22 AM, Vladimir Ozerov 
wrote:

> IGNITE-5620 is about error codes thrown from drivers. This is completely
> different story, as every driver has specification with it's own specific
> error codes. There is no common denominator.
>

Vova, I am not sure I understand. I would expect that drivers should
provide the same SQL error codes as the underlying database. Perhaps,
drivers have their custom codes for the errors in the driver itself, not in
SQL.

Can you please clarify?


>
> On Thu, Aug 17, 2017 at 11:10 PM, Denis Magda  wrote:
>
> > Vladimir,
> >
> > I would disagree. In IGNITE-5620 we’re going to introduce some constant
> > error codes and prepare a sheet that will elaborate on every error.
> That’s
> > a part of bigger endeavor when the whole platform should be covered by
> > special unique IDs for errors, warning and events.
> >
> > Now, we need to agree at least on the IDs range for SQL.
> >
> > —
> > Denis
> >
> > > On Aug 15, 2017, at 11:10 PM, Vladimir Ozerov 
> > wrote:
> > >
> > > Denis,
> > >
> > > IGNITE-5620 is completely different thing. Let's do not mix cluster
> > > monitoring and parser errors.
> > >
> > > ср, 16 авг. 2017 г. в 2:57, Denis Magda :
> > >
> > >> Alexey,
> > >>
> > >> Didn’t know that such an improvement as consistent IDs for errors and
> > >> events can be used as an integration point with the DevOps tools.
> Thanks
> > >> for sharing your experience with us.
> > >>
> > >> Would you step in as a architect for this task and make out a JIRA
> > ticket
> > >> with all the required information.
> > >>
> > >> In general, we’ve already planned to do something around this starting
> > >> with SQL:
> > >> https://issues.apache.org/jira/browse/IGNITE-5620 <
> > >> https://issues.apache.org/jira/browse/IGNITE-5620>
> > >>
> > >> It makes sense to consider your input before the work on IGNITE-5620
> is
> > >> started.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Aug 15, 2017, at 10:56 AM, Alexey Kukushkin <
> > >> alexeykukush...@yahoo.com.INVALID> wrote:
> > >>>
> > >>> Hi Alexey,
> > >>> A nice thing about delegating alerting to 3rd party enterprise
> systems
> > >> is that those systems already deal with lots of things including
> > >> distributed apps.
> > >>> What is needed from Ignite is to consistently write to log files
> (again
> > >> that means stable event IDs, proper event granularity, no repetition,
> > >> documentation). This would be 3rd party monitoring system's
> > responsibility
> > >> to monitor log files on all nodes, filter, aggregate, process,
> visualize
> > >> and notify on events.
> > >>> How a monitoring tool would deal with an event like "node left":
> > >>> The only thing needed from Ignite is to write an entry like below to
> > log
> > >> files on all Ignite servers. In this example 3300 identifies this
> "node
> > >> left" event and will never change in the future even if text
> description
> > >> changes:
> > >>> [2017-09-01 10:00:14] [WARN] 3300 Node DF2345F-XCVDS4-34ETJH left the
> > >> cluster
> > >>> Then we document somewhere on the web that Ignite has event 3300 and
> it
> > >> means a node left the cluster. Maybe provide documentation how to deal
> > with
> > >> it. Some examples:Oracle Web Cache events:
> > >> https://docs.oracle.com/cd/B14099_19/caching.1012/b14046/
> > event.htm#sthref2393MS
> > >> SQL Server events:
> > >> https://msdn.microsoft.com/en-us/library/cc645603(v=sql.105).aspx
> > >>> That is all for Ignite! Everything else is handled by specific
> > >> monitoring system configured by DevOps on the customer side.
> > >>> Basing on the Ignite documentation similar to above, DevOps of a
> > company
> > >> where Ignite is going to be used will configure their monitoring
> system
> > to
> > >> understand Ignite events. Consider the "node left" event as an
> example.
> > >>> - This event is output on every node but DevOps do not want to be
> > >> notified many times. To address this, they will build an "Ignite
> model"
> > >> where there will be a parent-child dependency between components
> "Ignite
> > >> Cluster" and "Ignite Node". For example, this is how you do it in
> > Nagios:
> > >> https://assets.nagios.com/downloads/nagioscore/docs/
> > nagioscore/4/en/dependencies.html
> > >> and this is how you do it in Microsoft SCSM:
> > >> https://docs.microsoft.com/en-us/system-center/scsm/auth-classes.
> Then
> > >> DevOps will configure "node left" monitors in SCSM (or a "checks" in
> > >> Nagios) for parent "Ignite Cluster" and child "Ignite Service"
> > components.
> > >> State change (OK -> WARNING) and notification (email, SMS, whatever)
> > will
> > >> be configured only for the "Ignite Cluster"'s "node left" monitor.-
> Now
> > >> suppose a node left. The "node left" monitor (that uses log file
> > monitoring
> > >> plugin) on "Ignite Node" will detect the event and pass it to the
> > parent.
> > >> This will trigger 

[GitHub] ignite pull request #1650: IGNITE-3592 Provide some kind of pluggable compre...

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1951: IGNITE-5226 Annotated fields compression

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1951


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] Urgent Ignite bug fix release

[Dmitriy Setrakyan]

Igor, can you please describe the algorithm with all the thresholds?

On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor 
wrote:

> The suggestion here is based on initial settings, and it's so because there
> is no other nodes on the host in the example.
>
> The algorithm tries to preserve the original ratio of memory policies
> keeping numbers reasonable (for example after some thresshold it will
> suggest not to use several memory policies if there is not enough of memory
> for all of them) and taking into consideration nodes count on the host,
> each jvm heap, needed memory for OS, etc
>
> 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :
>
> > Looks good, but why in the example provided are we suggesting 8GB? 2
> nodes
> > with 8GB will completely exhaust the available memory. I would suggest 6
> or
> > 7GB.
> >
> > Also, why 100MB for default policy. Anything under 1GB seems too small.
> >
> > Can you please comment?
> >
> > D.
> >
> > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor 
> > wrote:
> >
> > > One more example of possible warning:
> > >
> > > -
> > > Excessive memory usage by Ignite node process (performance may drop)
> > > [requested=44613MB, available=15942MB].
> > >
> > > Please tune the folowing settings as suggested:
> > >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> > >   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> > >
> > > Current settings:
> > >   Java Heap  maxSize: 3543MB
> > >   Java Heap initSize: 250MB
> > >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> > >   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> > >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> > >   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> > >   The overall expected memory usage by all Ignite nodes on the host:
> > > 44613MB
> > > -
> > >
> > > Your thoughts?
> > >
> > > 2017-08-28 5:06 GMT+03:00 Denis Magda :
> > >
> > > > Guys,
> > > >
> > > > ML lib profile is missing in 2.1 release! That must be fixed and
> rolled
> > > > out in this emergency release:
> > > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > > >
> > > > Oleg, Yuri, please step in and handle the issue.
> > > >
> > > > BTW, who is considered to be the release manager of this release?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > > wrote:
> > > > >
> > > > > I like the format proposed by Denis, very clear.
> > > > >
> > > > > However, I also do not understand why a user should change the size
> > of
> > > > some
> > > > > system cache. How would a user ever know what value to put there?
> > This
> > > > > value should be configured by Ignite automatically.
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda 
> > > wrote:
> > > > >
> > > > >> Igor,
> > > > >>
> > > > >> Let me suggest this format.
> > > > >>
> > > > >> -
> > > > >> Excessive memory usage by Ignite node process (performance may
> drop)
> > > > >> [requested=29251MB, available=15942MB]
> > > > >>
> > > > >> Please tune the following settings:
> > > > >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > > > >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> > > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1} =
> suggested
> > > > >> value]
> > > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2} =
> suggested
> > > > >> value]
> > > > >>
> > > > >> Current settings:
> > > > >>   [DefaultMemoryPolicySize = value]
> > > > >>   [{policy_name_1} size = value]
> > > > >>   [{policy_name_1} size = value]
> > > > >>   SystemCacheInitialSize = value
> > > > >>   SystemCacheMaxSize = value
> > > > >>   Java Heap Init Size = value
> > > > >>   Java Heap Max Size = value
> > > > >>
> > > > >> The overall memory usage by all Ignite nodes on the host: value
> > > > >> ---
> > > > >>
> > > > >> Records in […] are optional. If custom memory policy is not set or
> > the
> > > > >> default memory policy is overridden the output will miss some of
> the
> > > > rows.
> > > > >>
> > > > >> As for systemCacheMaxSize, it should be show ONLY if the parameter
> > was
> > > > set
> > > > >> explicitly by user code. Otherwise, the platform should be wise
> > enough
> > > > to
> > > > >> instantiate it properly depending on the host memory usage.
> > > > >>
> > > > >> —
> > > > >> Denis
> > > > >>
> > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> > 

[jira] [Created] (IGNITE-6194) SQL: implement lazy query execution for SQL with multiple map queries

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6194:
---

 Summary: SQL: implement lazy query execution for SQL with multiple 
map queries
 Key: IGNITE-6194
 URL: https://issues.apache.org/jira/browse/IGNITE-6194
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
 Fix For: 2.2


Currently we do not support lazy query execution for cases when there are 
several map queries, e.g. in case of {{UNION}}. Need to support this case as 
well. Specifically, parallel queries should be assigned to different laze 
worker threads.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

The suggestion here is based on initial settings, and it's so because there
is no other nodes on the host in the example.

The algorithm tries to preserve the original ratio of memory policies
keeping numbers reasonable (for example after some thresshold it will
suggest not to use several memory policies if there is not enough of memory
for all of them) and taking into consideration nodes count on the host,
each jvm heap, needed memory for OS, etc

2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan :

> Looks good, but why in the example provided are we suggesting 8GB? 2 nodes
> with 8GB will completely exhaust the available memory. I would suggest 6 or
> 7GB.
>
> Also, why 100MB for default policy. Anything under 1GB seems too small.
>
> Can you please comment?
>
> D.
>
> On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor 
> wrote:
>
> > One more example of possible warning:
> >
> > -
> > Excessive memory usage by Ignite node process (performance may drop)
> > [requested=44613MB, available=15942MB].
> >
> > Please tune the folowing settings as suggested:
> >   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
> >   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
> >   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
> >   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
> >
> > Current settings:
> >   Java Heap  maxSize: 3543MB
> >   Java Heap initSize: 250MB
> >   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
> >   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
> >   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
> >   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
> >   The overall expected memory usage by all Ignite nodes on the host:
> > 44613MB
> > -
> >
> > Your thoughts?
> >
> > 2017-08-28 5:06 GMT+03:00 Denis Magda :
> >
> > > Guys,
> > >
> > > ML lib profile is missing in 2.1 release! That must be fixed and rolled
> > > out in this emergency release:
> > > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > > https://issues.apache.org/jira/browse/IGNITE-6193>
> > >
> > > Oleg, Yuri, please step in and handle the issue.
> > >
> > > BTW, who is considered to be the release manager of this release?
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > > wrote:
> > > >
> > > > I like the format proposed by Denis, very clear.
> > > >
> > > > However, I also do not understand why a user should change the size
> of
> > > some
> > > > system cache. How would a user ever know what value to put there?
> This
> > > > value should be configured by Ignite automatically.
> > > >
> > > > D.
> > > >
> > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda 
> > wrote:
> > > >
> > > >> Igor,
> > > >>
> > > >> Let me suggest this format.
> > > >>
> > > >> -
> > > >> Excessive memory usage by Ignite node process (performance may drop)
> > > >> [requested=29251MB, available=15942MB]
> > > >>
> > > >> Please tune the following settings:
> > > >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > > >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> > > >> value]
> > > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> > > >> value]
> > > >>
> > > >> Current settings:
> > > >>   [DefaultMemoryPolicySize = value]
> > > >>   [{policy_name_1} size = value]
> > > >>   [{policy_name_1} size = value]
> > > >>   SystemCacheInitialSize = value
> > > >>   SystemCacheMaxSize = value
> > > >>   Java Heap Init Size = value
> > > >>   Java Heap Max Size = value
> > > >>
> > > >> The overall memory usage by all Ignite nodes on the host: value
> > > >> ---
> > > >>
> > > >> Records in […] are optional. If custom memory policy is not set or
> the
> > > >> default memory policy is overridden the output will miss some of the
> > > rows.
> > > >>
> > > >> As for systemCacheMaxSize, it should be show ONLY if the parameter
> was
> > > set
> > > >> explicitly by user code. Otherwise, the platform should be wise
> enough
> > > to
> > > >> instantiate it properly depending on the host memory usage.
> > > >>
> > > >> —
> > > >> Denis
> > > >>
> > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor <
> gvvinbl...@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>> The message without logging layout:
> > > >>>
> > > >>> Not enough memory for current process [required=29251MB,
> > > >> available=15942MB].
> > > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> > > allocated
> > > >>> for each node.
> > > >>>
> > > >>> Current settings:
> > > >>>  

Re: [DISCUSSION] Urgent Ignite bug fix release

[Dmitriy Setrakyan]

Looks good, but why in the example provided are we suggesting 8GB? 2 nodes
with 8GB will completely exhaust the available memory. I would suggest 6 or
7GB.

Also, why 100MB for default policy. Anything under 1GB seems too small.

Can you please comment?

D.

On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor 
wrote:

> One more example of possible warning:
>
> -
> Excessive memory usage by Ignite node process (performance may drop)
> [requested=44613MB, available=15942MB].
>
> Please tune the folowing settings as suggested:
>   MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
>   MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
>   MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
>   MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB
>
> Current settings:
>   Java Heap  maxSize: 3543MB
>   Java Heap initSize: 250MB
>   MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
>   MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
>   MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
>   MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
>   The overall expected memory usage by all Ignite nodes on the host:
> 44613MB
> -
>
> Your thoughts?
>
> 2017-08-28 5:06 GMT+03:00 Denis Magda :
>
> > Guys,
> >
> > ML lib profile is missing in 2.1 release! That must be fixed and rolled
> > out in this emergency release:
> > https://issues.apache.org/jira/browse/IGNITE-6193 <
> > https://issues.apache.org/jira/browse/IGNITE-6193>
> >
> > Oleg, Yuri, please step in and handle the issue.
> >
> > BTW, who is considered to be the release manager of this release?
> >
> > —
> > Denis
> >
> > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan 
> > wrote:
> > >
> > > I like the format proposed by Denis, very clear.
> > >
> > > However, I also do not understand why a user should change the size of
> > some
> > > system cache. How would a user ever know what value to put there? This
> > > value should be configured by Ignite automatically.
> > >
> > > D.
> > >
> > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda 
> wrote:
> > >
> > >> Igor,
> > >>
> > >> Let me suggest this format.
> > >>
> > >> -
> > >> Excessive memory usage by Ignite node process (performance may drop)
> > >> [requested=29251MB, available=15942MB]
> > >>
> > >> Please tune the following settings:
> > >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> > >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> > >> value]
> > >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> > >> value]
> > >>
> > >> Current settings:
> > >>   [DefaultMemoryPolicySize = value]
> > >>   [{policy_name_1} size = value]
> > >>   [{policy_name_1} size = value]
> > >>   SystemCacheInitialSize = value
> > >>   SystemCacheMaxSize = value
> > >>   Java Heap Init Size = value
> > >>   Java Heap Max Size = value
> > >>
> > >> The overall memory usage by all Ignite nodes on the host: value
> > >> ---
> > >>
> > >> Records in […] are optional. If custom memory policy is not set or the
> > >> default memory policy is overridden the output will miss some of the
> > rows.
> > >>
> > >> As for systemCacheMaxSize, it should be show ONLY if the parameter was
> > set
> > >> explicitly by user code. Otherwise, the platform should be wise enough
> > to
> > >> instantiate it properly depending on the host memory usage.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor 
> > >> wrote:
> > >>>
> > >>> The message without logging layout:
> > >>>
> > >>> Not enough memory for current process [required=29251MB,
> > >> available=15942MB].
> > >>> Please change MemoryConfiguration.systemCacheMaxSize and
> > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> > allocated
> > >>> for each node.
> > >>>
> > >>> Current settings:
> > >>>  HeapInit=250MB
> > >>>  HeapMax=3543MB
> > >>>  DefaultMemoryPolicySize=12753MB
> > >>>  SystemCacheInitialSize=40MB
> > >>>  SystemCacheMaxSize=100MB
> > >>>
> > >>> Other ignite instanses on the server require: 12853MB
> > >>>
> > >>> I think it's make sense to describe what these numbers are consist
> of.
> > >>>
> > >>> We simple say which parameters have an impact on how much memory the
> > >>> instance needs and their (parameters) actual values.
> > >>>
> > >>> Also we notice that more than one Ignite instance are ran on the
> server
> > >> or
> > >>> workstation and it also consumes memory.
> > >>>
> > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > >>> dsetrak...@apache.org> написал:
> > >>>
> >  Igor, what is this flood of WARN messaging coming 

[GitHub] ignite pull request #2527: IGNITE-5814 Remove non Ignite classes from marsha...

[ezhuravl]

GitHub user ezhuravl opened a pull request:

https://github.com/apache/ignite/pull/2527

IGNITE-5814 Remove non Ignite classes from marshaller exclusions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5814

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2527.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2527


commit fdb15c144c94e127061fcd9e5b6289561459c501
Author: Evgenii Zhuravlev 
Date:   2017-08-28T11:16:56Z

IGNITE-5814 Remove non Ignite classes from marshaller exclusions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2526: IGNITE-6184 Add checkClusterState() call to destr...

[alamar]

GitHub user alamar opened a pull request:

https://github.com/apache/ignite/pull/2526

IGNITE-6184 Add checkClusterState() call to destroyCachesAsync() and …

…getOrCreateCaches()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6184

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2526.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2526


commit bd8c6e8cd43810ff51f4ffd70ec21240b488131d
Author: Ilya Kasnacheev 
Date:   2017-08-28T10:33:21Z

IGNITE-6184 Add checkClusterState() call to destroyCachesAsync() and 
getOrCreateCaches()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

One more example of possible warning:

-
Excessive memory usage by Ignite node process (performance may drop)
[requested=44613MB, available=15942MB].

Please tune the folowing settings as suggested:
  MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB
  MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB
  MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB
  MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB

Current settings:
  Java Heap  maxSize: 3543MB
  Java Heap initSize: 250MB
  MemoryPolicyConfiguration.initialSize for bigPlc: 256MB
  MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB
  MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB
  MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB
  The overall expected memory usage by all Ignite nodes on the host: 44613MB
-

Your thoughts?

2017-08-28 5:06 GMT+03:00 Denis Magda :

> Guys,
>
> ML lib profile is missing in 2.1 release! That must be fixed and rolled
> out in this emergency release:
> https://issues.apache.org/jira/browse/IGNITE-6193 <
> https://issues.apache.org/jira/browse/IGNITE-6193>
>
> Oleg, Yuri, please step in and handle the issue.
>
> BTW, who is considered to be the release manager of this release?
>
> —
> Denis
>
> > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan 
> wrote:
> >
> > I like the format proposed by Denis, very clear.
> >
> > However, I also do not understand why a user should change the size of
> some
> > system cache. How would a user ever know what value to put there? This
> > value should be configured by Ignite automatically.
> >
> > D.
> >
> > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda  wrote:
> >
> >> Igor,
> >>
> >> Let me suggest this format.
> >>
> >> -
> >> Excessive memory usage by Ignite node process (performance may drop)
> >> [requested=29251MB, available=15942MB]
> >>
> >> Please tune the following settings:
> >>  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
> >>  MemoryConfiguration.systemCacheMaxSize = suggested value
> >>  [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> >> value]
> >>  [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> >> value]
> >>
> >> Current settings:
> >>   [DefaultMemoryPolicySize = value]
> >>   [{policy_name_1} size = value]
> >>   [{policy_name_1} size = value]
> >>   SystemCacheInitialSize = value
> >>   SystemCacheMaxSize = value
> >>   Java Heap Init Size = value
> >>   Java Heap Max Size = value
> >>
> >> The overall memory usage by all Ignite nodes on the host: value
> >> ---
> >>
> >> Records in […] are optional. If custom memory policy is not set or the
> >> default memory policy is overridden the output will miss some of the
> rows.
> >>
> >> As for systemCacheMaxSize, it should be show ONLY if the parameter was
> set
> >> explicitly by user code. Otherwise, the platform should be wise enough
> to
> >> instantiate it properly depending on the host memory usage.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor 
> >> wrote:
> >>>
> >>> The message without logging layout:
> >>>
> >>> Not enough memory for current process [required=29251MB,
> >> available=15942MB].
> >>> Please change MemoryConfiguration.systemCacheMaxSize and
> >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory
> allocated
> >>> for each node.
> >>>
> >>> Current settings:
> >>>  HeapInit=250MB
> >>>  HeapMax=3543MB
> >>>  DefaultMemoryPolicySize=12753MB
> >>>  SystemCacheInitialSize=40MB
> >>>  SystemCacheMaxSize=100MB
> >>>
> >>> Other ignite instanses on the server require: 12853MB
> >>>
> >>> I think it's make sense to describe what these numbers are consist of.
> >>>
> >>> We simple say which parameters have an impact on how much memory the
> >>> instance needs and their (parameters) actual values.
> >>>
> >>> Also we notice that more than one Ignite instance are ran on the server
> >> or
> >>> workstation and it also consumes memory.
> >>>
> >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> >>> dsetrak...@apache.org> написал:
> >>>
>  Igor, what is this flood of WARN messaging coming after the text? Are
> we
>  really going to print this whole thing out?
> 
>  On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor <
> gvvinbl...@gmail.com
> >>>
>  wrote:
> 
> > This message appears on topology change in case the available memory
> is
> > exceeded
> >
> > 2017-08-25 19:47 GMT+03:00 Seliverstov Igor :
> >
> >> An example of current impl:
> >>
> >>
> >> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> >> GridHomePathSelfTest0%][GridDiscoveryManager]
> >> [2017-08-25 

[GitHub] ignite pull request #2525: IGNITE-5425 JDBC thin: don't throw unsupported ex...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/2525

IGNITE-5425 JDBC thin: don't throw unsupported exception on ClientInf…

…o methods

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5425

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2525.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2525


commit c9e48fb11bbed66e183a8dd9a9474d8a66eabdb5
Author: tledkov-gridgain 
Date:   2017-08-28T09:14:26Z

IGNITE-5425 JDBC thin: don't throw unsupported exception on ClientInfo 
methods




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2524: Ignite 5714

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2524

Ignite 5714



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite ignite-5714

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2524.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2524


commit 4c811c5eff82052edf9d25651d3cd4db493799db
Author: voipp 
Date:   2017-08-25T14:21:23Z

ignite-5714 removed threadId

commit 2250a2e81d8530321b868dd8d09c0c1875d737f2
Author: voipp 
Date:   2017-08-28T09:11:54Z

ignite-5714 removed thredId from lock request




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2184: IGNITE-5425 JDBC thin: support client info

[tledkov-gridgain]

Github user tledkov-gridgain closed the pull request at:

https://github.com/apache/ignite/pull/2184


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2523: IGNITE-6115: Ignore page eviction mode if Ignite ...

[shroman]

GitHub user shroman opened a pull request:

https://github.com/apache/ignite/pull/2523

IGNITE-6115: Ignore page eviction mode if Ignite persistence is enabled.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shroman/ignite IGNITE-6115

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2523.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2523


commit 15036f94a849fc9b47a0fe5cd000b09c48ab794a
Author: shroman 
Date:   2017-08-28T08:59:20Z

IGNITE-6115: Ignore page eviction mode if Ignite persistence is enabled.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ignite not friendly for Monitoring

[Vladimir Ozerov]

IGNITE-5620 is about error codes thrown from drivers. This is completely
different story, as every driver has specification with it's own specific
error codes. There is no common denominator.

On Thu, Aug 17, 2017 at 11:10 PM, Denis Magda  wrote:

> Vladimir,
>
> I would disagree. In IGNITE-5620 we’re going to introduce some constant
> error codes and prepare a sheet that will elaborate on every error. That’s
> a part of bigger endeavor when the whole platform should be covered by
> special unique IDs for errors, warning and events.
>
> Now, we need to agree at least on the IDs range for SQL.
>
> —
> Denis
>
> > On Aug 15, 2017, at 11:10 PM, Vladimir Ozerov 
> wrote:
> >
> > Denis,
> >
> > IGNITE-5620 is completely different thing. Let's do not mix cluster
> > monitoring and parser errors.
> >
> > ср, 16 авг. 2017 г. в 2:57, Denis Magda :
> >
> >> Alexey,
> >>
> >> Didn’t know that such an improvement as consistent IDs for errors and
> >> events can be used as an integration point with the DevOps tools. Thanks
> >> for sharing your experience with us.
> >>
> >> Would you step in as a architect for this task and make out a JIRA
> ticket
> >> with all the required information.
> >>
> >> In general, we’ve already planned to do something around this starting
> >> with SQL:
> >> https://issues.apache.org/jira/browse/IGNITE-5620 <
> >> https://issues.apache.org/jira/browse/IGNITE-5620>
> >>
> >> It makes sense to consider your input before the work on IGNITE-5620 is
> >> started.
> >>
> >> —
> >> Denis
> >>
> >>> On Aug 15, 2017, at 10:56 AM, Alexey Kukushkin <
> >> alexeykukush...@yahoo.com.INVALID> wrote:
> >>>
> >>> Hi Alexey,
> >>> A nice thing about delegating alerting to 3rd party enterprise systems
> >> is that those systems already deal with lots of things including
> >> distributed apps.
> >>> What is needed from Ignite is to consistently write to log files (again
> >> that means stable event IDs, proper event granularity, no repetition,
> >> documentation). This would be 3rd party monitoring system's
> responsibility
> >> to monitor log files on all nodes, filter, aggregate, process, visualize
> >> and notify on events.
> >>> How a monitoring tool would deal with an event like "node left":
> >>> The only thing needed from Ignite is to write an entry like below to
> log
> >> files on all Ignite servers. In this example 3300 identifies this "node
> >> left" event and will never change in the future even if text description
> >> changes:
> >>> [2017-09-01 10:00:14] [WARN] 3300 Node DF2345F-XCVDS4-34ETJH left the
> >> cluster
> >>> Then we document somewhere on the web that Ignite has event 3300 and it
> >> means a node left the cluster. Maybe provide documentation how to deal
> with
> >> it. Some examples:Oracle Web Cache events:
> >> https://docs.oracle.com/cd/B14099_19/caching.1012/b14046/
> event.htm#sthref2393MS
> >> SQL Server events:
> >> https://msdn.microsoft.com/en-us/library/cc645603(v=sql.105).aspx
> >>> That is all for Ignite! Everything else is handled by specific
> >> monitoring system configured by DevOps on the customer side.
> >>> Basing on the Ignite documentation similar to above, DevOps of a
> company
> >> where Ignite is going to be used will configure their monitoring system
> to
> >> understand Ignite events. Consider the "node left" event as an example.
> >>> - This event is output on every node but DevOps do not want to be
> >> notified many times. To address this, they will build an "Ignite model"
> >> where there will be a parent-child dependency between components "Ignite
> >> Cluster" and "Ignite Node". For example, this is how you do it in
> Nagios:
> >> https://assets.nagios.com/downloads/nagioscore/docs/
> nagioscore/4/en/dependencies.html
> >> and this is how you do it in Microsoft SCSM:
> >> https://docs.microsoft.com/en-us/system-center/scsm/auth-classes. Then
> >> DevOps will configure "node left" monitors in SCSM (or a "checks" in
> >> Nagios) for parent "Ignite Cluster" and child "Ignite Service"
> components.
> >> State change (OK -> WARNING) and notification (email, SMS, whatever)
> will
> >> be configured only for the "Ignite Cluster"'s "node left" monitor.- Now
> >> suppose a node left. The "node left" monitor (that uses log file
> monitoring
> >> plugin) on "Ignite Node" will detect the event and pass it to the
> parent.
> >> This will trigger "Ignite Cluster" state change from OK to WARNING and
> send
> >> a notification. No more notification will be sent unless the "Ignite
> >> Cluster" state is reset back to OK, which happens either manually or on
> >> timeout or automatically on "node joined".
> >>> This was just FYI. We, Ignite developers, do not care about how
> >> monitoring works - this is responsibility of customer's DevOps. Our
> >> responsibility is consistent event logging.
> >>> Thank you!
> >>>
> >>>
> >>> Best regards, Alexey
> >>>
> >>>
> >>> On Tuesday, August 15, 2017, 6:16:25 PM 

Re: Data compression in Ignite 2.0

[Vladimir Ozerov]

Hi Vyacheslav,

Yes, I would suggest you to do so.

On Fri, Aug 25, 2017 at 2:51 PM, Vyacheslav Daradur 
wrote:

> Hi, should I close the initial ticket [1] as "Won't Fix" and add link to
> the new discusion about storage compression [2] in comments?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-3592
> [2] http://apache-ignite-developers.2346864.n4.nabble.
> com/Data-compression-in-Ignite-td20679.html
>
> 2017-08-09 23:05 GMT+03:00 Vyacheslav Daradur :
>
>> Vladimir, thank you for detailed explanation.
>>
>> I think I've understanded the main idea of described storage compression.
>>
>> I'll join the new discussion after researching of given material and
>> comlpetion of varint-optimization [1].
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-5097
>>
>> 2017-08-02 15:43 GMT+03:00 Alexey Kuznetsov :
>>
>>> Vova,
>>>
>>> Finally we back to my initial idea - to look how "big databases compress"
>>> data :)
>>>
>>>
>>> Just to remind how IBM DB2 do this[1].
>>>
>>> [1] http://www.ibm.com/developerworks/data/library/techarticle/dm-
>>> 1205db210compression/
>>> 
>>>
>>> On Tue, Aug 1, 2017 at 4:15 PM, Vladimir Ozerov 
>>> wrote:
>>>
>>> > Vyacheslav,
>>> >
>>> > This is not about my needs, but about the product :-) BinaryObject is a
>>> > central entity used for both data transfer and data storage. This is
>>> both
>>> > good and bad at the same time.
>>> >
>>> > Good thing is that as we optimize binary protocol, we improve both
>>> network
>>> > and storage performance at the same time. We have at least 3 things
>>> which
>>> > will be included into the product soon: varint encoding [1], optimized
>>> > string encoding [2] and null-field optimization [3]. Bad thing is that
>>> > binary object format is not well suited for data storage optimizations,
>>> > including compression. For example, one good compression technique is
>>> to
>>> > organize data in column-store format, or to introduce shared
>>> "dictionary"
>>> > with unique values on cache level. In both cases N equal values are not
>>> > stored N times. Instead, we store one value and N references to it, or
>>> so.
>>> > This way 2x-10x compression is possible depending on workload type.
>>> Binary
>>> > object protocol with some compression on top of it cannot give such
>>> > improvement, because it will compress data in individual objects,
>>> instead
>>> > of compressing the whole cache data in a single context.
>>> >
>>> > That said, I propose to give up adding compression to BinaryObject.
>>> This is
>>> > a dead end. Instead, we should:
>>> > 1) Optimize protocol itself to be more compact, as described in
>>> > aforementioned Ignite tickets
>>> > 2) Start new discussion about storage compression
>>> >
>>> > You can read papers of other vendors to get better understanding on
>>> > possible compression options. E.g. Oracle has a lot of compression
>>> > techniques, including heat maps, background compression, per-block
>>> > compression, data dictionaries, etc. [4].
>>> >
>>> > [1] https://issues.apache.org/jira/browse/IGNITE-5097
>>> > [2] https://issues.apache.org/jira/browse/IGNITE-5655
>>> > [3] https://issues.apache.org/jira/browse/IGNITE-3939
>>> > [4] http://www.oracle.com/technetwork/database/options/
>>> > compression/advanced-
>>> > compression-wp-12c-1896128.pdf
>>> >
>>> > Vladimir.
>>> >
>>> >
>>>
>>> --
>>> Alexey Kuznetsov
>>>
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: How a new index is built in runtime?

[Vladimir Ozerov]

Denis,

We iterate over the whole cache and build the index entry-by-entry. Control
is returned back to the user when index is ready.

On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov  wrote:

> Of course, iteration should have an option to be run from more than 1
> thread. What will really help, IMO, is ability to insert presorted batches
> in a single tree operation.
>
> --
> Yakov Zhdanov
>