[jira] [Created] (IGNITE-7863) Cleanup spark and spark_2.10 dependencies
Nikolay Izhikov created IGNITE-7863: --- Summary: Cleanup spark and spark_2.10 dependencies Key: IGNITE-7863 URL: https://issues.apache.org/jira/browse/IGNITE-7863 Project: Ignite Issue Type: Sub-task Reporter: Nikolay Izhikov Assignee: Nikolay Izhikov After solving issue with flatten plugin it possible to cleanup dependency list in spark and spark_2.10 modules. Transitive dependencies should be resolved using standart maven mechanism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-7862) Update flatten-plugin to 1.0.1 version
Nikolay Izhikov created IGNITE-7862:
---
Summary: Update flatten-plugin to 1.0.1 version
Key: IGNITE-7862
URL: https://issues.apache.org/jira/browse/IGNITE-7862
Project: Ignite
Issue Type: Improvement
Components: general
Affects Versions: 2.4
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
Fix For: 2.5
With flatten-plugin version {{1.0.0-beta-3}} we has to enlist all transitive
dependencies.
dev-list discussion -
http://apache-ignite-developers.2346864.n4.nabble.com/Maven-Issues-with-flatten-plugin-td27537.html
Solution is to update plugin to {{1.0.1}} version
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Transparent Data Encryption (TDE) in Apache Ignite
Dima, great job! Looking forward to the feature completion! On Fri, Mar 2, 2018 at 9:23 AM, Denis Magda wrote: > Dmitriy R., Nilokay, > > Thanks for the analysis and handout of the architectural design. No doubts, > it would be a valuable addition to Ignite. > > I would encourage you creating an IEP on the wiki and break the work into > pieces discussing specific part with the community. > > -- > Denis > > > On Thu, Mar 1, 2018 at 9:29 PM, Nikolay Izhikov wrote: > >> Hello, Dmitriy. >> >> Thank you for feedback! >> >> > Will it be supported? >> >> Yes. >> >> TDE shouldn't broke any of existing Ignite features. >> It adds some encrypt/decrypt level when we writing and reading pages >> in/from PDS. >> >> В Пт, 02/03/2018 в 07:29 +0300, Dmitriy Setrakyan пишет: >> > I have looked at the design, but could not find anything about running >> SQL >> > queries against the encrypted data. Will it be supported? >> > >> > D. >> > >> > On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov >> wrote: >> > >> > > Hell, Dima! >> > > >> > > Thank you for document! >> > > >> > > I'm ready to implement this feature with you. >> > > >> > > Igniters, please, share you thoughts about proposed design >> > > >> > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf >> > > >> > > В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет: >> > > > Hello, Igniters! >> > > > >> > > > I investigated the issue and wrote some details in a draft document >> > > > [1]. I think we should made IEP for TDE because it is a big change >> and >> > > > should be described in a single place, but not in a message >> > > > conversation. >> > > > Please, look it and write your thoughts. What is not understandable, >> > > > what should be detailed or described? >> > > > >> > > > > Where are we going to store keys (MEK) physically? Would it be >> PKCS#11 >> > > > > storage? Where we will store passwords to unlock storage or it >> will be >> > > > > responibilty of user? >> > > > >> > > > I think we should provide interface for MEK storage to let users use >> > > > storages they want. I suppose at the first step we should provide >> very >> > > > simple implementation, which will store MEK on every node and MEK >> will >> > > > be extracted by administrator during cluster activation process. Once >> > > > MEK is extracted from key store, we decrypt CEKs and destroy open >> MEK, >> > > > leaving open only cache keys. >> > > > >> > > > I think external storage is user's worry and we shouldn't give users >> > > > built-in external storage like Oracle Wallet or Microsoft Azure Key >> > > > Vault because it will increase Ignite's complexity too much. >> > > > >> > > > And yes, we should to comply with the standards like PKCS#11. >> > > > >> > > > > One more thing is how "node gets MEK from coordinator", if we send >> > > > > cleartext MEK, such security becomes useless also. >> > > > >> > > > Yeah, that's why we should use secured connection. As I know, we have >> > > > SSL implementation over JDK implementation, am I right? But we must >> > > > ensure to use latest SSL/TLS version. >> > > > >> > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf >> -- Best Regards, Vyacheslav D.
[GitHub] ignite pull request #3592: IGNITE-7862: flatten plugin updated to version 1....
GitHub user nizhikov opened a pull request: https://github.com/apache/ignite/pull/3592 IGNITE-7862: flatten plugin updated to version 1.0.1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/nizhikov/ignite IGNITE-7862 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3592.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3592 commit 4b9536ac6621761d538a6133c9ac6c7f0dae4696 Author: Nikolay Izhikov Date: 2018-03-02T08:22:14Z IGNITE-7862: flatten plugin updated to version 1.0.1 ---
[jira] [Created] (IGNITE-7864) Control utility: Add confirm on dangerous operations
Alexey Kuznetsov created IGNITE-7864: Summary: Control utility: Add confirm on dangerous operations Key: IGNITE-7864 URL: https://issues.apache.org/jira/browse/IGNITE-7864 Project: Ignite Issue Type: Improvement Reporter: Alexey Kuznetsov Fix For: 2.5 control.sh can deactivate cluster. It could be very dangerous in some cases. Lets add manual confirmation for dangerous operations: deactivate, change base line, ... Also, lets add "-force" option to not ask for confirmation (will be useful in scripts). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Maven. Issues with flatten plugin
Hi Petr, Thank you, it is great that you found the solution with low impact. Lets create ticket and merge PR. пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : > The problem is solved by updating flatten-maven-plugin version to 1.0.1. > > Nikolay, please, double check it. > If it really solves the problem, please, fill the ticket (or point to > existing one), so I can update it and check impact on release procedure. > > > > > On 1 Mar 2018, at 17:04, Nikolay Izhikov wrote: > > > > Petr. > > > > Thank you for trying! > > > > Did you remove 'test' dependencies before running commands? > > > > Because I commit in master only correct pom.xml for current build > process, > > of course. > > > > But, to make things works I have to copy paste transitive dependencies > from > > spark-core. > > > > 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" > > написал: > > > >> > >>> > >>> I don't get what is the point. > >>> Did you try to reproduce issue? > >>> Or should I provide full traces to you? > >>> > >> > >> My point in inability to fully understand and describe the problem in > >> terms of mechanism which causes it. > >> For now I can see only indirect guessing. > >> > >> > >> And yes, I’ve run your commands and both times (in spark module > directory > >> and in root) it produces the same result: > >> > >> 1. > >> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn > >> install -U -Plgpl,examples,scala,-clean-libs,-release,ml > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > >> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > >> [WARNING] The requested profile "ml" could not be activated because it > >> does not exist. > >> > >> 2. > >> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v > >> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > >> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > >> [WARNING] The requested profile "lgpl" could not be activated because it > >> does not exist. > >> [WARNING] The requested profile "examples" could not be activated > because > >> it does not exist. > >> [WARNING] The requested profile "scala" could not be activated because > it > >> does not exist. > >> [WARNING] The requested profile "ml" could not be activated because it > >> does not exist. > >
[jira] [Created] (IGNITE-7865) Need to provide method WAL manager for return serialize version
Dmitriy Govorukhin created IGNITE-7865:
--
Summary: Need to provide method WAL manager for return serialize
version
Key: IGNITE-7865
URL: https://issues.apache.org/jira/browse/IGNITE-7865
Project: Ignite
Issue Type: Improvement
Reporter: Dmitriy Govorukhin
{code}
public interface IgniteWriteAheadLogManager {
.
/**
* @return Current serializer version.
*/
public int serializerVersion();
.
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Maven. Issues with flatten plugin
Dmitry. I'm already done it. Will return with PR and TC results soon 2 марта 2018 г. 11:53 AM пользователь "Dmitry Pavlov" написал: > Hi Petr, > > Thank you, it is great that you found the solution with low impact. > > Lets create ticket and merge PR. > > пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : > > > The problem is solved by updating flatten-maven-plugin version to 1.0.1. > > > > Nikolay, please, double check it. > > If it really solves the problem, please, fill the ticket (or point to > > existing one), so I can update it and check impact on release procedure. > > > > > > > > > On 1 Mar 2018, at 17:04, Nikolay Izhikov wrote: > > > > > > Petr. > > > > > > Thank you for trying! > > > > > > Did you remove 'test' dependencies before running commands? > > > > > > Because I commit in master only correct pom.xml for current build > > process, > > > of course. > > > > > > But, to make things works I have to copy paste transitive dependencies > > from > > > spark-core. > > > > > > 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" < > mr.wei...@gmail.com> > > > написал: > > > > > >> > > >>> > > >>> I don't get what is the point. > > >>> Did you try to reproduce issue? > > >>> Or should I provide full traces to you? > > >>> > > >> > > >> My point in inability to fully understand and describe the problem in > > >> terms of mechanism which causes it. > > >> For now I can see only indirect guessing. > > >> > > >> > > >> And yes, I’ve run your commands and both times (in spark module > > directory > > >> and in root) it produces the same result: > > >> > > >> 1. > > >> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn > > >> install -U -Plgpl,examples,scala,-clean-libs,-release,ml > > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false > > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > > >> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > > >> [WARNING] The requested profile "ml" could not be activated because it > > >> does not exist. > > >> > > >> 2. > > >> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home > -v > > >> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml > > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false > > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > > >> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > > >> [WARNING] The requested profile "lgpl" could not be activated because > it > > >> does not exist. > > >> [WARNING] The requested profile "examples" could not be activated > > because > > >> it does not exist. > > >> [WARNING] The requested profile "scala" could not be activated because > > it > > >> does not exist. > > >> [WARNING] The requested profile "ml" could not be activated because it > > >> does not exist. > > > > >
[jira] [Created] (IGNITE-7866) TcpCommunicationSpi metrics causes cache operations slowdown
Vladimir Ozerov created IGNITE-7866:
---
Summary: TcpCommunicationSpi metrics causes cache operations
slowdown
Key: IGNITE-7866
URL: https://issues.apache.org/jira/browse/IGNITE-7866
Project: Ignite
Issue Type: Task
Components: general
Affects Versions: 2.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Fix For: 2.4
Extended TCP metrics were added as a part of IGNITE-6868 ticket. Unfortunately,
they causes ~5-10% performance drop for cache operations when working in
in-memory mode. Causes:
1) Contention on shared data structures (LongAdder, CMH)
2) A lot of string comparisons because we use class names to track particulat
message types
Suggested fix:
1) Move all processing to NIO threads
2) Introcduce thread-local state and update it from NIO threads
3) Metrics readers should merge state from all NIO threads
4) User {{Message}} type ID instead of class names
5) When returning final result we should use {{Class.getName()}} instead of
{{Class.getSimpleName()}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Maven. Issues with flatten plugin
Sorry - IGNITE-7862 is the ticket. 2 марта 2018 г. 12:14 PM пользователь "Nikolay Izhikov" написал: > Dmitry. > I'm already done it. > Will return with PR and TC results soon > > 2 марта 2018 г. 11:53 AM пользователь "Dmitry Pavlov" < > dpavlov@gmail.com> написал: > >> Hi Petr, >> >> Thank you, it is great that you found the solution with low impact. >> >> Lets create ticket and merge PR. >> >> пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : >> >> > The problem is solved by updating flatten-maven-plugin version to 1.0.1. >> > >> > Nikolay, please, double check it. >> > If it really solves the problem, please, fill the ticket (or point to >> > existing one), so I can update it and check impact on release procedure. >> > >> > >> > >> > > On 1 Mar 2018, at 17:04, Nikolay Izhikov wrote: >> > > >> > > Petr. >> > > >> > > Thank you for trying! >> > > >> > > Did you remove 'test' dependencies before running commands? >> > > >> > > Because I commit in master only correct pom.xml for current build >> > process, >> > > of course. >> > > >> > > But, to make things works I have to copy paste transitive dependencies >> > from >> > > spark-core. >> > > >> > > 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" < >> mr.wei...@gmail.com> >> > > написал: >> > > >> > >> >> > >>> >> > >>> I don't get what is the point. >> > >>> Did you try to reproduce issue? >> > >>> Or should I provide full traces to you? >> > >>> >> > >> >> > >> My point in inability to fully understand and describe the problem in >> > >> terms of mechanism which causes it. >> > >> For now I can see only indirect guessing. >> > >> >> > >> >> > >> And yes, I’ve run your commands and both times (in spark module >> > directory >> > >> and in root) it produces the same result: >> > >> >> > >> 1. >> > >> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn >> > >> install -U -Plgpl,examples,scala,-clean-libs,-release,ml >> > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite >> > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false >> > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time >> elapsed: >> > >> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite >> > >> [WARNING] The requested profile "ml" could not be activated because >> it >> > >> does not exist. >> > >> >> > >> 2. >> > >> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home >> -v >> > >> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml >> > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite >> > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false >> > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time >> elapsed: >> > >> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite >> > >> [WARNING] The requested profile "lgpl" could not be activated >> because it >> > >> does not exist. >> > >> [WARNING] The requested profile "examples" could not be activated >> > because >> > >> it does not exist. >> > >> [WARNING] The requested profile "scala" could not be activated >> because >> > it >> > >> does not exist. >> > >> [WARNING] The requested profile "ml" could not be activated because >> it >> > >> does not exist. >> > >> > >> >
[GitHub] ignite pull request #3593: IGNITE-7866
GitHub user devozerov opened a pull request: https://github.com/apache/ignite/pull/3593 IGNITE-7866 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-7866 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3593.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3593 commit 4925ffc10ce8e8287980eaec38b587512568a302 Author: Alexey Goncharuk Date: 2018-01-17T12:26:03Z IGNITE-7453 Use GridUnsafe.cleanDirectBuffer in PageSnapshot commit bcd68a058683b2f17b7ac60471b6e7aab3e4f6de Author: Pavel Tupitsyn Date: 2018-01-17T12:38:20Z IGNITE-7301 .NET: Baseline topology This closes #3352 commit 66b96316a7775ce8a6e2ff4475185d5929e4998b Author: devozerov Date: 2018-01-17T12:54:17Z Merge branch 'master' into ignite-2.4 commit 268481c1cf7fe57df24be130eb67c3e3a13afe01 Author: Alexey Goncharuk Date: 2018-01-17T13:50:34Z IGNITE-7453 Use GridUnsafe.cleanDirectBuffer in WalStat commit db0cd105719c8ae713b13b34d9dca0a8cd45d377 Author: Pavel Tupitsyn Date: 2018-01-17T14:05:25Z IGNITE-6776 .NET: Thin client: Add SQL & LINQ example This closes #3390 commit c214db879101aa5660e2a50b11cd20964c0bc114 Author: Andrey Gura Date: 2018-01-17T12:42:41Z ignite-7450 FileWriteAheadLogManager always uses RandomAccessFileIOFactory now commit 75c27d5e49d7458e46eb46e6f87a445c3f1320ea Author: Alexey Kuznetsov Date: 2018-01-18T02:25:19Z IGNITE-7274 Web Console: Support multiple statements on Queries screen. (cherry picked from commit 1926783) commit 36cc822935387b2150e690c29bc6992dca0563f7 Author: Dmitriy Shabalin Date: 2018-01-18T04:49:08Z IGNITE-7306 Web Console: Fixed export data from tables. (cherry picked from commit 1bb60ec) commit d753298b4012894b56f5c9218325211cd84a21d5 Author: Peter Ivanov Date: 2018-01-18T06:18:53Z IGNITE-7107 Apache Ignite RPM packages * added changelog to package specification This closes #3396 commit 63445893f1bc75dc9777184499f7eabc1d4e51b1 Author: Denis Mekhanikov Date: 2018-01-18T08:36:18Z IGNITE-3935 Use PeerDeployAware for streamer transformer - Fixes #3378. Signed-off-by: Alexey Goncharuk commit f3f9f2a24b23027cf0c835140322e41a788932ae Author: Pavel Tupitsyn Date: 2018-01-18T09:05:12Z IGNITE-7413 Fix SqlDmlExample This closes #3389 commit 1daa7c41bf1460a4d9a2b0c26a7a317f2fca3fb7 Author: Alexey Kuznetsov Date: 2018-01-18T10:14:53Z IGNITE-7461 UI tools: Actualized data storage configuration. (cherry picked from commit 577e632) commit cf0080210d24d9dd8b057f959446fac5f8a4ca01 Author: dpavlov Date: 2018-01-18T10:53:29Z IGNITE-7380 Implemented pluggable Direct IO - Fixes #3226. Signed-off-by: Alexey Goncharuk commit dd06d0bd7ef266bfbe156e858b312d1ac86e8982 Author: Pavel Tupitsyn Date: 2018-01-18T12:55:49Z IGNITE-7465 .NET: Fix SqlDdlExample failure with standalone node commit 57479ec564e1761716da3d5f9feb7a64c396a9f9 Author: Pavel Tupitsyn Date: 2018-01-18T13:45:54Z .NET: Fix CacheLocalTest.TestTxDeadlockDetection commit bd6be8a4653322905a3b63850c7e033ce3801ce5 Author: Pavel Tupitsyn Date: 2018-01-18T18:25:05Z .NET: Thin client: Fix OP_BINARY_TYPE_GET schema passing format commit 97564d160586d6d57d300937e6b8877994e35fc7 Author: rkondakov Date: 2018-01-19T08:24:51Z IGNITE-6456: Ability to separately enable or disable JDBC, ODBC and thin client endpoints. This closes #3309. commit d50274ca8875c9680c12e8786ac355a787ba95e0 Author: Yakov Zhdanov Date: 2018-01-18T17:57:17Z Javadoc enhancements - added @see commit cb2d3cf22388ab19fb2d34ae5bdfc8f1b608db75 Author: Dmitriy Govorukhin Date: 2018-01-18T14:14:26Z IGNITE-7471 Use soft reference for checkpoint entry contents to avoid excessive memory usage commit 3965923369870bb4e8e57e3332c1a1eb1e5f5ed3 Author: rkondakov Date: 2018-01-19T09:00:55Z IGNITE-6772: SQL exception messages became more informative. This closes #3342. commit ba68cb0fa87f776fcd0499d030c333f182611f41 Author: devozerov Date: 2018-01-19T09:03:52Z Merge remote-tracking branch 'origin/ignite-2.4' into ignite-2.4 commit b54c0c8786bd74aa0abb208f537c29f0c4be4b1e Author: tledkov-gridgain Date: 2018-01-19T09:09:34Z IGNITE-7248: JDBC: fixed schema resolution for streaming mode. This closes #3384. commit 2f5997788ccff265a088921210f561985f640517 Author: Dmitriy Govorukhin Date: 2018-01-19T11:46:38Z IGNITE-7471 fix npe commit 7adce10750704cc50cbf54fa7020aa3b20da87cb Author: Oleg Ignatenko Date: 2018-01-19T11:59:01Z IGNITE-7454 Wrong package in IgniteExamplesMLTestSuite this closes #3393 (cherry picked from commit cb1233a) commit 25e91b60694c589d8bf50c63a0d89
[GitHub] ignite pull request #3594: IGNITE-7865 Supported serializerVersion method fo...
GitHub user DmitriyGovorukhin opened a pull request: https://github.com/apache/ignite/pull/3594 IGNITE-7865 Supported serializerVersion method for WAL manager You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-7865 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3594.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3594 commit 569836532000c0224132dc01c6b07366e4796f75 Author: Dmitriy Govorukhin Date: 2018-03-02T09:16:27Z IGNITE-7865 Supported serializerVersion method for WAL manager ---
Re: Enabling swap space and Ignite Persistence
Prachi, Swap is legacy lightweight version of Ignite Native Persistence. In swap mode, we fully rely on OS in storing offheap memory into memory-mapped file. We don't provide durability guarantees in this mode. From my point of view, after 2.1 release there's no reason to prefer swap mode over Ignite Native Persistence. Igniters, please correct me if there are still any actual cases. Right now we indeed can configure both Ignite Native Persistence and swapping, but this makes even less sense. Node will just perform extra job by persisting data twice. Best Regards, Ivan Rakov On 02.03.2018 7:20, Prachi Garg wrote: Engineers, How does persistence and swap work when both are enabled? I was under the impression that for a data region you can either have swap or persistence configured at a time, but not both. Please clarify. Thanks, -Prachi
[jira] [Created] (IGNITE-7867) Document new feature in REST: possibility to get objects inserted via API or SQL
Alexey Kuznetsov created IGNITE-7867: Summary: Document new feature in REST: possibility to get objects inserted via API or SQL Key: IGNITE-7867 URL: https://issues.apache.org/jira/browse/IGNITE-7867 Project: Ignite Issue Type: Task Components: documentation Reporter: Alexey Kuznetsov Assignee: Prachi Garg Fix For: 2.5 [~pgarg] Please document changes from IGNITE-7803. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: REST: support getting objects from cache.
I merged IGNITE-7803 into master. Created ticket for documentation: https://issues.apache.org/jira/browse/IGNITE-7867 Prachi, could you update documentation for REST? On Thu, Mar 1, 2018 at 2:41 PM, Alexey Kuznetsov wrote: > Vladimir, > > I finished with this feature: https://issues. > apache.org/jira/browse/IGNITE-7803 > > Could you please review my changes in branch ignite-7803? > > DIFF: https://github.com/apache/ignite/commit/ > da03195ef5c39deabfc4961b2a0336fdbc98aa31 > > -- > Alexey Kuznetsov > -- Alexey Kuznetsov GridGain Systems www.gridgain.com
Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite
Turik, Also please remove pom.good.xml from ml module. Regards, Yury -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[GitHub] ignite pull request #3595: IGNITE-7851: .NET: linq query throws Hexadecimal ...
GitHub user apopovgg opened a pull request: https://github.com/apache/ignite/pull/3595 IGNITE-7851: .NET: linq query throws Hexadecimal string with odd numb⦠â¦er of characters exception You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-7851 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3595.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3595 commit f0575df148e7c6e9a952a9da601adb09536914db Author: apopov Date: 2018-03-02T09:54:39Z IGNITE-7851: .NET: linq query throws Hexadecimal string with odd number of characters exception ---
Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts
Сlear and intuitive API is the strength of Ignite, so I am also +1 for removing the unobvious settings. пт, 2 мар. 2018 г. в 4:11, Valentin Kulichenko < valentin.kuliche...@gmail.com>: > +1. Low level timeouts that we still have in discovery and communication > are very hard to explain and I doubt there is anyone who fully understands > how they currently work. They bring a lot of complexity and almost zero > value. Let's deprecate them and leave only failureDetectionTimeout plus > other high level settings that Alexey mentioned. > > -Val > > On Thu, Mar 1, 2018 at 6:06 AM, Ilya Kasnacheev > > wrote: > > > I agree with you. > > > > I think we could restrict usage of e.g. > setConnectTimeout/setSocketTimeout > > to people extending SPIs, since different implementations may need > > different values. > > > > However, for user configurations we should only expose timeouts we can > > explain, everything else should have reasonable values. > > > > -- > > Ilya Kasnacheev > > > > 2018-03-01 17:01 GMT+03:00 Alexey Popov : > > > > > Hi Igniters, > > > > > > We often see similar questions from users and customers related to > > > IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts and > > > their > > > relations. And we see several side-effects after incorrect timeout > > > configuration. > > > > > > I tried to briefly describe these timeout settings (please see below) > and > > > found out that the most of them do not have sense in terms of cluster > > > functions/operations and could not be explained to the users. > > > > > > I propose to deprecate most of them and leave only the timeouts we can > > > explain in common terms ( (setFailureDetectionTimeout, > setNetworkTimeout, > > > setJoinTimeout and some others). > > > > > > Please let me know your thoughts. > > > > > > Thanks, > > > Alexey > > > > > > GLOBAL: > > > > > > IgniteConfiguration.setNetworkTimeout: > > > It is a global timeout for high-level operations where a network is > > > involved. For instance, IgniteMessaging delivery uses this timeout or > > > DiscoverySpi handshake. > > > > > > IgniteConfiguration.setFailureDetectionTimeout: > > > It is a global timeout for detecting failures at IgniteSpi > > implementations > > > (including DiscoverySpi and CommunicationSpi). > > > The failure detection algorithm actually limits a range of simple > network > > > operations related to a single logical operation (for instance, a > > reliable > > > delivery of some DiscoverySpi message within a cluster). > > > Failure detection timeout is a cumulative timeout for a socket > > connection, > > > sending and receiving data bytes and all possible socket retries (if > some > > > failure happens). > > > This timeout is intended to simplify the failure detection condition > > from a > > > user perspective. > > > > > > IgniteConfiguration.setClientFailureDetectionTimeout: - it is a special > > > case > > > for DiscoverySpi client-node Ignite. > > > > > > TCP DISCOVERY SPI: > > > > > > If you need more control over failure detection algorithm for > > > TcpDiscoverySpi you can explicitly use the following low-level options > > > (that > > > will disable failureDetectoinTimeout logic): > > > > > > 1. TcpDiscoverySpi.setConnectTimeout - socket connection timeout > > > 2. TcpDiscoverySpi.setReconnectCount - number of reconnect attempts > used > > > when establishing connection with the remote node and sending messages > to > > > it > > > 3. TcpDiscoverySpi.setSocketTimeout - socket write timeout. The write > > > operation will be repeated getReconnectCount() times if it exceeds this > > > timeout > > > 4. TcpDiscoverySpi.setAckTimeout - message acknowledgment timeout. If a > > > message acknowledgment is not received within this timeout, sending is > > > considered as failed and SPI will try to repeat send operation. It is > > > automatically doubled for simultaneous retries up to getMaxAckTimeout > > > value. > > > 5. TcpDiscoverySpi.setMaxAckTimeout - maximum connection timeout, if > the > > > getAckTimeout reaches getMaxAckTimeout then SPI give up sending retries > > > > > > Another important TcpDiscoverySpi timeouts: > > > > > > TcpDiscoverySpi.setJoinTimeout - It is a timeout for join process when > a > > > new/restarted node joins a cluster. The node tries to connect to all > > > available IP addresses provided by ipFinder within this timeout. > > > If the timeout is exceeded, the node will give up and throw an > exception > > > from Ignition.start(). > > > > > > TcpDiscoverySpi.setNetworkTimeout - timeout for high-level operations > > like > > > handshake. It looks like it should be deprecated and the > > > IgniteConfiguration.getNetworkTimeout should be used here. > > > > > > TCP COMMUNICATION SPI: > > > > > > If you need more control over failure detection algorithm for > > > TcpCommunicationSpi you can explicitly use the following low-level > > options > > > (that will disable failureDetectoinTimeout logic): > > > > > > 1. TcpCommunicationSpi.set
Re: Maven. Issues with flatten plugin
Hello, Petr. I run TC for my PR [1] and have some issues on Team City: "Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.0.1:flatten (flatten) on project ignite-tools: The plugin org.codehaus.mojo:flatten-maven-plugin:1.0.1 requires Maven version 3.2.5" Can we update maven to version 3.2.5 or higher on all Team city agents? [1] https://github.com/apache/ignite/pull/3592 [2] https://ci.ignite.apache.org/viewLog.html?buildId=1117882&buildTypeId=IgniteTests24Java8_IgniteVisorConsoleScala&tab=buildLog&_focus=10143 В Пт, 02/03/2018 в 12:15 +0300, Nikolay Izhikov пишет: > Sorry - IGNITE-7862 is the ticket. > > 2 марта 2018 г. 12:14 PM пользователь "Nikolay Izhikov" > написал: > > Dmitry. > > I'm already done it. > > Will return with PR and TC results soon > > > > 2 марта 2018 г. 11:53 AM пользователь "Dmitry Pavlov" > > написал: > > > Hi Petr, > > > > > > Thank you, it is great that you found the solution with low impact. > > > > > > Lets create ticket and merge PR. > > > > > > пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : > > > > > > > The problem is solved by updating flatten-maven-plugin version to 1.0.1. > > > > > > > > Nikolay, please, double check it. > > > > If it really solves the problem, please, fill the ticket (or point to > > > > existing one), so I can update it and check impact on release procedure. > > > > > > > > > > > > > > > > > On 1 Mar 2018, at 17:04, Nikolay Izhikov wrote: > > > > > > > > > > Petr. > > > > > > > > > > Thank you for trying! > > > > > > > > > > Did you remove 'test' dependencies before running commands? > > > > > > > > > > Because I commit in master only correct pom.xml for current build > > > > process, > > > > > of course. > > > > > > > > > > But, to make things works I have to copy paste transitive dependencies > > > > from > > > > > spark-core. > > > > > > > > > > 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" > > > > > > > > > > написал: > > > > > > > > > >> > > > > >>> > > > > >>> I don't get what is the point. > > > > >>> Did you try to reproduce issue? > > > > >>> Or should I provide full traces to you? > > > > >>> > > > > >> > > > > >> My point in inability to fully understand and describe the problem in > > > > >> terms of mechanism which causes it. > > > > >> For now I can see only indirect guessing. > > > > >> > > > > >> > > > > >> And yes, I’ve run your commands and both times (in spark module > > > > directory > > > > >> and in root) it produces the same result: > > > > >> > > > > >> 1. > > > > >> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn > > > > >> install -U -Plgpl,examples,scala,-clean-libs,-release,ml > > > > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false > > > > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time > > > > >> elapsed: > > > > >> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > >> [WARNING] The requested profile "ml" could not be activated because > > > > >> it > > > > >> does not exist. > > > > >> > > > > >> 2. > > > > >> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home > > > > >> -v > > > > >> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml > > > > >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false > > > > >> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time > > > > >> elapsed: > > > > >> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > >> [WARNING] The requested profile "lgpl" could not be activated > > > > >> because it > > > > >> does not exist. > > > > >> [WARNING] The requested profile "examples" could not be activated > > > > because > > > > >> it does not exist. > > > > >> [WARNING] The requested profile "scala" could not be activated > > > > >> because > > > > it > > > > >> does not exist. > > > > >> [WARNING] The requested profile "ml" could not be activated because > > > > >> it > > > > >> does not exist. > > > > > > > > signature.asc Description: This is a digitally signed message part
Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts
Alexey, generally I agree. However, I don't understand what exactly you suggest. Can you please list the list of parameters you want to deprecate (1), internal logic changes (2) and updates to the javadocs/description of the params you want to keep (3)? --Yakov
[GitHub] ignite pull request #3596: For TeamCity test: default-readFromBackup-true fo...
GitHub user daradurvs opened a pull request: https://github.com/apache/ignite/pull/3596 For TeamCity test: default-readFromBackup-true for all cache's tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/daradurvs/ignite ignite-5357-default-readFromBackup-true Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3596.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3596 commit fc6cdf8cff5236254c9f3d4805561ca0baa8518d Author: Vyacheslav Daradur Date: 2018-02-27T15:58:38Z ignite-5357: neineighborhoods selection logic was added commit 7fe3b609bd5c5adfcca26516958da32796fc0be6 Author: Vyacheslav Daradur Date: 2018-02-27T16:01:02Z ignite-5357: fix statement with 'canRemap' flag commit 9c07ad741c1fcc2a051b77207d7f939b03a7d668 Author: Vyacheslav Daradur Date: 2018-02-27T16:09:29Z ignite-5357: code style fix commit 65546ae4f60d13bcbd49fa2c526ed00d63dd2b90 Author: Vyacheslav Daradur Date: 2018-02-27T16:30:09Z ignite-5357: replaced parallelStream with stream commit f6197a0e3994a671debb802474cde042f4a1749d Author: Vyacheslav Daradur Date: 2018-02-27T17:18:22Z ignite-5357: fix nullable logic commit b35ddb9cf8d15e3739ee55ce830cf9e442687a7e Author: Vyacheslav Daradur Date: 2018-02-27T17:20:12Z ignite-5357: remove unnecessary imports commit 518a083913d7a4dbd26faf96cc0d3ce1f89dc60d Author: Vyacheslav Daradur Date: 2018-02-27T17:36:16Z ignite-5357: skip primary node condition was added commit ded2441863ff9582947ae20a48161b366b793ca7 Author: Vyacheslav Daradur Date: 2018-02-28T15:36:24Z ignite-5357: refactoring commit 883dbbff5d257c4eb8a3de65020c84c5c444ae54 Author: Vyacheslav Daradur Date: 2018-02-28T17:50:01Z ignite-5357: javadoc fix commit e76013f40f17fc5ee07370b3d0d1ee7b73580460 Author: Vyacheslav Daradur Date: 2018-02-28T19:22:13Z ignite-5357: refactoring commit eaaa0318dc0242dc901ad9b447bfdbd96e9c7698 Author: Vyacheslav Daradur Date: 2018-02-28T19:33:37Z ignite-5357: back logic commit 66097c4998e799df980591d1fa0126096a27267e Author: Vyacheslav Daradur Date: 2018-02-28T19:36:24Z ignite-5357: changed logic commit f6f0db629bc19b414ae15f7aea63cec3e08f9749 Author: Vyacheslav Daradur Date: 2018-02-28T19:37:45Z ignite-5357: changed logic 2 commit 91c75ef77093e8747fd90d1b505d47aedacd711f Author: Vyacheslav Daradur Date: 2018-02-28T19:54:36Z ignite-5357: removed utils method commit 26c639ef9ba30b58b18fb21a56a644f14021dbb1 Author: Vyacheslav Daradur Date: 2018-02-28T20:43:09Z Merge remote-tracking branch 'origin/ignite-5357-2' into ignite-5357 commit 09ba7b2effd29fb9c6202958d4d019daf22e9800 Author: Vyacheslav Daradur Date: 2018-03-01T23:59:16Z ignite-5357: requests distribution tests were added commit 8a685e10f15a89eb48b4db731d6035c31dae9176 Author: Vyacheslav Daradur Date: 2018-03-02T06:02:33Z ignite-5357: fix tests classes names commit 56579c8f5194744e6ce83103d33d034f95ee219b Author: Vyacheslav Daradur Date: 2018-03-02T09:32:39Z ignite-5357: fix node for getting primary keys commit a5fca8ab25bc66c39ec46843a0c65ce09ba9334a Author: Vyacheslav Daradur Date: 2018-03-02T10:08:00Z ignite-5357: code style fix: long line split commit 688cd3e25639b9301deb6327de6be1a8b52a041f Author: Vyacheslav Daradur Date: 2018-03-02T10:32:43Z ignite-5357: changed flag CacheConfiguration to 'true' ---
Re: Maven. Issues with flatten plugin
Made some changes — 3.3.9 is now default maven. Please, rerun failed tests. > On 2 Mar 2018, at 13:21, Nikolay Izhikov wrote: > > Hello, Petr. > > I run TC for my PR [1] and have some issues on Team City: > > "Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.0.1:flatten > (flatten) on project ignite-tools: > The plugin org.codehaus.mojo:flatten-maven-plugin:1.0.1 requires Maven > version 3.2.5" > > Can we update maven to version 3.2.5 or higher on all Team city agents? > > [1] https://github.com/apache/ignite/pull/3592 > [2] > https://ci.ignite.apache.org/viewLog.html?buildId=1117882&buildTypeId=IgniteTests24Java8_IgniteVisorConsoleScala&tab=buildLog&_focus=10143 > > В Пт, 02/03/2018 в 12:15 +0300, Nikolay Izhikov пишет: >> Sorry - IGNITE-7862 is the ticket. >> >> 2 марта 2018 г. 12:14 PM пользователь "Nikolay Izhikov" >> написал: >>> Dmitry. >>> I'm already done it. >>> Will return with PR and TC results soon >>> >>> 2 марта 2018 г. 11:53 AM пользователь "Dmitry Pavlov" >>> написал: Hi Petr, Thank you, it is great that you found the solution with low impact. Lets create ticket and merge PR. пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : > The problem is solved by updating flatten-maven-plugin version to 1.0.1. > > Nikolay, please, double check it. > If it really solves the problem, please, fill the ticket (or point to > existing one), so I can update it and check impact on release procedure. > > > >> On 1 Mar 2018, at 17:04, Nikolay Izhikov wrote: >> >> Petr. >> >> Thank you for trying! >> >> Did you remove 'test' dependencies before running commands? >> >> Because I commit in master only correct pom.xml for current build > process, >> of course. >> >> But, to make things works I have to copy paste transitive dependencies > from >> spark-core. >> >> 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" >> написал: >> >>> I don't get what is the point. Did you try to reproduce issue? Or should I provide full traces to you? >>> >>> My point in inability to fully understand and describe the problem in >>> terms of mechanism which causes it. >>> For now I can see only indirect guessing. >>> >>> >>> And yes, I’ve run your commands and both times (in spark module > directory >>> and in root) it produces the same result: >>> >>> 1. >>> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn >>> install -U -Plgpl,examples,scala,-clean-libs,-release,ml >>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite >>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false >>> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: >>> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite >>> [WARNING] The requested profile "ml" could not be activated because it >>> does not exist. >>> >>> 2. >>> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v >>> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml >>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite >>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false >>> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: >>> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite >>> [WARNING] The requested profile "lgpl" could not be activated because it >>> does not exist. >>> [WARNING] The requested profile "examples" could not be activated > because >>> it does not exist. >>> [WARNING] The requested profile "scala" could not be activated because > it >>> does not exist. >>> [WARNING] The requested profile "ml" could not be activated because it >>> does not exist. >
Re: Ignite Thin clients for Node.js, Python, PHP
Hi Igniters Due to some delay for release 2.4 and personal circumstances I decided to public the prototype of Python Client [1]. Feel free to use it as a base if it makes sense. [1] https://github.com/skozlov-gridgain/apache-ignite-python-thin-client On Thu, Mar 1, 2018 at 8:17 PM, Pavel Tupitsyn wrote: > Agree with Vladimir and Denis. > > I don't think JSON has any place in the thin client protocol, > which is a binary protocol designed to be efficient, with clearly defined > data format. > > We already have JSON in "REST" client. > > > On Thu, Mar 1, 2018 at 8:15 PM, Denis Magda wrote: > > > Totally share Vladimir's stance. Let's support the scope that already > > exists in the protocol and think about the future later. The users will > > definitely guide us to a right direction :) > > > > -- > > Denis > > > > On Thu, Mar 1, 2018 at 7:12 AM, Vladimir Ozerov > > wrote: > > > > > I would extract compute tasks into separate scope. It is better to keep > > > focus on protocol things and basic language support for now. Once we > have > > > basic client API in production-ready state, we could consider adding > > > JavaScript to our core compute feature set and then extend it to the > > > clients (which should be trivial provided that core part is ready). We > > > should > > > be ready to spend considerable efforts to prior R&D because dynamic > code > > > execution is not very simple thing, especially in terms of security, > > native > > > compilation, etc.. > > > > > > > > > > > > On Thu, Mar 1, 2018 at 5:17 PM, Ilya Kasnacheev < > > ilya.kasnach...@gmail.com > > > > > > > wrote: > > > > > > > With regards of thin clients for dynamically typed languages, I think > > > > Ignite needs two following features to shine: > > > > > > > > - Ability to pass JSON to such clients, turn JSON Objects into a > > > > BinaryObjects, which will give ability to index top-level keys in > such > > > JSON > > > > with SQL Indexing. Of course this should be integrated with > > > QueryEntities. > > > > - Ability to pass JavaScript snippets to invoke() and affinityCall() > > > > families of calls. On Server node they should be interpreted by > Nashorn > > > > (since we've upgraded to Java 8). We should also cache such snippets > > > > pre-interpreted, in this case it can be pretty fast since Nashorn > > compile > > > > to JVM bytecode. > > > > > > > > WDYT? > > > > > > > > -- > > > > Ilya Kasnacheev > > > > > > > > 2018-02-20 0:23 GMT+03:00 Alexey Kosenchuk > > > com > > > > >: > > > > > > > > > Hi All! > > > > > > > > > > Let us join the party, please ;) > > > > > > > > > > As we see, there is Binary Client Protocol to communicate with > Ignite > > > > > cluster and a concept of Thin (lightweight) client. > > > > > > > > > > If there are no objections or duplicated plans, we would like to > > > develop > > > > > Thin Client libs for: > > > > > - Node.js > > > > > - Python > > > > > - PHP > > > > > > > > > > Please add us as contributors and provide access to the Ignite Jira > > > > > components. > > > > > > > > > > Usernames in the Apache Jira: > > > > > alexey.kosenchuk > > > > > ekaterina.vergizova > > > > > pavel.petroshenko > > > > > > > > > > Thanks! > > > > > -Alexey > > > > > > > > > > > > > > > -- Sergey Kozlov GridGain Systems www.gridgain.com
[GitHub] ignite pull request #3593: IGNITE-7866
Github user devozerov closed the pull request at: https://github.com/apache/ignite/pull/3593 ---
Re: [IGNITE-7789] Ignite Client Nodes testErrorOnDisconnect()
Maxim, Alexey G. thank you. Test now passes in nigth build, https://ci.ignite.apache.org/viewLog.html?buildId=1117170 Sincerely, Dmitriy Pavlov ср, 28 февр. 2018 г. в 18:11, Maxim Muzafarov : > Dmitry, Sergey > > I've found solution of fixing this test-case, but not sure about > correctness. > Can you review it? > > JIRA: https://issues.apache.org/jira/browse/IGNITE-7789 > Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-507 > PR: https://github.com/apache/ignite/pull/3588 > > > ср, 28 февр. 2018 г. в 18:02, Dmitry Pavlov : > > > Hi Maxim, > > > > First of all, thank you for being ready to help community with tests and > > TC. Contributions which helps to improve tests stability are highly > > appreciated. > > > > Feel free to assign any issue if it is currently unassigned. If issue > > became too hard to implement later, it may be returned to unassigned. > > > > I hope Sergey C. can provide some details about this test failure. > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 27 февр. 2018 г. в 15:59, Maxim Muzafarov : > > > > > Hi all, > > > > > > I just briefly look at this issue [1] and found that for: > > > > > > 1) CollectionConfiguration[2] we have default backup = 0; > > > 2) AtomicConfiguration[3] we have DFLT_BACKUPS = 1; > > > > > > Is this correct? Because, both of them used for testing > > > dataStructureOperationsTest(). After disconnet and recreation cache > > > GridCacheAtomicLongImpl the default value for backup used. > > > > > > We have doTestIgniteOperationOnDisconnect() and it's argument of > > operations > > > List> which calls asynch. So thats why > > we > > > have unstable test-case. > > > > > > OK - test case > > > [2018-02-27 15:51:05,065][INFO ][async-callable-runner-1][root] Queue > > > creation > > > [2018-02-27 15:51:05,065][INFO ][async-callable-runner-1][root] Set > > > creation > > > [2018-02-27 15:51:05,065][INFO ][async-callable-runner-1][root] Atomic > > > creation > > > > > > FAIL - test case > > > 2018-02-27 15:52:23,304][INFO ][async-callable-runner-1][root] Atomic > > > creation > > > [2018-02-27 15:52:23,304][INFO ][async-callable-runner-1][root] Queue > > > creation > > > [2018-02-27 15:52:23,303][INFO ][async-callable-runner-1][root] Set > > > creation > > > > > > Can i take this issue for mysefl? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7789 > > > [2] > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/CollectionConfiguration.java#L47 > > > [3] > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/AtomicConfiguration.java#L32 > > > > > >
Re: Apache Ignite 2.4 release
Hi, I merged performance fix for TCP metrics. Nikolay, I think it is OK to merge your fix today. Let's agree on a final code freeze on Monday, 5 Mar. On Thu, Mar 1, 2018 at 8:24 PM, Denis Magda wrote: > Hi Vladimir, > > The regression, as well as the performance, is definitely bad. Please work > with Sergey to see how much time he needs to rerun the whole test set if > you merge the changes. > > I would target the end of the next week as a release date if the vote goes > well. Probably, we can run the test suites over the weekend if your changes > get to the branch. > > Nikolay Izhikov, follow up with Vladimir and Sergey tomorrow. If they > decide to merge the fixes then you are free to merge your fixed for Spark > examples. > > -- > Denis > > On Thu, Mar 1, 2018 at 7:46 AM, Vladimir Ozerov > wrote: > > > Hi, > > > > We found one nasty regression in DROP COLUMN feature and fixed it today. > > Another problem is performance - we found that recent changes to TCP > > communication metrics caused considerable slowdown in in-memory mode. I > > almost fixed it and will merge the fix in the nearest day if benchmarks > are > > ok. > > > > That said I think we will be able to start vote in the beginning of the > > next week. > > > > Any objections? > > > > Vladimir. > > > > On Mon, Feb 26, 2018 at 11:22 PM, Denis Magda wrote: > > > > > Good question! > > > > > > We've been waiting for a sign-off from Alexey Goncharuk and Sergey > Kozlov > > > who are benchmarking and trying to address performance issues in > > > coopeartion. > > > > > > Guys, please share your results and forecast. > > > > > > -- > > > Denis > > > > > > On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn > > > > wrote: > > > > > > > Igniters, > > > > > > > > Any update on the 2.4 release status? Anything else to merge there? > > > > > > > > Pavel > > > > > > > > On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan < > > > dsetrak...@apache.org> > > > > wrote: > > > > > > > > > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov < > > voze...@gridgain.com > > > > > > > > > wrote: > > > > > > > > > > > Alex, > > > > > > > > > > > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any > > dramatic > > > > > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the > > box > > > in > > > > > > 2.3, and it is unsafe in 2.4 as well. > > > > > > > > > > > > The very problem is that we claim ourselves to be ACID, while in > > > > reality > > > > > we > > > > > > are only "AI" out of the box, because durability is not > guaranteed > > > due > > > > to > > > > > > zero backups and LOG_ONLY and consistency is not guaranteed due > to > > > > > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim > > > > themselves > > > > > > to be ACID, so it is not valid to refer to their defaults. > > > > > > > > > > > > > > > > Vladimir, > > > > > Ignite can be fully ACID, but at the same time have non-ACID > > defaults, > > > as > > > > > long as we clearly document how to get ACID behavior. I do not see > an > > > > issue > > > > > with it. > > > > > > > > > > D. > > > > > > > > > > > > > > >
[GitHub] ignite pull request #3594: IGNITE-7865 Supported serializerVersion method fo...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/3594 ---
Re: Maven. Issues with flatten plugin
Petr, thank you. But seems it doesn't help "Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.0.1:flatten (flatten) on project ignite-tools: The plugin org.codehaus.mojo:flatten-maven-plugin:1.0.1 requires Maven version 3.2.5" https://ci.ignite.apache.org/viewLog.html?buildId=1118290&buildTypeId=IgniteTests24Java8_IgniteActivateDeactivateCluster&tab=buildLog&_focus=189 В Пт, 02/03/2018 в 13:38 +0300, Petr Ivanov пишет: > Made some changes — 3.3.9 is now default maven. > Please, rerun failed tests. > > > > > On 2 Mar 2018, at 13:21, Nikolay Izhikov wrote: > > > > Hello, Petr. > > > > I run TC for my PR [1] and have some issues on Team City: > > > > "Failed to execute goal > > org.codehaus.mojo:flatten-maven-plugin:1.0.1:flatten (flatten) on project > > ignite-tools: > > The plugin org.codehaus.mojo:flatten-maven-plugin:1.0.1 requires Maven > > version 3.2.5" > > > > Can we update maven to version 3.2.5 or higher on all Team city agents? > > > > [1] https://github.com/apache/ignite/pull/3592 > > [2] > > https://ci.ignite.apache.org/viewLog.html?buildId=1117882&buildTypeId=IgniteTests24Java8_IgniteVisorConsoleScala&tab=buildLog&_focus=10143 > > > > В Пт, 02/03/2018 в 12:15 +0300, Nikolay Izhikov пишет: > > > Sorry - IGNITE-7862 is the ticket. > > > > > > 2 марта 2018 г. 12:14 PM пользователь "Nikolay Izhikov" > > > написал: > > > > Dmitry. > > > > I'm already done it. > > > > Will return with PR and TC results soon > > > > > > > > 2 марта 2018 г. 11:53 AM пользователь "Dmitry Pavlov" > > > > написал: > > > > > Hi Petr, > > > > > > > > > > Thank you, it is great that you found the solution with low impact. > > > > > > > > > > Lets create ticket and merge PR. > > > > > > > > > > пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : > > > > > > > > > > > The problem is solved by updating flatten-maven-plugin version to > > > > > > 1.0.1. > > > > > > > > > > > > Nikolay, please, double check it. > > > > > > If it really solves the problem, please, fill the ticket (or point > > > > > > to > > > > > > existing one), so I can update it and check impact on release > > > > > > procedure. > > > > > > > > > > > > > > > > > > > > > > > > > On 1 Mar 2018, at 17:04, Nikolay Izhikov > > > > > > > wrote: > > > > > > > > > > > > > > Petr. > > > > > > > > > > > > > > Thank you for trying! > > > > > > > > > > > > > > Did you remove 'test' dependencies before running commands? > > > > > > > > > > > > > > Because I commit in master only correct pom.xml for current build > > > > > > > > > > > > process, > > > > > > > of course. > > > > > > > > > > > > > > But, to make things works I have to copy paste transitive > > > > > > > dependencies > > > > > > > > > > > > from > > > > > > > spark-core. > > > > > > > > > > > > > > 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" > > > > > > > > > > > > > > написал: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't get what is the point. > > > > > > > > > Did you try to reproduce issue? > > > > > > > > > Or should I provide full traces to you? > > > > > > > > > > > > > > > > > > > > > > > > > My point in inability to fully understand and describe the > > > > > > > > problem in > > > > > > > > terms of mechanism which causes it. > > > > > > > > For now I can see only indirect guessing. > > > > > > > > > > > > > > > > > > > > > > > > And yes, I’ve run your commands and both times (in spark module > > > > > > > > > > > > directory > > > > > > > > and in root) it produces the same result: > > > > > > > > > > > > > > > > 1. > > > > > > > > ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" > > > > > > > > mvn > > > > > > > > install -U -Plgpl,examples,scala,-clean-libs,-release,ml > > > > > > > > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > > > > > -Dmaven.javadoc.skip=true -DfailIfNoTests=false > > > > > > > > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time > > > > > > > > elapsed: > > > > > > > > 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > > > > > [WARNING] The requested profile "ml" could not be activated > > > > > > > > because it > > > > > > > > does not exist. > > > > > > > > > > > > > > > > 2. > > > > > > > > ignite/modules/spark (master) $ > > > > > > > > JAVA_HOME="$(/usr/libexec/java_home -v > > > > > > > > 1.8)" mvn install -U > > > > > > > > -Plgpl,examples,scala,-clean-libs,-release,ml > > > > > > > > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > > > > > -Dmaven.javadoc.skip=true -DfailIfNoTests=false > > > > > > > > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time > > > > > > > > elapsed: > > > > > > > > 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > > > > > > > > [WARNING] The requested profile "lgpl" could not be activated > > > > > > > > because it > > > > > > > > does not exist. > > > > > > > > [WARNING] The requested profile "examples" could not be > > > > > > > > a
Re: Maven. Issues with flatten plugin
Updated all maven definitions I’ve found in templates of test project. Please, try once more. > On 2 Mar 2018, at 16:36, Nikolay Izhikov wrote: > > Petr, thank you. > > But seems it doesn't help > > "Failed to execute goal org.codehaus.mojo:flatten-maven-plugin:1.0.1:flatten > (flatten) on project ignite-tools: > The plugin org.codehaus.mojo:flatten-maven-plugin:1.0.1 requires Maven > version 3.2.5" > > https://ci.ignite.apache.org/viewLog.html?buildId=1118290&buildTypeId=IgniteTests24Java8_IgniteActivateDeactivateCluster&tab=buildLog&_focus=189 > > В Пт, 02/03/2018 в 13:38 +0300, Petr Ivanov пишет: >> Made some changes — 3.3.9 is now default maven. >> Please, rerun failed tests. >> >> >> >>> On 2 Mar 2018, at 13:21, Nikolay Izhikov wrote: >>> >>> Hello, Petr. >>> >>> I run TC for my PR [1] and have some issues on Team City: >>> >>> "Failed to execute goal >>> org.codehaus.mojo:flatten-maven-plugin:1.0.1:flatten (flatten) on project >>> ignite-tools: >>> The plugin org.codehaus.mojo:flatten-maven-plugin:1.0.1 requires Maven >>> version 3.2.5" >>> >>> Can we update maven to version 3.2.5 or higher on all Team city agents? >>> >>> [1] https://github.com/apache/ignite/pull/3592 >>> [2] >>> https://ci.ignite.apache.org/viewLog.html?buildId=1117882&buildTypeId=IgniteTests24Java8_IgniteVisorConsoleScala&tab=buildLog&_focus=10143 >>> >>> В Пт, 02/03/2018 в 12:15 +0300, Nikolay Izhikov пишет: Sorry - IGNITE-7862 is the ticket. 2 марта 2018 г. 12:14 PM пользователь "Nikolay Izhikov" написал: > Dmitry. > I'm already done it. > Will return with PR and TC results soon > > 2 марта 2018 г. 11:53 AM пользователь "Dmitry Pavlov" > написал: >> Hi Petr, >> >> Thank you, it is great that you found the solution with low impact. >> >> Lets create ticket and merge PR. >> >> пт, 2 мар. 2018 г. в 10:06, Petr Ivanov : >> >>> The problem is solved by updating flatten-maven-plugin version to 1.0.1. >>> >>> Nikolay, please, double check it. >>> If it really solves the problem, please, fill the ticket (or point to >>> existing one), so I can update it and check impact on release procedure. >>> >>> >>> On 1 Mar 2018, at 17:04, Nikolay Izhikov wrote: Petr. Thank you for trying! Did you remove 'test' dependencies before running commands? Because I commit in master only correct pom.xml for current build >>> >>> process, of course. But, to make things works I have to copy paste transitive dependencies >>> >>> from spark-core. 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" написал: > >> >> I don't get what is the point. >> Did you try to reproduce issue? >> Or should I provide full traces to you? >> > > My point in inability to fully understand and describe the problem in > terms of mechanism which causes it. > For now I can see only indirect guessing. > > > And yes, I’ve run your commands and both times (in spark module >>> >>> directory > and in root) it produces the same result: > > 1. > ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn > install -U -Plgpl,examples,scala,-clean-libs,-release,ml > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > -Dmaven.javadoc.skip=true -DfailIfNoTests=false > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > [WARNING] The requested profile "ml" could not be activated because it > does not exist. > > 2. > ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v > 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite > -Dmaven.javadoc.skip=true -DfailIfNoTests=false > [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: > 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite > [WARNING] The requested profile "lgpl" could not be activated because > it > does not exist. > [WARNING] The requested profile "examples" could not be activated >>> >>> because > it does not exist. > [WARNING] The requested profile "scala" could not be activated because >>> >>> it > does not exist. > [WARNING] The requested profile "ml" could not be activated because it > does not exist. >>
[jira] [Created] (IGNITE-7869) Dynamic start cache by stored cache data
Anton Kalashnikov created IGNITE-7869: - Summary: Dynamic start cache by stored cache data Key: IGNITE-7869 URL: https://issues.apache.org/jira/browse/IGNITE-7869 Project: Ignite Issue Type: Improvement Reporter: Anton Kalashnikov Assignee: Anton Kalashnikov Dynamic start cache by stored cache data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #3596: For TeamCity test: default-readFromBackup-true fo...
Github user daradurvs closed the pull request at: https://github.com/apache/ignite/pull/3596 ---
[GitHub] ignite pull request #3597: IGNITE-7869 Dynamic start cache by stored cache d...
GitHub user akalash opened a pull request: https://github.com/apache/ignite/pull/3597 IGNITE-7869 Dynamic start cache by stored cache data You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-gg-13535-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3597.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3597 commit f8f701ea110a0dbd655b4a0ff23a85f1549ebd59 Author: Anton Kalashnikov Date: 2018-03-02T14:45:49Z GG-13535 add storedCacheData to snapshot ---
[jira] [Created] (IGNITE-7870) Update Discovery and Communication Technical Docs
Denis Magda created IGNITE-7870: --- Summary: Update Discovery and Communication Technical Docs Key: IGNITE-7870 URL: https://issues.apache.org/jira/browse/IGNITE-7870 Project: Ignite Issue Type: Task Components: documentation Reporter: Denis Magda Assignee: Prachi Garg Fix For: 2.5 Attachments: IgniteNetworkingOverview.md Existing discovery and communication pages are not in the best shape and require an overhaul. Use the revisited version (attached) contributed by a community member and see how to improve or mix discovery, communication, servers & clients docs from readme. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-7871) Partition update counters may be different during exchange
Pavel Kovalenko created IGNITE-7871: --- Summary: Partition update counters may be different during exchange Key: IGNITE-7871 URL: https://issues.apache.org/jira/browse/IGNITE-7871 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 2.4 Reporter: Pavel Kovalenko Using validation implemented in IGNITE-7467 we can observe the following situation: Let's we have some partition and nodes which owning it N1 (primary) and N2 (backup) 1) Exchange is started 2) N2 finished waiting for partitions release and started to create Single message (with update counters). 3) N1 waits for partitions release. 4) We have pending cache update N1 -> N2. This update is done after step 2. 5) This update increments update counters both on N1 and N2. 6) N1 finished waiting for partitions release, while N2 already sent Single message to coordinator with outdated update counter. 7) Coordinator sees different partition update counters for N1 and N2. Validation is failed, while data is equal. Possible solutions: 1) Cancel transactions and atomic updates on backups if topology version on them is already changed (or waiting for partitions release is finished). 2) Each node participating in exchange should wait for partitions release of other nodes not only self (like distributed countdown latch right after waiting for partitions release). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-7872) Actualize code style rules in Ignite
Dmitriy Pavlov created IGNITE-7872: -- Summary: Actualize code style rules in Ignite Key: IGNITE-7872 URL: https://issues.apache.org/jira/browse/IGNITE-7872 Project: Ignite Issue Type: Task Reporter: Dmitriy Pavlov Code style used by Ignite contributors ( ignite\idea\ignite_codeStyle.xml ) is outdated now. Need to update code style with new suggestions Operators (newline) Lamdas (Java 8) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
TeamCity user mapping to github user
Hi Igniters, Recently Ignite Basic Test suite notification on failures was enabled (draft of desctiption and details can be found here https://cwiki.apache.org/confluence/display/IGNITE/Continuous+Integration ). But unfortunately notification may not work if TC username differs from github username. Could you please ensure your github user is correctly mapped to TeamCity user registered in VCS usert https://ci.ignite.apache.org/profile.html?item=vcsUsernames Thank you! Sincerely, Dmitriy Pavlov
[jira] [Created] (IGNITE-7873) Partition update counters and sizes may be different if cache is using readThrough
Pavel Kovalenko created IGNITE-7873: --- Summary: Partition update counters and sizes may be different if cache is using readThrough Key: IGNITE-7873 URL: https://issues.apache.org/jira/browse/IGNITE-7873 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 2.4 Reporter: Pavel Kovalenko Tracking partition update counters and cache sizes may not properly work if cache is using readThrough behavior. Read requests to such cache can increment update counters or cache sizes not on all nodes serving such cache in case if data in underlying storage is changed. It means that update counter or cache size will be incremented only on partition where we followed such request (primary or any random node). BackupPostProcessingClosure should use preload=false for entry. In other case it can increment update counter for read request while data is not changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Race on IgniteAtomicLong#close
Thank you, Anton.
If it is expected behaviour I will fix test but anyway in my opinion this
behaviour is not straight, I mean when you do same action and you getting
differents result it is not good. At least, exception of missing value should
be one. I don't have solution right now but I will think about it. But if you
have any thoughts about it, please, share them.
Nevertheless my target was understand the problem and it was achived. Now I can
fix the test.
--
Best regards,
Anton Kalashnikov
01.03.2018, 00:33, "Anton Vinogradov" :
> Anton,
>
> dsMap allows you to share any datastructure between local node's threads
> without requesting it using retryTopologySafe method (which is heavy).
>
> T dataStructure = cast(dsMap.get(key), cls);
>
> if (dataStructure != null)
> return dataStructure;
>
> return retryTopologySafe( ...
>
> I see no problems at current behavour since you'll never gain obsolete
> value.
> You'll gain null once continuous query update dsMap.
>
> Looks like problem with test, not with atomicLong implementation.
> After datastructure closing you should be ready to see non null value some
> time, but any get method on this value should cause exception.
>
> 2018-02-28 18:15 GMT+03:00 Anton Kalashnikov :
>
>> Hello Igniters.
>>
>> During investigation flaky test
>> IgniteClientDataStructuresTest.testAtomicLong.
>> I found a issue and now I need your help because I don't have enough
>> knowledge about IgniteAtomicLong.
>> I created the task for it - https://issues.apache.org/
>> jira/browse/IGNITE-7845. And also I duplicate information below:
>>
>> Given:
>>
>> IgniteAtomicLong was created e.g. atomicLong = ignite.atomicLong("long1",
>> 1L, true)
>>
>> When:
>>
>> IgniteAtomicLong was closed e.g. atomicLong.close()
>>
>> Then:
>>
>> If you try to get this value again - sometimes you will get null
>> IgniteAtomicLong value and sometimes you will get not null IgniteAtomicLong
>> value e.g. ignite.atomicLong("long1", 1L, false) sometimes null, sometimes
>> not null
>>
>> But if we will get not null value IgniteAtomicLong and we will call method
>> "get" on it, we will have one of two exceptions
>> IllegalStateException("Sequence
>> was removed from cache: " + name) if it already marked as deleted, and
>> IgniteException("Failed to find atomic long: " + name) if it sill no marked
>> as deleted but already deleted from cache.
>>
>> Expected:
>>
>> IgniteAtomicLong value always should be null(or not?)
>>
>> Why it's happend:
>>
>> When we close IgniteAtomicLong we removing value from cache in transaction
>> but removing from internal storage(dsMap) happen asynchroniously in
>> DataStructuresEntryListener for all nodes include local node. And when we
>> try get value after close we still sometimes able to get IgniteAtomicLong
>> from internal local storage.
>>
>> Solution(In my opinion):
>>
>> I guess in common case we don't need call Ignite#atomicLong every time
>> when we need value, but we should use IgniteAtomicLong object received
>> after first call. And if it is true, we can remove receiving
>> IgniteAtomicLong from local storage(dsMap) - this changes will fix problem
>>
>> --
>> Best regards,
>> Anton Kalashnikov
Re: Enabling swap space and Ignite Persistence
Hi Ivan, > Swap is legacy lightweight version of Ignite Native Persistence. In swap > mode, we fully rely on OS in storing offheap memory into memory-mapped > file. We don't provide durability guarantees in this mode. From my point of > view, after 2.1 release there's no reason to prefer swap mode over Ignite > Native Persistence. > Igniters, please correct me if there are still any actual cases. There is a business use case for the swap space I've come across with recently. Some applications want to store data entirely in RAM avoiding any persistence in general (Ignite persistence or 3rd party DB). It's ok for them to lose a data set in case of a cluster shutdown. However, they want to avoid OOM exception that might happen if they don't scale out the cluster in time. And here is the swap space comes to rescue. If a node is running out of RAM, the OS begins the swap-out/in the process putting off OOM and DevOps will have much more time to scale the cluster and rebalance the data. Right now we indeed can configure both Ignite Native Persistence and > swapping, but this makes even less sense. Node will just perform extra job > by persisting data twice. Guess, that's the point Prachi tried to point out. Could we throw an exception if a user tries to configure both? As we agreed, it's error-prone and doesn't make sense in general. -- Denis On Fri, Mar 2, 2018 at 1:28 AM, Ivan Rakov wrote: > Prachi, > > Swap is legacy lightweight version of Ignite Native Persistence. In swap > mode, we fully rely on OS in storing offheap memory into memory-mapped > file. We don't provide durability guarantees in this mode. From my point of > view, after 2.1 release there's no reason to prefer swap mode over Ignite > Native Persistence. > Igniters, please correct me if there are still any actual cases. > > Right now we indeed can configure both Ignite Native Persistence and > swapping, but this makes even less sense. Node will just perform extra job > by persisting data twice. > > Best Regards, > Ivan Rakov > > > On 02.03.2018 7:20, Prachi Garg wrote: > >> Engineers, >> >> How does persistence and swap work when both are enabled? I was under the >> impression that for a data region you can either have swap or persistence >> configured at a time, but not both. Please clarify. >> >> Thanks, >> -Prachi >> >> >
Re: Ignite Thin clients for Node.js, Python, PHP
Thanks, Sergey! It's a really robust foundation. My guess that the guys can take your client, support the rest of APIs the protocol has and eventually merge it to Ignite code base. -- Denis On Fri, Mar 2, 2018 at 2:40 AM, Sergey Kozlov wrote: > Hi Igniters > > Due to some delay for release 2.4 and personal circumstances I decided to > public the prototype of Python Client [1]. > > Feel free to use it as a base if it makes sense. > > [1] https://github.com/skozlov-gridgain/apache-ignite-python-thin-client > > > On Thu, Mar 1, 2018 at 8:17 PM, Pavel Tupitsyn > wrote: > > > Agree with Vladimir and Denis. > > > > I don't think JSON has any place in the thin client protocol, > > which is a binary protocol designed to be efficient, with clearly defined > > data format. > > > > We already have JSON in "REST" client. > > > > > > On Thu, Mar 1, 2018 at 8:15 PM, Denis Magda wrote: > > > > > Totally share Vladimir's stance. Let's support the scope that already > > > exists in the protocol and think about the future later. The users will > > > definitely guide us to a right direction :) > > > > > > -- > > > Denis > > > > > > On Thu, Mar 1, 2018 at 7:12 AM, Vladimir Ozerov > > > wrote: > > > > > > > I would extract compute tasks into separate scope. It is better to > keep > > > > focus on protocol things and basic language support for now. Once we > > have > > > > basic client API in production-ready state, we could consider adding > > > > JavaScript to our core compute feature set and then extend it to the > > > > clients (which should be trivial provided that core part is ready). > We > > > > should > > > > be ready to spend considerable efforts to prior R&D because dynamic > > code > > > > execution is not very simple thing, especially in terms of security, > > > native > > > > compilation, etc.. > > > > > > > > > > > > > > > > On Thu, Mar 1, 2018 at 5:17 PM, Ilya Kasnacheev < > > > ilya.kasnach...@gmail.com > > > > > > > > > wrote: > > > > > > > > > With regards of thin clients for dynamically typed languages, I > think > > > > > Ignite needs two following features to shine: > > > > > > > > > > - Ability to pass JSON to such clients, turn JSON Objects into a > > > > > BinaryObjects, which will give ability to index top-level keys in > > such > > > > JSON > > > > > with SQL Indexing. Of course this should be integrated with > > > > QueryEntities. > > > > > - Ability to pass JavaScript snippets to invoke() and > affinityCall() > > > > > families of calls. On Server node they should be interpreted by > > Nashorn > > > > > (since we've upgraded to Java 8). We should also cache such > snippets > > > > > pre-interpreted, in this case it can be pretty fast since Nashorn > > > compile > > > > > to JVM bytecode. > > > > > > > > > > WDYT? > > > > > > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > 2018-02-20 0:23 GMT+03:00 Alexey Kosenchuk > > > > > com > > > > > >: > > > > > > > > > > > Hi All! > > > > > > > > > > > > Let us join the party, please ;) > > > > > > > > > > > > As we see, there is Binary Client Protocol to communicate with > > Ignite > > > > > > cluster and a concept of Thin (lightweight) client. > > > > > > > > > > > > If there are no objections or duplicated plans, we would like to > > > > develop > > > > > > Thin Client libs for: > > > > > > - Node.js > > > > > > - Python > > > > > > - PHP > > > > > > > > > > > > Please add us as contributors and provide access to the Ignite > Jira > > > > > > components. > > > > > > > > > > > > Usernames in the Apache Jira: > > > > > > alexey.kosenchuk > > > > > > ekaterina.vergizova > > > > > > pavel.petroshenko > > > > > > > > > > > > Thanks! > > > > > > -Alexey > > > > > > > > > > > > > > > > > > > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com >
Re: REST: support getting objects from cache.
Thanks, Alexey! I'll update the docs. -P On Fri, Mar 2, 2018 at 1:42 AM, Alexey Kuznetsov wrote: > I merged IGNITE-7803 into master. > Created ticket for documentation: https://issues. > apache.org/jira/browse/IGNITE-7867 > > Prachi, could you update documentation for REST? > > On Thu, Mar 1, 2018 at 2:41 PM, Alexey Kuznetsov > wrote: > >> Vladimir, >> >> I finished with this feature: https://issues.apache >> .org/jira/browse/IGNITE-7803 >> >> Could you please review my changes in branch ignite-7803? >> >> DIFF: https://github.com/apache/ignite/commit/da03195ef5c39deabfc4 >> 961b2a0336fdbc98aa31 >> >> -- >> Alexey Kuznetsov >> > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com >
Re: Enabling swap space and Ignite Persistence
I stumbled across couple of use cases where swap space was more suitable than persistence. However, enabling both for a same region definitely doesn't make sense to me, I would throw an exception in this case. -Val On Fri, Mar 2, 2018 at 9:46 AM, Denis Magda wrote: > Hi Ivan, > > > > Swap is legacy lightweight version of Ignite Native Persistence. In swap > > mode, we fully rely on OS in storing offheap memory into memory-mapped > > file. We don't provide durability guarantees in this mode. From my point > of > > view, after 2.1 release there's no reason to prefer swap mode over Ignite > > Native Persistence. > > Igniters, please correct me if there are still any actual cases. > > > There is a business use case for the swap space I've come across with > recently. Some applications want to store data entirely in RAM avoiding any > persistence in general (Ignite persistence or 3rd party DB). It's ok for > them to lose a data set in case of a cluster shutdown. However, they want > to avoid OOM exception that might happen if they don't scale out the > cluster in time. And here is the swap space comes to rescue. If a node is > running out of RAM, the OS begins the swap-out/in the process putting off > OOM and DevOps will have much more time to scale the cluster and rebalance > the data. > > Right now we indeed can configure both Ignite Native Persistence and > > swapping, but this makes even less sense. Node will just perform extra > job > > by persisting data twice. > > > Guess, that's the point Prachi tried to point out. Could we throw an > exception if a user tries to configure both? As we agreed, it's error-prone > and doesn't make sense in general. > > -- > Denis > > On Fri, Mar 2, 2018 at 1:28 AM, Ivan Rakov wrote: > > > Prachi, > > > > Swap is legacy lightweight version of Ignite Native Persistence. In swap > > mode, we fully rely on OS in storing offheap memory into memory-mapped > > file. We don't provide durability guarantees in this mode. From my point > of > > view, after 2.1 release there's no reason to prefer swap mode over Ignite > > Native Persistence. > > Igniters, please correct me if there are still any actual cases. > > > > Right now we indeed can configure both Ignite Native Persistence and > > swapping, but this makes even less sense. Node will just perform extra > job > > by persisting data twice. > > > > Best Regards, > > Ivan Rakov > > > > > > On 02.03.2018 7:20, Prachi Garg wrote: > > > >> Engineers, > >> > >> How does persistence and swap work when both are enabled? I was under > the > >> impression that for a data region you can either have swap or > persistence > >> configured at a time, but not both. Please clarify. > >> > >> Thanks, > >> -Prachi > >> > >> > > >
IGNITE-5357 is ready for review (Replicated cache reads load balancing)
Hi, Igniters! This task [1] is about 'get' requests distribution between primary and backup nodes in the replicated cache if 'readFromBackup' flag is enabled. I've prepared a solution [2] suggested by Alexei Scherbakov in Jira comments. It passed prereviews by Alexei and Nikolay Izhikov. TeamCity tests look similar with the master branch. Could someone of core module maintainers do the final review [2][3]? [1] https://issues.apache.org/jira/browse/IGNITE-5357 [2] https://github.com/apache/ignite/pull/3578 [3] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-509 -- Best Regards, Vyacheslav D.
Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite
Yury, I have removed pom.good.xml from ml module. Regards Turik -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[jira] [Created] (IGNITE-7874) Update of an enum through jdbc yields a flag mismatch when object is retrieved through get
Danny Mui created IGNITE-7874: - Summary: Update of an enum through jdbc yields a flag mismatch when object is retrieved through get Key: IGNITE-7874 URL: https://issues.apache.org/jira/browse/IGNITE-7874 Project: Ignite Issue Type: Bug Components: binary, jdbc Affects Versions: 2.3 Environment: Sample maven project: https://github.com/dannymui/igniteReport mvn test Reporter: Danny Mui JDBC query update is issued and then the object fails to be deserialized with the stack trace below. one is the binaryEnum and the other is the standard enum. javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Unexpected field type [pos=24, expected=Enum, actual=Enum] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1287) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1648) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:831) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:662) at com.mycompany.app.AppTest.enum_update(AppTest.java:55) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206) Caused by: class org.apache.ignite.IgniteCheckedException: Unexpected field type [pos=24, expected=Enum, actual=Enum] at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7252) at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:171) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4512) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4493) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1326) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:828) ... 25 more Caused by: class org.apache.ignite.binary.BinaryObjectException: Unexpected field type [pos=24, expected=Enum, actual=Enum] at org.apache.ignite.internal.binary.BinaryReaderExImpl.checkFlagNoHandles(BinaryReaderExImpl.java:1676) at org.apache.ignite.internal.binary.BinaryReaderExImpl.readEnum0(BinaryReaderExImpl.java:1403) at org.apache.ignite.internal.binary.BinaryReaderExImpl.readEnum(BinaryReaderExImpl.java:1387) at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.readFixedType(BinaryFieldAccessor.java:862) at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(BinaryFieldAccessor.java:679) at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:164) at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:843) at org.apache.ignite.internal.binary.BinaryReade
